* Re: Linux 6.1.191
2026-10-30 12:57 Linux 6.1.191 Greg Kroah-Hartman
@ 2026-10-30 12:57 ` Greg Kroah-Hartman
0 siblings, 0 replies; 2+ messages in thread
From: Greg Kroah-Hartman @ 2026-10-30 12:57 UTC (permalink / raw)
To: linux-kernel, akpm, torvalds, stable; +Cc: lwn, jslaby, Greg Kroah-Hartman
diff --git a/CREDITS b/CREDITS
index 198f675c419e..f6c04d03c145 100644
--- a/CREDITS
+++ b/CREDITS
@@ -67,6 +67,11 @@ D: dosfs, LILO, some fd features, ATM, various other hacks here and there
S: Buenos Aires
S: Argentina
+NTFS FILESYSTEM
+N: Anton Altaparmakov
+E: anton@tuxera.com
+D: NTFS filesystem
+
N: Tim Alpaerts
E: tim_alpaerts@toyota-motor-europe.com
D: 802.2 class II logical link control layer,
diff --git a/Documentation/arm64/silicon-errata.rst b/Documentation/arm64/silicon-errata.rst
index fe7b90ac52d7..74ae9e4f637a 100644
--- a/Documentation/arm64/silicon-errata.rst
+++ b/Documentation/arm64/silicon-errata.rst
@@ -155,6 +155,8 @@ stable kernels.
+----------------+-----------------+-----------------+-----------------------------+
| ARM | Cortex-A725 | #3456106 | ARM64_ERRATUM_3194386 |
+----------------+-----------------+-----------------+-----------------------------+
+| ARM | Cortex-A725 | #3821522 | ARM64_ERRATUM_3821522 |
++----------------+-----------------+-----------------+-----------------------------+
| ARM | Cortex-X1 | #3324344 | ARM64_ERRATUM_3194386 |
+----------------+-----------------+-----------------+-----------------------------+
| ARM | Cortex-X1 | #4193791 | ARM64_ERRATUM_4118414 |
diff --git a/Documentation/filesystems/index.rst b/Documentation/filesystems/index.rst
index bee63d42e5ec..14904178be86 100644
--- a/Documentation/filesystems/index.rst
+++ b/Documentation/filesystems/index.rst
@@ -99,7 +99,6 @@ Documentation for filesystem implementations.
isofs
nilfs2
nfs/index
- ntfs
ntfs3
ocfs2
ocfs2-online-filecheck
diff --git a/Documentation/filesystems/ntfs.rst b/Documentation/filesystems/ntfs.rst
deleted file mode 100644
index 5bb093a26485..000000000000
--- a/Documentation/filesystems/ntfs.rst
+++ /dev/null
@@ -1,466 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0
-
-================================
-The Linux NTFS filesystem driver
-================================
-
-
-.. Table of contents
-
- - Overview
- - Web site
- - Features
- - Supported mount options
- - Known bugs and (mis-)features
- - Using NTFS volume and stripe sets
- - The Device-Mapper driver
- - The Software RAID / MD driver
- - Limitations when using the MD driver
-
-
-Overview
-========
-
-Linux-NTFS comes with a number of user-space programs known as ntfsprogs.
-These include mkntfs, a full-featured ntfs filesystem format utility,
-ntfsundelete used for recovering files that were unintentionally deleted
-from an NTFS volume and ntfsresize which is used to resize an NTFS partition.
-See the web site for more information.
-
-To mount an NTFS 1.2/3.x (Windows NT4/2000/XP/2003) volume, use the file
-system type 'ntfs'. The driver currently supports read-only mode (with no
-fault-tolerance, encryption or journalling) and very limited, but safe, write
-support.
-
-For fault tolerance and raid support (i.e. volume and stripe sets), you can
-use the kernel's Software RAID / MD driver. See section "Using Software RAID
-with NTFS" for details.
-
-
-Web site
-========
-
-There is plenty of additional information on the linux-ntfs web site
-at http://www.linux-ntfs.org/
-
-The web site has a lot of additional information, such as a comprehensive
-FAQ, documentation on the NTFS on-disk format, information on the Linux-NTFS
-userspace utilities, etc.
-
-
-Features
-========
-
-- This is a complete rewrite of the NTFS driver that used to be in the 2.4 and
- earlier kernels. This new driver implements NTFS read support and is
- functionally equivalent to the old ntfs driver and it also implements limited
- write support. The biggest limitation at present is that files/directories
- cannot be created or deleted. See below for the list of write features that
- are so far supported. Another limitation is that writing to compressed files
- is not implemented at all. Also, neither read nor write access to encrypted
- files is so far implemented.
-- The new driver has full support for sparse files on NTFS 3.x volumes which
- the old driver isn't happy with.
-- The new driver supports execution of binaries due to mmap() now being
- supported.
-- The new driver supports loopback mounting of files on NTFS which is used by
- some Linux distributions to enable the user to run Linux from an NTFS
- partition by creating a large file while in Windows and then loopback
- mounting the file while in Linux and creating a Linux filesystem on it that
- is used to install Linux on it.
-- A comparison of the two drivers using::
-
- time find . -type f -exec md5sum "{}" \;
-
- run three times in sequence with each driver (after a reboot) on a 1.4GiB
- NTFS partition, showed the new driver to be 20% faster in total time elapsed
- (from 9:43 minutes on average down to 7:53). The time spent in user space
- was unchanged but the time spent in the kernel was decreased by a factor of
- 2.5 (from 85 CPU seconds down to 33).
-- The driver does not support short file names in general. For backwards
- compatibility, we implement access to files using their short file names if
- they exist. The driver will not create short file names however, and a
- rename will discard any existing short file name.
-- The new driver supports exporting of mounted NTFS volumes via NFS.
-- The new driver supports async io (aio).
-- The new driver supports fsync(2), fdatasync(2), and msync(2).
-- The new driver supports readv(2) and writev(2).
-- The new driver supports access time updates (including mtime and ctime).
-- The new driver supports truncate(2) and open(2) with O_TRUNC. But at present
- only very limited support for highly fragmented files, i.e. ones which have
- their data attribute split across multiple extents, is included. Another
- limitation is that at present truncate(2) will never create sparse files,
- since to mark a file sparse we need to modify the directory entry for the
- file and we do not implement directory modifications yet.
-- The new driver supports write(2) which can both overwrite existing data and
- extend the file size so that you can write beyond the existing data. Also,
- writing into sparse regions is supported and the holes are filled in with
- clusters. But at present only limited support for highly fragmented files,
- i.e. ones which have their data attribute split across multiple extents, is
- included. Another limitation is that write(2) will never create sparse
- files, since to mark a file sparse we need to modify the directory entry for
- the file and we do not implement directory modifications yet.
-
-Supported mount options
-=======================
-
-In addition to the generic mount options described by the manual page for the
-mount command (man 8 mount, also see man 5 fstab), the NTFS driver supports the
-following mount options:
-
-======================= =======================================================
-iocharset=name Deprecated option. Still supported but please use
- nls=name in the future. See description for nls=name.
-
-nls=name Character set to use when returning file names.
- Unlike VFAT, NTFS suppresses names that contain
- unconvertible characters. Note that most character
- sets contain insufficient characters to represent all
- possible Unicode characters that can exist on NTFS.
- To be sure you are not missing any files, you are
- advised to use nls=utf8 which is capable of
- representing all Unicode characters.
-
-utf8=<bool> Option no longer supported. Currently mapped to
- nls=utf8 but please use nls=utf8 in the future and
- make sure utf8 is compiled either as module or into
- the kernel. See description for nls=name.
-
-uid=
-gid=
-umask= Provide default owner, group, and access mode mask.
- These options work as documented in mount(8). By
- default, the files/directories are owned by root and
- he/she has read and write permissions, as well as
- browse permission for directories. No one else has any
- access permissions. I.e. the mode on all files is by
- default rw------- and for directories rwx------, a
- consequence of the default fmask=0177 and dmask=0077.
- Using a umask of zero will grant all permissions to
- everyone, i.e. all files and directories will have mode
- rwxrwxrwx.
-
-fmask=
-dmask= Instead of specifying umask which applies both to
- files and directories, fmask applies only to files and
- dmask only to directories.
-
-sloppy=<BOOL> If sloppy is specified, ignore unknown mount options.
- Otherwise the default behaviour is to abort mount if
- any unknown options are found.
-
-show_sys_files=<BOOL> If show_sys_files is specified, show the system files
- in directory listings. Otherwise the default behaviour
- is to hide the system files.
- Note that even when show_sys_files is specified, "$MFT"
- will not be visible due to bugs/mis-features in glibc.
- Further, note that irrespective of show_sys_files, all
- files are accessible by name, i.e. you can always do
- "ls -l \$UpCase" for example to specifically show the
- system file containing the Unicode upcase table.
-
-case_sensitive=<BOOL> If case_sensitive is specified, treat all file names as
- case sensitive and create file names in the POSIX
- namespace. Otherwise the default behaviour is to treat
- file names as case insensitive and to create file names
- in the WIN32/LONG name space. Note, the Linux NTFS
- driver will never create short file names and will
- remove them on rename/delete of the corresponding long
- file name.
- Note that files remain accessible via their short file
- name, if it exists. If case_sensitive, you will need
- to provide the correct case of the short file name.
-
-disable_sparse=<BOOL> If disable_sparse is specified, creation of sparse
- regions, i.e. holes, inside files is disabled for the
- volume (for the duration of this mount only). By
- default, creation of sparse regions is enabled, which
- is consistent with the behaviour of traditional Unix
- filesystems.
-
-errors=opt What to do when critical filesystem errors are found.
- Following values can be used for "opt":
-
- ======== =========================================
- continue DEFAULT, try to clean-up as much as
- possible, e.g. marking a corrupt inode as
- bad so it is no longer accessed, and then
- continue.
- recover At present only supported is recovery of
- the boot sector from the backup copy.
- If read-only mount, the recovery is done
- in memory only and not written to disk.
- ======== =========================================
-
- Note that the options are additive, i.e. specifying::
-
- errors=continue,errors=recover
-
- means the driver will attempt to recover and if that
- fails it will clean-up as much as possible and
- continue.
-
-mft_zone_multiplier= Set the MFT zone multiplier for the volume (this
- setting is not persistent across mounts and can be
- changed from mount to mount but cannot be changed on
- remount). Values of 1 to 4 are allowed, 1 being the
- default. The MFT zone multiplier determines how much
- space is reserved for the MFT on the volume. If all
- other space is used up, then the MFT zone will be
- shrunk dynamically, so this has no impact on the
- amount of free space. However, it can have an impact
- on performance by affecting fragmentation of the MFT.
- In general use the default. If you have a lot of small
- files then use a higher value. The values have the
- following meaning:
-
- ===== =================================
- Value MFT zone size (% of volume size)
- ===== =================================
- 1 12.5%
- 2 25%
- 3 37.5%
- 4 50%
- ===== =================================
-
- Note this option is irrelevant for read-only mounts.
-======================= =======================================================
-
-
-Known bugs and (mis-)features
-=============================
-
-- The link count on each directory inode entry is set to 1, due to Linux not
- supporting directory hard links. This may well confuse some user space
- applications, since the directory names will have the same inode numbers.
- This also speeds up ntfs_read_inode() immensely. And we haven't found any
- problems with this approach so far. If you find a problem with this, please
- let us know.
-
-
-Please send bug reports/comments/feedback/abuse to the Linux-NTFS development
-list at sourceforge: linux-ntfs-dev@lists.sourceforge.net
-
-
-Using NTFS volume and stripe sets
-=================================
-
-For support of volume and stripe sets, you can either use the kernel's
-Device-Mapper driver or the kernel's Software RAID / MD driver. The former is
-the recommended one to use for linear raid. But the latter is required for
-raid level 5. For striping and mirroring, either driver should work fine.
-
-
-The Device-Mapper driver
-------------------------
-
-You will need to create a table of the components of the volume/stripe set and
-how they fit together and load this into the kernel using the dmsetup utility
-(see man 8 dmsetup).
-
-Linear volume sets, i.e. linear raid, has been tested and works fine. Even
-though untested, there is no reason why stripe sets, i.e. raid level 0, and
-mirrors, i.e. raid level 1 should not work, too. Stripes with parity, i.e.
-raid level 5, unfortunately cannot work yet because the current version of the
-Device-Mapper driver does not support raid level 5. You may be able to use the
-Software RAID / MD driver for raid level 5, see the next section for details.
-
-To create the table describing your volume you will need to know each of its
-components and their sizes in sectors, i.e. multiples of 512-byte blocks.
-
-For NT4 fault tolerant volumes you can obtain the sizes using fdisk. So for
-example if one of your partitions is /dev/hda2 you would do::
-
- $ fdisk -ul /dev/hda
-
- Disk /dev/hda: 81.9 GB, 81964302336 bytes
- 255 heads, 63 sectors/track, 9964 cylinders, total 160086528 sectors
- Units = sectors of 1 * 512 = 512 bytes
-
- Device Boot Start End Blocks Id System
- /dev/hda1 * 63 4209029 2104483+ 83 Linux
- /dev/hda2 4209030 37768814 16779892+ 86 NTFS
- /dev/hda3 37768815 46170809 4200997+ 83 Linux
-
-And you would know that /dev/hda2 has a size of 37768814 - 4209030 + 1 =
-33559785 sectors.
-
-For Win2k and later dynamic disks, you can for example use the ldminfo utility
-which is part of the Linux LDM tools (the latest version at the time of
-writing is linux-ldm-0.0.8.tar.bz2). You can download it from:
-
- http://www.linux-ntfs.org/
-
-Simply extract the downloaded archive (tar xvjf linux-ldm-0.0.8.tar.bz2), go
-into it (cd linux-ldm-0.0.8) and change to the test directory (cd test). You
-will find the precompiled (i386) ldminfo utility there. NOTE: You will not be
-able to compile this yourself easily so use the binary version!
-
-Then you would use ldminfo in dump mode to obtain the necessary information::
-
- $ ./ldminfo --dump /dev/hda
-
-This would dump the LDM database found on /dev/hda which describes all of your
-dynamic disks and all the volumes on them. At the bottom you will see the
-VOLUME DEFINITIONS section which is all you really need. You may need to look
-further above to determine which of the disks in the volume definitions is
-which device in Linux. Hint: Run ldminfo on each of your dynamic disks and
-look at the Disk Id close to the top of the output for each (the PRIVATE HEADER
-section). You can then find these Disk Ids in the VBLK DATABASE section in the
-<Disk> components where you will get the LDM Name for the disk that is found in
-the VOLUME DEFINITIONS section.
-
-Note you will also need to enable the LDM driver in the Linux kernel. If your
-distribution did not enable it, you will need to recompile the kernel with it
-enabled. This will create the LDM partitions on each device at boot time. You
-would then use those devices (for /dev/hda they would be /dev/hda1, 2, 3, etc)
-in the Device-Mapper table.
-
-You can also bypass using the LDM driver by using the main device (e.g.
-/dev/hda) and then using the offsets of the LDM partitions into this device as
-the "Start sector of device" when creating the table. Once again ldminfo would
-give you the correct information to do this.
-
-Assuming you know all your devices and their sizes things are easy.
-
-For a linear raid the table would look like this (note all values are in
-512-byte sectors)::
-
- # Offset into Size of this Raid type Device Start sector
- # volume device of device
- 0 1028161 linear /dev/hda1 0
- 1028161 3903762 linear /dev/hdb2 0
- 4931923 2103211 linear /dev/hdc1 0
-
-For a striped volume, i.e. raid level 0, you will need to know the chunk size
-you used when creating the volume. Windows uses 64kiB as the default, so it
-will probably be this unless you changes the defaults when creating the array.
-
-For a raid level 0 the table would look like this (note all values are in
-512-byte sectors)::
-
- # Offset Size Raid Number Chunk 1st Start 2nd Start
- # into of the type of size Device in Device in
- # volume volume stripes device device
- 0 2056320 striped 2 128 /dev/hda1 0 /dev/hdb1 0
-
-If there are more than two devices, just add each of them to the end of the
-line.
-
-Finally, for a mirrored volume, i.e. raid level 1, the table would look like
-this (note all values are in 512-byte sectors)::
-
- # Ofs Size Raid Log Number Region Should Number Source Start Target Start
- # in of the type type of log size sync? of Device in Device in
- # vol volume params mirrors Device Device
- 0 2056320 mirror core 2 16 nosync 2 /dev/hda1 0 /dev/hdb1 0
-
-If you are mirroring to multiple devices you can specify further targets at the
-end of the line.
-
-Note the "Should sync?" parameter "nosync" means that the two mirrors are
-already in sync which will be the case on a clean shutdown of Windows. If the
-mirrors are not clean, you can specify the "sync" option instead of "nosync"
-and the Device-Mapper driver will then copy the entirety of the "Source Device"
-to the "Target Device" or if you specified multiple target devices to all of
-them.
-
-Once you have your table, save it in a file somewhere (e.g. /etc/ntfsvolume1),
-and hand it over to dmsetup to work with, like so::
-
- $ dmsetup create myvolume1 /etc/ntfsvolume1
-
-You can obviously replace "myvolume1" with whatever name you like.
-
-If it all worked, you will now have the device /dev/device-mapper/myvolume1
-which you can then just use as an argument to the mount command as usual to
-mount the ntfs volume. For example::
-
- $ mount -t ntfs -o ro /dev/device-mapper/myvolume1 /mnt/myvol1
-
-(You need to create the directory /mnt/myvol1 first and of course you can use
-anything you like instead of /mnt/myvol1 as long as it is an existing
-directory.)
-
-It is advisable to do the mount read-only to see if the volume has been setup
-correctly to avoid the possibility of causing damage to the data on the ntfs
-volume.
-
-
-The Software RAID / MD driver
------------------------------
-
-An alternative to using the Device-Mapper driver is to use the kernel's
-Software RAID / MD driver. For which you need to set up your /etc/raidtab
-appropriately (see man 5 raidtab).
-
-Linear volume sets, i.e. linear raid, as well as stripe sets, i.e. raid level
-0, have been tested and work fine (though see section "Limitations when using
-the MD driver with NTFS volumes" especially if you want to use linear raid).
-Even though untested, there is no reason why mirrors, i.e. raid level 1, and
-stripes with parity, i.e. raid level 5, should not work, too.
-
-You have to use the "persistent-superblock 0" option for each raid-disk in the
-NTFS volume/stripe you are configuring in /etc/raidtab as the persistent
-superblock used by the MD driver would damage the NTFS volume.
-
-Windows by default uses a stripe chunk size of 64k, so you probably want the
-"chunk-size 64k" option for each raid-disk, too.
-
-For example, if you have a stripe set consisting of two partitions /dev/hda5
-and /dev/hdb1 your /etc/raidtab would look like this::
-
- raiddev /dev/md0
- raid-level 0
- nr-raid-disks 2
- nr-spare-disks 0
- persistent-superblock 0
- chunk-size 64k
- device /dev/hda5
- raid-disk 0
- device /dev/hdb1
- raid-disk 1
-
-For linear raid, just change the raid-level above to "raid-level linear", for
-mirrors, change it to "raid-level 1", and for stripe sets with parity, change
-it to "raid-level 5".
-
-Note for stripe sets with parity you will also need to tell the MD driver
-which parity algorithm to use by specifying the option "parity-algorithm
-which", where you need to replace "which" with the name of the algorithm to
-use (see man 5 raidtab for available algorithms) and you will have to try the
-different available algorithms until you find one that works. Make sure you
-are working read-only when playing with this as you may damage your data
-otherwise. If you find which algorithm works please let us know (email the
-linux-ntfs developers list linux-ntfs-dev@lists.sourceforge.net or drop in on
-IRC in channel #ntfs on the irc.freenode.net network) so we can update this
-documentation.
-
-Once the raidtab is setup, run for example raid0run -a to start all devices or
-raid0run /dev/md0 to start a particular md device, in this case /dev/md0.
-
-Then just use the mount command as usual to mount the ntfs volume using for
-example::
-
- mount -t ntfs -o ro /dev/md0 /mnt/myntfsvolume
-
-It is advisable to do the mount read-only to see if the md volume has been
-setup correctly to avoid the possibility of causing damage to the data on the
-ntfs volume.
-
-
-Limitations when using the Software RAID / MD driver
------------------------------------------------------
-
-Using the md driver will not work properly if any of your NTFS partitions have
-an odd number of sectors. This is especially important for linear raid as all
-data after the first partition with an odd number of sectors will be offset by
-one or more sectors so if you mount such a partition with write support you
-will cause massive damage to the data on the volume which will only become
-apparent when you try to use the volume again under Windows.
-
-So when using linear raid, make sure that all your partitions have an even
-number of sectors BEFORE attempting to use it. You have been warned!
-
-Even better is to simply use the Device-Mapper for linear raid and then you do
-not have this problem with odd numbers of sectors.
diff --git a/Documentation/networking/ip-sysctl.rst b/Documentation/networking/ip-sysctl.rst
index 0a5ae93b4461..3fd6e0f0d3c9 100644
--- a/Documentation/networking/ip-sysctl.rst
+++ b/Documentation/networking/ip-sysctl.rst
@@ -2000,6 +2000,12 @@ nexthop_compat_mode - BOOLEAN
understands the new API, this sysctl can be disabled to achieve full
performance benefits of the new API by disabling the nexthop expansion
and extraneous notifications.
+
+ Note that as a backward-compatible mode, dumping of modern features
+ might be incomplete or wrong. For example, resilient groups will not be
+ shown as such, but rather as just a list of next hops. Also weights that
+ do not fit into 8 bits will show incorrectly.
+
Default: true (backward compat mode)
fib_notify_on_flag_change - INTEGER
diff --git a/Documentation/networking/strparser.rst b/Documentation/networking/strparser.rst
index 7f623d1db72a..29144a9c4e83 100644
--- a/Documentation/networking/strparser.rst
+++ b/Documentation/networking/strparser.rst
@@ -77,7 +77,7 @@ Functions
strp_done is called to release any resources held by the stream
parser instance. This must be called after the stream processor
- has been stopped.
+ has been stopped. The socket lock must not be held.
::
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index 1bc61bf804f1..1f0f84501e59 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -1382,6 +1382,12 @@ The KVM_SET_MEMORY_REGION does not allow fine grained control over memory
allocation and is deprecated.
+S390:
+^^^^^
+
+Returns -EINVAL if the VM has the KVM_VM_S390_UCONTROL flag set.
+Returns -EINVAL if called on a protected VM.
+
4.36 KVM_SET_TSS_ADDR
---------------------
diff --git a/MAINTAINERS b/MAINTAINERS
index a02522a3ec08..bf4a79088f50 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -5293,6 +5293,8 @@ S: Maintained
T: git git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git
F: Documentation/admin-guide/cgroup-v1/cpusets.rst
F: include/linux/cpuset.h
+F: kernel/cgroup/cpuset-internal.h
+F: kernel/cgroup/cpuset-v1.c
F: kernel/cgroup/cpuset.c
CONTROL GROUP - MEMORY RESOURCE CONTROLLER (MEMCG)
@@ -14669,15 +14671,6 @@ W: https://github.com/davejiang/linux/wiki
T: git https://github.com/davejiang/linux.git
F: drivers/ntb/hw/intel/
-NTFS FILESYSTEM
-M: Anton Altaparmakov <anton@tuxera.com>
-L: linux-ntfs-dev@lists.sourceforge.net
-S: Supported
-W: http://www.tuxera.com/
-T: git git://git.kernel.org/pub/scm/linux/kernel/git/aia21/ntfs.git
-F: Documentation/filesystems/ntfs.rst
-F: fs/ntfs/
-
NTFS3 FILESYSTEM
M: Konstantin Komarov <almaz.alexandrovich@paragon-software.com>
L: ntfs3@lists.linux.dev
diff --git a/Makefile b/Makefile
index e4c0c25b58ed..7efb7366912b 100644
--- a/Makefile
+++ b/Makefile
@@ -1,7 +1,7 @@
# SPDX-License-Identifier: GPL-2.0
VERSION = 6
PATCHLEVEL = 1
-SUBLEVEL = 190
+SUBLEVEL = 191
EXTRAVERSION =
NAME = Curry Ramen
@@ -760,12 +760,6 @@ endif # KBUILD_EXTMOD
# Defaults to vmlinux, but the arch makefile usually adds further targets
all: vmlinux
-CFLAGS_GCOV := -fprofile-arcs -ftest-coverage
-ifdef CONFIG_CC_IS_GCC
-CFLAGS_GCOV += -fno-tree-loop-im
-endif
-export CFLAGS_GCOV
-
# The arch Makefiles can override CC_FLAGS_FTRACE. We may also append it later.
ifdef CONFIG_FUNCTION_TRACER
CC_FLAGS_FTRACE := -pg
@@ -1090,6 +1084,27 @@ KBUILD_CFLAGS += $(call cc-option,-Werror=designated-init)
# Ensure compilers do not transform certain loops into calls to wcslen()
KBUILD_CFLAGS += -fno-builtin-wcslen
+CFLAGS_GCOV := -fprofile-arcs -ftest-coverage
+ifdef CONFIG_CC_IS_GCC
+CFLAGS_GCOV += -fno-tree-loop-im
+# Use atomic counter updates to avoid concurrent-access crashes in GCOV.
+# Only enable if -fprofile-update=prefer-atomic does not introduce new
+# undefined symbols (e.g. libatomic calls that the kernel cannot link).
+CFLAGS_GCOV += $(call try-run,\
+ echo 'long long x; void f(void){x++;}' | \
+ $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) -w -fprofile-arcs \
+ -ftest-coverage -x c - -c -o "$$TMP.base" && \
+ echo 'long long x; void f(void){x++;}' | \
+ $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) -w -fprofile-arcs \
+ -ftest-coverage -fprofile-update=prefer-atomic \
+ -x c - -c -o "$$TMP" && \
+ $(NM) "$$TMP.base" | grep ' U ' > "$$TMP.ubase" || true ; \
+ $(NM) "$$TMP" | grep ' U ' > "$$TMP.utest" || true ; \
+ cmp -s "$$TMP.ubase" "$$TMP.utest",\
+ -fprofile-update=prefer-atomic)
+endif
+export CFLAGS_GCOV
+
# change __FILE__ to the relative path from the srctree
KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
diff --git a/arch/alpha/include/asm/pgtable.h b/arch/alpha/include/asm/pgtable.h
index 6fe4e9deeb5e..5aa8cb53e18c 100644
--- a/arch/alpha/include/asm/pgtable.h
+++ b/arch/alpha/include/asm/pgtable.h
@@ -17,6 +17,7 @@
#include <asm/processor.h> /* For TASK_SIZE */
#include <asm/machvec.h>
#include <asm/setup.h>
+#include <linux/page_table_check.h>
struct mm_struct;
struct vm_area_struct;
@@ -210,6 +211,9 @@ extern inline void pud_set(pud_t * pudp, pmd_t * pmdp)
{ pud_val(*pudp) = _PAGE_TABLE | ((((unsigned long) pmdp) - PAGE_OFFSET) << (32-PAGE_SHIFT)); }
+extern void migrate_flush_tlb_page(struct vm_area_struct *vma,
+ unsigned long addr);
+
extern inline unsigned long
pmd_page_vaddr(pmd_t pmd)
{
@@ -229,7 +233,7 @@ extern inline int pte_none(pte_t pte) { return !pte_val(pte); }
extern inline int pte_present(pte_t pte) { return pte_val(pte) & _PAGE_VALID; }
extern inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep)
{
- pte_val(*ptep) = 0;
+ WRITE_ONCE(pte_val(*ptep), 0);
}
extern inline int pmd_none(pmd_t pmd) { return !pmd_val(pmd); }
@@ -291,6 +295,33 @@ extern inline pte_t * pte_offset_kernel(pmd_t * dir, unsigned long address)
extern pgd_t swapper_pg_dir[1024];
+#ifdef CONFIG_COMPACTION
+#define __HAVE_ARCH_PTEP_GET_AND_CLEAR
+
+static inline pte_t ptep_get_and_clear(struct mm_struct *mm,
+ unsigned long address,
+ pte_t *ptep)
+{
+ pte_t pte = READ_ONCE(*ptep);
+
+ pte_clear(mm, address, ptep);
+ return pte;
+}
+
+#define __HAVE_ARCH_PTEP_CLEAR_FLUSH
+
+static inline pte_t ptep_clear_flush(struct vm_area_struct *vma,
+ unsigned long addr, pte_t *ptep)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ pte_t pte = ptep_get_and_clear(mm, addr, ptep);
+
+ page_table_check_pte_clear(mm, pte);
+ migrate_flush_tlb_page(vma, addr);
+ return pte;
+}
+
+#endif
/*
* The Alpha doesn't have any external MMU info: the kernel page
* tables contain all the necessary information.
diff --git a/arch/alpha/include/asm/tlbflush.h b/arch/alpha/include/asm/tlbflush.h
index 94dc37cf873a..e78de2456ffa 100644
--- a/arch/alpha/include/asm/tlbflush.h
+++ b/arch/alpha/include/asm/tlbflush.h
@@ -91,7 +91,9 @@ flush_tlb_other(struct mm_struct *mm)
unsigned long *mmc = &mm->context[smp_processor_id()];
/* Check it's not zero first to avoid cacheline ping pong
when possible. */
- if (*mmc) *mmc = 0;
+
+ if (READ_ONCE(*mmc))
+ WRITE_ONCE(*mmc, 0);
}
#ifndef CONFIG_SMP
diff --git a/arch/alpha/lib/Makefile b/arch/alpha/lib/Makefile
index 1cc74f7b50ef..6a779b9018fd 100644
--- a/arch/alpha/lib/Makefile
+++ b/arch/alpha/lib/Makefile
@@ -4,7 +4,6 @@
#
asflags-y := $(KBUILD_CFLAGS)
-ccflags-y := -Werror
# Many of these routines have implementations tuned for ev6.
# Choose them iff we're targeting ev6 specifically.
diff --git a/arch/alpha/mm/Makefile b/arch/alpha/mm/Makefile
index bd770302eb82..2d05664058f6 100644
--- a/arch/alpha/mm/Makefile
+++ b/arch/alpha/mm/Makefile
@@ -3,6 +3,4 @@
# Makefile for the linux alpha-specific parts of the memory manager.
#
-ccflags-y := -Werror
-
-obj-y := init.o fault.o
+obj-y := init.o fault.o tlbflush.o
diff --git a/arch/alpha/mm/tlbflush.c b/arch/alpha/mm/tlbflush.c
new file mode 100644
index 000000000000..239c72b8a741
--- /dev/null
+++ b/arch/alpha/mm/tlbflush.c
@@ -0,0 +1,96 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Alpha TLB shootdown helpers
+ *
+ * Copyright (C) 2025 Magnus Lindholm <linmag7@gmail.com>
+ *
+ * Alpha-specific TLB flush helpers that cannot be expressed purely
+ * as inline functions.
+ *
+ * These helpers provide combined MM context handling (ASN rollover)
+ * and immediate TLB invalidation for page migration and memory
+ * compaction paths, where lazy shootdowns are insufficient.
+ */
+
+#include <linux/mm.h>
+#include <linux/smp.h>
+#include <linux/sched.h>
+#include <asm/tlbflush.h>
+#include <asm/pal.h>
+#include <asm/mmu_context.h>
+
+#define asn_locked() (cpu_data[smp_processor_id()].asn_lock)
+
+/*
+ * Migration/compaction helper: combine mm context (ASN) handling with an
+ * immediate per-page TLB invalidate and (for exec) an instruction barrier.
+ *
+ * This mirrors the SMP combined IPI handler semantics, but runs locally on UP.
+ */
+#ifndef CONFIG_SMP
+void migrate_flush_tlb_page(struct vm_area_struct *vma,
+ unsigned long addr)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ int tbi_type = (vma->vm_flags & VM_EXEC) ? 3 : 2;
+
+ /*
+ * First do the mm-context side:
+ * If we're currently running this mm, reload a fresh context ASN.
+ * Otherwise, mark context invalid.
+ *
+ * On UP, this is mostly about matching the SMP semantics and ensuring
+ * exec/i-cache tagging assumptions hold when compaction migrates pages.
+ */
+ if (mm == current->active_mm)
+ flush_tlb_current(mm);
+ else
+ flush_tlb_other(mm);
+
+ /*
+ * Then do the immediate translation kill for this VA.
+ * For exec mappings, order instruction fetch after invalidation.
+ */
+ tbi(tbi_type, addr);
+}
+
+#else
+struct tlb_mm_and_addr {
+ struct mm_struct *mm;
+ unsigned long addr;
+ int tbi_type; /* 2 = DTB, 3 = ITB+DTB */
+};
+
+static void ipi_flush_mm_and_page(void *x)
+{
+ struct tlb_mm_and_addr *d = x;
+
+ /* Part 1: mm context side (Alpha uses ASN/context as a key mechanism). */
+ if (d->mm == current->active_mm && !asn_locked())
+ __load_new_mm_context(d->mm);
+ else
+ flush_tlb_other(d->mm);
+
+ /* Part 2: immediate per-VA invalidation on this CPU. */
+ tbi(d->tbi_type, d->addr);
+}
+
+void migrate_flush_tlb_page(struct vm_area_struct *vma, unsigned long addr)
+{
+ struct mm_struct *mm = vma->vm_mm;
+ struct tlb_mm_and_addr d = {
+ .mm = mm,
+ .addr = addr,
+ .tbi_type = (vma->vm_flags & VM_EXEC) ? 3 : 2,
+ };
+
+ /*
+ * One synchronous rendezvous: every CPU runs ipi_flush_mm_and_page().
+ * This is the "combined" version of flush_tlb_mm + per-page invalidate.
+ */
+ preempt_disable();
+ on_each_cpu(ipi_flush_mm_and_page, &d, 1);
+ preempt_enable();
+}
+
+#endif
diff --git a/arch/arm/mm/flush.c b/arch/arm/mm/flush.c
index 7ff9feea13a6..b19ecc307ad4 100644
--- a/arch/arm/mm/flush.c
+++ b/arch/arm/mm/flush.c
@@ -280,6 +280,8 @@ void __sync_icache_dcache(pte_t pteval)
return;
page = pfn_to_page(pfn);
+ if (PageReserved(page))
+ return;
if (cache_is_vipt_aliasing())
mapping = page_mapping_file(page);
else
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 9cb8867eb120..1b8ee28f2383 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -966,9 +966,17 @@ config ARM64_ERRATUM_1902691
If unsure, say Y.
+config ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT
+ bool
+ # Targeting errata affecting AMEVCNTR01, the AMU constant counter.
+ # On affected CPUs the counter may not increment at the expected rate,
+ # making it unreliable. As a result, AMU users should treat it as
+ # unavailable rather than relying on misleading counter values.
+
config ARM64_ERRATUM_2457168
bool "Cortex-A510: 2457168: workaround for AMEVCNTR01 incrementing incorrectly"
depends on ARM64_AMU_EXTN
+ select ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT
default y
help
This option adds the workaround for ARM Cortex-A510 erratum 2457168.
@@ -983,6 +991,28 @@ config ARM64_ERRATUM_2457168
If unsure, say Y.
+config ARM64_ERRATUM_3821522
+ bool "Cortex-A725: 3821522: workaround for possible CNT_CYCLES increment error due to WFE/WFI"
+ depends on ARM64_AMU_EXTN
+ select ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT
+ default y
+ help
+ This option adds the workaround for ARM Cortex-A725 erratum 3821522.
+
+ On affected A725 cores, the CNT_CYCLES event may incur a significant
+ increment error when entering and subsequently exiting WFx.
+ As a result, the CNT_CYCLES may diverge from the system counter
+ frequency at which it is expected to increment.
+ This renders the AMU counter AMEVCNTR01, that implements CNT_CYCLES,
+ being unreliable and unsuitable for use.
+
+ Since there is no hardware workaround, reads of the affected CNT_CYCLES
+ counter return 0 in the relevant paths. This causes users of the counter
+ to treat it as unavailable and is functionally equivalent to firmware
+ disabling the affected counter.
+
+ If unsure, say Y.
+
config ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD
bool
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 0935f9849510..f57f02a52a84 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -64,8 +64,6 @@ enum kvm_mode {
};
enum kvm_mode kvm_get_mode(void);
-DECLARE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
-
extern unsigned int kvm_sve_max_vl;
extern unsigned int kvm_host_sve_max_vl;
int kvm_arm_init_sve(void);
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index 2f361a883d8c..b09039be892e 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -25,6 +25,9 @@ KASAN_SANITIZE_stacktrace.o := n
KCOV_INSTRUMENT_entry-common.o := n
KCOV_INSTRUMENT_idle.o := n
+# Softirq stack switching can run outside interrupt context.
+KCOV_INSTRUMENT_irq.o := n
+
# Object file lists.
obj-y := debug-monitors.o entry.o irq.o fpsimd.o \
entry-common.o entry-fpsimd.o process.o ptrace.o \
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index cff85a371c35..660b3ff16ebc 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -275,6 +275,20 @@ static const struct arm64_cpu_capabilities arm64_repeat_tlbi_list[] = {
};
#endif
+#ifdef CONFIG_ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT
+static const struct midr_range workaround_amu_constcnt_list[] = {
+#ifdef CONFIG_ARM64_ERRATUM_2457168
+ /* Cortex-A510 r0p0-r1p1 */
+ MIDR_RANGE(MIDR_CORTEX_A510, 0, 0, 1, 1),
+#endif
+#ifdef CONFIG_ARM64_ERRATUM_3821522
+ /* Cortex-A725 r0p0 - r0p2 */
+ MIDR_RANGE(MIDR_CORTEX_A725, 0, 0, 0, 2),
+#endif
+ {}
+};
+#endif /* CONFIG_ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT */
+
#ifdef CONFIG_CAVIUM_ERRATUM_23154
static const struct midr_range cavium_erratum_23154_cpus[] = {
MIDR_ALL_VERSIONS(MIDR_THUNDERX),
@@ -737,14 +751,12 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
ERRATA_MIDR_REV_RANGE(MIDR_CORTEX_A510, 0, 0, 2)
},
#endif
-#ifdef CONFIG_ARM64_ERRATUM_2457168
+#ifdef CONFIG_ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT
{
- .desc = "ARM erratum 2457168",
- .capability = ARM64_WORKAROUND_2457168,
+ .desc = "Broken AMU AMEVCNTR01 (const counter)",
+ .capability = ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT,
.type = ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE,
-
- /* Cortex-A510 r0p0-r1p1 */
- CAP_MIDR_RANGE(MIDR_CORTEX_A510, 0, 0, 1, 1)
+ CAP_MIDR_RANGE_LIST(workaround_amu_constcnt_list)
},
#endif
#ifdef CONFIG_ARM64_ERRATUM_2038923
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 547c47358133..467737615bf3 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -1907,7 +1907,7 @@ static void cpu_amu_enable(struct arm64_cpu_capabilities const *cap)
cpumask_set_cpu(smp_processor_id(), &amu_cpus);
/* 0 reference values signal broken/disabled counters */
- if (!this_cpu_has_cap(ARM64_WORKAROUND_2457168))
+ if (!this_cpu_has_cap(ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT))
update_freq_counters_refs();
}
}
diff --git a/arch/arm64/kernel/entry.S b/arch/arm64/kernel/entry.S
index 4b4a9aa76e1a..6df98546605d 100644
--- a/arch/arm64/kernel/entry.S
+++ b/arch/arm64/kernel/entry.S
@@ -419,16 +419,9 @@ alternative_else_nop_endif
ldp x28, x29, [sp, #16 * 14]
.if \el == 0
-alternative_if ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD
- tlbi vale1, xzr
- dsb nsh
-alternative_else_nop_endif
-alternative_if_not ARM64_UNMAP_KERNEL_AT_EL0
- ldr lr, [sp, #S_LR]
- add sp, sp, #PT_REGS_SIZE // restore sp
- eret
-alternative_else_nop_endif
#ifdef CONFIG_UNMAP_KERNEL_AT_EL0
+ alternative_insn "b .L_skip_tramp_exit_\@", nop, ARM64_UNMAP_KERNEL_AT_EL0
+
bne 4f
msr far_el1, x29
tramp_alias x30, tramp_exit_native, x29
@@ -436,7 +429,18 @@ alternative_else_nop_endif
4:
tramp_alias x30, tramp_exit_compat, x29
br x30
+
+.L_skip_tramp_exit_\@:
#endif
+ ldr lr, [sp, #S_LR]
+ add sp, sp, #PT_REGS_SIZE // restore sp
+
+ /* This must be after the last explicit memory access */
+alternative_if ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD
+ tlbi vale1, xzr
+ dsb nsh
+alternative_else_nop_endif
+ eret
.else
ldr lr, [sp, #S_LR]
add sp, sp, #PT_REGS_SIZE // restore sp
diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c
index 3025d14c253d..6fcbde1ed943 100644
--- a/arch/arm64/kernel/ptrace.c
+++ b/arch/arm64/kernel/ptrace.c
@@ -564,6 +564,42 @@ static int gpr_get(struct task_struct *target,
return membuf_write(&to, uregs, sizeof(*uregs));
}
+static void update_syscall_orig_x0_after_ptrace(struct task_struct *target)
+{
+ struct pt_regs *regs = task_pt_regs(target);
+ struct kernel_siginfo *info = target->last_siginfo;
+
+ /*
+ * Skip the update for NO_SYSCALL (set either by the user or the
+ * tracer), as regs[0] holds the return value (see the comment in
+ * el0_svc_common()) and can be unwound using syscall_rollback().
+ */
+ if (regs->syscallno == NO_SYSCALL)
+ return;
+
+ /* We should only be called when target is in a ptrace stop */
+ if (WARN_ON_ONCE(!info))
+ return;
+
+ /*
+ * For compat tasks, orig_r0 is provided directly through GPR index
+ * 17.
+ */
+ if (is_compat_thread(task_thread_info(target)))
+ return;
+
+ /*
+ * Don't update orig_x0 for a syscall-exit-stop, as x0 now contains the
+ * return value of the system call.
+ */
+ if ((info->si_code & ~0x80) == SIGTRAP &&
+ target->ptrace_message == PTRACE_EVENTMSG_SYSCALL_EXIT) {
+ return;
+ }
+
+ regs->orig_x0 = regs->regs[0];
+}
+
static int gpr_set(struct task_struct *target, const struct user_regset *regset,
unsigned int pos, unsigned int count,
const void *kbuf, const void __user *ubuf)
@@ -579,6 +615,14 @@ static int gpr_set(struct task_struct *target, const struct user_regset *regset,
return -EINVAL;
task_pt_regs(target)->user_regs = newregs;
+
+ /*
+ * Keep orig_x0 authoritative so that seccomp (via
+ * syscall_get_arguments()), audit and the restart path all see the same
+ * first argument the syscall is dispatched with, even if it has been
+ * updated by a tracer.
+ */
+ update_syscall_orig_x0_after_ptrace(target);
return 0;
}
@@ -722,6 +766,12 @@ static int system_call_set(struct task_struct *target,
return ret;
task_pt_regs(target)->syscallno = syscallno;
+
+ /*
+ * Re-sync orig_x0 in case the syscall number has been changed
+ * from NO_SYSCALL.
+ */
+ update_syscall_orig_x0_after_ptrace(target);
return ret;
}
diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c
index 9fd7f78d11d1..b00bf1a41540 100644
--- a/arch/arm64/kernel/signal.c
+++ b/arch/arm64/kernel/signal.c
@@ -271,22 +271,37 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user)
unsigned int vl, vq;
struct user_fpsimd_state fpsimd;
struct sve_context sve;
+ bool fpsimd_only;
if (__copy_from_user(&sve, user->sve, sizeof(sve)))
return -EFAULT;
+ fpsimd_only = (sve.head.size <= sizeof(*user->sve));
+
if (sve.flags & SVE_SIG_FLAG_SM) {
if (!system_supports_sme())
return -EINVAL;
+ /*
+ * Streaming SVE state is always preserved with an SVE payload.
+ * Only accept streaming state which has an SVE payload.
+ */
+ if (fpsimd_only)
+ return -EINVAL;
+
vl = task_get_sme_vl(current);
} else {
/*
- * A SME only system use SVE for streaming mode so can
- * have a SVE formatted context with a zero VL and no
- * payload data.
+ * Non-streaming SVE state may be preserved without an SVE
+ * payload, in which case all state is saved in the FPSIMD
+ * context.
+ *
+ * On SME-only systems, non-streaming (FPSIMD-only) state is
+ * always preserved without an SVE payload, and with VL==0. On
+ * such systems, only accept non-streaming state without an SVE
+ * payload.
*/
- if (!system_supports_sve() && !system_supports_sme())
+ if (!system_supports_sve() && !fpsimd_only)
return -EINVAL;
vl = task_get_sve_vl(current);
@@ -295,7 +310,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user)
if (sve.vl != vl)
return -EINVAL;
- if (sve.head.size <= sizeof(*user->sve)) {
+ if (fpsimd_only) {
clear_thread_flag(TIF_SVE);
current->thread.svcr &= ~SVCR_SM_MASK;
current->thread.fp_type = FP_STATE_FPSIMD;
diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c
index 817d788cd866..61df0918585b 100644
--- a/arch/arm64/kernel/topology.c
+++ b/arch/arm64/kernel/topology.c
@@ -268,12 +268,13 @@ static void cpu_read_corecnt(void *val)
static void cpu_read_constcnt(void *val)
{
/*
- * Return 0 if the current CPU is affected by erratum 2457168. A value
- * of 0 is also returned if the current CPU does not support AMUs or if
- * the counter is disabled. A return value of 0 at counter read is
- * properly handled as an error case by the users of the counter.
+ * Return 0 if the current CPU is affected by a HW erratum.
+ * A value of 0 is also returned if the current CPU does not
+ * support AMUs or if the counter is disabled. A return
+ * value of 0 at counter read is properly handled as an error
+ * case by the users of the counter.
*/
- *(u64 *)val = this_cpu_has_cap(ARM64_WORKAROUND_2457168) ?
+ *(u64 *)val = this_cpu_has_cap(ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT) ?
0UL : read_constcnt();
}
diff --git a/arch/arm64/kvm/arch_timer.c b/arch/arm64/kvm/arch_timer.c
index bb24a76b4224..52df63aa73a1 100644
--- a/arch/arm64/kvm/arch_timer.c
+++ b/arch/arm64/kvm/arch_timer.c
@@ -161,8 +161,7 @@ static void get_timer_map(struct kvm_vcpu *vcpu, struct timer_map *map)
static inline bool userspace_irqchip(struct kvm *kvm)
{
- return static_branch_unlikely(&userspace_irqchip_in_use) &&
- unlikely(!irqchip_in_kernel(kvm));
+ return unlikely(!irqchip_in_kernel(kvm));
}
static void soft_timer_start(struct hrtimer *hrt, u64 ns)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index b589ed95c89e..4ef70acabb9e 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -56,7 +56,6 @@ DECLARE_KVM_NVHE_PER_CPU(struct kvm_nvhe_init_params, kvm_init_params);
static bool vgic_present;
static DEFINE_PER_CPU(unsigned char, kvm_arm_hardware_enabled);
-DEFINE_STATIC_KEY_FALSE(userspace_irqchip_in_use);
int kvm_arch_vcpu_should_kick(struct kvm_vcpu *vcpu)
{
@@ -384,9 +383,6 @@ void kvm_arch_vcpu_postcreate(struct kvm_vcpu *vcpu)
void kvm_arch_vcpu_destroy(struct kvm_vcpu *vcpu)
{
- if (vcpu_has_run_once(vcpu) && unlikely(!irqchip_in_kernel(vcpu->kvm)))
- static_branch_dec(&userspace_irqchip_in_use);
-
kvm_mmu_free_memory_cache(&vcpu->arch.mmu_page_cache);
kvm_timer_vcpu_terminate(vcpu);
kvm_pmu_vcpu_destroy(vcpu);
@@ -609,14 +605,6 @@ int kvm_arch_vcpu_run_pid_change(struct kvm_vcpu *vcpu)
if (ret)
return ret;
- if (!irqchip_in_kernel(kvm)) {
- /*
- * Tell the rest of the code that there are userspace irqchip
- * VMs in the wild.
- */
- static_branch_inc(&userspace_irqchip_in_use);
- }
-
/*
* Initialize traps for protected VMs.
* NOTE: Move to run in EL2 directly, rather than via a hypercall, once
@@ -829,7 +817,7 @@ static bool kvm_vcpu_exit_request(struct kvm_vcpu *vcpu, int *ret)
* state gets updated in kvm_timer_update_run and
* kvm_pmu_update_run below).
*/
- if (static_branch_unlikely(&userspace_irqchip_in_use)) {
+ if (unlikely(!irqchip_in_kernel(vcpu->kvm))) {
if (kvm_timer_should_notify_user(vcpu) ||
kvm_pmu_should_notify_user(vcpu)) {
*ret = -EINTR;
@@ -948,7 +936,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
vcpu->mode = OUTSIDE_GUEST_MODE;
isb(); /* Ensure work in x_flush_hwstate is committed */
kvm_pmu_sync_hwstate(vcpu);
- if (static_branch_unlikely(&userspace_irqchip_in_use))
+ if (unlikely(!irqchip_in_kernel(vcpu->kvm)))
kvm_timer_sync_user(vcpu);
kvm_vgic_sync_hwstate(vcpu);
local_irq_enable();
@@ -994,7 +982,7 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu)
* we don't want vtimer interrupts to race with syncing the
* timer virtual interrupt state.
*/
- if (static_branch_unlikely(&userspace_irqchip_in_use))
+ if (unlikely(!irqchip_in_kernel(vcpu->kvm)))
kvm_timer_sync_user(vcpu);
kvm_arch_vcpu_ctxsync_fp(vcpu);
diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 54ccdcc2dbdf..2ccc62379642 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1077,32 +1077,11 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot,
return PAGE_SIZE;
}
-static int get_vma_page_shift(struct vm_area_struct *vma, unsigned long hva)
+static int get_vma_page_shift(struct vm_area_struct *vma)
{
- unsigned long pa;
-
if (is_vm_hugetlb_page(vma) && !(vma->vm_flags & VM_PFNMAP))
return huge_page_shift(hstate_vma(vma));
- if (!(vma->vm_flags & VM_PFNMAP))
- return PAGE_SHIFT;
-
- VM_BUG_ON(is_vm_hugetlb_page(vma));
-
- pa = (vma->vm_pgoff << PAGE_SHIFT) + (hva - vma->vm_start);
-
-#ifndef __PAGETABLE_PMD_FOLDED
- if ((hva & (PUD_SIZE - 1)) == (pa & (PUD_SIZE - 1)) &&
- ALIGN_DOWN(hva, PUD_SIZE) >= vma->vm_start &&
- ALIGN(hva, PUD_SIZE) <= vma->vm_end)
- return PUD_SHIFT;
-#endif
-
- if ((hva & (PMD_SIZE - 1)) == (pa & (PMD_SIZE - 1)) &&
- ALIGN_DOWN(hva, PMD_SIZE) >= vma->vm_start &&
- ALIGN(hva, PMD_SIZE) <= vma->vm_end)
- return PMD_SHIFT;
-
return PAGE_SHIFT;
}
@@ -1214,7 +1193,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa,
use_read_lock = (fault_status == FSC_PERM && write_fault &&
fault_granule == PAGE_SIZE);
} else {
- vma_shift = get_vma_page_shift(vma, hva);
+ vma_shift = get_vma_page_shift(vma);
}
shared = (vma->vm_flags & VM_SHARED);
diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c
index 15c047d7539d..88a01ceb8ce2 100644
--- a/arch/arm64/kvm/vgic/vgic-its.c
+++ b/arch/arm64/kvm/vgic/vgic-its.c
@@ -2218,6 +2218,14 @@ static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev,
u32 next_offset;
u64 val;
+ /*
+ * MAPC with V=0 keeps the ITEs mapped but drops their collection,
+ * and with it the ICID. Save a zeroed entry, which the restore path
+ * reads back as invalid.
+ */
+ if (!ite->collection)
+ return vgic_its_write_entry_lock(its, gpa, 0ULL, ite_esz);
+
next_offset = compute_next_eventid_offset(&dev->itt_head, ite);
val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) |
((u64)ite->irq->intid << KVM_ITS_ITE_PINTID_SHIFT) |
diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c
index 832fa90f48e8..7c49b113b1f6 100644
--- a/arch/arm64/kvm/vgic/vgic.c
+++ b/arch/arm64/kvm/vgic/vgic.c
@@ -160,6 +160,7 @@ void vgic_flush_pending_lpis(struct kvm_vcpu *vcpu)
list_for_each_entry_safe(irq, tmp, &vgic_cpu->ap_list_head, ap_list) {
if (irq->intid >= VGIC_MIN_LPI) {
raw_spin_lock(&irq->irq_lock);
+ irq->pending_latch = false;
list_del(&irq->ap_list);
irq->vcpu = NULL;
raw_spin_unlock(&irq->irq_lock);
@@ -663,7 +664,11 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu)
continue;
}
- /* This interrupt looks like it has to be migrated. */
+ /*
+ * This interrupt looks like it has to be migrated,
+ * make sure it is kept alive while locks are dropped.
+ */
+ vgic_get_irq_kref(irq);
raw_spin_unlock(&irq->irq_lock);
raw_spin_unlock(&vgic_cpu->ap_list_lock);
@@ -708,6 +713,8 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu)
raw_spin_unlock(&vcpuB->arch.vgic_cpu.ap_list_lock);
raw_spin_unlock(&vcpuA->arch.vgic_cpu.ap_list_lock);
+ vgic_put_irq(vcpu->kvm, irq);
+
if (target_vcpu_needs_kick) {
kvm_make_request(KVM_REQ_IRQ_PENDING, target_vcpu);
kvm_vcpu_kick(target_vcpu);
diff --git a/arch/arm64/mm/fault.c b/arch/arm64/mm/fault.c
index 0776c98ad27f..5bd5e10c49f1 100644
--- a/arch/arm64/mm/fault.c
+++ b/arch/arm64/mm/fault.c
@@ -15,6 +15,7 @@
#include <linux/mm.h>
#include <linux/hardirq.h>
#include <linux/init.h>
+#include <linux/irqflags.h>
#include <linux/kasan.h>
#include <linux/kprobes.h>
#include <linux/uaccess.h>
@@ -145,6 +146,9 @@ static void show_pte(unsigned long addr)
pr_alert("%s pgtable: %luk pages, %llu-bit VAs, pgdp=%016lx\n",
mm == &init_mm ? "swapper" : "user", PAGE_SIZE / SZ_1K,
vabits_actual, mm_to_pgd_phys(mm));
+
+ guard(irqsave)();
+
pgdp = pgd_offset(mm, addr);
pgd = READ_ONCE(*pgdp);
pr_alert("[%016lx] pgd=%016llx", addr, pgd_val(pgd));
@@ -158,25 +162,28 @@ static void show_pte(unsigned long addr)
if (pgd_none(pgd) || pgd_bad(pgd))
break;
- p4dp = p4d_offset(pgdp, addr);
+ p4dp = p4d_offset_lockless(pgdp, pgd, addr);
p4d = READ_ONCE(*p4dp);
pr_cont(", p4d=%016llx", p4d_val(p4d));
if (p4d_none(p4d) || p4d_bad(p4d))
break;
- pudp = pud_offset(p4dp, addr);
+ pudp = pud_offset_lockless(p4dp, p4d, addr);
pud = READ_ONCE(*pudp);
pr_cont(", pud=%016llx", pud_val(pud));
if (pud_none(pud) || pud_bad(pud))
break;
- pmdp = pmd_offset(pudp, addr);
+ pmdp = pmd_offset_lockless(pudp, pud, addr);
pmd = READ_ONCE(*pmdp);
pr_cont(", pmd=%016llx", pmd_val(pmd));
if (pmd_none(pmd) || pmd_bad(pmd))
break;
- ptep = pte_offset_map(pmdp, addr);
+ ptep = pte_offset_map(&pmd, addr);
+ if (!ptep)
+ break;
+
pte = READ_ONCE(*ptep);
pr_cont(", pte=%016llx", pte_val(pte));
pte_unmap(ptep);
diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c
index e66218aed831..1a8627ab4f2c 100644
--- a/arch/arm64/net/bpf_jit_comp.c
+++ b/arch/arm64/net/bpf_jit_comp.c
@@ -1589,7 +1589,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog)
extable_offset = round_up(prog_size + PLT_TARGET_SIZE, extable_align);
image_size = extable_offset + extable_size;
header = bpf_jit_binary_alloc(image_size, &image_ptr,
- sizeof(u32), jit_fill_hole);
+ sizeof(u64), jit_fill_hole);
if (header == NULL) {
prog = orig_prog;
goto out_off;
diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps
index f5c9a4f5f722..f609e8d93937 100644
--- a/arch/arm64/tools/cpucaps
+++ b/arch/arm64/tools/cpucaps
@@ -69,7 +69,6 @@ WORKAROUND_1902691
WORKAROUND_2038923
WORKAROUND_2064142
WORKAROUND_2077057
-WORKAROUND_2457168
WORKAROUND_2658417
WORKAROUND_AMPERE_AC03_CPU_38
WORKAROUND_TRBE_OVERWRITE_FILL_MODE
@@ -88,3 +87,4 @@ WORKAROUND_REPEAT_TLBI
WORKAROUND_SPECULATIVE_AT
WORKAROUND_SPECULATIVE_SSBS
WORKAROUND_SPECULATIVE_UNPRIV_LOAD
+WORKAROUND_BROKEN_AMU_CONSTCNT
diff --git a/arch/loongarch/kernel/machine_kexec.c b/arch/loongarch/kernel/machine_kexec.c
index ce4e132ac137..7c93e9d1f020 100644
--- a/arch/loongarch/kernel/machine_kexec.c
+++ b/arch/loongarch/kernel/machine_kexec.c
@@ -62,6 +62,7 @@ static void kexec_image_info(const struct kimage *kimage)
int machine_kexec_prepare(struct kimage *kimage)
{
int i;
+ char head[8];
char *bootloader = "kexec";
void *cmdline_ptr = (void *)KEXEC_CMDLINE_ADDR;
@@ -72,7 +73,9 @@ int machine_kexec_prepare(struct kimage *kimage)
/* Find the command line */
for (i = 0; i < kimage->nr_segments; i++) {
- if (!strncmp(bootloader, (char __user *)kimage->segment[i].buf, strlen(bootloader))) {
+ if (copy_from_user(head, kimage->segment[i].buf, strlen(bootloader)))
+ continue;
+ if (!strncmp(bootloader, head, strlen(bootloader))) {
if (!copy_from_user(cmdline_ptr, kimage->segment[i].buf, COMMAND_LINE_SIZE))
kimage->arch.cmdline_ptr = (unsigned long)cmdline_ptr;
break;
diff --git a/arch/loongarch/kernel/module-sections.c b/arch/loongarch/kernel/module-sections.c
index d296a70b758f..10969d242f44 100644
--- a/arch/loongarch/kernel/module-sections.c
+++ b/arch/loongarch/kernel/module-sections.c
@@ -6,6 +6,7 @@
#include <linux/elf.h>
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/sort.h>
Elf_Addr module_emit_got_entry(struct module *mod, Elf_Addr val)
{
@@ -59,39 +60,38 @@ Elf_Addr module_emit_plt_entry(struct module *mod, Elf_Addr val)
return (Elf_Addr)&plt[nr];
}
-static int is_rela_equal(const Elf_Rela *x, const Elf_Rela *y)
-{
- return x->r_info == y->r_info && x->r_addend == y->r_addend;
-}
+#define cmp_3way(a, b) ((a) < (b) ? -1 : (a) > (b))
-static bool duplicate_rela(const Elf_Rela *rela, int idx)
+static int compare_rela(const void *x, const void *y)
{
- int i;
+ int ret;
+ const Elf_Rela *rela_x = x, *rela_y = y;
- for (i = 0; i < idx; i++) {
- if (is_rela_equal(&rela[i], &rela[idx]))
- return true;
- }
+ ret = cmp_3way(rela_x->r_info, rela_y->r_info);
+ if (ret == 0)
+ ret = cmp_3way(rela_x->r_addend, rela_y->r_addend);
- return false;
+ return ret;
}
static void count_max_entries(Elf_Rela *relas, int num,
unsigned int *plts, unsigned int *gots)
{
- unsigned int i, type;
+ unsigned int i;
+
+ sort(relas, num, sizeof(Elf_Rela), compare_rela, NULL);
for (i = 0; i < num; i++) {
- type = ELF_R_TYPE(relas[i].r_info);
- switch (type) {
+ if (i && !compare_rela(&relas[i-1], &relas[i]))
+ continue;
+
+ switch (ELF_R_TYPE(relas[i].r_info)) {
case R_LARCH_SOP_PUSH_PLT_PCREL:
case R_LARCH_B26:
- if (!duplicate_rela(relas, i))
- (*plts)++;
+ (*plts)++;
break;
case R_LARCH_GOT_PC_HI20:
- if (!duplicate_rela(relas, i))
- (*gots)++;
+ (*gots)++;
break;
default:
break; /* Do nothing. */
diff --git a/arch/loongarch/kernel/setup.c b/arch/loongarch/kernel/setup.c
index ed47cf6714df..439658695f7b 100644
--- a/arch/loongarch/kernel/setup.c
+++ b/arch/loongarch/kernel/setup.c
@@ -428,6 +428,7 @@ void __init setup_arch(char **cmdline_p)
efi_init();
memblock_init();
pagetable_init();
+ jump_label_init(); /* Initialise the static keys for early params */
parse_early_param();
reserve_initrd_mem();
diff --git a/arch/loongarch/kernel/smp.c b/arch/loongarch/kernel/smp.c
index 356918d5aa28..671e660746c0 100644
--- a/arch/loongarch/kernel/smp.c
+++ b/arch/loongarch/kernel/smp.c
@@ -89,6 +89,77 @@ void show_ipi_list(struct seq_file *p, int prec)
}
}
+static inline void set_cpu_core_map(int cpu)
+{
+ int i;
+
+ cpumask_set_cpu(cpu, &cpu_core_setup_map);
+
+ for_each_cpu(i, &cpu_core_setup_map) {
+ if (cpu_data[cpu].package == cpu_data[i].package) {
+ cpumask_set_cpu(i, &cpu_core_map[cpu]);
+ cpumask_set_cpu(cpu, &cpu_core_map[i]);
+ }
+ }
+}
+
+static inline void set_cpu_sibling_map(int cpu)
+{
+ int i;
+
+ cpumask_set_cpu(cpu, &cpu_sibling_setup_map);
+
+ if (smp_num_siblings <= 1)
+ cpumask_set_cpu(cpu, &cpu_sibling_map[cpu]);
+ else {
+ for_each_cpu(i, &cpu_sibling_setup_map) {
+ if (cpus_are_siblings(cpu, i)) {
+ cpumask_set_cpu(i, &cpu_sibling_map[cpu]);
+ cpumask_set_cpu(cpu, &cpu_sibling_map[i]);
+ }
+ }
+ }
+}
+
+static inline void clear_cpu_sibling_map(int cpu)
+{
+ int i;
+
+ for_each_cpu(i, &cpu_sibling_setup_map) {
+ if (cpus_are_siblings(cpu, i)) {
+ cpumask_clear_cpu(i, &cpu_sibling_map[cpu]);
+ cpumask_clear_cpu(cpu, &cpu_sibling_map[i]);
+ }
+ }
+
+ cpumask_clear_cpu(cpu, &cpu_sibling_setup_map);
+}
+
+/*
+ * Calculate a new cpu_foreign_map mask whenever a
+ * new cpu appears or disappears.
+ */
+void calculate_cpu_foreign_map(void)
+{
+ int i, k, core_present;
+ cpumask_t temp_foreign_map;
+
+ /* Re-calculate the mask */
+ cpumask_clear(&temp_foreign_map);
+ for_each_online_cpu(i) {
+ core_present = 0;
+ for_each_cpu(k, &temp_foreign_map)
+ if (cpus_are_siblings(i, k))
+ core_present = 1;
+ if (!core_present)
+ cpumask_set_cpu(i, &temp_foreign_map);
+ }
+
+ for_each_online_cpu(i)
+ cpumask_andnot(&cpu_foreign_map[i],
+ &temp_foreign_map, &cpu_sibling_map[i]);
+}
+
/* Send mailbox buffer via Mail_Send */
static void csr_mail_send(uint64_t data, int cpu, int mailbox)
{
@@ -263,6 +334,7 @@ int loongson_cpu_disable(void)
numa_remove_cpu(cpu);
#endif
set_cpu_online(cpu, false);
+ clear_cpu_sibling_map(cpu);
calculate_cpu_foreign_map();
local_irq_save(flags);
irq_migrate_all_off_this_cpu();
@@ -340,63 +412,6 @@ static int __init ipi_pm_init(void)
core_initcall(ipi_pm_init);
#endif
-static inline void set_cpu_sibling_map(int cpu)
-{
- int i;
-
- cpumask_set_cpu(cpu, &cpu_sibling_setup_map);
-
- if (smp_num_siblings <= 1)
- cpumask_set_cpu(cpu, &cpu_sibling_map[cpu]);
- else {
- for_each_cpu(i, &cpu_sibling_setup_map) {
- if (cpus_are_siblings(cpu, i)) {
- cpumask_set_cpu(i, &cpu_sibling_map[cpu]);
- cpumask_set_cpu(cpu, &cpu_sibling_map[i]);
- }
- }
- }
-}
-
-static inline void set_cpu_core_map(int cpu)
-{
- int i;
-
- cpumask_set_cpu(cpu, &cpu_core_setup_map);
-
- for_each_cpu(i, &cpu_core_setup_map) {
- if (cpu_data[cpu].package == cpu_data[i].package) {
- cpumask_set_cpu(i, &cpu_core_map[cpu]);
- cpumask_set_cpu(cpu, &cpu_core_map[i]);
- }
- }
-}
-
-/*
- * Calculate a new cpu_foreign_map mask whenever a
- * new cpu appears or disappears.
- */
-void calculate_cpu_foreign_map(void)
-{
- int i, k, core_present;
- cpumask_t temp_foreign_map;
-
- /* Re-calculate the mask */
- cpumask_clear(&temp_foreign_map);
- for_each_online_cpu(i) {
- core_present = 0;
- for_each_cpu(k, &temp_foreign_map)
- if (cpus_are_siblings(i, k))
- core_present = 1;
- if (!core_present)
- cpumask_set_cpu(i, &temp_foreign_map);
- }
-
- for_each_online_cpu(i)
- cpumask_andnot(&cpu_foreign_map[i],
- &temp_foreign_map, &cpu_sibling_map[i]);
-}
-
/* Preload SMP state for boot cpu */
void smp_prepare_boot_cpu(void)
{
diff --git a/arch/loongarch/mm/hugetlbpage.c b/arch/loongarch/mm/hugetlbpage.c
index 70b4a51885c2..9bc1ac6ad762 100644
--- a/arch/loongarch/mm/hugetlbpage.c
+++ b/arch/loongarch/mm/hugetlbpage.c
@@ -47,7 +47,8 @@ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr,
pmd = pmd_offset(pud, addr);
}
}
- return pmd_none(*pmd) ? NULL : (pte_t *) pmd;
+
+ return (!pmd || pmd_none(*pmd)) ? NULL : (pte_t *) pmd;
}
/*
diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c
index bd601c269a94..88772302f952 100644
--- a/arch/loongarch/net/bpf_jit.c
+++ b/arch/loongarch/net/bpf_jit.c
@@ -775,7 +775,10 @@ static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, bool ext
move_addr(ctx, t1, func_addr);
emit_insn(ctx, jirl, t1, LOONGARCH_GPR_RA, 0);
- move_reg(ctx, regmap[BPF_REG_0], LOONGARCH_GPR_A0);
+
+ if (insn->src_reg != BPF_PSEUDO_CALL)
+ move_reg(ctx, regmap[BPF_REG_0], LOONGARCH_GPR_A0);
+
break;
/* tail call */
diff --git a/arch/mips/Kbuild b/arch/mips/Kbuild
index 9e8071f0e58f..ed09e894749b 100644
--- a/arch/mips/Kbuild
+++ b/arch/mips/Kbuild
@@ -1,10 +1,4 @@
# SPDX-License-Identifier: GPL-2.0
-# Fail on warnings - also for files referenced in subdirs
-# -Werror can be disabled for specific files using:
-# CFLAGS_<file.o> := -Wno-error
-ifeq ($(W),)
-subdir-ccflags-y := -Werror
-endif
# platform specific definitions
include arch/mips/Kbuild.platforms
diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h
index 296bcf31abb5..b43bfd445252 100644
--- a/arch/mips/include/asm/uasm.h
+++ b/arch/mips/include/asm/uasm.h
@@ -193,9 +193,7 @@ struct uasm_label {
void uasm_build_label(struct uasm_label **lab, u32 *addr,
int lid);
-#ifdef CONFIG_64BIT
int uasm_in_compat_space_p(long addr);
-#endif
int uasm_rel_hi(long val);
int uasm_rel_lo(long val);
void UASM_i_LA_mostly(u32 **buf, unsigned int rs, long addr);
diff --git a/arch/mips/kernel/ftrace.c b/arch/mips/kernel/ftrace.c
index f39e85fd58fa..b15615b28569 100644
--- a/arch/mips/kernel/ftrace.c
+++ b/arch/mips/kernel/ftrace.c
@@ -54,10 +54,20 @@ static inline void ftrace_dyn_arch_init_insns(void)
u32 *buf;
unsigned int v1;
- /* la v1, _mcount */
- v1 = 3;
- buf = (u32 *)&insn_la_mcount[0];
- UASM_i_LA(&buf, v1, MCOUNT_ADDR);
+ /* If we are not in compat space, the number of generated
+ * instructions will exceed the maximum expected limit of 2.
+ * To prevent buffer overflow, we avoid generating them.
+ * insn_la_mcount will not be used later in ftrace_make_call.
+ */
+ if (uasm_in_compat_space_p(MCOUNT_ADDR)) {
+ /* la v1, _mcount */
+ v1 = 3;
+ buf = (u32 *)&insn_la_mcount[0];
+ UASM_i_LA(&buf, v1, MCOUNT_ADDR);
+ } else {
+ pr_warn("ftrace: mcount address beyond 32 bits is not supported (%lX)\n",
+ MCOUNT_ADDR);
+ }
/* jal (ftrace_caller + 8), jump over the first two instruction */
buf = (u32 *)&insn_jal_ftrace_caller;
@@ -189,6 +199,13 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr)
unsigned int new;
unsigned long ip = rec->ip;
+ /* When the code to patch does not belong to the kernel code
+ * space, we must use insn_la_mcount. However, if MCOUNT_ADDR
+ * is not in compat space, insn_la_mcount is not usable.
+ */
+ if (!core_kernel_text(ip) && !uasm_in_compat_space_p(MCOUNT_ADDR))
+ return -EFAULT;
+
new = core_kernel_text(ip) ? insn_jal_ftrace_caller : insn_la_mcount[0];
#ifdef CONFIG_64BIT
diff --git a/arch/nios2/include/asm/checksum.h b/arch/nios2/include/asm/checksum.h
index 69004e07a1ba..b6b415ae17a0 100644
--- a/arch/nios2/include/asm/checksum.h
+++ b/arch/nios2/include/asm/checksum.h
@@ -27,7 +27,7 @@ static inline __sum16 csum_fold(__wsum sum)
"add %0, %0, r8\n"
"nor %0, %0, %0\n"
: "=r" (sum)
- : "r" (sum << 16), "0" (sum)
+ : "r" ((__force u32)sum << 16), "0" (sum)
: "r8");
return (__force __sum16) sum;
}
diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h
index fdbe0b381f3a..c51473f290dc 100644
--- a/arch/powerpc/include/asm/book3s/64/pgtable.h
+++ b/arch/powerpc/include/asm/book3s/64/pgtable.h
@@ -1285,6 +1285,22 @@ extern int pmdp_test_and_clear_young(struct vm_area_struct *vma,
static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm,
unsigned long addr, pmd_t *pmdp)
{
+ pmd_t old_pmd;
+
+ /*
+ * Non-present PMDs can be migration entries or device-private THP
+ * entries. This can happen at 2 places:
+ * - When the address space is being unmapped zap_huge_pmd(), and we
+ * encounter non-present pmds.
+ * - migrate_vma_collect_huge_pmd() could calls this during migration
+ * of device-private pmd entries.
+ */
+ if (!pmd_present(*pmdp)) {
+ old_pmd = READ_ONCE(*pmdp);
+ pmd_clear(pmdp);
+ return old_pmd;
+ }
+
if (radix_enabled())
return radix__pmdp_huge_get_and_clear(mm, addr, pmdp);
return hash__pmdp_huge_get_and_clear(mm, addr, pmdp);
diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h
index 92930b0b5d0e..8ac7c1d3564f 100644
--- a/arch/powerpc/include/asm/xive.h
+++ b/arch/powerpc/include/asm/xive.h
@@ -91,7 +91,7 @@ static inline bool xive_enabled(void) { return __xive_enabled; }
bool xive_spapr_init(void);
bool xive_native_init(void);
-void xive_smp_probe(void);
+int xive_smp_probe(void);
int xive_smp_prepare_cpu(unsigned int cpu);
void xive_smp_setup_cpu(void);
void xive_smp_disable_cpu(void);
@@ -154,7 +154,7 @@ static inline bool xive_enabled(void) { return false; }
static inline bool xive_spapr_init(void) { return false; }
static inline bool xive_native_init(void) { return false; }
-static inline void xive_smp_probe(void) { }
+static inline int xive_smp_probe(void) { return -EINVAL; }
static inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; }
static inline void xive_smp_setup_cpu(void) { }
static inline void xive_smp_disable_cpu(void) { }
diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c
index 81bd60c8094c..e10df1b5b630 100644
--- a/arch/powerpc/kernel/iommu.c
+++ b/arch/powerpc/kernel/iommu.c
@@ -685,7 +685,7 @@ static void iommu_table_clear(struct iommu_table *tbl)
static void iommu_table_reserve_pages(struct iommu_table *tbl,
unsigned long res_start, unsigned long res_end)
{
- int i;
+ unsigned long i;
WARN_ON_ONCE(res_end < res_start);
/*
diff --git a/arch/powerpc/kernel/setup-common.c b/arch/powerpc/kernel/setup-common.c
index 41c03c8e41b4..fa8c14e890c8 100644
--- a/arch/powerpc/kernel/setup-common.c
+++ b/arch/powerpc/kernel/setup-common.c
@@ -600,7 +600,6 @@ struct seq_buf ppc_hw_desc __initdata = {
.buffer = ppc_hw_desc_buf,
.size = sizeof(ppc_hw_desc_buf),
.len = 0,
- .readpos = 0,
};
static __init void probe_machine(void)
diff --git a/arch/powerpc/kexec/ranges.c b/arch/powerpc/kexec/ranges.c
index 563e9989a5bf..d169a862158c 100644
--- a/arch/powerpc/kexec/ranges.c
+++ b/arch/powerpc/kexec/ranges.c
@@ -20,6 +20,7 @@
#include <linux/kexec.h>
#include <linux/of_device.h>
#include <linux/slab.h>
+#include <linux/minmax.h>
#include <asm/sections.h>
#include <asm/kexec_ranges.h>
@@ -101,19 +102,16 @@ static void __merge_memory_ranges(struct crash_mem *mem_rngs)
struct crash_mem_range *ranges;
int i, idx;
- if (!mem_rngs)
+ if (!mem_rngs || mem_rngs->nr_ranges <= 1)
return;
idx = 0;
- ranges = &(mem_rngs->ranges[0]);
+ ranges = mem_rngs->ranges;
for (i = 1; i < mem_rngs->nr_ranges; i++) {
- if (ranges[i].start <= (ranges[i-1].end + 1))
- ranges[idx].end = ranges[i].end;
+ if (ranges[i].start <= (ranges[idx].end + 1))
+ ranges[idx].end = max(ranges[idx].end, ranges[i].end);
else {
idx++;
- if (i == idx)
- continue;
-
ranges[idx] = ranges[i];
}
}
diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c
index a3bce0895b7d..70488cc3ef11 100644
--- a/arch/powerpc/mm/book3s64/pgtable.c
+++ b/arch/powerpc/mm/book3s64/pgtable.c
@@ -134,16 +134,21 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma,
unsigned long addr, pmd_t *pmdp, int full)
{
pmd_t pmd;
+ bool was_present = pmd_present(*pmdp);
+
VM_BUG_ON(addr & ~HPAGE_PMD_MASK);
- VM_BUG_ON((pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) &&
- !pmd_devmap(*pmdp)) || !pmd_present(*pmdp));
+ VM_BUG_ON(was_present && !pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp));
+ /*
+ * Check pmdp_huge_get_and_clear() for non-present pmd case.
+ */
pmd = pmdp_huge_get_and_clear(vma->vm_mm, addr, pmdp);
/*
* if it not a fullmm flush, then we can possibly end up converting
* this PMD pte entry to a regular level 0 PTE by a parallel page fault.
- * Make sure we flush the tlb in this case.
+ * Make sure we flush the tlb in this case. TLB flush not needed for
+ * non-present case.
*/
- if (!full)
+ if (was_present && !full)
flush_pmd_tlb_range(vma, addr, addr + HPAGE_PMD_SIZE);
return pmd;
}
diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c
index 9e1a25398f98..4887f9176aa4 100644
--- a/arch/powerpc/platforms/powernv/smp.c
+++ b/arch/powerpc/platforms/powernv/smp.c
@@ -329,10 +329,12 @@ static void pnv_cause_ipi(int cpu)
static void __init pnv_smp_probe(void)
{
- if (xive_enabled())
- xive_smp_probe();
- else
+ if (xive_enabled()) {
+ if (xive_smp_probe() < 0)
+ return;
+ } else {
xics_smp_probe();
+ }
if (cpu_has_feature(CPU_FTR_DBELL)) {
ic_cause_ipi = smp_ops->cause_ipi;
diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c
index d5b5d1a4ec16..41e7400bc319 100644
--- a/arch/powerpc/platforms/pseries/iommu.c
+++ b/arch/powerpc/platforms/pseries/iommu.c
@@ -1510,7 +1510,8 @@ static bool enable_ddw(struct pci_dev *dev, struct device_node *pdn)
iommu_table_setparms_common(newtbl, pci->phb->bus->number, create.liobn, win_addr,
1UL << len, page_shift, NULL, &iommu_table_lpar_multi_ops);
- iommu_init_table(newtbl, pci->phb->node, start, end);
+ iommu_init_table(newtbl, pci->phb->node,
+ start >> page_shift, end >> page_shift);
pci->table_group->tables[1] = newtbl;
diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c
index fd2174edfa1d..52afd1427d1e 100644
--- a/arch/powerpc/platforms/pseries/smp.c
+++ b/arch/powerpc/platforms/pseries/smp.c
@@ -194,10 +194,12 @@ static int pseries_cause_nmi_ipi(int cpu)
static __init void pSeries_smp_probe(void)
{
- if (xive_enabled())
- xive_smp_probe();
- else
+ if (xive_enabled()) {
+ if (xive_smp_probe() < 0)
+ return;
+ } else {
xics_smp_probe();
+ }
/* No doorbell facility, must use the interrupt controller for IPIs */
if (!cpu_has_feature(CPU_FTR_DBELL))
diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c
index a289cb97c1d7..a5da09c1895f 100644
--- a/arch/powerpc/sysdev/xive/common.c
+++ b/arch/powerpc/sysdev/xive/common.c
@@ -1270,15 +1270,19 @@ noinstr static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc)
xive_ops->put_ipi(cpu, xc);
}
-void __init xive_smp_probe(void)
+int __init xive_smp_probe(void)
{
+ int ret;
+
smp_ops->cause_ipi = xive_cause_ipi;
/* Register the IPI */
- xive_init_ipis();
+ ret = xive_init_ipis();
+ if (ret < 0)
+ return ret;
/* Allocate and setup IPI for the boot CPU */
- xive_setup_cpu_ipi(smp_processor_id());
+ return xive_setup_cpu_ipi(smp_processor_id());
}
#endif /* CONFIG_SMP */
diff --git a/arch/riscv/include/asm/asm-prototypes.h b/arch/riscv/include/asm/asm-prototypes.h
index ef386fcf3939..4a06fa0f6493 100644
--- a/arch/riscv/include/asm/asm-prototypes.h
+++ b/arch/riscv/include/asm/asm-prototypes.h
@@ -25,7 +25,6 @@ DECLARE_DO_ERROR_INFO(do_trap_ecall_s);
DECLARE_DO_ERROR_INFO(do_trap_ecall_m);
DECLARE_DO_ERROR_INFO(do_trap_break);
-asmlinkage unsigned long get_overflow_stack(void);
asmlinkage void handle_bad_stack(struct pt_regs *regs);
#endif /* _ASM_RISCV_PROTOTYPES_H */
diff --git a/arch/riscv/include/asm/asm.h b/arch/riscv/include/asm/asm.h
index 816e753de636..c61f56fa3ff3 100644
--- a/arch/riscv/include/asm/asm.h
+++ b/arch/riscv/include/asm/asm.h
@@ -69,6 +69,7 @@
#endif
#ifdef __ASSEMBLY__
+#include <asm/asm-offsets.h>
/* Common assembly source macros */
@@ -81,6 +82,28 @@
.endr
.endm
+#ifdef CONFIG_SMP
+#ifdef CONFIG_32BIT
+#define PER_CPU_OFFSET_SHIFT 2
+#else
+#define PER_CPU_OFFSET_SHIFT 3
+#endif
+
+.macro asm_per_cpu dst sym tmp
+ lw \tmp, TASK_TI_CPU_NUM(tp)
+ slli \tmp, \tmp, PER_CPU_OFFSET_SHIFT
+ la \dst, __per_cpu_offset
+ add \dst, \dst, \tmp
+ REG_L \tmp, 0(\dst)
+ la \dst, \sym
+ add \dst, \dst, \tmp
+.endm
+#else /* CONFIG_SMP */
+.macro asm_per_cpu dst sym tmp
+ la \dst, \sym
+.endm
+#endif /* CONFIG_SMP */
+
#endif /* __ASSEMBLY__ */
#endif /* _ASM_RISCV_ASM_H */
diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h
index 94a0590c6971..2a22ee93c54c 100644
--- a/arch/riscv/include/asm/processor.h
+++ b/arch/riscv/include/asm/processor.h
@@ -39,6 +39,7 @@ struct thread_struct {
unsigned long s[12]; /* s[0]: frame pointer */
struct __riscv_d_ext_state fstate;
unsigned long bad_cause;
+ unsigned long status;
};
/* Whitelist the fstate from the task_struct for hardened usercopy */
diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/asm/thread_info.h
index f704c8dd57e0..e7572d0ea29a 100644
--- a/arch/riscv/include/asm/thread_info.h
+++ b/arch/riscv/include/asm/thread_info.h
@@ -38,13 +38,9 @@
#define THREAD_SHIFT (PAGE_SHIFT + THREAD_SIZE_ORDER)
#define OVERFLOW_STACK_SIZE SZ_4K
-#define SHADOW_OVERFLOW_STACK_SIZE (1024)
#ifndef __ASSEMBLY__
-extern long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE / sizeof(long)];
-extern unsigned long spin_shadow_stack;
-
#include <asm/processor.h>
#include <asm/csr.h>
diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c
index 21f034b3fdbe..1c228973971f 100644
--- a/arch/riscv/kernel/asm-offsets.c
+++ b/arch/riscv/kernel/asm-offsets.c
@@ -33,10 +33,12 @@ void asm_offsets(void)
OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]);
OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]);
OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags);
+ OFFSET(TASK_THREAD_STATUS, task_struct, thread.status);
OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count);
OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp);
OFFSET(TASK_TI_USER_SP, task_struct, thread_info.user_sp);
+ OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu);
OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]);
OFFSET(TASK_THREAD_F1, task_struct, thread.fstate.f[1]);
OFFSET(TASK_THREAD_F2, task_struct, thread.fstate.f[2]);
@@ -330,6 +332,10 @@ void asm_offsets(void)
offsetof(struct task_struct, thread.s[11])
- offsetof(struct task_struct, thread.ra)
);
+ DEFINE(TASK_THREAD_STATUS_RA,
+ offsetof(struct task_struct, thread.status)
+ - offsetof(struct task_struct, thread.ra)
+ );
DEFINE(TASK_THREAD_F0_F0,
offsetof(struct task_struct, thread.fstate.f[0])
diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S
index 99d38fdf8b18..2f8bc50c3b4a 100644
--- a/arch/riscv/kernel/entry.S
+++ b/arch/riscv/kernel/entry.S
@@ -10,9 +10,11 @@
#include <asm/asm.h>
#include <asm/csr.h>
#include <asm/unistd.h>
+#include <asm/page.h>
#include <asm/thread_info.h>
#include <asm/asm-offsets.h>
#include <asm/errata_list.h>
+#include <linux/sizes.h>
#if !IS_ENABLED(CONFIG_PREEMPTION)
.set resume_kernel, restore_all
@@ -397,67 +399,15 @@ handle_syscall_trace_exit:
#ifdef CONFIG_VMAP_STACK
handle_kernel_stack_overflow:
- /*
- * Takes the psuedo-spinlock for the shadow stack, in case multiple
- * harts are concurrently overflowing their kernel stacks. We could
- * store any value here, but since we're overflowing the kernel stack
- * already we only have SP to use as a scratch register. So we just
- * swap in the address of the spinlock, as that's definately non-zero.
- *
- * Pairs with a store_release in handle_bad_stack().
- */
-1: la sp, spin_shadow_stack
- REG_AMOSWAP_AQ sp, sp, (sp)
- bnez sp, 1b
-
- la sp, shadow_stack
- addi sp, sp, SHADOW_OVERFLOW_STACK_SIZE
-
- //save caller register to shadow stack
- addi sp, sp, -(PT_SIZE_ON_STACK)
- REG_S x1, PT_RA(sp)
- REG_S x5, PT_T0(sp)
- REG_S x6, PT_T1(sp)
- REG_S x7, PT_T2(sp)
- REG_S x10, PT_A0(sp)
- REG_S x11, PT_A1(sp)
- REG_S x12, PT_A2(sp)
- REG_S x13, PT_A3(sp)
- REG_S x14, PT_A4(sp)
- REG_S x15, PT_A5(sp)
- REG_S x16, PT_A6(sp)
- REG_S x17, PT_A7(sp)
- REG_S x28, PT_T3(sp)
- REG_S x29, PT_T4(sp)
- REG_S x30, PT_T5(sp)
- REG_S x31, PT_T6(sp)
-
- la ra, restore_caller_reg
- tail get_overflow_stack
+ /* we reach here from kernel context, sscratch must be 0 */
+ csrrw x31, CSR_SCRATCH, x31
+ asm_per_cpu sp, overflow_stack, x31
+ li x31, OVERFLOW_STACK_SIZE
+ add sp, sp, x31
+ /* zero out x31 again and restore x31 */
+ xor x31, x31, x31
+ csrrw x31, CSR_SCRATCH, x31
-restore_caller_reg:
- //save per-cpu overflow stack
- REG_S a0, -8(sp)
- //restore caller register from shadow_stack
- REG_L x1, PT_RA(sp)
- REG_L x5, PT_T0(sp)
- REG_L x6, PT_T1(sp)
- REG_L x7, PT_T2(sp)
- REG_L x10, PT_A0(sp)
- REG_L x11, PT_A1(sp)
- REG_L x12, PT_A2(sp)
- REG_L x13, PT_A3(sp)
- REG_L x14, PT_A4(sp)
- REG_L x15, PT_A5(sp)
- REG_L x16, PT_A6(sp)
- REG_L x17, PT_A7(sp)
- REG_L x28, PT_T3(sp)
- REG_L x29, PT_T4(sp)
- REG_L x30, PT_T5(sp)
- REG_L x31, PT_T6(sp)
-
- //load per-cpu overflow stack
- REG_L sp, -8(sp)
addi sp, sp, -(PT_SIZE_ON_STACK)
//save context to overflow stack
@@ -552,7 +502,16 @@ ENTRY(__switch_to)
REG_S s9, TASK_THREAD_S9_RA(a3)
REG_S s10, TASK_THREAD_S10_RA(a3)
REG_S s11, TASK_THREAD_S11_RA(a3)
+
+ /* save the user space access flag */
+ csrr s0, CSR_STATUS
+ REG_S s0, TASK_THREAD_STATUS_RA(a3)
+
/* Restore context from next->thread */
+ REG_L s0, TASK_THREAD_STATUS_RA(a4)
+ li s1, SR_SUM
+ and s0, s0, s1
+ csrs CSR_STATUS, s0
REG_L ra, TASK_THREAD_RA_RA(a4)
REG_L sp, TASK_THREAD_SP_RA(a4)
REG_L s0, TASK_THREAD_S0_RA(a4)
diff --git a/arch/riscv/kernel/probes/kprobes_trampoline.S b/arch/riscv/kernel/probes/kprobes_trampoline.S
index 7bdb09ded39b..b1f0fc933e0b 100644
--- a/arch/riscv/kernel/probes/kprobes_trampoline.S
+++ b/arch/riscv/kernel/probes/kprobes_trampoline.S
@@ -41,6 +41,9 @@
REG_S x29, PT_T4(sp)
REG_S x30, PT_T5(sp)
REG_S x31, PT_T6(sp)
+ /* save original sp */
+ addi a0, sp, PT_SIZE_ON_STACK
+ REG_S a0, PT_SP(sp)
.endm
.macro restore_all_base_regs
diff --git a/arch/riscv/kernel/traps.c b/arch/riscv/kernel/traps.c
index 7a6c02e17f4f..ffa37f4f6a10 100644
--- a/arch/riscv/kernel/traps.c
+++ b/arch/riscv/kernel/traps.c
@@ -209,42 +209,14 @@ int is_valid_bugaddr(unsigned long pc)
#endif /* CONFIG_GENERIC_BUG */
#ifdef CONFIG_VMAP_STACK
-static DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)],
+DEFINE_PER_CPU(unsigned long [OVERFLOW_STACK_SIZE/sizeof(long)],
overflow_stack)__aligned(16);
-/*
- * shadow stack, handled_ kernel_ stack_ overflow(in kernel/entry.S) is used
- * to get per-cpu overflow stack(get_overflow_stack).
- */
-long shadow_stack[SHADOW_OVERFLOW_STACK_SIZE/sizeof(long)] __aligned(16);
-asmlinkage unsigned long get_overflow_stack(void)
-{
- return (unsigned long)this_cpu_ptr(overflow_stack) +
- OVERFLOW_STACK_SIZE;
-}
-
-/*
- * A pseudo spinlock to protect the shadow stack from being used by multiple
- * harts concurrently. This isn't a real spinlock because the lock side must
- * be taken without a valid stack and only a single register, it's only taken
- * while in the process of panicing anyway so the performance and error
- * checking a proper spinlock gives us doesn't matter.
- */
-unsigned long spin_shadow_stack;
asmlinkage void handle_bad_stack(struct pt_regs *regs)
{
unsigned long tsk_stk = (unsigned long)current->stack;
unsigned long ovf_stk = (unsigned long)this_cpu_ptr(overflow_stack);
- /*
- * We're done with the shadow stack by this point, as we're on the
- * overflow stack. Tell any other concurrent overflowing harts that
- * they can proceed with panicing by releasing the pseudo-spinlock.
- *
- * This pairs with an amoswap.aq in handle_kernel_stack_overflow.
- */
- smp_store_release(&spin_shadow_stack, 0);
-
console_verbose();
pr_emerg("Insufficient stack space to handle exception!\n");
diff --git a/arch/riscv/kvm/vmid.c b/arch/riscv/kvm/vmid.c
index 6cd93995fb65..dc5d318fe91c 100644
--- a/arch/riscv/kvm/vmid.c
+++ b/arch/riscv/kvm/vmid.c
@@ -26,7 +26,7 @@ void kvm_riscv_gstage_vmid_detect(void)
/* Figure-out number of VMID bits in HW */
old = csr_read(CSR_HGATP);
- csr_write(CSR_HGATP, old | HGATP_VMID_MASK);
+ csr_write(CSR_HGATP, (kvm_riscv_gstage_mode() << HGATP_MODE_SHIFT) | HGATP_VMID_MASK);
vmid_bits = csr_read(CSR_HGATP);
vmid_bits = (vmid_bits & HGATP_VMID_MASK) >> HGATP_VMID_SHIFT;
vmid_bits = fls_long(vmid_bits);
diff --git a/arch/s390/appldata/appldata_base.c b/arch/s390/appldata/appldata_base.c
index d74a4c7d5df6..4880af67dc6f 100644
--- a/arch/s390/appldata/appldata_base.c
+++ b/arch/s390/appldata/appldata_base.c
@@ -26,12 +26,10 @@
#include <linux/notifier.h>
#include <linux/cpu.h>
#include <linux/workqueue.h>
-#include <linux/suspend.h>
-#include <linux/platform_device.h>
+#include <linux/uaccess.h>
+#include <linux/io.h>
#include <asm/appldata.h>
#include <asm/vtimer.h>
-#include <linux/uaccess.h>
-#include <asm/io.h>
#include <asm/smp.h>
#include "appldata.h"
@@ -44,8 +42,6 @@
#define TOD_MICRO 0x01000 /* nr. of TOD clock units
for 1 microsecond */
-static struct platform_device *appldata_pdev;
-
/*
* /proc entries (sysctl)
*/
@@ -88,7 +84,6 @@ static struct vtimer_list appldata_timer;
static DEFINE_SPINLOCK(appldata_timer_lock);
static int appldata_interval = APPLDATA_CPU_INTERVAL;
static int appldata_timer_active;
-static int appldata_timer_suspended = 0;
/*
* Work queue
@@ -412,88 +407,6 @@ void appldata_unregister_ops(struct appldata_ops *ops)
/********************** module-ops management <END> **************************/
-/**************************** suspend / resume *******************************/
-static int appldata_freeze(struct device *dev)
-{
- struct appldata_ops *ops;
- int rc;
- struct list_head *lh;
-
- spin_lock(&appldata_timer_lock);
- if (appldata_timer_active) {
- __appldata_vtimer_setup(APPLDATA_DEL_TIMER);
- appldata_timer_suspended = 1;
- }
- spin_unlock(&appldata_timer_lock);
-
- mutex_lock(&appldata_ops_mutex);
- list_for_each(lh, &appldata_ops_list) {
- ops = list_entry(lh, struct appldata_ops, list);
- if (ops->active == 1) {
- rc = appldata_diag(ops->record_nr, APPLDATA_STOP_REC,
- (unsigned long) ops->data, ops->size,
- ops->mod_lvl);
- if (rc != 0)
- pr_err("Stopping the data collection for %s "
- "failed with rc=%d\n", ops->name, rc);
- }
- }
- mutex_unlock(&appldata_ops_mutex);
- return 0;
-}
-
-static int appldata_restore(struct device *dev)
-{
- struct appldata_ops *ops;
- int rc;
- struct list_head *lh;
-
- spin_lock(&appldata_timer_lock);
- if (appldata_timer_suspended) {
- __appldata_vtimer_setup(APPLDATA_ADD_TIMER);
- appldata_timer_suspended = 0;
- }
- spin_unlock(&appldata_timer_lock);
-
- mutex_lock(&appldata_ops_mutex);
- list_for_each(lh, &appldata_ops_list) {
- ops = list_entry(lh, struct appldata_ops, list);
- if (ops->active == 1) {
- ops->callback(ops->data); // init record
- rc = appldata_diag(ops->record_nr,
- APPLDATA_START_INTERVAL_REC,
- (unsigned long) ops->data, ops->size,
- ops->mod_lvl);
- if (rc != 0) {
- pr_err("Starting the data collection for %s "
- "failed with rc=%d\n", ops->name, rc);
- }
- }
- }
- mutex_unlock(&appldata_ops_mutex);
- return 0;
-}
-
-static int appldata_thaw(struct device *dev)
-{
- return appldata_restore(dev);
-}
-
-static const struct dev_pm_ops appldata_pm_ops = {
- .freeze = appldata_freeze,
- .thaw = appldata_thaw,
- .restore = appldata_restore,
-};
-
-static struct platform_driver appldata_pdrv = {
- .driver = {
- .name = "appldata",
- .pm = &appldata_pm_ops,
- },
-};
-/************************* suspend / resume <END> ****************************/
-
-
/******************************* init / exit *********************************/
/*
@@ -503,36 +416,14 @@ static struct platform_driver appldata_pdrv = {
*/
static int __init appldata_init(void)
{
- int rc;
-
init_virt_timer(&appldata_timer);
appldata_timer.function = appldata_timer_function;
appldata_timer.data = (unsigned long) &appldata_work;
-
- rc = platform_driver_register(&appldata_pdrv);
- if (rc)
- return rc;
-
- appldata_pdev = platform_device_register_simple("appldata", -1, NULL,
- 0);
- if (IS_ERR(appldata_pdev)) {
- rc = PTR_ERR(appldata_pdev);
- goto out_driver;
- }
appldata_wq = alloc_ordered_workqueue("appldata", 0);
- if (!appldata_wq) {
- rc = -ENOMEM;
- goto out_device;
- }
-
+ if (!appldata_wq)
+ return -ENOMEM;
appldata_sysctl_header = register_sysctl_table(appldata_dir_table);
return 0;
-
-out_device:
- platform_device_unregister(appldata_pdev);
-out_driver:
- platform_driver_unregister(&appldata_pdrv);
- return rc;
}
__initcall(appldata_init);
diff --git a/arch/s390/appldata/appldata_mem.c b/arch/s390/appldata/appldata_mem.c
index 21c3147bd92a..fc608f9b79ab 100644
--- a/arch/s390/appldata/appldata_mem.c
+++ b/arch/s390/appldata/appldata_mem.c
@@ -15,7 +15,7 @@
#include <linux/pagemap.h>
#include <linux/swap.h>
#include <linux/slab.h>
-#include <asm/io.h>
+#include <linux/io.h>
#include "appldata.h"
diff --git a/arch/s390/include/asm/appldata.h b/arch/s390/include/asm/appldata.h
index c5bd9f4437e5..f2240392c708 100644
--- a/arch/s390/include/asm/appldata.h
+++ b/arch/s390/include/asm/appldata.h
@@ -8,8 +8,8 @@
#ifndef _ASM_S390_APPLDATA_H
#define _ASM_S390_APPLDATA_H
+#include <linux/io.h>
#include <asm/diag.h>
-#include <asm/io.h>
#define APPLDATA_START_INTERVAL_REC 0x80
#define APPLDATA_STOP_REC 0x81
diff --git a/arch/s390/include/asm/cpu_mf.h b/arch/s390/include/asm/cpu_mf.h
index efa103b52a1a..7e417d7de568 100644
--- a/arch/s390/include/asm/cpu_mf.h
+++ b/arch/s390/include/asm/cpu_mf.h
@@ -42,7 +42,6 @@ static inline int cpum_sf_avail(void)
return test_facility(40) && test_facility(68);
}
-
struct cpumf_ctr_info {
u16 cfvn;
u16 auth_ctl;
@@ -275,56 +274,4 @@ static inline int lsctl(struct hws_lsctl_request_block *req)
return cc ? -EINVAL : 0;
}
-
-/* Sampling control helper functions */
-
-#include <linux/time.h>
-
-static inline unsigned long freq_to_sample_rate(struct hws_qsi_info_block *qsi,
- unsigned long freq)
-{
- return (USEC_PER_SEC / freq) * qsi->cpu_speed;
-}
-
-static inline unsigned long sample_rate_to_freq(struct hws_qsi_info_block *qsi,
- unsigned long rate)
-{
- return USEC_PER_SEC * qsi->cpu_speed / rate;
-}
-
-/* Return TOD timestamp contained in an trailer entry */
-static inline unsigned long long trailer_timestamp(struct hws_trailer_entry *te)
-{
- /* TOD in STCKE format */
- if (te->header.t)
- return *((unsigned long long *) &te->timestamp[1]);
-
- /* TOD in STCK format */
- return *((unsigned long long *) &te->timestamp[0]);
-}
-
-/* Return pointer to trailer entry of an sample data block */
-static inline unsigned long *trailer_entry_ptr(unsigned long v)
-{
- void *ret;
-
- ret = (void *) v;
- ret += PAGE_SIZE;
- ret -= sizeof(struct hws_trailer_entry);
-
- return (unsigned long *) ret;
-}
-
-/* Return true if the entry in the sample data block table (sdbt)
- * is a link to the next sdbt */
-static inline int is_link_entry(unsigned long *s)
-{
- return *s & 0x1ul ? 1 : 0;
-}
-
-/* Return pointer to the linked sdbt */
-static inline unsigned long *get_next_sdbt(unsigned long *s)
-{
- return (unsigned long *) (*s & ~0x1ul);
-}
#endif /* _ASM_S390_CPU_MF_H */
diff --git a/arch/s390/include/asm/dma.h b/arch/s390/include/asm/dma.h
index dec1c4ce628c..c260adb25997 100644
--- a/arch/s390/include/asm/dma.h
+++ b/arch/s390/include/asm/dma.h
@@ -2,7 +2,7 @@
#ifndef _ASM_S390_DMA_H
#define _ASM_S390_DMA_H
-#include <asm/io.h>
+#include <linux/io.h>
/*
* MAX_DMA_ADDRESS is ambiguous because on s390 its completely unrelated
diff --git a/arch/s390/include/asm/nmi.h b/arch/s390/include/asm/nmi.h
index af1cd3a6f406..3f37440700c9 100644
--- a/arch/s390/include/asm/nmi.h
+++ b/arch/s390/include/asm/nmi.h
@@ -22,6 +22,7 @@
#define MCCK_CODE_SYSTEM_DAMAGE BIT(63)
#define MCCK_CODE_EXT_DAMAGE BIT(63 - 5)
#define MCCK_CODE_CP BIT(63 - 9)
+#define MCCK_CODE_CK BIT(63 - 11)
#define MCCK_CODE_STG_ERROR BIT(63 - 16)
#define MCCK_CODE_STG_KEY_ERROR BIT(63 - 18)
#define MCCK_CODE_STG_DEGRAD BIT(63 - 19)
@@ -33,6 +34,8 @@
#define MCCK_CODE_FC_VALID BIT(63 - 43)
#define MCCK_CODE_CPU_TIMER_VALID BIT(63 - 46)
+#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE | MCCK_CODE_CK)
+
#ifndef __ASSEMBLY__
union mci {
diff --git a/arch/s390/kernel/cpcmd.c b/arch/s390/kernel/cpcmd.c
index 72e106cfd8c7..b210a29d3ee9 100644
--- a/arch/s390/kernel/cpcmd.c
+++ b/arch/s390/kernel/cpcmd.c
@@ -16,10 +16,10 @@
#include <linux/stddef.h>
#include <linux/string.h>
#include <linux/mm.h>
+#include <linux/io.h>
#include <asm/diag.h>
#include <asm/ebcdic.h>
#include <asm/cpcmd.h>
-#include <asm/io.h>
static DEFINE_SPINLOCK(cpcmd_lock);
static char cpcmd_buf[241];
diff --git a/arch/s390/kernel/dis.c b/arch/s390/kernel/dis.c
index 90bbb4ea1d08..51d6e6dcadcd 100644
--- a/arch/s390/kernel/dis.c
+++ b/arch/s390/kernel/dis.c
@@ -24,8 +24,8 @@
#include <linux/kdebug.h>
#include <linux/uaccess.h>
#include <linux/atomic.h>
+#include <linux/io.h>
#include <asm/dis.h>
-#include <asm/io.h>
#include <asm/cpcmd.h>
#include <asm/lowcore.h>
#include <asm/debug.h>
diff --git a/arch/s390/kernel/nmi.c b/arch/s390/kernel/nmi.c
index 31cb9b00a36b..1c92cc2dd437 100644
--- a/arch/s390/kernel/nmi.c
+++ b/arch/s390/kernel/nmi.c
@@ -372,8 +372,7 @@ static void notrace s390_backup_mcck_info(struct pt_regs *regs)
sie_page = container_of(sie_block, struct sie_page, sie_block);
mcck_backup = &sie_page->mcck_info;
- mcck_backup->mcic = S390_lowcore.mcck_interruption_code &
- ~(MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE);
+ mcck_backup->mcic = S390_lowcore.mcck_interruption_code & ~MCCK_CODE_NO_GUEST;
mcck_backup->ext_damage_code = S390_lowcore.external_damage_code;
mcck_backup->failing_storage_address
= S390_lowcore.failing_storage_address;
@@ -386,8 +385,6 @@ NOKPROBE_SYMBOL(s390_backup_mcck_info);
#define ED_STP_ISLAND 6 /* External damage STP island check */
#define ED_STP_SYNC 7 /* External damage STP sync check */
-#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE)
-
/*
* machine check handler.
*/
diff --git a/arch/s390/kernel/perf_cpum_sf.c b/arch/s390/kernel/perf_cpum_sf.c
index 96fd3654abda..0d71370b5781 100644
--- a/arch/s390/kernel/perf_cpum_sf.c
+++ b/arch/s390/kernel/perf_cpum_sf.c
@@ -22,6 +22,7 @@
#include <asm/irq.h>
#include <asm/debug.h>
#include <asm/timex.h>
+#include <linux/io.h>
/* Minimum number of sample-data-block-tables:
* At least one table is required for the sampling buffer structure.
@@ -99,6 +100,57 @@ static DEFINE_PER_CPU(struct cpu_hw_sf, cpu_hw_sf);
/* Debug feature */
static debug_info_t *sfdbg;
+/* Sampling control helper functions */
+static inline unsigned long freq_to_sample_rate(struct hws_qsi_info_block *qsi,
+ unsigned long freq)
+{
+ return (USEC_PER_SEC / freq) * qsi->cpu_speed;
+}
+
+static inline unsigned long sample_rate_to_freq(struct hws_qsi_info_block *qsi,
+ unsigned long rate)
+{
+ return USEC_PER_SEC * qsi->cpu_speed / rate;
+}
+
+/* Return TOD timestamp contained in an trailer entry */
+static inline unsigned long long trailer_timestamp(struct hws_trailer_entry *te)
+{
+ /* TOD in STCKE format */
+ if (te->header.t)
+ return *((unsigned long long *)&te->timestamp[1]);
+
+ /* TOD in STCK format */
+ return *((unsigned long long *)&te->timestamp[0]);
+}
+
+/* Return pointer to trailer entry of an sample data block */
+static inline unsigned long *trailer_entry_ptr(unsigned long v)
+{
+ void *ret;
+
+ ret = (void *)v;
+ ret += PAGE_SIZE;
+ ret -= sizeof(struct hws_trailer_entry);
+
+ return (unsigned long *)ret;
+}
+
+/*
+ * Return true if the entry in the sample data block table (sdbt)
+ * is a link to the next sdbt
+ */
+static inline int is_link_entry(unsigned long *s)
+{
+ return *s & 0x1UL ? 1 : 0;
+}
+
+/* Return pointer to the linked sdbt */
+static inline unsigned long *get_next_sdbt(unsigned long *s)
+{
+ return (unsigned long *)(*s & ~0x1UL);
+}
+
/*
* sf_disable() - Switch off sampling facility
*/
diff --git a/arch/s390/kernel/process.c b/arch/s390/kernel/process.c
index 407500eca532..c08f02d667e6 100644
--- a/arch/s390/kernel/process.c
+++ b/arch/s390/kernel/process.c
@@ -30,8 +30,8 @@
#include <linux/export.h>
#include <linux/init_task.h>
#include <linux/entry-common.h>
+#include <linux/io.h>
#include <asm/cpu_mf.h>
-#include <asm/io.h>
#include <asm/processor.h>
#include <asm/vtimer.h>
#include <asm/exec.h>
diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c
index 6be9c79383dd..cd5f387c9b76 100644
--- a/arch/s390/kvm/interrupt.c
+++ b/arch/s390/kvm/interrupt.c
@@ -2858,9 +2858,7 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e,
void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
struct mcck_volatile_info *mcck_info)
{
- struct kvm_s390_interrupt_info inti;
struct kvm_s390_irq irq = {};
- struct kvm_s390_mchk_info *mchk;
union mci mci;
__u64 cr14 = 0; /* upper bits are not used */
int rc;
@@ -2873,20 +2871,14 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu,
if (mci.w)
cr14 |= CR14_WARNING_SUBMASK;
- mchk = mci.ck ? &inti.mchk : &irq.u.mchk;
- mchk->cr14 = cr14;
- mchk->mcic = mcck_info->mcic;
- mchk->ext_damage_code = mcck_info->ext_damage_code;
- mchk->failing_storage_address = mcck_info->failing_storage_address;
- if (mci.ck) {
- /* Inject the floating machine check */
- inti.type = KVM_S390_MCHK;
- rc = __inject_vm(vcpu->kvm, &inti);
- } else {
- /* Inject the machine check to specified vcpu */
- irq.type = KVM_S390_MCHK;
- rc = kvm_s390_inject_vcpu(vcpu, &irq);
- }
+ irq.u.mchk.cr14 = cr14;
+ irq.u.mchk.mcic = mcck_info->mcic;
+ irq.u.mchk.ext_damage_code = mcck_info->ext_damage_code;
+ irq.u.mchk.failing_storage_address = mcck_info->failing_storage_address;
+
+ /* Inject the machine check to specified vcpu */
+ irq.type = KVM_S390_MCHK;
+ rc = kvm_s390_inject_vcpu(vcpu, &irq);
WARN_ON_ONCE(rc);
}
diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c
index c7b4c0d37c87..0d4a05f3cdd9 100644
--- a/arch/s390/kvm/kvm-s390.c
+++ b/arch/s390/kvm/kvm-s390.c
@@ -4430,8 +4430,8 @@ long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable)
static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token,
unsigned long token)
{
- struct kvm_s390_interrupt inti;
- struct kvm_s390_irq irq;
+ struct kvm_s390_interrupt inti = {};
+ struct kvm_s390_irq irq = {};
if (start_token) {
irq.u.ext.ext_params2 = token;
@@ -5587,6 +5587,9 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
{
gpa_t size;
+ if (kvm_is_ucontrol(kvm))
+ return -EINVAL;
+
/* When we are protected, we should not change the memory slots */
if (kvm_s390_pv_get_handle(kvm))
return -EINVAL;
diff --git a/arch/s390/kvm/priv.c b/arch/s390/kvm/priv.c
index 9af826d093ef..beae85549ebe 100644
--- a/arch/s390/kvm/priv.c
+++ b/arch/s390/kvm/priv.c
@@ -13,7 +13,7 @@
#include <linux/errno.h>
#include <linux/mm_types.h>
#include <linux/pgtable.h>
-
+#include <linux/io.h>
#include <asm/asm-offsets.h>
#include <asm/facility.h>
#include <asm/current.h>
@@ -22,7 +22,6 @@
#include <asm/sysinfo.h>
#include <asm/page-states.h>
#include <asm/gmap.h>
-#include <asm/io.h>
#include <asm/ptrace.h>
#include <asm/sclp.h>
#include <asm/ap.h>
diff --git a/arch/s390/lib/spinlock.c b/arch/s390/lib/spinlock.c
index 04d4c6cf898e..81c53440b3e6 100644
--- a/arch/s390/lib/spinlock.c
+++ b/arch/s390/lib/spinlock.c
@@ -13,8 +13,8 @@
#include <linux/init.h>
#include <linux/smp.h>
#include <linux/percpu.h>
+#include <linux/io.h>
#include <asm/alternative.h>
-#include <asm/io.h>
int spin_retry = -1;
diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c
index 2ab388179833..dc0aa423e142 100644
--- a/arch/s390/mm/fault.c
+++ b/arch/s390/mm/fault.c
@@ -818,6 +818,8 @@ void do_secure_storage_access(struct pt_regs *regs)
switch (get_fault_type(regs)) {
case GMAP_FAULT:
+ if (faulthandler_disabled())
+ return do_fault_error(regs, VM_FAULT_BADMAP);
mm = current->mm;
gmap = (struct gmap *)S390_lowcore.gmap;
mmap_read_lock(mm);
@@ -829,6 +831,8 @@ void do_secure_storage_access(struct pt_regs *regs)
}
fallthrough;
case USER_FAULT:
+ if (faulthandler_disabled())
+ return do_fault_error(regs, VM_FAULT_BADMAP);
mm = current->mm;
mmap_read_lock(mm);
vma = find_vma(mm, addr);
@@ -851,10 +855,17 @@ void do_secure_storage_access(struct pt_regs *regs)
page = phys_to_page(addr);
if (unlikely(!try_get_page(page)))
break;
- rc = arch_make_page_accessible(page);
+ rc = uv_convert_from_secure(page_to_phys(page));
+ if (!rc)
+ clear_bit(PG_arch_1, &page->flags);
put_page(page);
+ /*
+ * There are some valid fixup types for kernel
+ * accesses to donated secure memory. zeropad is one
+ * of them.
+ */
if (rc)
- BUG();
+ return do_no_context(regs, 0);
break;
default:
do_fault_error(regs, VM_FAULT_BADMAP);
diff --git a/arch/s390/mm/maccess.c b/arch/s390/mm/maccess.c
index 4824d1cd33d8..b2d65afbb0ff 100644
--- a/arch/s390/mm/maccess.c
+++ b/arch/s390/mm/maccess.c
@@ -13,9 +13,9 @@
#include <linux/gfp.h>
#include <linux/cpu.h>
#include <linux/uio.h>
+#include <linux/io.h>
#include <asm/asm-extable.h>
#include <asm/ctl_reg.h>
-#include <asm/io.h>
#include <asm/abs_lowcore.h>
#include <asm/stacktrace.h>
#include <asm/maccess.h>
diff --git a/arch/s390/pci/pci_sysfs.c b/arch/s390/pci/pci_sysfs.c
index cae280e5c047..f2b615c597bc 100644
--- a/arch/s390/pci/pci_sysfs.c
+++ b/arch/s390/pci/pci_sysfs.c
@@ -146,6 +146,9 @@ static ssize_t report_error_write(struct file *filp, struct kobject *kobj,
if (off || (count < sizeof(*report)))
return -EINVAL;
+ if (count < (report->length + sizeof(*report)))
+ return -EINVAL;
+
ret = sclp_pci_report(report, zdev->fh, zdev->fid);
return ret ? ret : count;
diff --git a/arch/sparc/kernel/Makefile b/arch/sparc/kernel/Makefile
index 0984bb6f0f17..58ea4ef9b622 100644
--- a/arch/sparc/kernel/Makefile
+++ b/arch/sparc/kernel/Makefile
@@ -5,7 +5,6 @@
#
asflags-y := -ansi
-ccflags-y := -Werror
# Undefine sparc when processing vmlinux.lds - it is used
# And teach CPP we are doing $(BITS) builds (for this case)
diff --git a/arch/sparc/lib/Makefile b/arch/sparc/lib/Makefile
index 063556fe2cb1..59669ebddd4e 100644
--- a/arch/sparc/lib/Makefile
+++ b/arch/sparc/lib/Makefile
@@ -3,7 +3,6 @@
#
asflags-y := -ansi -DST_DIV0=0x02
-ccflags-y := -Werror
lib-$(CONFIG_SPARC32) += ashrdi3.o
lib-$(CONFIG_SPARC32) += memcpy.o memset.o
diff --git a/arch/sparc/mm/Makefile b/arch/sparc/mm/Makefile
index 871354aa3c00..809d993f6d88 100644
--- a/arch/sparc/mm/Makefile
+++ b/arch/sparc/mm/Makefile
@@ -3,7 +3,6 @@
#
asflags-y := -ansi
-ccflags-y := -Werror
obj-$(CONFIG_SPARC64) += ultra.o tlb.o tsb.o
obj-y += fault_$(BITS).o
diff --git a/arch/sparc/prom/Makefile b/arch/sparc/prom/Makefile
index 397b79af77f7..a1adc75d8055 100644
--- a/arch/sparc/prom/Makefile
+++ b/arch/sparc/prom/Makefile
@@ -3,7 +3,6 @@
# Linux.
#
asflags := -ansi
-ccflags := -Werror
lib-y := bootstr_$(BITS).o
lib-y += init_$(BITS).o
diff --git a/arch/um/kernel/skas/syscall.c b/arch/um/kernel/skas/syscall.c
index 9ee19e566da3..b09e85279d2b 100644
--- a/arch/um/kernel/skas/syscall.c
+++ b/arch/um/kernel/skas/syscall.c
@@ -12,23 +12,13 @@
#include <sysdep/syscalls.h>
#include <linux/time-internal.h>
#include <asm/unistd.h>
+#include <asm/delay.h>
void handle_syscall(struct uml_pt_regs *r)
{
struct pt_regs *regs = container_of(r, struct pt_regs, regs);
int syscall;
- /*
- * If we have infinite CPU resources, then make every syscall also a
- * preemption point, since we don't have any other preemption in this
- * case, and kernel threads would basically never run until userspace
- * went to sleep, even if said userspace interacts with the kernel in
- * various ways.
- */
- if (time_travel_mode == TT_MODE_INFCPU ||
- time_travel_mode == TT_MODE_EXTERNAL)
- schedule();
-
/* Initialize the syscall number and default return value. */
UPT_SYSCALL_NR(r) = PT_SYSCALL_NR(r->gp);
PT_REGS_SET_SYSCALL_RETURN(regs, -ENOSYS);
@@ -41,9 +31,25 @@ void handle_syscall(struct uml_pt_regs *r)
goto out;
syscall = UPT_SYSCALL_NR(r);
- if (syscall >= 0 && syscall < __NR_syscalls)
- PT_REGS_SET_SYSCALL_RETURN(regs,
- EXECUTE_SYSCALL(syscall, regs));
+ if (syscall >= 0 && syscall < __NR_syscalls) {
+ unsigned long ret = EXECUTE_SYSCALL(syscall, regs);
+
+ PT_REGS_SET_SYSCALL_RETURN(regs, ret);
+
+ /*
+ * An error value here can be some form of -ERESTARTSYS
+ * and then we'd just loop. Make any error syscalls take
+ * some time, so that it won't just loop if something is
+ * not ready, and hopefully other things will make some
+ * progress.
+ */
+ if (IS_ERR_VALUE(ret) &&
+ (time_travel_mode == TT_MODE_INFCPU ||
+ time_travel_mode == TT_MODE_EXTERNAL)) {
+ um_udelay(1);
+ schedule();
+ }
+ }
out:
syscall_trace_leave(regs);
diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c
index d1f882cf418d..4acf9adad16b 100644
--- a/arch/x86/events/amd/core.c
+++ b/arch/x86/events/amd/core.c
@@ -719,13 +719,11 @@ static void amd_pmu_enable_event(struct perf_event *event)
x86_pmu_enable_event(event);
}
-static void amd_pmu_enable_all(int added)
+static void __amd_pmu_enable_all(void)
{
struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events);
int idx;
- amd_brs_enable_all();
-
for (idx = 0; idx < x86_pmu.num_counters; idx++) {
/* only activate events which are marked as active */
if (!test_bit(idx, cpuc->active_mask))
@@ -740,6 +738,12 @@ static void amd_pmu_enable_all(int added)
}
}
+static void amd_pmu_enable_all(int added)
+{
+ amd_brs_enable_all();
+ __amd_pmu_enable_all();
+}
+
static void amd_pmu_v2_enable_event(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
@@ -1493,7 +1497,7 @@ static inline void amd_pmu_reload_virt(void)
* set global enable bits once again
*/
amd_pmu_v2_disable_all();
- amd_pmu_enable_all(0);
+ __amd_pmu_enable_all();
amd_pmu_v2_enable_all(0);
return;
}
diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c
index 48f4095f500d..6096fb6925f6 100644
--- a/arch/x86/events/amd/lbr.c
+++ b/arch/x86/events/amd/lbr.c
@@ -124,7 +124,8 @@ static void amd_pmu_lbr_filter(void)
}
/* If type does not correspond, then discard */
- if (type == X86_BR_NONE || (br_sel & type) != type) {
+ if (type == X86_BR_NONE || (br_sel & type) != type ||
+ (!(br_sel & X86_BR_KERNEL) && kernel_ip(cpuc->lbr_entries[i].from))) {
cpuc->lbr_entries[i].from = 0; /* mark invalid */
compress = true;
}
diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c
index 4ddd967a9d3d..6f5805ad723b 100644
--- a/arch/x86/events/intel/core.c
+++ b/arch/x86/events/intel/core.c
@@ -5990,12 +5990,6 @@ __init int intel_pmu_init(void)
x86_add_quirk(intel_arch_events_quirk); /* Install first, so it runs last */
- if (version >= 5) {
- x86_pmu.intel_cap.anythread_deprecated = edx.split.anythread_deprecated;
- if (x86_pmu.intel_cap.anythread_deprecated)
- pr_cont(" AnyThread deprecated, ");
- }
-
/*
* Install the hw-cache-events table:
*/
@@ -6201,7 +6195,7 @@ __init int intel_pmu_init(void)
x86_pmu.flags |= PMU_FL_INSTR_LATENCY;
intel_pmu_pebs_data_source_grt();
- x86_pmu.pebs_latency_data = adl_latency_data_small;
+ x86_pmu.pebs_latency_data = grt_latency_data;
x86_pmu.get_event_constraints = tnt_get_event_constraints;
x86_pmu.limit_period = spr_limit_period;
td_attr = tnt_events_attrs;
@@ -6584,7 +6578,7 @@ __init int intel_pmu_init(void)
x86_pmu.flags |= PMU_FL_MEM_LOADS_AUX;
x86_pmu.lbr_pt_coexist = true;
intel_pmu_pebs_data_source_adl();
- x86_pmu.pebs_latency_data = adl_latency_data_small;
+ x86_pmu.pebs_latency_data = grt_latency_data;
x86_pmu.num_topdown_events = 8;
static_call_update(intel_pmu_update_topdown_event,
&adl_update_topdown_event);
@@ -6734,8 +6728,10 @@ __init int intel_pmu_init(void)
(u64)fixed_mask);
/* AnyThread may be deprecated on arch perfmon v5 or later */
- if (x86_pmu.intel_cap.anythread_deprecated)
+ if (version >= 5 && edx.split.anythread_deprecated) {
x86_pmu.format_attrs = intel_arch_formats_attr;
+ pr_cont("AnyThread deprecated, ");
+ }
intel_pmu_check_event_constraints(x86_pmu.event_constraints,
x86_pmu.num_counters,
diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c
index 4b7faa3180e5..d05db7aeef78 100644
--- a/arch/x86/events/intel/ds.c
+++ b/arch/x86/events/intel/ds.c
@@ -221,40 +221,52 @@ static inline void pebs_set_tlb_lock(u64 *val, bool tlb, bool lock)
}
/* Retrieve the latency data for e-core of ADL */
-u64 adl_latency_data_small(struct perf_event *event, u64 status)
+static u64 __grt_latency_data(struct perf_event *event, u64 status,
+ u8 dse, bool tlb, bool lock, bool blk)
{
- union intel_x86_pebs_dse dse;
+ union perf_mem_data_src src;
u64 val;
- WARN_ON_ONCE(hybrid_pmu(event->pmu)->cpu_type == hybrid_big);
-
- dse.val = status;
+ WARN_ON_ONCE(is_hybrid() &&
+ hybrid_pmu(event->pmu)->cpu_type == hybrid_big);
- val = hybrid_var(event->pmu, pebs_data_source)[dse.ld_dse];
+ val = hybrid_var(event->pmu, pebs_data_source)[dse];
- /*
- * For the atom core on ADL,
- * bit 4: lock, bit 5: TLB access.
- */
- pebs_set_tlb_lock(&val, dse.ld_locked, dse.ld_stlb_miss);
+ pebs_set_tlb_lock(&val, tlb, lock);
- if (dse.ld_data_blk)
+ if (blk)
val |= P(BLK, DATA);
else
val |= P(BLK, NA);
- return val;
+ src.val = val;
+
+ if (event->hw.flags &
+ (PERF_X86_EVENT_PEBS_LDLAT | PERF_X86_EVENT_PEBS_LD_HSW))
+ src.mem_op = P(OP, LOAD);
+ if (event->hw.flags &
+ (PERF_X86_EVENT_PEBS_STLAT | PERF_X86_EVENT_PEBS_ST_HSW))
+ src.mem_op = P(OP, STORE);
+
+ return src.val;
}
-static u64 load_latency_data(struct perf_event *event, u64 status)
+u64 grt_latency_data(struct perf_event *event, u64 status)
{
union intel_x86_pebs_dse dse;
u64 val;
dse.val = status;
+ /* Gracemont uses bit 4 for lock and bit 5 for TLB access. */
+ if (event->hw.flags & PERF_X86_EVENT_PEBS_LAT_HYBRID)
+ return __grt_latency_data(event, status, dse.ld_dse,
+ dse.ld_locked, dse.ld_stlb_miss,
+ dse.ld_data_blk);
+
/*
- * use the mapping table for bit 0-3
+ * For non-Gracemont load-latency events, use the mapping table
+ * for bit 0-3 and retain the legacy TLB, lock and block decoding.
*/
val = hybrid_var(event->pmu, pebs_data_source)[dse.ld_dse];
@@ -837,8 +849,8 @@ struct event_constraint intel_glm_pebs_event_constraints[] = {
struct event_constraint intel_grt_pebs_event_constraints[] = {
/* Allow all events as PEBS with no flags */
- INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0x3),
- INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0xf),
+ INTEL_HYBRID_LDLAT_CONSTRAINT(0x5d0, 0x3),
+ INTEL_HYBRID_STLAT_CONSTRAINT(0x6d0, 0x3f),
EVENT_CONSTRAINT_END
};
@@ -1511,7 +1523,7 @@ static u64 get_data_src(struct perf_event *event, u64 aux)
bool fst = fl & (PERF_X86_EVENT_PEBS_ST | PERF_X86_EVENT_PEBS_HSW_PREC);
if (fl & PERF_X86_EVENT_PEBS_LDLAT)
- val = load_latency_data(event, aux);
+ val = grt_latency_data(event, aux);
else if (fl & PERF_X86_EVENT_PEBS_STLAT)
val = store_latency_data(event, aux);
else if (fl & PERF_X86_EVENT_PEBS_LAT_HYBRID)
diff --git a/arch/x86/events/intel/uncore_snbep.c b/arch/x86/events/intel/uncore_snbep.c
index 831c4e3f371a..8a5e47655974 100644
--- a/arch/x86/events/intel/uncore_snbep.c
+++ b/arch/x86/events/intel/uncore_snbep.c
@@ -1458,11 +1458,6 @@ static int snbep_pci2phy_map_init(int devid, int nodeid_loc, int idmap_loc, bool
}
}
raw_spin_unlock(&pci2phy_map_lock);
-
- if (WARN_ON_ONCE(die_id == -1)) {
- err = -EINVAL;
- break;
- }
}
}
diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h
index b95cab9846dd..897d0abd9410 100644
--- a/arch/x86/events/perf_event.h
+++ b/arch/x86/events/perf_event.h
@@ -468,6 +468,14 @@ struct cpu_hw_events {
__EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \
HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_LAT_HYBRID)
+#define INTEL_HYBRID_LDLAT_CONSTRAINT(c, n) \
+ __EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \
+ HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_LAT_HYBRID|PERF_X86_EVENT_PEBS_LD_HSW)
+
+#define INTEL_HYBRID_STLAT_CONSTRAINT(c, n) \
+ __EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \
+ HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_LAT_HYBRID|PERF_X86_EVENT_PEBS_ST_HSW)
+
/* Event constraint, but match on all event flags too. */
#define INTEL_FLAGS_EVENT_CONSTRAINT(c, n) \
EVENT_CONSTRAINT(c, n, ARCH_PERFMON_EVENTSEL_EVENT|X86_ALL_EVENT_FLAGS)
@@ -606,7 +614,7 @@ union perf_capabilities {
u64 pebs_baseline:1;
u64 perf_metrics:1;
u64 pebs_output_pt_available:1;
- u64 anythread_deprecated:1;
+ u64 __reserved:1;
};
u64 capabilities;
};
@@ -1497,7 +1505,7 @@ void intel_pmu_disable_bts(void);
int intel_pmu_drain_bts_buffer(void);
-u64 adl_latency_data_small(struct perf_event *event, u64 status);
+u64 grt_latency_data(struct perf_event *event, u64 status);
extern struct event_constraint intel_core2_pebs_event_constraints[];
diff --git a/arch/x86/kernel/alternative.c b/arch/x86/kernel/alternative.c
index 3bc6be6404d2..af47ab50e7c1 100644
--- a/arch/x86/kernel/alternative.c
+++ b/arch/x86/kernel/alternative.c
@@ -1976,11 +1976,6 @@ void __ref text_poke_queue(void *addr, const void *opcode, size_t len, const voi
{
struct text_poke_loc *tp;
- if (unlikely(system_state == SYSTEM_BOOTING)) {
- text_poke_early(addr, opcode, len);
- return;
- }
-
text_poke_flush(addr);
tp = &tp_vec[tp_vec_nr++];
@@ -2002,11 +1997,6 @@ void __ref text_poke_bp(void *addr, const void *opcode, size_t len, const void *
{
struct text_poke_loc tp;
- if (unlikely(system_state == SYSTEM_BOOTING)) {
- text_poke_early(addr, opcode, len);
- return;
- }
-
text_poke_loc_init(&tp, addr, opcode, len, emulate);
text_poke_bp_batch(&tp, 1);
}
diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c
index a61606a614bd..8d9c067992e5 100644
--- a/arch/x86/kernel/cpu/amd.c
+++ b/arch/x86/kernel/cpu/amd.c
@@ -1161,6 +1161,17 @@ static void init_amd_zen3(struct cpuinfo_x86 *c)
static void init_amd_zen4(struct cpuinfo_x86 *c)
{
init_amd_zen_common();
+
+ /*
+ * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE
+ * in some BIOS versions but they can lead to random host reboots.
+ */
+ switch (c->x86_model) {
+ case 0x18 ... 0x1f:
+ case 0x60 ... 0x7f:
+ clear_cpu_cap(c, X86_FEATURE_V_VMSAVE_VMLOAD);
+ break;
+ }
}
static void init_amd(struct cpuinfo_x86 *c)
diff --git a/arch/x86/kernel/cpu/mce/severity.c b/arch/x86/kernel/cpu/mce/severity.c
index 00483d1c27e4..7793dfd30052 100644
--- a/arch/x86/kernel/cpu/mce/severity.c
+++ b/arch/x86/kernel/cpu/mce/severity.c
@@ -283,14 +283,12 @@ static noinstr int error_context(struct mce *m, struct pt_regs *regs)
copy_user = is_copy_from_user(regs);
instrumentation_end();
- switch (fixup_type) {
- case EX_TYPE_UACCESS:
- case EX_TYPE_COPY:
- if (!copy_user)
- return IN_KERNEL;
- m->kflags |= MCE_IN_KERNEL_COPYIN;
- fallthrough;
+ if (copy_user) {
+ m->kflags |= MCE_IN_KERNEL_COPYIN | MCE_IN_KERNEL_RECOV;
+ return IN_KERNEL_RECOV;
+ }
+ switch (fixup_type) {
case EX_TYPE_FAULT_MCE_SAFE:
case EX_TYPE_DEFAULT_MCE_SAFE:
m->kflags |= MCE_IN_KERNEL_RECOV;
diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
index 1100747ffebf..6de03440d66a 100644
--- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c
+++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c
@@ -2068,20 +2068,23 @@ void rdtgroup_kn_unlock(struct kernfs_node *kn)
{
struct rdtgroup *rdtgrp = kernfs_to_rdtgroup(kn);
+ bool needs_free;
+
if (!rdtgrp)
return;
+ needs_free = atomic_dec_and_test(&rdtgrp->waitcount) &&
+ (rdtgrp->flags & RDT_DELETED);
+
mutex_unlock(&rdtgroup_mutex);
- if (atomic_dec_and_test(&rdtgrp->waitcount) &&
- (rdtgrp->flags & RDT_DELETED)) {
+ kernfs_unbreak_active_protection(kn);
+
+ if (needs_free) {
if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP ||
rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED)
rdtgroup_pseudo_lock_remove(rdtgrp);
- kernfs_unbreak_active_protection(kn);
rdtgroup_remove(rdtgrp);
- } else {
- kernfs_unbreak_active_protection(kn);
}
}
diff --git a/arch/x86/kernel/ftrace.c b/arch/x86/kernel/ftrace.c
index 48cf91625a16..c20422367f82 100644
--- a/arch/x86/kernel/ftrace.c
+++ b/arch/x86/kernel/ftrace.c
@@ -415,8 +415,7 @@ create_trampoline(struct ftrace_ops *ops, unsigned int *tramp_size)
/* ALLOC_TRAMP flags lets us know we created it */
ops->flags |= FTRACE_OPS_FL_ALLOC_TRAMP;
- if (likely(system_state != SYSTEM_BOOTING))
- set_memory_ro((unsigned long)trampoline, npages);
+ set_memory_ro((unsigned long)trampoline, npages);
set_memory_x((unsigned long)trampoline, npages);
return (unsigned long)trampoline;
fail:
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index d7f1fbf8a83d..33e89555bb09 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -2395,7 +2395,9 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp,
* avoids retaining a large number of stale nested SPs.
*/
if (tdp_enabled && invalid_list &&
- child->role.guest_mode && !child->parent_ptes.val)
+ child->role.guest_mode &&
+ !child->root_count &&
+ !child->parent_ptes.val)
return kvm_mmu_prepare_zap_page(kvm, child,
invalid_list);
}
diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c
index 6a4a0d15d4a3..a35fc16ba384 100644
--- a/arch/x86/kvm/svm/nested.c
+++ b/arch/x86/kvm/svm/nested.c
@@ -589,10 +589,12 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm, struct vmcb *vmcb12
*/
svm_copy_lbrs(vmcb02, vmcb12);
vmcb02->save.dbgctl &= ~DEBUGCTL_RESERVED_BITS;
+ vmcb_mark_dirty(vmcb02, VMCB_LBR);
svm_update_lbrv(&svm->vcpu);
} else if (unlikely(vmcb01->control.virt_ext & LBR_CTL_ENABLE_MASK)) {
svm_copy_lbrs(vmcb02, vmcb01);
+ vmcb_mark_dirty(vmcb02, VMCB_LBR);
}
}
@@ -684,24 +686,30 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm,
vmcb02->control.event_inj_err = svm->nested.ctl.event_inj_err;
/*
- * next_rip is consumed on VMRUN as the return address pushed on the
+ * NextRIP is consumed on VMRUN as the return address pushed on the
* stack for injected soft exceptions/interrupts. If nrips is exposed
- * to L1, take it verbatim from vmcb12. If nrips is supported in
- * hardware but not exposed to L1, stuff the actual L2 RIP to emulate
- * what a nrips=0 CPU would do (L1 is responsible for advancing RIP
- * prior to injecting the event).
+ * to L1, take it verbatim from vmcb12.
+ *
+ * If nrips is supported in hardware but not exposed to L1, stuff the
+ * actual L2 RIP to emulate what a nrips=0 CPU would do (L1 is
+ * responsible for advancing RIP prior to injecting the event). This is
+ * only the case for the first L2 run after VMRUN. After that (e.g.
+ * during save/restore), NextRIP is updated by the CPU and/or KVM, and
+ * the value of the L2 RIP from vmcb12 should not be used.
*/
- if (svm->nrips_enabled)
- vmcb02->control.next_rip = svm->nested.ctl.next_rip;
- else if (boot_cpu_has(X86_FEATURE_NRIPS))
- vmcb02->control.next_rip = vmcb12_rip;
+ if (boot_cpu_has(X86_FEATURE_NRIPS)) {
+ if (svm->nrips_enabled || !svm->nested.nested_run_pending)
+ vmcb02->control.next_rip = svm->nested.ctl.next_rip;
+ else
+ vmcb02->control.next_rip = vmcb12_rip;
+ }
svm->nmi_l1_to_l2 = is_evtinj_nmi(vmcb02->control.event_inj);
if (is_evtinj_soft(vmcb02->control.event_inj)) {
svm->soft_int_injected = true;
svm->soft_int_csbase = vmcb12_csbase;
svm->soft_int_old_rip = vmcb12_rip;
- if (svm->nrips_enabled)
+ if (svm->nrips_enabled || !svm->nested.nested_run_pending)
svm->soft_int_next_rip = svm->nested.ctl.next_rip;
else
svm->soft_int_next_rip = vmcb12_rip;
@@ -1006,6 +1014,7 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
svm_update_lbrv(vcpu);
} else if (unlikely(vmcb01->control.virt_ext & LBR_CTL_ENABLE_MASK)) {
svm_copy_lbrs(vmcb01, vmcb02);
+ vmcb_mark_dirty(vmcb01, VMCB_LBR);
svm_update_lbrv(vcpu);
}
@@ -1057,9 +1066,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm)
nested_svm_uninit_mmu_context(vcpu);
- rc = nested_svm_load_cr3(vcpu, vmcb01->save.cr3, false, true);
- if (rc)
- return 1;
+ if (nested_svm_load_cr3(vcpu, vmcb01->save.cr3, false, true))
+ kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu);
/*
* Drop what we picked up for L2 via svm_complete_interrupts() so it
diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c
index 53b37f3d770b..52592ebc156d 100644
--- a/arch/x86/kvm/svm/sev.c
+++ b/arch/x86/kvm/svm/sev.c
@@ -393,7 +393,7 @@ static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp)
static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
unsigned long ulen, unsigned long *n,
- int write)
+ unsigned int flags)
{
struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info;
unsigned long npages, size;
@@ -437,7 +437,7 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr,
return ERR_PTR(-ENOMEM);
/* Pin the user virtual address. */
- npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages);
+ npinned = pin_user_pages_fast(uaddr, npages, flags, pages);
if (npinned != npages) {
pr_err("SEV: Failure locking %lu pages.\n", npages);
ret = -ENOMEM;
@@ -525,7 +525,7 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
vaddr_end = vaddr + size;
/* Lock the user memory. */
- inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1);
+ inpages = sev_pin_memory(kvm, vaddr, size, &npages, FOLL_WRITE);
if (IS_ERR(inpages))
return PTR_ERR(inpages);
@@ -958,7 +958,7 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec)
if (IS_ERR(src_p))
return PTR_ERR(src_p);
- dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, 1);
+ dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, FOLL_WRITE);
if (IS_ERR(dst_p)) {
sev_unpin_memory(kvm, src_p, n);
return PTR_ERR(dst_p);
@@ -1025,7 +1025,7 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp)
if (copy_from_user(¶ms, (void __user *)(uintptr_t)argp->data, sizeof(params)))
return -EFAULT;
- pages = sev_pin_memory(kvm, params.guest_uaddr, params.guest_len, &n, 1);
+ pages = sev_pin_memory(kvm, params.guest_uaddr, params.guest_len, &n, FOLL_WRITE);
if (IS_ERR(pages))
return PTR_ERR(pages);
@@ -1507,7 +1507,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp)
/* Pin guest memory */
guest_page = sev_pin_memory(kvm, params.guest_uaddr & PAGE_MASK,
- PAGE_SIZE, &n, 1);
+ PAGE_SIZE, &n, FOLL_WRITE);
if (IS_ERR(guest_page)) {
ret = PTR_ERR(guest_page);
goto e_free_trans;
@@ -1956,6 +1956,8 @@ int sev_mem_enc_register_region(struct kvm *kvm,
struct enc_region *region;
int ret = 0;
+ guard(mutex)(&kvm->lock);
+
if (!sev_guest(kvm))
return -ENOTTY;
@@ -1970,11 +1972,14 @@ int sev_mem_enc_register_region(struct kvm *kvm,
if (!region)
return -ENOMEM;
- mutex_lock(&kvm->lock);
- region->pages = sev_pin_memory(kvm, range->addr, range->size, ®ion->npages, 1);
+ /*
+ * Do NOT specify FOLL_WRITE, as KVM isn't using the pinned pages to
+ * write memory, and FOLL_LONGTERM itself triggers CoW unshare.
+ */
+ region->pages = sev_pin_memory(kvm, range->addr, range->size, ®ion->npages,
+ FOLL_LONGTERM);
if (IS_ERR(region->pages)) {
ret = PTR_ERR(region->pages);
- mutex_unlock(&kvm->lock);
goto e_free;
}
@@ -1992,8 +1997,6 @@ int sev_mem_enc_register_region(struct kvm *kvm,
region->size = range->size;
list_add_tail(®ion->list, &sev->regions_list);
- mutex_unlock(&kvm->lock);
-
return ret;
e_free:
@@ -2953,6 +2956,11 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu)
control->exit_info_1, control->exit_info_2);
ret = -EINVAL;
break;
+ case SVM_EXIT_IOIO:
+ if (!((control->exit_info_1 & SVM_IOIO_SIZE_MASK) >> SVM_IOIO_SIZE_SHIFT))
+ return 1;
+
+ fallthrough;
default:
ret = svm_invoke_exit_handler(vcpu, exit_code);
}
@@ -2973,6 +2981,9 @@ int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in)
if (unlikely(check_mul_overflow(count, size, &bytes)))
return -EINVAL;
+ if (!bytes)
+ return 1;
+
r = setup_vmgexit_scratch(svm, in, bytes);
if (r)
return r;
diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c
index 88979fefaa28..7b6110ca9f68 100644
--- a/arch/x86/kvm/svm/svm.c
+++ b/arch/x86/kvm/svm/svm.c
@@ -732,17 +732,7 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr)
u32 offset;
u32 *msrpm;
- /*
- * For non-nested case:
- * If the L01 MSR bitmap does not intercept the MSR, then we need to
- * save it.
- *
- * For nested case:
- * If the L02 MSR bitmap does not intercept the MSR, then we need to
- * save it.
- */
- msrpm = is_guest_mode(vcpu) ? to_svm(vcpu)->nested.msrpm:
- to_svm(vcpu)->msrpm;
+ msrpm = __va(__sme_clr(to_svm(vcpu)->vmcb->control.msrpm_base_pa));
offset = svm_msrpm_offset(msr);
bit_write = 2 * (msr & 0x0f) + 1;
@@ -920,8 +910,6 @@ void svm_copy_lbrs(struct vmcb *to_vmcb, struct vmcb *from_vmcb)
to_vmcb->save.br_to = from_vmcb->save.br_to;
to_vmcb->save.last_excp_from = from_vmcb->save.last_excp_from;
to_vmcb->save.last_excp_to = from_vmcb->save.last_excp_to;
-
- vmcb_mark_dirty(to_vmcb, VMCB_LBR);
}
static void svm_enable_lbrv(struct kvm_vcpu *vcpu)
@@ -935,8 +923,10 @@ static void svm_enable_lbrv(struct kvm_vcpu *vcpu)
set_msr_interception(vcpu, svm->msrpm, MSR_IA32_LASTINTTOIP, 1, 1);
/* Move the LBR msrs to the vmcb02 so that the guest can see them. */
- if (is_guest_mode(vcpu))
+ if (is_guest_mode(vcpu)) {
svm_copy_lbrs(svm->vmcb, svm->vmcb01.ptr);
+ vmcb_mark_dirty(svm->vmcb, VMCB_LBR);
+ }
}
static void svm_disable_lbrv(struct kvm_vcpu *vcpu)
@@ -953,8 +943,10 @@ static void svm_disable_lbrv(struct kvm_vcpu *vcpu)
* Move the LBR msrs back to the vmcb01 to avoid copying them
* on nested guest entries.
*/
- if (is_guest_mode(vcpu))
+ if (is_guest_mode(vcpu)) {
svm_copy_lbrs(svm->vmcb01.ptr, svm->vmcb);
+ vmcb_mark_dirty(svm->vmcb01.ptr, VMCB_LBR);
+ }
}
static int svm_get_lbr_msr(struct vcpu_svm *svm, u32 index)
@@ -2140,6 +2132,8 @@ static int shutdown_interception(struct kvm_vcpu *vcpu)
* in response to shutdown.
*/
clear_page(svm->vmcb);
+ if (is_smm(vcpu))
+ kvm_smm_changed(vcpu, false);
kvm_vcpu_reset(vcpu, true);
kvm_run->exit_reason = KVM_EXIT_SHUTDOWN;
diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c
index 7febe4f2ee06..ca610108c4b7 100644
--- a/arch/x86/kvm/vmx/nested.c
+++ b/arch/x86/kvm/vmx/nested.c
@@ -3540,8 +3540,15 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu,
vmentry_fail_vmexit:
vmx_switch_vmcs(vcpu, &vmx->vmcs01);
- if (!from_vmentry)
+ if (!from_vmentry) {
+ nested_ept_uninit_mmu_context(vcpu);
return NVMX_VMENTRY_VMEXIT;
+ }
+
+ kvm_vcpu_unmap(vcpu, &vmx->nested.apic_access_page_map, false);
+ kvm_vcpu_unmap(vcpu, &vmx->nested.virtual_apic_map, true);
+ kvm_vcpu_unmap(vcpu, &vmx->nested.pi_desc_map, true);
+ vmx->nested.pi_desc = NULL;
load_vmcs12_host_state(vcpu, vmcs12);
vmcs12->vm_exit_reason = exit_reason.full;
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index ffbf532baf7c..daa52c04630a 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -5165,8 +5165,6 @@ static void kvm_vcpu_ioctl_x86_get_vcpu_events(struct kvm_vcpu *vcpu,
memset(&events->reserved, 0, sizeof(events->reserved));
}
-static void kvm_smm_changed(struct kvm_vcpu *vcpu, bool entering_smm);
-
static int kvm_vcpu_ioctl_x86_set_vcpu_events(struct kvm_vcpu *vcpu,
struct kvm_vcpu_events *events)
{
@@ -8676,7 +8674,7 @@ static bool retry_instruction(struct x86_emulate_ctxt *ctxt,
static int complete_emulated_mmio(struct kvm_vcpu *vcpu);
static int complete_emulated_pio(struct kvm_vcpu *vcpu);
-static void kvm_smm_changed(struct kvm_vcpu *vcpu, bool entering_smm)
+void kvm_smm_changed(struct kvm_vcpu *vcpu, bool entering_smm)
{
trace_kvm_smm_transition(vcpu->vcpu_id, vcpu->arch.smbase, entering_smm);
@@ -8698,6 +8696,7 @@ static void kvm_smm_changed(struct kvm_vcpu *vcpu, bool entering_smm)
kvm_mmu_reset_context(vcpu);
}
+EXPORT_SYMBOL_GPL(kvm_smm_changed);
static int kvm_vcpu_check_hw_bp(unsigned long addr, u32 type, u32 dr7,
unsigned long *db)
diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h
index 9bb2f237b0fc..f3b83d726f4e 100644
--- a/arch/x86/kvm/x86.h
+++ b/arch/x86/kvm/x86.h
@@ -82,6 +82,7 @@ static inline unsigned int __shrink_ple_window(unsigned int val,
void kvm_service_local_tlb_flush_requests(struct kvm_vcpu *vcpu);
int kvm_check_nested_events(struct kvm_vcpu *vcpu);
+void kvm_smm_changed(struct kvm_vcpu *vcpu, bool entering_smm);
static inline bool kvm_is_exception_pending(struct kvm_vcpu *vcpu)
{
@@ -204,7 +205,7 @@ static inline bool is_noncanonical_address(u64 la, struct kvm_vcpu *vcpu)
static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu,
gva_t gva, gfn_t gfn, unsigned access)
{
- u64 gen = kvm_memslots(vcpu->kvm)->generation;
+ u64 gen = kvm_vcpu_memslots(vcpu)->generation;
if (unlikely(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS))
return;
@@ -221,7 +222,7 @@ static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu,
static inline bool vcpu_match_mmio_gen(struct kvm_vcpu *vcpu)
{
- return vcpu->arch.mmio_gen == kvm_memslots(vcpu->kvm)->generation;
+ return vcpu->arch.mmio_gen == kvm_vcpu_memslots(vcpu)->generation;
}
/*
diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c
index 30c6a021abca..803fa8b249e3 100644
--- a/arch/x86/net/bpf_jit_comp.c
+++ b/arch/x86/net/bpf_jit_comp.c
@@ -2131,10 +2131,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i
u8 *prog;
bool save_ret;
- /* extra registers for struct arguments */
for (i = 0; i < m->nr_args; i++)
- if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG)
- nr_regs += (m->arg_size[i] + 7) / 8 - 1;
+ nr_regs += (m->arg_size[i] + 7) / 8 - 1;
/* x86-64 supports up to 6 arguments. 7+ can be added in the future */
if (nr_regs > 6)
diff --git a/arch/x86/pci/fixup.c b/arch/x86/pci/fixup.c
index 6ea3a64764b9..7d06107ab982 100644
--- a/arch/x86/pci/fixup.c
+++ b/arch/x86/pci/fixup.c
@@ -3,9 +3,11 @@
* Exceptions for specific devices. Usually work-arounds for fatal design flaws.
*/
+#include <linux/bitfield.h>
#include <linux/delay.h>
#include <linux/dmi.h>
#include <linux/pci.h>
+#include <linux/suspend.h>
#include <linux/vgaarb.h>
#include <asm/amd_nb.h>
#include <asm/hpet.h>
@@ -944,3 +946,90 @@ DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1122,
quirk_amd_nbio_7_11_disable_enhanced_atomic);
#endif
+
+#ifdef CONFIG_SUSPEND
+/*
+ * Root Ports on some AMD SoCs advertise PME_Support for D3hot and D3cold, but
+ * if the SoC is put into a hardware sleep state by the amd-pmc driver, the
+ * Root Ports don't generate wakeup interrupts for USB devices.
+ *
+ * When suspending, remove D3hot and D3cold from the PME_Support advertised
+ * by the Root Port so we don't use those states if we're expecting wakeup
+ * interrupts. Restore the advertised PME_Support when resuming.
+ */
+static void amd_rp_pme_suspend(struct pci_dev *dev)
+{
+ struct pci_dev *rp;
+
+ /*
+ * PM_SUSPEND_ON means we're doing runtime suspend, which means
+ * amd-pmc will not be involved so PMEs during D3 work as advertised.
+ *
+ * The PMEs *do* work if amd-pmc doesn't put the SoC in the hardware
+ * sleep state, but we assume amd-pmc is always present.
+ */
+ if (pm_suspend_target_state == PM_SUSPEND_ON)
+ return;
+
+ rp = pcie_find_root_port(dev);
+ if (!rp->pm_cap)
+ return;
+
+ rp->pme_support &= ~((PCI_PM_CAP_PME_D3hot|PCI_PM_CAP_PME_D3cold) >>
+ PCI_PM_CAP_PME_SHIFT);
+ dev_info_once(&rp->dev, "quirk: disabling D3cold for suspend\n");
+}
+
+static void amd_rp_pme_resume(struct pci_dev *dev)
+{
+ struct pci_dev *rp;
+ u16 pmc;
+
+ rp = pcie_find_root_port(dev);
+ if (!rp->pm_cap)
+ return;
+
+ pci_read_config_word(rp, rp->pm_cap + PCI_PM_PMC, &pmc);
+ rp->pme_support = FIELD_GET(PCI_PM_CAP_PME_MASK, pmc);
+}
+/* Rembrandt (yellow_carp) */
+DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_AMD, 0x162e, amd_rp_pme_suspend);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x162e, amd_rp_pme_resume);
+DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_AMD, 0x162f, amd_rp_pme_suspend);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x162f, amd_rp_pme_resume);
+/* Phoenix (pink_sardine) */
+DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_AMD, 0x1668, amd_rp_pme_suspend);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1668, amd_rp_pme_resume);
+DECLARE_PCI_FIXUP_SUSPEND(PCI_VENDOR_ID_AMD, 0x1669, amd_rp_pme_suspend);
+DECLARE_PCI_FIXUP_RESUME(PCI_VENDOR_ID_AMD, 0x1669, amd_rp_pme_resume);
+
+/*
+ * Putting PCIe root ports on Ryzen SoCs with USB4 controllers into D3hot
+ * may cause problems when the system attempts wake up from s2idle.
+ *
+ * On the TUXEDO Sirius 16 Gen 1 with a specific old BIOS this manifests as
+ * a system hang.
+ */
+static const struct dmi_system_id quirk_tuxeo_rp_d3_dmi_table[] = {
+ {
+ .matches = {
+ DMI_EXACT_MATCH(DMI_SYS_VENDOR, "TUXEDO"),
+ DMI_EXACT_MATCH(DMI_BOARD_NAME, "APX958"),
+ DMI_EXACT_MATCH(DMI_BIOS_VERSION, "V1.00A00_20240108"),
+ },
+ },
+ {}
+};
+
+static void quirk_tuxeo_rp_d3(struct pci_dev *pdev)
+{
+ struct pci_dev *root_pdev;
+
+ if (dmi_check_system(quirk_tuxeo_rp_d3_dmi_table)) {
+ root_pdev = pcie_find_root_port(pdev);
+ if (root_pdev)
+ root_pdev->dev_flags |= PCI_DEV_FLAGS_NO_D3;
+ }
+}
+DECLARE_PCI_FIXUP_FINAL(PCI_VENDOR_ID_AMD, 0x1502, quirk_tuxeo_rp_d3);
+#endif /* CONFIG_SUSPEND */
diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
index 8e2478efd669..136a61a1458b 100644
--- a/block/bfq-cgroup.c
+++ b/block/bfq-cgroup.c
@@ -316,14 +316,12 @@ struct bfq_group *bfqq_group(struct bfq_queue *bfqq)
static void bfqg_get(struct bfq_group *bfqg)
{
- bfqg->ref++;
+ atomic_inc(&bfqg->ref);
}
static void bfqg_put(struct bfq_group *bfqg)
{
- bfqg->ref--;
-
- if (bfqg->ref == 0)
+ if (atomic_dec_and_test(&bfqg->ref))
kfree(bfqg);
}
@@ -556,14 +554,21 @@ static void bfq_pd_init(struct blkg_policy_data *pd)
bfqg->rq_pos_tree = RB_ROOT;
}
-static void bfq_pd_free(struct blkg_policy_data *pd)
+static void bfqg_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct bfq_group *bfqg = pd_to_bfqg(pd);
bfqg_stats_exit(&bfqg->stats);
bfqg_put(bfqg);
}
+static void bfq_pd_free(struct blkg_policy_data *pd)
+{
+ call_rcu(&pd->rcu_head, bfqg_release);
+}
+
static void bfq_pd_reset_stats(struct blkg_policy_data *pd)
{
struct bfq_group *bfqg = pd_to_bfqg(pd);
diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
index d1c62a4e8c35..e9fee4d9d08a 100644
--- a/block/bfq-iosched.c
+++ b/block/bfq-iosched.c
@@ -5496,16 +5496,16 @@ bfq_set_next_ioprio_data(struct bfq_queue *bfqq, struct bfq_io_cq *bic)
bfqq->new_ioprio_class = task_nice_ioclass(tsk);
break;
case IOPRIO_CLASS_RT:
- bfqq->new_ioprio = IOPRIO_PRIO_DATA(bic->ioprio);
+ bfqq->new_ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio);
bfqq->new_ioprio_class = IOPRIO_CLASS_RT;
break;
case IOPRIO_CLASS_BE:
- bfqq->new_ioprio = IOPRIO_PRIO_DATA(bic->ioprio);
+ bfqq->new_ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio);
bfqq->new_ioprio_class = IOPRIO_CLASS_BE;
break;
case IOPRIO_CLASS_IDLE:
bfqq->new_ioprio_class = IOPRIO_CLASS_IDLE;
- bfqq->new_ioprio = 7;
+ bfqq->new_ioprio = IOPRIO_NR_LEVELS - 1;
break;
}
@@ -5791,7 +5791,7 @@ static struct bfq_queue *bfq_get_queue(struct bfq_data *bfqd,
struct bfq_io_cq *bic,
bool respawn)
{
- const int ioprio = IOPRIO_PRIO_DATA(bic->ioprio);
+ const int ioprio = IOPRIO_PRIO_LEVEL(bic->ioprio);
const int ioprio_class = IOPRIO_PRIO_CLASS(bic->ioprio);
struct bfq_queue **async_bfqq = NULL;
struct bfq_queue *bfqq;
diff --git a/block/bfq-iosched.h b/block/bfq-iosched.h
index 2b413ddffbb9..278fb51919b5 100644
--- a/block/bfq-iosched.h
+++ b/block/bfq-iosched.h
@@ -953,7 +953,7 @@ struct bfq_group {
char blkg_path[128];
/* reference counter (see comments in bfq_bic_update_cgroup) */
- int ref;
+ atomic_t ref;
/* Is bfq_group still online? */
bool online;
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 7d7c99d93bdb..4c86003bf444 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -94,6 +94,8 @@ static void blkg_free_workfn(struct work_struct *work)
if (blkg->pd[i])
blkcg_policy[i]->pd_free_fn(blkg->pd[i]);
+ if (blkg->parent)
+ blkg_put(blkg->parent);
if (blkg->q)
blk_put_queue(blkg->q);
free_percpu(blkg->iostat_cpu);
@@ -128,8 +130,6 @@ static void __blkg_release(struct rcu_head *rcu)
/* release the blkcg and parent blkg refs this blkg has been holding */
css_put(&blkg->blkcg->css);
- if (blkg->parent)
- blkg_put(blkg->parent);
blkg_free(blkg);
}
@@ -656,6 +656,13 @@ struct block_device *blkcg_conf_open_bdev(char **inputp)
return ERR_PTR(-ENODEV);
}
+ mutex_lock(&bdev->bd_queue->rq_qos_mutex);
+ if (!disk_live(bdev->bd_disk)) {
+ mutex_unlock(&bdev->bd_queue->rq_qos_mutex);
+ blkdev_put_no_open(bdev);
+ return ERR_PTR(-ENODEV);
+ }
+
*inputp = input;
return bdev;
}
@@ -779,6 +786,7 @@ int blkg_conf_prep(struct blkcg *blkcg, const struct blkcg_policy *pol,
fail_exit_queue:
blk_queue_exit(q);
fail:
+ mutex_unlock(&q->rq_qos_mutex);
blkdev_put_no_open(bdev);
/*
* If queue was bypassing, we should retry. Do so after a
@@ -803,9 +811,11 @@ EXPORT_SYMBOL_GPL(blkg_conf_prep);
*/
void blkg_conf_finish(struct blkg_conf_ctx *ctx)
__releases(&ctx->bdev->bd_queue->queue_lock) __releases(rcu)
+ __releases(&ctx->bdev->bd_queue->rq_qos_mutex)
{
spin_unlock_irq(&bdev_get_queue(ctx->bdev)->queue_lock);
rcu_read_unlock();
+ mutex_unlock(&ctx->bdev->bd_queue->rq_qos_mutex);
blkdev_put_no_open(ctx->bdev);
}
EXPORT_SYMBOL_GPL(blkg_conf_finish);
@@ -1297,7 +1307,9 @@ int blkcg_init_disk(struct gendisk *disk)
if (ret)
goto err_ioprio_exit;
+ mutex_lock(&q->rq_qos_mutex);
ret = blk_iolatency_init(disk);
+ mutex_unlock(&q->rq_qos_mutex);
if (ret)
goto err_throtl_exit;
@@ -2022,7 +2034,7 @@ void blk_cgroup_bio_start(struct bio *bio)
u64_stats_update_end_irqrestore(&bis->sync, flags);
if (cgroup_subsys_on_dfl(io_cgrp_subsys))
- cgroup_rstat_updated(bio->bi_blkg->blkcg->css.cgroup, cpu);
+ __cgroup_rstat_updated(bio->bi_blkg->blkcg->css.cgroup, cpu);
put_cpu();
}
diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h
index 59815b269a20..840deb32746c 100644
--- a/block/blk-cgroup.h
+++ b/block/blk-cgroup.h
@@ -126,6 +126,8 @@ struct blkg_policy_data {
struct blkcg_gq *blkg;
int plid;
bool online;
+
+ struct rcu_head rcu_head;
};
/*
diff --git a/block/blk-core.c b/block/blk-core.c
index 1fcd80a1c54f..641bc0fa8a4a 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -412,6 +412,7 @@ struct request_queue *blk_alloc_queue(int node_id, bool alloc_srcu)
mutex_init(&q->debugfs_mutex);
mutex_init(&q->sysfs_lock);
mutex_init(&q->sysfs_dir_lock);
+ mutex_init(&q->rq_qos_mutex);
spin_lock_init(&q->queue_lock);
init_waitqueue_head(&q->mq_freeze_wq);
diff --git a/block/blk-iocost.c b/block/blk-iocost.c
index 93482cc420c9..814ad657cc56 100644
--- a/block/blk-iocost.c
+++ b/block/blk-iocost.c
@@ -3003,9 +3003,20 @@ static void ioc_pd_init(struct blkg_policy_data *pd)
spin_unlock_irqrestore(&ioc->lock, flags);
}
+static void iocg_release(struct rcu_head *rcu)
+{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
+ struct ioc_gq *iocg = pd_to_iocg(pd);
+
+ free_percpu(iocg->pcpu_stat);
+ kfree(iocg);
+}
+
static void ioc_pd_free(struct blkg_policy_data *pd)
{
struct ioc_gq *iocg = pd_to_iocg(pd);
+ struct blkcg_gq *blkg = pd_to_blkg(pd);
struct ioc *ioc = iocg->ioc;
unsigned long flags;
@@ -3027,8 +3038,14 @@ static void ioc_pd_free(struct blkg_policy_data *pd)
hrtimer_cancel(&iocg->waitq_timer);
}
- free_percpu(iocg->pcpu_stat);
- kfree(iocg);
+
+ /* off ->active_iocgs and timer gone, so nothing can re-arm the delay */
+ iocg->delay = 0;
+ iocg->indelay_since = 0;
+ if (blkg)
+ blkcg_clear_delay(blkg);
+
+ call_rcu(&pd->rcu_head, iocg_release);
}
static void ioc_pd_stat(struct blkg_policy_data *pd, struct seq_file *s)
@@ -3309,11 +3326,13 @@ static ssize_t ioc_qos_write(struct kernfs_open_file *of, char *input,
ioc_refresh_params(ioc, true);
spin_unlock_irq(&ioc->lock);
+ mutex_unlock(&bdev->bd_queue->rq_qos_mutex);
blkdev_put_no_open(bdev);
return nbytes;
einval:
ret = -EINVAL;
err:
+ mutex_unlock(&bdev->bd_queue->rq_qos_mutex);
blkdev_put_no_open(bdev);
return ret;
}
@@ -3434,12 +3453,14 @@ static ssize_t ioc_cost_model_write(struct kernfs_open_file *of, char *input,
ioc_refresh_params(ioc, true);
spin_unlock_irq(&ioc->lock);
+ mutex_unlock(&bdev->bd_queue->rq_qos_mutex);
blkdev_put_no_open(bdev);
return nbytes;
einval:
ret = -EINVAL;
err:
+ mutex_unlock(&bdev->bd_queue->rq_qos_mutex);
blkdev_put_no_open(bdev);
return ret;
}
diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c
index 571fa95aafe9..10d0ffccc874 100644
--- a/block/blk-iolatency.c
+++ b/block/blk-iolatency.c
@@ -1025,13 +1025,30 @@ static void iolatency_pd_offline(struct blkg_policy_data *pd)
iolatency_clear_scaling(blkg);
}
-static void iolatency_pd_free(struct blkg_policy_data *pd)
+static void iolat_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct iolatency_grp *iolat = pd_to_lat(pd);
+
free_percpu(iolat->stats);
kfree(iolat);
}
+static void iolatency_pd_free(struct blkg_policy_data *pd)
+{
+ struct blkcg_gq *blkg = pd_to_blkg(pd);
+
+ /*
+ * Groups throttled as collateral have min_lat_nsec == 0, so
+ * iolatency_pd_offline() leaves their delay set. Drop it here, where
+ * no in-flight bio can re-arm it via check_scale_change().
+ */
+ if (blkg)
+ blkcg_clear_delay(blkg);
+ call_rcu(&pd->rcu_head, iolat_release);
+}
+
static struct cftype iolatency_files[] = {
{
.name = "latency",
diff --git a/block/blk-mq.c b/block/blk-mq.c
index bf69079f6bd5..bc619519dd66 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -2953,16 +2953,7 @@ static bool blk_mq_can_use_cached_rq(struct request *rq, struct blk_plug *plug,
if (op_is_flush(rq->cmd_flags) != op_is_flush(bio->bi_opf))
return false;
- /*
- * If any qos ->throttle() end up blocking, we will have flushed the
- * plug and hence killed the cached_rq list as well. Pop this entry
- * before we throttle.
- */
plug->cached_rq = rq_list_next(rq);
- rq_qos_throttle(rq->q, bio);
-
- rq->cmd_flags = bio->bi_opf;
- INIT_LIST_HEAD(&rq->queuelist);
return true;
}
@@ -2994,20 +2985,23 @@ void blk_mq_submit_bio(struct bio *bio)
rq = rq_list_peek(&plug->cached_rq);
if (rq && rq->q != q)
rq = NULL;
+ if (rq && !blk_mq_can_use_cached_rq(rq, plug, bio))
+ rq = NULL;
}
if (rq) {
if (unlikely(bio_may_exceed_limits(bio, &q->limits))) {
bio = __bio_split_to_limits(bio, &q->limits, &nr_segs);
if (!bio)
- return;
+ goto free_rq;
}
if (!bio_integrity_prep(bio))
- return;
+ goto free_rq;
if (blk_mq_attempt_bio_merge(q, bio, nr_segs))
- return;
- if (blk_mq_can_use_cached_rq(rq, plug, bio))
- goto done;
- percpu_ref_get(&q->q_usage_counter);
+ goto free_rq;
+ rq_qos_throttle(q, bio);
+ rq->cmd_flags = bio->bi_opf;
+ INIT_LIST_HEAD(&rq->queuelist);
+ goto done;
} else {
if (unlikely(bio_queue_enter(bio)))
return;
@@ -3056,6 +3050,10 @@ void blk_mq_submit_bio(struct bio *bio)
else
blk_mq_run_dispatch_ops(rq->q,
blk_mq_try_issue_directly(rq->mq_hctx, rq));
+ return;
+
+free_rq:
+ blk_mq_free_request(rq);
}
#ifdef CONFIG_BLK_MQ_STACKING
diff --git a/block/blk-rq-qos.c b/block/blk-rq-qos.c
index a319311d1ced..aa45f073d51b 100644
--- a/block/blk-rq-qos.c
+++ b/block/blk-rq-qos.c
@@ -288,9 +288,11 @@ void rq_qos_wait(struct rq_wait *rqw, void *private_data,
void rq_qos_exit(struct request_queue *q)
{
+ mutex_lock(&q->rq_qos_mutex);
while (q->rq_qos) {
struct rq_qos *rqos = q->rq_qos;
q->rq_qos = rqos->next;
rqos->ops->exit(rqos);
}
+ mutex_unlock(&q->rq_qos_mutex);
}
diff --git a/block/blk-rq-qos.h b/block/blk-rq-qos.h
index 1ef1f7d4bc3c..98be0e02a86c 100644
--- a/block/blk-rq-qos.h
+++ b/block/blk-rq-qos.h
@@ -87,21 +87,18 @@ static inline void rq_wait_init(struct rq_wait *rq_wait)
static inline int rq_qos_add(struct request_queue *q, struct rq_qos *rqos)
{
+ lockdep_assert_held(&q->rq_qos_mutex);
+
/*
* No IO can be in-flight when adding rqos, so freeze queue, which
* is fine since we only support rq_qos for blk-mq queue.
- *
- * Reuse ->queue_lock for protecting against other concurrent
- * rq_qos adding/deleting
*/
blk_mq_freeze_queue(q);
- spin_lock_irq(&q->queue_lock);
if (rq_qos_id(q, rqos->id))
goto ebusy;
rqos->next = q->rq_qos;
q->rq_qos = rqos;
- spin_unlock_irq(&q->queue_lock);
blk_mq_unfreeze_queue(q);
@@ -113,7 +110,6 @@ static inline int rq_qos_add(struct request_queue *q, struct rq_qos *rqos)
return 0;
ebusy:
- spin_unlock_irq(&q->queue_lock);
blk_mq_unfreeze_queue(q);
return -EBUSY;
@@ -123,21 +119,15 @@ static inline void rq_qos_del(struct request_queue *q, struct rq_qos *rqos)
{
struct rq_qos **cur;
- /*
- * See comment in rq_qos_add() about freezing queue & using
- * ->queue_lock.
- */
- blk_mq_freeze_queue(q);
+ lockdep_assert_held(&q->rq_qos_mutex);
- spin_lock_irq(&q->queue_lock);
+ blk_mq_freeze_queue(q);
for (cur = &q->rq_qos; *cur; cur = &(*cur)->next) {
if (*cur == rqos) {
*cur = rqos->next;
break;
}
}
- spin_unlock_irq(&q->queue_lock);
-
blk_mq_unfreeze_queue(q);
mutex_lock(&q->debugfs_mutex);
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 62a3f62316df..7ae61c3f887e 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -486,16 +486,25 @@ static void throtl_pd_offline(struct blkg_policy_data *pd)
throtl_upgrade_state(tg->td);
}
-static void throtl_pd_free(struct blkg_policy_data *pd)
+static void tg_release(struct rcu_head *rcu)
{
+ struct blkg_policy_data *pd =
+ container_of(rcu, struct blkg_policy_data, rcu_head);
struct throtl_grp *tg = pd_to_tg(pd);
- del_timer_sync(&tg->service_queue.pending_timer);
blkg_rwstat_exit(&tg->stat_bytes);
blkg_rwstat_exit(&tg->stat_ios);
kfree(tg);
}
+static void throtl_pd_free(struct blkg_policy_data *pd)
+{
+ struct throtl_grp *tg = pd_to_tg(pd);
+
+ del_timer_sync(&tg->service_queue.pending_timer);
+ call_rcu(&pd->rcu_head, tg_release);
+}
+
static struct throtl_grp *
throtl_rb_first(struct throtl_service_queue *parent_sq)
{
diff --git a/block/blk-wbt.c b/block/blk-wbt.c
index c293e08b301f..46534c877c12 100644
--- a/block/blk-wbt.c
+++ b/block/blk-wbt.c
@@ -850,7 +850,9 @@ int wbt_init(struct request_queue *q)
/*
* Assign rwb and add the stats callback.
*/
+ mutex_lock(&q->rq_qos_mutex);
ret = rq_qos_add(q, &rwb->rqos);
+ mutex_unlock(&q->rq_qos_mutex);
if (ret)
goto err_free;
diff --git a/block/ioprio.c b/block/ioprio.c
index 32a456b45804..b5a942519a79 100644
--- a/block/ioprio.c
+++ b/block/ioprio.c
@@ -33,7 +33,7 @@
int ioprio_check_cap(int ioprio)
{
int class = IOPRIO_PRIO_CLASS(ioprio);
- int data = IOPRIO_PRIO_DATA(ioprio);
+ int level = IOPRIO_PRIO_LEVEL(ioprio);
switch (class) {
case IOPRIO_CLASS_RT:
@@ -49,15 +49,16 @@ int ioprio_check_cap(int ioprio)
fallthrough;
/* rt has prio field too */
case IOPRIO_CLASS_BE:
- if (data >= IOPRIO_NR_LEVELS || data < 0)
+ if (level >= IOPRIO_NR_LEVELS)
return -EINVAL;
break;
case IOPRIO_CLASS_IDLE:
break;
case IOPRIO_CLASS_NONE:
- if (data)
+ if (level)
return -EINVAL;
break;
+ case IOPRIO_CLASS_INVALID:
default:
return -EINVAL;
}
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 2e2978e2939f..ea1585f307bc 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -1279,6 +1279,7 @@ endif # if CRYPTO_DRBG_MENU
config CRYPTO_JITTERENTROPY
tristate "CPU Jitter Non-Deterministic RNG (Random Number Generator)"
select CRYPTO_RNG
+ select CRYPTO_SHA3
help
CPU Jitter RNG (Random Number Generator) from the Jitterentropy library
diff --git a/crypto/af_alg.c b/crypto/af_alg.c
index 80a8d3a4a3cf..3cfdd7f1479e 100644
--- a/crypto/af_alg.c
+++ b/crypto/af_alg.c
@@ -1126,6 +1126,8 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags,
seglen = min_t(size_t, (maxsize - len),
msg_data_left(msg));
+ /* Never pin more pages than the remaining RX accounting budget. */
+ seglen = min_t(size_t, seglen, af_alg_rcvbuf(sk));
if (list_empty(&areq->rsgl_list)) {
rsgl = &areq->first_rsgl;
diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c
index 6a1a71284fe8..0fe62be7470f 100644
--- a/crypto/algif_skcipher.c
+++ b/crypto/algif_skcipher.c
@@ -82,8 +82,14 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg,
* If more buffers are to be expected to be processed, process only
* full block size buffers.
*/
- if (ctx->more || len < ctx->used)
+ if (ctx->more || len < ctx->used) {
+ if (len < bs) {
+ err = -EINVAL;
+ goto free;
+ }
+
len -= len % bs;
+ }
/*
* Create a per request TX SGL for this request which tracks the
diff --git a/crypto/asymmetric_keys/restrict.c b/crypto/asymmetric_keys/restrict.c
index 6b1ac5f5896a..4d73b7dc97d4 100644
--- a/crypto/asymmetric_keys/restrict.c
+++ b/crypto/asymmetric_keys/restrict.c
@@ -152,10 +152,14 @@ static int key_or_keyring_common(struct key *dest_keyring,
if (IS_ERR(key))
key = NULL;
} else if (trusted->type == &key_type_asymmetric) {
+ const struct asymmetric_key_ids *kids;
const struct asymmetric_key_id **signer_ids;
- signer_ids = (const struct asymmetric_key_id **)
- asymmetric_key_ids(trusted)->id;
+ kids = asymmetric_key_ids(trusted);
+ if (!kids)
+ return -ENOKEY;
+
+ signer_ids = (const struct asymmetric_key_id **)kids->id;
/*
* The auth_ids come from the candidate key (the
diff --git a/crypto/drbg.c b/crypto/drbg.c
index 211757f3f972..8d6dff6f8c9d 100644
--- a/crypto/drbg.c
+++ b/crypto/drbg.c
@@ -2001,7 +2001,6 @@ static int drbg_kcapi_seed(struct crypto_rng *tfm,
*/
static inline int __init drbg_healthcheck_sanity(void)
{
- int len = 0;
#define OUTBUFLEN 16
unsigned char buf[OUTBUFLEN];
struct drbg_state *drbg = NULL;
@@ -2043,12 +2042,12 @@ static inline int __init drbg_healthcheck_sanity(void)
max_addtllen = drbg_max_addtl(drbg);
max_request_bytes = drbg_max_request_bytes(drbg);
drbg_string_fill(&addtl, buf, max_addtllen + 1);
- /* overflow addtllen with additonal info string */
- len = drbg_generate(drbg, buf, OUTBUFLEN, &addtl);
- BUG_ON(0 < len);
+ /* overflow addtllen with additional info string */
+ ret = drbg_generate(drbg, buf, OUTBUFLEN, &addtl);
+ BUG_ON(ret == 0);
/* overflow max_bits */
- len = drbg_generate(drbg, buf, (max_request_bytes + 1), NULL);
- BUG_ON(0 < len);
+ ret = drbg_generate(drbg, buf, max_request_bytes + 1, NULL);
+ BUG_ON(ret == 0);
/* overflow max addtllen with personalization string */
ret = drbg_seed(drbg, &addtl, false);
diff --git a/crypto/jitterentropy-kcapi.c b/crypto/jitterentropy-kcapi.c
index b9edfaa51b27..74d5c8da0e8a 100644
--- a/crypto/jitterentropy-kcapi.c
+++ b/crypto/jitterentropy-kcapi.c
@@ -2,7 +2,7 @@
* Non-physical true random number generator based on timing jitter --
* Linux Kernel Crypto API specific code
*
- * Copyright Stephan Mueller <smueller@chronox.de>, 2015
+ * Copyright Stephan Mueller <smueller@chronox.de>, 2015 - 2023
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
@@ -37,15 +37,20 @@
* DAMAGE.
*/
+#include <crypto/hash.h>
+#include <crypto/sha3.h>
#include <linux/fips.h>
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/time.h>
#include <crypto/internal/rng.h>
#include "jitterentropy.h"
+#define JENT_CONDITIONING_HASH "sha3-256-generic"
+
/***************************************************************************
* Helper function
***************************************************************************/
@@ -60,11 +65,6 @@ void jent_zfree(void *ptr)
kfree_sensitive(ptr);
}
-void jent_memcpy(void *dest, const void *src, unsigned int n)
-{
- memcpy(dest, src, n);
-}
-
/*
* Obtain a high-resolution time stamp value. The time stamp is used to measure
* the execution time of a given code path and its variations. Hence, the time
@@ -91,37 +91,171 @@ void jent_get_nstime(__u64 *out)
*out = tmp;
}
+int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
+ unsigned int addtl_len, __u64 hash_loop_cnt,
+ unsigned int stuck)
+{
+ struct shash_desc *hash_state_desc = (struct shash_desc *)hash_state;
+ SHASH_DESC_ON_STACK(desc, hash_state_desc->tfm);
+ u8 intermediary[SHA3_256_DIGEST_SIZE];
+ __u64 j = 0;
+ int ret;
+
+ desc->tfm = hash_state_desc->tfm;
+
+ if (sizeof(intermediary) != crypto_shash_digestsize(desc->tfm)) {
+ pr_warn_ratelimited("Unexpected digest size\n");
+ return -EINVAL;
+ }
+
+ /*
+ * This loop fills a buffer which is injected into the entropy pool.
+ * The main reason for this loop is to execute something over which we
+ * can perform a timing measurement. The injection of the resulting
+ * data into the pool is performed to ensure the result is used and
+ * the compiler cannot optimize the loop away in case the result is not
+ * used at all. Yet that data is considered "additional information"
+ * considering the terminology from SP800-90A without any entropy.
+ *
+ * Note, it does not matter which or how much data you inject, we are
+ * interested in one Keccack1600 compression operation performed with
+ * the crypto_shash_final.
+ */
+ for (j = 0; j < hash_loop_cnt; j++) {
+ ret = crypto_shash_init(desc) ?:
+ crypto_shash_update(desc, intermediary,
+ sizeof(intermediary)) ?:
+ crypto_shash_finup(desc, addtl, addtl_len, intermediary);
+ if (ret)
+ goto err;
+ }
+
+ /*
+ * Inject the data from the previous loop into the pool. This data is
+ * not considered to contain any entropy, but it stirs the pool a bit.
+ */
+ ret = crypto_shash_update(desc, intermediary, sizeof(intermediary));
+ if (ret)
+ goto err;
+
+ /*
+ * Insert the time stamp into the hash context representing the pool.
+ *
+ * If the time stamp is stuck, do not finally insert the value into the
+ * entropy pool. Although this operation should not do any harm even
+ * when the time stamp has no entropy, SP800-90B requires that any
+ * conditioning operation to have an identical amount of input data
+ * according to section 3.1.5.
+ */
+ if (!stuck) {
+ ret = crypto_shash_update(hash_state_desc, (u8 *)&time,
+ sizeof(__u64));
+ }
+
+err:
+ shash_desc_zero(desc);
+ memzero_explicit(intermediary, sizeof(intermediary));
+
+ return ret;
+}
+
+int jent_read_random_block(void *hash_state, char *dst, unsigned int dst_len)
+{
+ struct shash_desc *hash_state_desc = (struct shash_desc *)hash_state;
+ u8 jent_block[SHA3_256_DIGEST_SIZE];
+ /* Obtain data from entropy pool and re-initialize it */
+ int ret = crypto_shash_final(hash_state_desc, jent_block) ?:
+ crypto_shash_init(hash_state_desc) ?:
+ crypto_shash_update(hash_state_desc, jent_block,
+ sizeof(jent_block));
+
+ if (!ret && dst_len)
+ memcpy(dst, jent_block, dst_len);
+
+ memzero_explicit(jent_block, sizeof(jent_block));
+ return ret;
+}
+
/***************************************************************************
* Kernel crypto API interface
***************************************************************************/
struct jitterentropy {
- spinlock_t jent_lock;
+ struct mutex jent_lock;
struct rand_data *entropy_collector;
+ struct crypto_shash *tfm;
+ struct shash_desc *sdesc;
};
-static int jent_kcapi_init(struct crypto_tfm *tfm)
+static void jent_kcapi_cleanup(struct crypto_tfm *tfm)
{
struct jitterentropy *rng = crypto_tfm_ctx(tfm);
- int ret = 0;
- rng->entropy_collector = jent_entropy_collector_alloc(1, 0);
- if (!rng->entropy_collector)
- ret = -ENOMEM;
+ mutex_lock(&rng->jent_lock);
- spin_lock_init(&rng->jent_lock);
- return ret;
-}
+ if (rng->sdesc) {
+ shash_desc_zero(rng->sdesc);
+ kfree(rng->sdesc);
+ }
+ rng->sdesc = NULL;
-static void jent_kcapi_cleanup(struct crypto_tfm *tfm)
-{
- struct jitterentropy *rng = crypto_tfm_ctx(tfm);
+ if (rng->tfm)
+ crypto_free_shash(rng->tfm);
+ rng->tfm = NULL;
- spin_lock(&rng->jent_lock);
if (rng->entropy_collector)
jent_entropy_collector_free(rng->entropy_collector);
rng->entropy_collector = NULL;
- spin_unlock(&rng->jent_lock);
+ mutex_unlock(&rng->jent_lock);
+}
+
+static int jent_kcapi_init(struct crypto_tfm *tfm)
+{
+ struct jitterentropy *rng = crypto_tfm_ctx(tfm);
+ struct crypto_shash *hash;
+ struct shash_desc *sdesc;
+ int size, ret = 0;
+
+ mutex_init(&rng->jent_lock);
+
+ /*
+ * Use SHA3-256 as conditioner. We allocate only the generic
+ * implementation as we are not interested in high-performance. The
+ * execution time of the SHA3 operation is measured and adds to the
+ * Jitter RNG's unpredictable behavior. If we have a slower hash
+ * implementation, the execution timing variations are larger. When
+ * using a fast implementation, we would need to call it more often
+ * as its variations are lower.
+ */
+ hash = crypto_alloc_shash(JENT_CONDITIONING_HASH, 0, 0);
+ if (IS_ERR(hash)) {
+ pr_err("Cannot allocate conditioning digest\n");
+ return PTR_ERR(hash);
+ }
+ rng->tfm = hash;
+
+ size = sizeof(struct shash_desc) + crypto_shash_descsize(hash);
+ sdesc = kmalloc(size, GFP_KERNEL);
+ if (!sdesc) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ sdesc->tfm = hash;
+ crypto_shash_init(sdesc);
+ rng->sdesc = sdesc;
+
+ rng->entropy_collector = jent_entropy_collector_alloc(1, 0, sdesc);
+ if (!rng->entropy_collector) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ return 0;
+
+err:
+ jent_kcapi_cleanup(tfm);
+ return ret;
}
static int jent_kcapi_random(struct crypto_rng *tfm,
@@ -131,7 +265,7 @@ static int jent_kcapi_random(struct crypto_rng *tfm,
struct jitterentropy *rng = crypto_rng_ctx(tfm);
int ret = 0;
- spin_lock(&rng->jent_lock);
+ mutex_lock(&rng->jent_lock);
ret = jent_read_entropy(rng->entropy_collector, rdata, dlen);
@@ -157,7 +291,7 @@ static int jent_kcapi_random(struct crypto_rng *tfm,
ret = -EINVAL;
}
- spin_unlock(&rng->jent_lock);
+ mutex_unlock(&rng->jent_lock);
return ret;
}
@@ -180,15 +314,24 @@ static struct rng_alg jent_alg = {
.cra_module = THIS_MODULE,
.cra_init = jent_kcapi_init,
.cra_exit = jent_kcapi_cleanup,
-
}
};
static int __init jent_mod_init(void)
{
+ SHASH_DESC_ON_STACK(desc, tfm);
+ struct crypto_shash *tfm;
int ret = 0;
- ret = jent_entropy_init();
+ tfm = crypto_alloc_shash(JENT_CONDITIONING_HASH, 0, 0);
+ if (IS_ERR(tfm))
+ return PTR_ERR(tfm);
+
+ desc->tfm = tfm;
+ crypto_shash_init(desc);
+ ret = jent_entropy_init(desc);
+ shash_desc_zero(desc);
+ crypto_free_shash(tfm);
if (ret) {
/* Handle permanent health test error */
if (fips_enabled)
diff --git a/crypto/jitterentropy.c b/crypto/jitterentropy.c
index 227cedfa4f0a..c7d7f2caa779 100644
--- a/crypto/jitterentropy.c
+++ b/crypto/jitterentropy.c
@@ -2,7 +2,7 @@
* Non-physical true random number generator based on timing jitter --
* Jitter RNG standalone code.
*
- * Copyright Stephan Mueller <smueller@chronox.de>, 2015 - 2020
+ * Copyright Stephan Mueller <smueller@chronox.de>, 2015 - 2023
*
* Design
* ======
@@ -47,7 +47,7 @@
/*
* This Jitterentropy RNG is based on the jitterentropy library
- * version 2.2.0 provided at https://www.chronox.de/jent.html
+ * version 3.4.0 provided at https://www.chronox.de/jent.html
*/
#ifdef __OPTIMIZE__
@@ -57,21 +57,22 @@
typedef unsigned long long __u64;
typedef long long __s64;
typedef unsigned int __u32;
+typedef unsigned char u8;
#define NULL ((void *) 0)
/* The entropy pool */
struct rand_data {
+ /* SHA3-256 is used as conditioner */
+#define DATA_SIZE_BITS 256
/* all data values that are vital to maintain the security
* of the RNG are marked as SENSITIVE. A user must not
* access that information while the RNG executes its loops to
* calculate the next random value. */
- __u64 data; /* SENSITIVE Actual random number */
- __u64 old_data; /* SENSITIVE Previous random number */
- __u64 prev_time; /* SENSITIVE Previous time stamp */
-#define DATA_SIZE_BITS ((sizeof(__u64)) * 8)
- __u64 last_delta; /* SENSITIVE stuck test */
- __s64 last_delta2; /* SENSITIVE stuck test */
- unsigned int osr; /* Oversample rate */
+ void *hash_state; /* SENSITIVE hash state entropy pool */
+ __u64 prev_time; /* SENSITIVE Previous time stamp */
+ __u64 last_delta; /* SENSITIVE stuck test */
+ __s64 last_delta2; /* SENSITIVE stuck test */
+ unsigned int osr; /* Oversample rate */
#define JENT_MEMORY_BLOCKS 64
#define JENT_MEMORY_BLOCKSIZE 32
#define JENT_MEMORY_ACCESSLOOPS 128
@@ -301,15 +302,13 @@ static int jent_permanent_health_failure(struct rand_data *ec)
* an entropy collection.
*
* Input:
- * @ec entropy collector struct -- may be NULL
* @bits is the number of low bits of the timer to consider
* @min is the number of bits we shift the timer value to the right at
* the end to make sure we have a guaranteed minimum value
*
* @return Newly calculated loop counter
*/
-static __u64 jent_loop_shuffle(struct rand_data *ec,
- unsigned int bits, unsigned int min)
+static __u64 jent_loop_shuffle(unsigned int bits, unsigned int min)
{
__u64 time = 0;
__u64 shuffle = 0;
@@ -317,12 +316,7 @@ static __u64 jent_loop_shuffle(struct rand_data *ec,
unsigned int mask = (1<<bits) - 1;
jent_get_nstime(&time);
- /*
- * Mix the current state of the random number into the shuffle
- * calculation to balance that shuffle a bit more.
- */
- if (ec)
- time ^= ec->data;
+
/*
* We fold the time value as much as possible to ensure that as many
* bits of the time stamp are included as possible.
@@ -344,81 +338,32 @@ static __u64 jent_loop_shuffle(struct rand_data *ec,
* execution time jitter
*
* This function injects the individual bits of the time value into the
- * entropy pool using an LFSR.
+ * entropy pool using a hash.
*
- * The code is deliberately inefficient with respect to the bit shifting
- * and shall stay that way. This function is the root cause why the code
- * shall be compiled without optimization. This function not only acts as
- * folding operation, but this function's execution is used to measure
- * the CPU execution time jitter. Any change to the loop in this function
- * implies that careful retesting must be done.
- *
- * @ec [in] entropy collector struct
- * @time [in] time stamp to be injected
- * @loop_cnt [in] if a value not equal to 0 is set, use the given value as
- * number of loops to perform the folding
- * @stuck [in] Is the time stamp identified as stuck?
+ * ec [in] entropy collector
+ * time [in] time stamp to be injected
+ * stuck [in] Is the time stamp identified as stuck?
*
* Output:
- * updated ec->data
- *
- * @return Number of loops the folding operation is performed
+ * updated hash context in the entropy collector or error code
*/
-static void jent_lfsr_time(struct rand_data *ec, __u64 time, __u64 loop_cnt,
- int stuck)
+static int jent_condition_data(struct rand_data *ec, __u64 time, int stuck)
{
- unsigned int i;
- __u64 j = 0;
- __u64 new = 0;
-#define MAX_FOLD_LOOP_BIT 4
-#define MIN_FOLD_LOOP_BIT 0
- __u64 fold_loop_cnt =
- jent_loop_shuffle(ec, MAX_FOLD_LOOP_BIT, MIN_FOLD_LOOP_BIT);
-
- /*
- * testing purposes -- allow test app to set the counter, not
- * needed during runtime
- */
- if (loop_cnt)
- fold_loop_cnt = loop_cnt;
- for (j = 0; j < fold_loop_cnt; j++) {
- new = ec->data;
- for (i = 1; (DATA_SIZE_BITS) >= i; i++) {
- __u64 tmp = time << (DATA_SIZE_BITS - i);
-
- tmp = tmp >> (DATA_SIZE_BITS - 1);
-
- /*
- * Fibonacci LSFR with polynomial of
- * x^64 + x^61 + x^56 + x^31 + x^28 + x^23 + 1 which is
- * primitive according to
- * http://poincare.matf.bg.ac.rs/~ezivkovm/publications/primpol1.pdf
- * (the shift values are the polynomial values minus one
- * due to counting bits from 0 to 63). As the current
- * position is always the LSB, the polynomial only needs
- * to shift data in from the left without wrap.
- */
- tmp ^= ((new >> 63) & 1);
- tmp ^= ((new >> 60) & 1);
- tmp ^= ((new >> 55) & 1);
- tmp ^= ((new >> 30) & 1);
- tmp ^= ((new >> 27) & 1);
- tmp ^= ((new >> 22) & 1);
- new <<= 1;
- new ^= tmp;
- }
- }
-
- /*
- * If the time stamp is stuck, do not finally insert the value into
- * the entropy pool. Although this operation should not do any harm
- * even when the time stamp has no entropy, SP800-90B requires that
- * any conditioning operation (SP800-90B considers the LFSR to be a
- * conditioning operation) to have an identical amount of input
- * data according to section 3.1.5.
- */
- if (!stuck)
- ec->data = new;
+#define SHA3_HASH_LOOP (1<<3)
+ struct {
+ int rct_count;
+ unsigned int apt_observations;
+ unsigned int apt_count;
+ unsigned int apt_base;
+ } addtl = {
+ ec->rct_count,
+ ec->apt_observations,
+ ec->apt_count,
+ ec->apt_base
+ };
+
+ return jent_hash_time(ec->hash_state, time, (u8 *)&addtl, sizeof(addtl),
+ SHA3_HASH_LOOP, stuck);
}
/*
@@ -452,7 +397,7 @@ static void jent_memaccess(struct rand_data *ec, __u64 loop_cnt)
#define MAX_ACC_LOOP_BIT 7
#define MIN_ACC_LOOP_BIT 0
__u64 acc_loop_cnt =
- jent_loop_shuffle(ec, MAX_ACC_LOOP_BIT, MIN_ACC_LOOP_BIT);
+ jent_loop_shuffle(MAX_ACC_LOOP_BIT, MIN_ACC_LOOP_BIT);
if (NULL == ec || NULL == ec->mem)
return;
@@ -520,14 +465,15 @@ static int jent_measure_jitter(struct rand_data *ec)
stuck = jent_stuck(ec, current_delta);
/* Now call the next noise sources which also injects the data */
- jent_lfsr_time(ec, current_delta, 0, stuck);
+ if (jent_condition_data(ec, current_delta, stuck))
+ stuck = 1;
return stuck;
}
/*
* Generator of one 64 bit random number
- * Function fills rand_data->data
+ * Function fills rand_data->hash_state
*
* @ec [in] Reference to entropy collector
*/
@@ -574,7 +520,7 @@ static void jent_gen_entropy(struct rand_data *ec)
* @return 0 when request is fulfilled or an error
*
* The following error codes can occur:
- * -1 entropy_collector is NULL
+ * -1 entropy_collector is NULL or the generation failed
* -2 Intermittent health failure
* -3 Permanent health failure
*/
@@ -604,7 +550,7 @@ int jent_read_entropy(struct rand_data *ec, unsigned char *data,
* Perform startup health tests and return permanent
* error if it fails.
*/
- if (jent_entropy_init())
+ if (jent_entropy_init(ec->hash_state))
return -3;
return -2;
@@ -614,7 +560,8 @@ int jent_read_entropy(struct rand_data *ec, unsigned char *data,
tocopy = (DATA_SIZE_BITS / 8);
else
tocopy = len;
- jent_memcpy(p, &ec->data, tocopy);
+ if (jent_read_random_block(ec->hash_state, p, tocopy))
+ return -1;
len -= tocopy;
p += tocopy;
@@ -628,7 +575,8 @@ int jent_read_entropy(struct rand_data *ec, unsigned char *data,
***************************************************************************/
struct rand_data *jent_entropy_collector_alloc(unsigned int osr,
- unsigned int flags)
+ unsigned int flags,
+ void *hash_state)
{
struct rand_data *entropy_collector;
@@ -655,6 +603,8 @@ struct rand_data *jent_entropy_collector_alloc(unsigned int osr,
osr = 1; /* minimum sampling rate is 1 */
entropy_collector->osr = osr;
+ entropy_collector->hash_state = hash_state;
+
/* fill the data pad with non-zero values */
jent_gen_entropy(entropy_collector);
@@ -668,7 +618,7 @@ void jent_entropy_collector_free(struct rand_data *entropy_collector)
jent_zfree(entropy_collector);
}
-int jent_entropy_init(void)
+int jent_entropy_init(void *hash_state)
{
int i;
__u64 delta_sum = 0;
@@ -681,6 +631,7 @@ int jent_entropy_init(void)
/* Required for RCT */
ec.osr = 1;
+ ec.hash_state = hash_state;
/* We could perform statistical tests here, but the problem is
* that we only have a few loop counts to do testing. These
@@ -718,7 +669,7 @@ int jent_entropy_init(void)
/* Invoke core entropy collection logic */
jent_get_nstime(&time);
ec.prev_time = time;
- jent_lfsr_time(&ec, time, 0, 0);
+ jent_condition_data(&ec, time, 0);
jent_get_nstime(&time2);
/* test whether timer works */
diff --git a/crypto/jitterentropy.h b/crypto/jitterentropy.h
index 5cc583f6bc6b..b3890ff26a02 100644
--- a/crypto/jitterentropy.h
+++ b/crypto/jitterentropy.h
@@ -2,14 +2,18 @@
extern void *jent_zalloc(unsigned int len);
extern void jent_zfree(void *ptr);
-extern void jent_memcpy(void *dest, const void *src, unsigned int n);
extern void jent_get_nstime(__u64 *out);
+extern int jent_hash_time(void *hash_state, __u64 time, u8 *addtl,
+ unsigned int addtl_len, __u64 hash_loop_cnt,
+ unsigned int stuck);
+int jent_read_random_block(void *hash_state, char *dst, unsigned int dst_len);
struct rand_data;
-extern int jent_entropy_init(void);
+extern int jent_entropy_init(void *hash_state);
extern int jent_read_entropy(struct rand_data *ec, unsigned char *data,
unsigned int len);
extern struct rand_data *jent_entropy_collector_alloc(unsigned int osr,
- unsigned int flags);
+ unsigned int flags,
+ void *hash_state);
extern void jent_entropy_collector_free(struct rand_data *entropy_collector);
diff --git a/drivers/acpi/acpi_tad.c b/drivers/acpi/acpi_tad.c
index 8383d6329c64..3ee041d5df59 100644
--- a/drivers/acpi/acpi_tad.c
+++ b/drivers/acpi/acpi_tad.c
@@ -23,6 +23,7 @@
#include <linux/acpi.h>
#include <linux/kernel.h>
#include <linux/module.h>
+#include <linux/mutex.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
#include <linux/suspend.h>
@@ -48,6 +49,9 @@ MODULE_AUTHOR("Rafael J. Wysocki");
/* Special value for disabled timer or expired timer wake policy. */
#define ACPI_TAD_WAKE_DISABLED (~(u32)0)
+/* ACPI TAD RTC */
+#define ACPI_TAD_TZ_UNSPEC 2047
+
struct acpi_tad_driver_data {
u32 capabilities;
};
@@ -66,6 +70,18 @@ struct acpi_tad_rt {
u8 padding[3]; /* must be 0 */
} __packed;
+static bool acpi_tad_rt_is_invalid(struct acpi_tad_rt *rt)
+{
+ return rt->year < 1900 || rt->year > 9999 ||
+ rt->month < 1 || rt->month > 12 ||
+ rt->hour > 23 || rt->minute > 59 || rt->second > 59 ||
+ rt->tz < -1440 ||
+ (rt->tz > 1440 && rt->tz != ACPI_TAD_TZ_UNSPEC) ||
+ rt->daylight > 3;
+}
+
+static DEFINE_MUTEX(acpi_tad_aml_lock);
+
static int acpi_tad_set_real_time(struct device *dev, struct acpi_tad_rt *rt)
{
acpi_handle handle = ACPI_HANDLE(dev);
@@ -79,18 +95,16 @@ static int acpi_tad_set_real_time(struct device *dev, struct acpi_tad_rt *rt)
unsigned long long retval;
acpi_status status;
- if (rt->year < 1900 || rt->year > 9999 ||
- rt->month < 1 || rt->month > 12 ||
- rt->hour > 23 || rt->minute > 59 || rt->second > 59 ||
- rt->tz < -1440 || (rt->tz > 1440 && rt->tz != 2047) ||
- rt->daylight > 3)
- return -ERANGE;
+ if (acpi_tad_rt_is_invalid(rt))
+ return -EINVAL;
args[0].buffer.pointer = (u8 *)rt;
args[0].buffer.length = sizeof(*rt);
pm_runtime_get_sync(dev);
+ guard(mutex)(&acpi_tad_aml_lock);
+
status = acpi_evaluate_integer(handle, "_SRT", &arg_list, &retval);
pm_runtime_put_sync(dev);
@@ -101,43 +115,115 @@ static int acpi_tad_set_real_time(struct device *dev, struct acpi_tad_rt *rt)
return 0;
}
-static int acpi_tad_get_real_time(struct device *dev, struct acpi_tad_rt *rt)
+static int acpi_tad_evaluate_grt(struct device *dev, struct acpi_tad_rt *rt)
{
acpi_handle handle = ACPI_HANDLE(dev);
struct acpi_buffer output = { ACPI_ALLOCATE_BUFFER };
- union acpi_object *out_obj;
- struct acpi_tad_rt *data;
acpi_status status;
int ret = -EIO;
- pm_runtime_get_sync(dev);
+ guard(mutex)(&acpi_tad_aml_lock);
status = acpi_evaluate_object(handle, "_GRT", NULL, &output);
+ if (ACPI_SUCCESS(status)) {
+ union acpi_object *out_obj;
+
+ out_obj = output.pointer;
+ if (out_obj->type == ACPI_TYPE_BUFFER &&
+ out_obj->buffer.length == sizeof(*rt)) {
+ struct acpi_tad_rt *data;
+
+ data = (struct acpi_tad_rt *)(out_obj->buffer.pointer);
+ if (data->valid) {
+ memcpy(rt, data, sizeof(*rt));
+ ret = 0;
+ }
+ }
+ }
+ ACPI_FREE(output.pointer);
+ return ret;
+}
- pm_runtime_put_sync(dev);
+static int __acpi_tad_get_real_time(struct device *dev, struct acpi_tad_rt *rt)
+{
+ int ret;
- if (ACPI_FAILURE(status))
- goto out_free;
+ ret = acpi_tad_evaluate_grt(dev, rt);
+ if (ret)
+ return ret;
- out_obj = output.pointer;
- if (out_obj->type != ACPI_TYPE_BUFFER)
- goto out_free;
+ if (acpi_tad_rt_is_invalid(rt))
+ return -ENODATA;
- if (out_obj->buffer.length != sizeof(*rt))
- goto out_free;
+ return 0;
+}
- data = (struct acpi_tad_rt *)(out_obj->buffer.pointer);
- if (!data->valid)
- goto out_free;
+static int acpi_tad_get_real_time(struct device *dev, struct acpi_tad_rt *rt)
+{
+ int ret;
- memcpy(rt, data, sizeof(*rt));
- ret = 0;
+ pm_runtime_get_sync(dev);
+
+ ret = __acpi_tad_get_real_time(dev, rt);
+
+ pm_runtime_put_sync(dev);
-out_free:
- ACPI_FREE(output.pointer);
return ret;
}
+static int __acpi_tad_wake_set(struct device *dev, char *method, u32 timer_id,
+ u32 value)
+{
+ acpi_handle handle = ACPI_HANDLE(dev);
+ union acpi_object args[] = {
+ { .type = ACPI_TYPE_INTEGER, },
+ { .type = ACPI_TYPE_INTEGER, },
+ };
+ struct acpi_object_list arg_list = {
+ .pointer = args,
+ .count = ARRAY_SIZE(args),
+ };
+ unsigned long long retval;
+ acpi_status status;
+
+ args[0].integer.value = timer_id;
+ args[1].integer.value = value;
+
+ guard(mutex)(&acpi_tad_aml_lock);
+
+ status = acpi_evaluate_integer(handle, method, &arg_list, &retval);
+ if (ACPI_FAILURE(status) || retval)
+ return -EIO;
+
+ return 0;
+}
+
+static int __acpi_tad_wake_read(struct device *dev, char *method, u32 timer_id,
+ unsigned long long *retval)
+{
+ acpi_handle handle = ACPI_HANDLE(dev);
+ union acpi_object args[] = {
+ { .type = ACPI_TYPE_INTEGER, },
+ };
+ struct acpi_object_list arg_list = {
+ .pointer = args,
+ .count = ARRAY_SIZE(args),
+ };
+ acpi_status status;
+
+ args[0].integer.value = timer_id;
+
+ guard(mutex)(&acpi_tad_aml_lock);
+
+ status = acpi_evaluate_integer(handle, method, &arg_list, retval);
+ if (ACPI_FAILURE(status))
+ return -EIO;
+
+ return 0;
+}
+
+/* sysfs interface */
+
static char *acpi_tad_rt_next_field(char *s, int *val)
{
char *p;
@@ -250,31 +336,15 @@ static const struct attribute_group acpi_tad_time_attr_group = {
static int acpi_tad_wake_set(struct device *dev, char *method, u32 timer_id,
u32 value)
{
- acpi_handle handle = ACPI_HANDLE(dev);
- union acpi_object args[] = {
- { .type = ACPI_TYPE_INTEGER, },
- { .type = ACPI_TYPE_INTEGER, },
- };
- struct acpi_object_list arg_list = {
- .pointer = args,
- .count = ARRAY_SIZE(args),
- };
- unsigned long long retval;
- acpi_status status;
-
- args[0].integer.value = timer_id;
- args[1].integer.value = value;
+ int ret;
pm_runtime_get_sync(dev);
- status = acpi_evaluate_integer(handle, method, &arg_list, &retval);
+ ret = __acpi_tad_wake_set(dev, method, timer_id, value);
pm_runtime_put_sync(dev);
- if (ACPI_FAILURE(status) || retval)
- return -EIO;
-
- return 0;
+ return ret;
}
static int acpi_tad_wake_write(struct device *dev, const char *buf, char *method,
@@ -300,27 +370,17 @@ static int acpi_tad_wake_write(struct device *dev, const char *buf, char *method
static ssize_t acpi_tad_wake_read(struct device *dev, char *buf, char *method,
u32 timer_id, const char *specval)
{
- acpi_handle handle = ACPI_HANDLE(dev);
- union acpi_object args[] = {
- { .type = ACPI_TYPE_INTEGER, },
- };
- struct acpi_object_list arg_list = {
- .pointer = args,
- .count = ARRAY_SIZE(args),
- };
unsigned long long retval;
- acpi_status status;
-
- args[0].integer.value = timer_id;
+ int ret;
pm_runtime_get_sync(dev);
- status = acpi_evaluate_integer(handle, method, &arg_list, &retval);
+ ret = __acpi_tad_wake_read(dev, method, timer_id, &retval);
pm_runtime_put_sync(dev);
- if (ACPI_FAILURE(status))
- return -EIO;
+ if (ret)
+ return ret;
if ((u32)retval == ACPI_TAD_WAKE_DISABLED)
return sprintf(buf, "%s\n", specval);
@@ -371,6 +431,8 @@ static int acpi_tad_clear_status(struct device *dev, u32 timer_id)
pm_runtime_get_sync(dev);
+ guard(mutex)(&acpi_tad_aml_lock);
+
status = acpi_evaluate_integer(handle, "_CWS", &arg_list, &retval);
pm_runtime_put_sync(dev);
@@ -412,6 +474,8 @@ static ssize_t acpi_tad_status_read(struct device *dev, char *buf, u32 timer_id)
pm_runtime_get_sync(dev);
+ guard(mutex)(&acpi_tad_aml_lock);
+
status = acpi_evaluate_integer(handle, "_GWS", &arg_list, &retval);
pm_runtime_put_sync(dev);
diff --git a/drivers/acpi/battery.c b/drivers/acpi/battery.c
index 5a4e02266241..9db4189ac267 100644
--- a/drivers/acpi/battery.c
+++ b/drivers/acpi/battery.c
@@ -701,7 +701,7 @@ static void battery_hook_unregister_unlocked(struct acpi_battery_hook *hook)
* de-register all the batteries that are registered.
*/
list_for_each_entry(battery, &acpi_battery_list, list) {
- hook->remove_battery(battery->bat);
+ hook->remove_battery(battery->bat, hook);
}
list_del_init(&hook->list);
@@ -736,7 +736,7 @@ void battery_hook_register(struct acpi_battery_hook *hook)
* its attributes.
*/
list_for_each_entry(battery, &acpi_battery_list, list) {
- if (hook->add_battery(battery->bat)) {
+ if (hook->add_battery(battery->bat, hook)) {
/*
* If a add-battery returns non-zero,
* the registration of the extension has failed,
@@ -774,7 +774,7 @@ static void battery_hook_add_battery(struct acpi_battery *battery)
* during the battery module initialization.
*/
list_for_each_entry_safe(hook_node, tmp, &battery_hook_list, list) {
- if (hook_node->add_battery(battery->bat)) {
+ if (hook_node->add_battery(battery->bat, hook_node)) {
/*
* The notification of the extensions has failed, to
* prevent further errors we will unload the extension.
@@ -797,7 +797,7 @@ static void battery_hook_remove_battery(struct acpi_battery *battery)
* custom attributes from the battery.
*/
list_for_each_entry(hook, &battery_hook_list, list) {
- hook->remove_battery(battery->bat);
+ hook->remove_battery(battery->bat, hook);
}
/* Then, just remove the battery from the list */
list_del(&battery->list);
diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c
index fafa15507b14..4ec726be27a6 100644
--- a/drivers/acpi/bus.c
+++ b/drivers/acpi/bus.c
@@ -643,30 +643,52 @@ static int __init acpi_setup_sb_notify_handler(void)
Device Matching
-------------------------------------------------------------------------- */
+
+static struct device *primary_physical_device(struct acpi_device *adev)
+{
+ struct acpi_device_physical_node *pn;
+
+ pn = list_first_entry_or_null(&adev->physical_node_list,
+ struct acpi_device_physical_node, node);
+ if (pn)
+ return pn->dev;
+
+ return NULL;
+}
+
/**
- * acpi_get_first_physical_node - Get first physical node of an ACPI device
+ * acpi_bus_get_primary_device - Get first physical device for a given ACPI one
+ * @adev: ACPI device to get the first physical device for.
+ *
+ * Find the first physical device for which @adev is the ACPI companion and
+ * reference count it if present.
+ *
+ * Return: Pointer to the first physical counterpart of @adev or NULL if there
+ * are none. Callers are responsible for invoking put_device() on the returned
+ * device.
+ */
+struct device *acpi_bus_get_primary_device(struct acpi_device *adev)
+{
+ if (!adev)
+ return NULL;
+
+ guard(mutex)(&adev->physical_node_lock);
+
+ return get_device(primary_physical_device(adev));
+}
+EXPORT_SYMBOL_GPL(acpi_bus_get_primary_device);
+
+/**
+ * acpi_get_first_physical_node - Find first physical node of an ACPI device
* @adev: ACPI device in question
*
* Return: First physical node of ACPI device @adev
*/
struct device *acpi_get_first_physical_node(struct acpi_device *adev)
{
- struct mutex *physical_node_lock = &adev->physical_node_lock;
- struct device *phys_dev;
-
- mutex_lock(physical_node_lock);
- if (list_empty(&adev->physical_node_list)) {
- phys_dev = NULL;
- } else {
- const struct acpi_device_physical_node *node;
+ guard(mutex)(&adev->physical_node_lock);
- node = list_first_entry(&adev->physical_node_list,
- struct acpi_device_physical_node, node);
-
- phys_dev = node->dev;
- }
- mutex_unlock(physical_node_lock);
- return phys_dev;
+ return primary_physical_device(adev);
}
EXPORT_SYMBOL_GPL(acpi_get_first_physical_node);
diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c
index 8df8e789a0db..90b9e8d725d4 100644
--- a/drivers/acpi/processor_idle.c
+++ b/drivers/acpi/processor_idle.c
@@ -1183,7 +1183,7 @@ static int acpi_processor_get_lpi_info(struct acpi_processor *pr)
return 0;
}
-int __weak acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi)
+int __weak __cpuidle acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi)
{
return -ENODEV;
}
@@ -1196,7 +1196,7 @@ int __weak acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi)
*
* Return: 0 for success or negative value for error
*/
-static int acpi_idle_lpi_enter(struct cpuidle_device *dev,
+static int __cpuidle acpi_idle_lpi_enter(struct cpuidle_device *dev,
struct cpuidle_driver *drv, int index)
{
struct acpi_processor *pr;
diff --git a/drivers/android/binder.c b/drivers/android/binder.c
index 819c7e0ece5e..a05f70bdad2c 100644
--- a/drivers/android/binder.c
+++ b/drivers/android/binder.c
@@ -2833,8 +2833,9 @@ static int binder_proc_transaction(struct binder_transaction *t,
t_outdated->buffer = NULL;
buffer->transaction = NULL;
trace_binder_transaction_update_buffer_release(buffer);
- binder_release_entire_buffer(proc, NULL, buffer, false);
+ binder_release_entire_buffer(proc, NULL, buffer, true);
binder_alloc_free_buf(&proc->alloc, buffer);
+ binder_free_txn_fixups(t_outdated);
kfree(t_outdated);
binder_stats_deleted(BINDER_STAT_TRANSACTION);
}
diff --git a/drivers/ata/libata-core.c b/drivers/ata/libata-core.c
index ce4dac033578..675eafc8a023 100644
--- a/drivers/ata/libata-core.c
+++ b/drivers/ata/libata-core.c
@@ -5499,6 +5499,7 @@ struct ata_port *ata_port_alloc(struct ata_host *host)
mutex_init(&ap->scsi_scan_mutex);
INIT_DELAYED_WORK(&ap->hotplug_task, ata_scsi_hotplug);
INIT_DELAYED_WORK(&ap->scsi_rescan_task, ata_scsi_dev_rescan);
+ INIT_WORK(&ap->deferred_qc_work, ata_scsi_deferred_qc_work);
INIT_LIST_HEAD(&ap->eh_done_q);
init_waitqueue_head(&ap->eh_wait_q);
init_completion(&ap->park_req_pending);
@@ -6124,9 +6125,11 @@ static void ata_port_detach(struct ata_port *ap)
/* wait till EH commits suicide */
ata_port_wait_eh(ap);
- /* it better be dead now */
+ /* It better be dead now and not have any remaining deferred qc. */
WARN_ON(!(ap->pflags & ATA_PFLAG_UNLOADED));
+ WARN_ON(ap->deferred_qc);
+ cancel_work_sync(&ap->deferred_qc_work);
cancel_delayed_work_sync(&ap->hotplug_task);
cancel_delayed_work_sync(&ap->scsi_rescan_task);
diff --git a/drivers/ata/libata-eh.c b/drivers/ata/libata-eh.c
index 586982a2a61f..21cb05e50c26 100644
--- a/drivers/ata/libata-eh.c
+++ b/drivers/ata/libata-eh.c
@@ -910,6 +910,12 @@ static void ata_eh_set_pending(struct ata_port *ap, int fastdrain)
ap->pflags |= ATA_PFLAG_EH_PENDING;
+ /*
+ * If we have a deferred qc, requeue it so that it is retried once EH
+ * completes.
+ */
+ ata_scsi_requeue_deferred_qc(ap);
+
if (!fastdrain)
return;
diff --git a/drivers/ata/libata-scsi.c b/drivers/ata/libata-scsi.c
index ff31b99dab36..3a495116ff0c 100644
--- a/drivers/ata/libata-scsi.c
+++ b/drivers/ata/libata-scsi.c
@@ -1690,6 +1690,75 @@ static void ata_qc_done(struct ata_queued_cmd *qc)
done(cmd);
}
+void ata_scsi_deferred_qc_work(struct work_struct *work)
+{
+ struct ata_port *ap =
+ container_of(work, struct ata_port, deferred_qc_work);
+ struct ata_queued_cmd *qc;
+ unsigned long flags;
+
+ spin_lock_irqsave(ap->lock, flags);
+
+ /*
+ * If we still have a deferred qc and we are not in EH, issue it. In
+ * such case, we should not need any more deferring the qc, so warn if
+ * qc_defer() says otherwise.
+ */
+ qc = ap->deferred_qc;
+ if (qc && !(ap->pflags &
+ (ATA_PFLAG_EH_PENDING | ATA_PFLAG_EH_IN_PROGRESS))) {
+ WARN_ON_ONCE(ap->ops->qc_defer(qc));
+ ap->deferred_qc = NULL;
+ ata_qc_issue(qc);
+ }
+
+ spin_unlock_irqrestore(ap->lock, flags);
+}
+
+void ata_scsi_requeue_deferred_qc(struct ata_port *ap)
+{
+ struct ata_queued_cmd *qc = ap->deferred_qc;
+ struct scsi_cmnd *scmd;
+
+ lockdep_assert_held(ap->lock);
+
+ /*
+ * If we have a deferred qc when a reset occurs or NCQ commands fail,
+ * do not try to be smart about what to do with this deferred command
+ * and simply retry it by completing it with DID_SOFT_ERROR.
+ */
+ if (!qc)
+ return;
+
+ scmd = qc->scsicmd;
+ ap->deferred_qc = NULL;
+ ata_qc_free(qc);
+ scmd->result = (DID_SOFT_ERROR << 16);
+ scsi_done(scmd);
+}
+
+static void ata_scsi_schedule_deferred_qc(struct ata_port *ap)
+{
+ struct ata_queued_cmd *qc = ap->deferred_qc;
+
+ lockdep_assert_held(ap->lock);
+
+ /*
+ * If we have a deferred qc, then qc_defer() is defined and we can use
+ * this callback to determine if this qc is good to go, unless EH has
+ * been scheduled.
+ */
+ if (!qc)
+ return;
+
+ if (ap->pflags & (ATA_PFLAG_EH_PENDING | ATA_PFLAG_EH_IN_PROGRESS)) {
+ ata_scsi_requeue_deferred_qc(ap);
+ return;
+ }
+ if (!ap->ops->qc_defer(qc))
+ queue_work(system_highpri_wq, &ap->deferred_qc_work);
+}
+
static void ata_scsi_qc_complete(struct ata_queued_cmd *qc)
{
struct ata_port *ap = qc->ap;
@@ -1720,6 +1789,8 @@ static void ata_scsi_qc_complete(struct ata_queued_cmd *qc)
ata_dump_status(ap, &qc->result_tf);
ata_qc_done(qc);
+
+ ata_scsi_schedule_deferred_qc(ap);
}
static int ata_scsi_qc_issue(struct ata_port *ap, struct ata_queued_cmd *qc)
@@ -1729,6 +1800,16 @@ static int ata_scsi_qc_issue(struct ata_port *ap, struct ata_queued_cmd *qc)
if (!ap->ops->qc_defer)
goto issue;
+ /*
+ * If we already have a deferred qc, then rely on the SCSI layer to
+ * requeue and defer all incoming commands until the deferred qc is
+ * processed, once all on-going commands complete.
+ */
+ if (ap->deferred_qc) {
+ ata_qc_free(qc);
+ return SCSI_MLQUEUE_DEVICE_BUSY;
+ }
+
/* Check if the command needs to be deferred. */
ret = ap->ops->qc_defer(qc);
switch (ret) {
@@ -1747,6 +1828,18 @@ static int ata_scsi_qc_issue(struct ata_port *ap, struct ata_queued_cmd *qc)
}
if (ret) {
+ /*
+ * We must defer this qc: if this is not an NCQ command, keep
+ * this qc as a deferred one and report to the SCSI layer that
+ * we issued it so that it is not requeued. The deferred qc will
+ * be issued with the port deferred_qc_work once all on-going
+ * commands complete.
+ */
+ if (!ata_is_ncq(qc->tf.protocol)) {
+ ap->deferred_qc = qc;
+ return 0;
+ }
+
/* Force a requeue of the command to defer its execution. */
ata_qc_free(qc);
return ret;
diff --git a/drivers/ata/libata.h b/drivers/ata/libata.h
index 25e18bcc0192..c60d9ce89b6c 100644
--- a/drivers/ata/libata.h
+++ b/drivers/ata/libata.h
@@ -142,6 +142,8 @@ extern int ata_scsi_user_scan(struct Scsi_Host *shost, unsigned int channel,
void ata_scsi_sdev_config(struct scsi_device *sdev);
int ata_scsi_dev_config(struct scsi_device *sdev, struct ata_device *dev);
int __ata_scsi_queuecmd(struct scsi_cmnd *scmd, struct ata_device *dev);
+void ata_scsi_deferred_qc_work(struct work_struct *work);
+void ata_scsi_requeue_deferred_qc(struct ata_port *ap);
/* libata-eh.c */
extern unsigned int ata_internal_cmd_timeout(struct ata_device *dev, u8 cmd);
diff --git a/drivers/ata/sata_dwc_460ex.c b/drivers/ata/sata_dwc_460ex.c
index 24bc1b755dc7..d3b8bbeef363 100644
--- a/drivers/ata/sata_dwc_460ex.c
+++ b/drivers/ata/sata_dwc_460ex.c
@@ -616,16 +616,11 @@ static irqreturn_t sata_dwc_isr(int irq, void *dev_instance)
status = ap->ops->sff_check_status(ap);
dev_dbg(ap->dev, "%s ATA status register=0x%x\n", __func__, status);
- tag = 0;
num_processed = 0;
while (tag_mask) {
num_processed++;
- while (!(tag_mask & 0x00000001)) {
- tag++;
- tag_mask <<= 1;
- }
-
- tag_mask &= (~0x00000001);
+ tag = __ffs(tag_mask);
+ tag_mask &= ~(1U << tag);
qc = ata_qc_from_tag(ap, tag);
if (unlikely(!qc)) {
dev_err(ap->dev, "failed to get qc");
diff --git a/drivers/base/core.c b/drivers/base/core.c
index 00737fc05157..e39910e68d82 100644
--- a/drivers/base/core.c
+++ b/drivers/base/core.c
@@ -1501,7 +1501,8 @@ static void __device_links_no_driver(struct device *dev)
if (link->supplier->links.status == DL_DEV_DRIVER_BOUND) {
WRITE_ONCE(link->status, DL_STATE_AVAILABLE);
} else {
- WARN_ON(!(link->flags & DL_FLAG_SYNC_STATE_ONLY));
+ WARN_ON(link->supplier->links.status != DL_DEV_UNBINDING &&
+ !(link->flags & DL_FLAG_SYNC_STATE_ONLY));
WRITE_ONCE(link->status, DL_STATE_DORMANT);
}
}
@@ -4076,6 +4077,41 @@ int device_for_each_child_reverse(struct device *parent, void *data,
}
EXPORT_SYMBOL_GPL(device_for_each_child_reverse);
+/**
+ * device_for_each_child_reverse_from - device child iterator in reversed order.
+ * @parent: parent struct device.
+ * @from: optional starting point in child list
+ * @fn: function to be called for each device.
+ * @data: data for the callback.
+ *
+ * Iterate over @parent's child devices, starting at @from, and call @fn
+ * for each, passing it @data. This helper is identical to
+ * device_for_each_child_reverse() when @from is NULL.
+ *
+ * @fn is checked each iteration. If it returns anything other than 0,
+ * iteration stop and that value is returned to the caller of
+ * device_for_each_child_reverse_from();
+ */
+int device_for_each_child_reverse_from(struct device *parent,
+ struct device *from, const void *data,
+ int (*fn)(struct device *, const void *))
+{
+ struct klist_iter i;
+ struct device *child;
+ int error = 0;
+
+ if (!parent->p)
+ return 0;
+
+ klist_iter_init_node(&parent->p->klist_children, &i,
+ (from ? &from->p->knode_parent : NULL));
+ while ((child = prev_device(&i)) && !error)
+ error = fn(child, data);
+ klist_iter_exit(&i);
+ return error;
+}
+EXPORT_SYMBOL_GPL(device_for_each_child_reverse_from);
+
/**
* device_find_child - device iterator for locating a particular device.
* @parent: parent struct device
diff --git a/drivers/block/drbd/drbd_receiver.c b/drivers/block/drbd/drbd_receiver.c
index 309d7bb028db..c3dee6265ca3 100644
--- a/drivers/block/drbd/drbd_receiver.c
+++ b/drivers/block/drbd/drbd_receiver.c
@@ -1030,6 +1030,9 @@ static int conn_connect(struct drbd_connection *connection)
sock.socket->sk->sk_allocation = GFP_NOIO;
msock.socket->sk->sk_allocation = GFP_NOIO;
+ sock.socket->sk->sk_use_task_frag = false;
+ msock.socket->sk->sk_use_task_frag = false;
+
sock.socket->sk->sk_priority = TC_PRIO_INTERACTIVE_BULK;
msock.socket->sk->sk_priority = TC_PRIO_INTERACTIVE;
diff --git a/drivers/block/loop.c b/drivers/block/loop.c
index e3e5d533a7ec..20ecd689c2e6 100644
--- a/drivers/block/loop.c
+++ b/drivers/block/loop.c
@@ -1472,19 +1472,36 @@ static int loop_set_dio(struct loop_device *lo, unsigned long arg)
return error;
}
-static int loop_set_block_size(struct loop_device *lo, unsigned long arg)
+static int loop_set_block_size(struct loop_device *lo, fmode_t mode,
+ struct block_device *bdev, unsigned long arg)
{
int err = 0;
- if (lo->lo_state != Lo_bound)
- return -ENXIO;
+ /*
+ * If we don't hold exclusive handle for the device, upgrade to it
+ * here to avoid changing device under exclusive owner.
+ */
+ if (!(mode & FMODE_EXCL)) {
+ err = bd_prepare_to_claim(bdev, loop_set_block_size);
+ if (err)
+ return err;
+ }
+
+ err = mutex_lock_killable(&lo->lo_mutex);
+ if (err)
+ goto abort_claim;
+
+ if (lo->lo_state != Lo_bound) {
+ err = -ENXIO;
+ goto unlock;
+ }
err = blk_validate_block_size(arg);
if (err)
- return err;
+ goto unlock;
if (lo->lo_queue->limits.logical_block_size == arg)
- return 0;
+ goto unlock;
sync_blockdev(lo->lo_device);
invalidate_bdev(lo->lo_device);
@@ -1496,6 +1513,11 @@ static int loop_set_block_size(struct loop_device *lo, unsigned long arg)
loop_update_dio(lo);
blk_mq_unfreeze_queue(lo->lo_queue);
+unlock:
+ mutex_unlock(&lo->lo_mutex);
+abort_claim:
+ if (!(mode & FMODE_EXCL))
+ bd_abort_claiming(bdev, loop_set_block_size);
return err;
}
@@ -1514,9 +1536,6 @@ static int lo_simple_ioctl(struct loop_device *lo, unsigned int cmd,
case LOOP_SET_DIRECT_IO:
err = loop_set_dio(lo, arg);
break;
- case LOOP_SET_BLOCK_SIZE:
- err = loop_set_block_size(lo, arg);
- break;
default:
err = -EINVAL;
}
@@ -1573,9 +1592,12 @@ static int lo_ioctl(struct block_device *bdev, fmode_t mode,
break;
case LOOP_GET_STATUS64:
return loop_get_status64(lo, argp);
+ case LOOP_SET_BLOCK_SIZE:
+ if (!(mode & FMODE_WRITE) && !capable(CAP_SYS_ADMIN))
+ return -EPERM;
+ return loop_set_block_size(lo, mode, bdev, arg);
case LOOP_SET_CAPACITY:
case LOOP_SET_DIRECT_IO:
- case LOOP_SET_BLOCK_SIZE:
if (!(mode & FMODE_WRITE) && !capable(CAP_SYS_ADMIN))
return -EPERM;
fallthrough;
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index 008534aad0a1..26971f58c684 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -520,6 +520,7 @@ static int __sock_xmit(struct nbd_device *nbd, struct socket *sock, int send,
noreclaim_flag = memalloc_noreclaim_save();
do {
sock->sk->sk_allocation = GFP_NOIO | __GFP_MEMALLOC;
+ sock->sk->sk_use_task_frag = false;
msg.msg_flags = msg_flags | MSG_NOSIGNAL;
if (send)
diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c
index 237e9a821374..971dbe77b294 100644
--- a/drivers/block/null_blk/main.c
+++ b/drivers/block/null_blk/main.c
@@ -552,8 +552,13 @@ static ssize_t nullb_device_zone_readonly_store(struct config_item *item,
const char *page, size_t count)
{
struct nullb_device *dev = to_nullb_device(item);
+ ssize_t ret;
+
+ mutex_lock(&lock);
+ ret = zone_cond_store(dev, page, count, BLK_ZONE_COND_READONLY);
+ mutex_unlock(&lock);
- return zone_cond_store(dev, page, count, BLK_ZONE_COND_READONLY);
+ return ret;
}
CONFIGFS_ATTR_WO(nullb_device_, zone_readonly);
@@ -561,8 +566,13 @@ static ssize_t nullb_device_zone_offline_store(struct config_item *item,
const char *page, size_t count)
{
struct nullb_device *dev = to_nullb_device(item);
+ ssize_t ret;
- return zone_cond_store(dev, page, count, BLK_ZONE_COND_OFFLINE);
+ mutex_lock(&lock);
+ ret = zone_cond_store(dev, page, count, BLK_ZONE_COND_OFFLINE);
+ mutex_unlock(&lock);
+
+ return ret;
}
CONFIGFS_ATTR_WO(nullb_device_, zone_offline);
@@ -746,7 +756,6 @@ static void null_free_dev(struct nullb_device *dev)
if (!dev)
return;
- null_free_zoned_dev(dev);
badblocks_exit(&dev->badblocks);
kfree(dev);
}
@@ -1825,6 +1834,7 @@ static void null_del_dev(struct nullb *nullb)
}
put_disk(nullb->disk);
+ null_free_zoned_dev(dev);
if (dev->queue_mode == NULL_Q_MQ &&
nullb->tag_set == &nullb->__tag_set)
blk_mq_free_tag_set(nullb->tag_set);
diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c
index c41bb08efb12..538827b68d88 100644
--- a/drivers/block/ublk_drv.c
+++ b/drivers/block/ublk_drv.c
@@ -1697,6 +1697,10 @@ static int ublk_ctrl_add_dev(struct io_uring_cmd *cmd)
if (copy_from_user(&info, argp, sizeof(info)))
return -EFAULT;
ublk_dump_dev_info(&info);
+
+ if (info.queue_depth > UBLK_MAX_QUEUE_DEPTH)
+ return -EINVAL;
+
if (header->dev_id != info.dev_id) {
pr_warn("%s: dev id not match %u %u\n",
__func__, header->dev_id, info.dev_id);
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index bef5388e4e28..b8a2cb2f901c 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -644,7 +644,7 @@ static ssize_t writeback_store(struct device *dev,
struct device_attribute *attr, const char *buf, size_t len)
{
struct zram *zram = dev_to_zram(dev);
- unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
+ unsigned long nr_pages;
unsigned long index = 0;
struct bio bio;
struct bio_vec bio_vec;
@@ -663,11 +663,9 @@ static ssize_t writeback_store(struct device *dev,
if (strncmp(buf, PAGE_WB_SIG, sizeof(PAGE_WB_SIG) - 1))
return -EINVAL;
- if (kstrtol(buf + sizeof(PAGE_WB_SIG) - 1, 10, &index) ||
- index >= nr_pages)
+ if (kstrtol(buf + sizeof(PAGE_WB_SIG) - 1, 10, &index))
return -EINVAL;
- nr_pages = 1;
mode = PAGE_WRITEBACK;
}
@@ -682,6 +680,15 @@ static ssize_t writeback_store(struct device *dev,
goto release_init_lock;
}
+ nr_pages = zram->disksize >> PAGE_SHIFT;
+ if (mode == PAGE_WRITEBACK) {
+ if (index >= nr_pages) {
+ ret = -EINVAL;
+ goto release_init_lock;
+ }
+ nr_pages = 1;
+ }
+
page = alloc_page(GFP_KERNEL);
if (!page) {
ret = -ENOMEM;
@@ -903,8 +910,7 @@ static ssize_t read_block_state(struct file *file, char __user *buf,
char *kbuf;
ssize_t index, written = 0;
struct zram *zram = file->private_data;
- unsigned long nr_pages = zram->disksize >> PAGE_SHIFT;
- struct timespec64 ts;
+ unsigned long nr_pages;
kbuf = kvmalloc(count, GFP_KERNEL);
if (!kbuf)
@@ -917,8 +923,11 @@ static ssize_t read_block_state(struct file *file, char __user *buf,
return -EINVAL;
}
+ nr_pages = zram->disksize >> PAGE_SHIFT;
+
for (index = *ppos; index < nr_pages; index++) {
int copied;
+ struct timespec64 ts;
zram_slot_lock(zram, index);
if (!zram_allocated(zram, index))
diff --git a/drivers/bluetooth/btintel.c b/drivers/bluetooth/btintel.c
index 1cba08e9403a..efc4788b3d30 100644
--- a/drivers/bluetooth/btintel.c
+++ b/drivers/bluetooth/btintel.c
@@ -248,7 +248,8 @@ static void btintel_hw_error(struct hci_dev *hdev, u8 code)
goto unlock;
}
- bt_dev_err(hdev, "Exception info %s", (char *)(skb->data + 1));
+ bt_dev_err(hdev, "Exception info %.*s", (int)(skb->len - 1),
+ (char *)(skb->data + 1));
kfree_skb(skb);
@@ -363,8 +364,16 @@ int btintel_load_ddc_config(struct hci_dev *hdev, const char *ddc_name)
/* DDC file contains one or more DDC structure which has
* Length (1 byte), DDC ID (2 bytes), and DDC value (Length - 2).
*/
- while (fw->size > fw_ptr - fw->data) {
- u8 cmd_plen = fw_ptr[0] + sizeof(u8);
+ while (fw->size > (size_t)(fw_ptr - fw->data)) {
+ size_t remaining = fw->size - (fw_ptr - fw->data);
+ unsigned int cmd_plen = fw_ptr[0] + 1U;
+
+ if (cmd_plen < 3 || cmd_plen > U8_MAX || cmd_plen > remaining) {
+ bt_dev_err(hdev, "Malformed DDC record (plen=%u, remaining=%zu)",
+ cmd_plen, remaining);
+ release_firmware(fw);
+ return -EINVAL;
+ }
skb = __hci_cmd_sync(hdev, 0xfc8b, cmd_plen, fw_ptr,
HCI_INIT_TIMEOUT);
diff --git a/drivers/bluetooth/btqcomsmd.c b/drivers/bluetooth/btqcomsmd.c
index 11c7e04bf394..4ce8354ebb86 100644
--- a/drivers/bluetooth/btqcomsmd.c
+++ b/drivers/bluetooth/btqcomsmd.c
@@ -188,7 +188,10 @@ static int btqcomsmd_probe(struct platform_device *pdev)
return 0;
hci_free_dev:
+ rpmsg_destroy_ept(btq->cmd_channel);
+ rpmsg_destroy_ept(btq->acl_channel);
hci_free_dev(hdev);
+ return ret;
destroy_cmd_channel:
rpmsg_destroy_ept(btq->cmd_channel);
destroy_acl_channel:
@@ -197,17 +200,16 @@ static int btqcomsmd_probe(struct platform_device *pdev)
return ret;
}
-static int btqcomsmd_remove(struct platform_device *pdev)
+static void btqcomsmd_remove(struct platform_device *pdev)
{
struct btqcomsmd *btq = platform_get_drvdata(pdev);
hci_unregister_dev(btq->hdev);
- hci_free_dev(btq->hdev);
rpmsg_destroy_ept(btq->cmd_channel);
rpmsg_destroy_ept(btq->acl_channel);
- return 0;
+ hci_free_dev(btq->hdev);
}
static const struct of_device_id btqcomsmd_of_match[] = {
@@ -218,7 +220,7 @@ MODULE_DEVICE_TABLE(of, btqcomsmd_of_match);
static struct platform_driver btqcomsmd_driver = {
.probe = btqcomsmd_probe,
- .remove = btqcomsmd_remove,
+ .remove_new = btqcomsmd_remove,
.driver = {
.name = "btqcomsmd",
.of_match_table = btqcomsmd_of_match,
diff --git a/drivers/bluetooth/btusb.c b/drivers/bluetooth/btusb.c
index 3c1d6a711338..5196731ce892 100644
--- a/drivers/bluetooth/btusb.c
+++ b/drivers/bluetooth/btusb.c
@@ -304,6 +304,8 @@ static const struct usb_device_id blacklist_table[] = {
BTUSB_WIDEBAND_SPEECH },
{ USB_DEVICE(0x13d3, 0x3501), .driver_info = BTUSB_QCA_ROME |
BTUSB_WIDEBAND_SPEECH },
+ { USB_DEVICE(0x13d3, 0x3503), .driver_info = BTUSB_QCA_ROME |
+ BTUSB_WIDEBAND_SPEECH },
/* QCA WCN6855 chipset */
{ USB_DEVICE(0x0cf3, 0xe600), .driver_info = BTUSB_QCA_WCN6855 |
@@ -2560,6 +2562,7 @@ static void btusb_mtk_wmt_recv(struct urb *urb)
return;
} else if (urb->status == -ENOENT) {
/* Avoid suspend failed when usb_kill_urb */
+ kfree(urb->setup_packet);
return;
}
@@ -2633,6 +2636,7 @@ static int btusb_mtk_submit_wmt_recv_urb(struct hci_dev *hdev)
if (err != -EPERM && err != -ENODEV)
bt_dev_err(hdev, "urb %p submission failed (%d)",
urb, -err);
+ kfree(dr);
usb_unanchor_urb(urb);
}
diff --git a/drivers/bus/mhi/ep/ring.c b/drivers/bus/mhi/ep/ring.c
index 115518ec76a4..926a158bb22d 100644
--- a/drivers/bus/mhi/ep/ring.c
+++ b/drivers/bus/mhi/ep/ring.c
@@ -125,19 +125,19 @@ int mhi_ep_ring_add_element(struct mhi_ep_ring *ring, struct mhi_ring_element *e
}
old_offset = ring->rd_offset;
- mhi_ep_ring_inc_index(ring);
dev_dbg(dev, "Adding an element to ring at offset (%zu)\n", ring->rd_offset);
-
- /* Update rp in ring context */
- rp = cpu_to_le64(ring->rd_offset * sizeof(*el) + ring->rbase);
- memcpy_toio((void __iomem *) &ring->ring_ctx->generic.rp, &rp, sizeof(u64));
-
ret = mhi_cntrl->write_to_host(mhi_cntrl, el, ring->rbase + (old_offset * sizeof(*el)),
sizeof(*el));
if (ret < 0)
return ret;
+ mhi_ep_ring_inc_index(ring);
+
+ /* Update rp in ring context */
+ rp = cpu_to_le64(ring->rd_offset * sizeof(*el) + ring->rbase);
+ memcpy_toio((void __iomem *) &ring->ring_ctx->generic.rp, &rp, sizeof(u64));
+
return 0;
}
diff --git a/drivers/bus/mhi/host/pci_generic.c b/drivers/bus/mhi/host/pci_generic.c
index 4bb4bcd45b0a..5a070ea4710c 100644
--- a/drivers/bus/mhi/host/pci_generic.c
+++ b/drivers/bus/mhi/host/pci_generic.c
@@ -77,22 +77,6 @@ struct mhi_pci_dev_info {
.doorbell_mode_switch = false, \
}
-#define MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(ch_num, ch_name, el_count, ev_ring) \
- { \
- .num = ch_num, \
- .name = ch_name, \
- .num_elements = el_count, \
- .event_ring = ev_ring, \
- .dir = DMA_FROM_DEVICE, \
- .ee_mask = BIT(MHI_EE_AMSS), \
- .pollcfg = 0, \
- .doorbell = MHI_DB_BRST_DISABLE, \
- .lpm_notify = false, \
- .offload_channel = false, \
- .doorbell_mode_switch = false, \
- .auto_queue = true, \
- }
-
#define MHI_EVENT_CONFIG_CTRL(ev_ring, el_count) \
{ \
.num_elements = el_count, \
@@ -231,7 +215,7 @@ static const struct mhi_channel_config modem_qcom_v1_mhi_channels[] = {
MHI_CHANNEL_CONFIG_UL(14, "QMI", 4, 0),
MHI_CHANNEL_CONFIG_DL(15, "QMI", 4, 0),
MHI_CHANNEL_CONFIG_UL(20, "IPCR", 8, 0),
- MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(21, "IPCR", 8, 0),
+ MHI_CHANNEL_CONFIG_DL(21, "IPCR", 8, 0),
MHI_CHANNEL_CONFIG_UL_FP(34, "FIREHOSE", 32, 0),
MHI_CHANNEL_CONFIG_DL_FP(35, "FIREHOSE", 32, 0),
MHI_CHANNEL_CONFIG_HW_UL(100, "IP_HW0", 128, 2),
@@ -467,7 +451,7 @@ static const struct mhi_channel_config mhi_telit_fn980_hw_v1_channels[] = {
MHI_CHANNEL_CONFIG_UL(14, "QMI", 32, 0),
MHI_CHANNEL_CONFIG_DL(15, "QMI", 32, 0),
MHI_CHANNEL_CONFIG_UL(20, "IPCR", 16, 0),
- MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(21, "IPCR", 16, 0),
+ MHI_CHANNEL_CONFIG_DL(21, "IPCR", 16, 0),
MHI_CHANNEL_CONFIG_HW_UL(100, "IP_HW0", 128, 1),
MHI_CHANNEL_CONFIG_HW_DL(101, "IP_HW0", 128, 2),
};
diff --git a/drivers/char/hw_random/core.c b/drivers/char/hw_random/core.c
index c319f9937709..4b1a0fed48ab 100644
--- a/drivers/char/hw_random/core.c
+++ b/drivers/char/hw_random/core.c
@@ -22,22 +22,24 @@
#include <linux/miscdevice.h>
#include <linux/module.h>
#include <linux/random.h>
+#include <linux/rcupdate.h>
#include <linux/sched.h>
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/uaccess.h>
+#include <linux/workqueue.h>
#define RNG_MODULE_NAME "hw_random"
#define RNG_BUFFER_SIZE (SMP_CACHE_BYTES < 32 ? 32 : SMP_CACHE_BYTES)
-static struct hwrng *current_rng;
+static struct hwrng __rcu *current_rng;
/* the current rng has been explicitly chosen by user via sysfs */
static int cur_rng_set_by_user;
static struct task_struct *hwrng_fill;
/* list of registered rngs */
static LIST_HEAD(rng_list);
-/* Protects rng_list and current_rng */
+/* Protects rng_list, hwrng_fill and updating on current_rng */
static DEFINE_MUTEX(rng_mutex);
/* Protects rng read functions, data_avail, rng_buffer and rng_fillbuf */
static DEFINE_MUTEX(reading_mutex);
@@ -76,18 +78,39 @@ static void add_early_randomness(struct hwrng *rng)
add_device_randomness(rng_fillbuf, bytes_read);
}
-static inline void cleanup_rng(struct kref *kref)
+static void cleanup_rng_work(struct work_struct *work)
{
- struct hwrng *rng = container_of(kref, struct hwrng, ref);
+ struct hwrng *rng = container_of(work, struct hwrng, cleanup_work);
+
+ /*
+ * Hold rng_mutex here so we serialize in case they set_current_rng
+ * on rng again immediately.
+ */
+ mutex_lock(&rng_mutex);
+
+ /* Skip if rng has been reinitialized. */
+ if (kref_read(&rng->ref)) {
+ mutex_unlock(&rng_mutex);
+ return;
+ }
if (rng->cleanup)
rng->cleanup(rng);
complete(&rng->cleanup_done);
+ mutex_unlock(&rng_mutex);
+}
+
+static inline void cleanup_rng(struct kref *kref)
+{
+ struct hwrng *rng = container_of(kref, struct hwrng, ref);
+
+ schedule_work(&rng->cleanup_work);
}
static int set_current_rng(struct hwrng *rng)
{
+ struct hwrng *old_rng;
int err;
BUG_ON(!mutex_is_locked(&rng_mutex));
@@ -96,8 +119,14 @@ static int set_current_rng(struct hwrng *rng)
if (err)
return err;
- drop_current_rng();
- current_rng = rng;
+ old_rng = rcu_dereference_protected(current_rng,
+ lockdep_is_held(&rng_mutex));
+ rcu_assign_pointer(current_rng, rng);
+
+ if (old_rng) {
+ synchronize_rcu();
+ kref_put(&old_rng->ref, cleanup_rng);
+ }
/* if necessary, start hwrng thread */
if (!hwrng_fill) {
@@ -113,47 +142,56 @@ static int set_current_rng(struct hwrng *rng)
static void drop_current_rng(void)
{
- BUG_ON(!mutex_is_locked(&rng_mutex));
- if (!current_rng)
+ struct hwrng *rng;
+
+ rng = rcu_dereference_protected(current_rng,
+ lockdep_is_held(&rng_mutex));
+ if (!rng)
return;
+ RCU_INIT_POINTER(current_rng, NULL);
+ synchronize_rcu();
+
+ if (hwrng_fill) {
+ kthread_stop(hwrng_fill);
+ hwrng_fill = NULL;
+ }
+
/* decrease last reference for triggering the cleanup */
- kref_put(¤t_rng->ref, cleanup_rng);
- current_rng = NULL;
+ kref_put(&rng->ref, cleanup_rng);
}
-/* Returns ERR_PTR(), NULL or refcounted hwrng */
+/* Returns NULL or refcounted hwrng */
static struct hwrng *get_current_rng_nolock(void)
{
- if (current_rng)
- kref_get(¤t_rng->ref);
+ struct hwrng *rng;
- return current_rng;
+ rng = rcu_dereference_protected(current_rng,
+ lockdep_is_held(&rng_mutex));
+ if (rng)
+ kref_get(&rng->ref);
+
+ return rng;
}
static struct hwrng *get_current_rng(void)
{
struct hwrng *rng;
- if (mutex_lock_interruptible(&rng_mutex))
- return ERR_PTR(-ERESTARTSYS);
+ rcu_read_lock();
+ rng = rcu_dereference(current_rng);
+ if (rng)
+ kref_get(&rng->ref);
- rng = get_current_rng_nolock();
+ rcu_read_unlock();
- mutex_unlock(&rng_mutex);
return rng;
}
static void put_rng(struct hwrng *rng)
{
- /*
- * Hold rng_mutex here so we serialize in case they set_current_rng
- * on rng again immediately.
- */
- mutex_lock(&rng_mutex);
if (rng)
kref_put(&rng->ref, cleanup_rng);
- mutex_unlock(&rng_mutex);
}
static int hwrng_init(struct hwrng *rng)
@@ -222,10 +260,6 @@ static ssize_t rng_dev_read(struct file *filp, char __user *buf,
while (size) {
rng = get_current_rng();
- if (IS_ERR(rng)) {
- err = PTR_ERR(rng);
- goto out;
- }
if (!rng) {
err = -ENODEV;
goto out;
@@ -312,7 +346,7 @@ static struct miscdevice rng_miscdev = {
static int enable_best_rng(void)
{
- struct hwrng *rng, *new_rng = NULL;
+ struct hwrng *rng, *cur_rng, *new_rng = NULL;
int ret = -ENODEV;
BUG_ON(!mutex_is_locked(&rng_mutex));
@@ -330,7 +364,9 @@ static int enable_best_rng(void)
new_rng = rng;
}
- ret = ((new_rng == current_rng) ? 0 : set_current_rng(new_rng));
+ cur_rng = rcu_dereference_protected(current_rng,
+ lockdep_is_held(&rng_mutex));
+ ret = ((new_rng == cur_rng) ? 0 : set_current_rng(new_rng));
if (!ret)
cur_rng_set_by_user = 0;
@@ -381,8 +417,6 @@ static ssize_t rng_current_show(struct device *dev,
struct hwrng *rng;
rng = get_current_rng();
- if (IS_ERR(rng))
- return PTR_ERR(rng);
ret = snprintf(buf, PAGE_SIZE, "%s\n", rng ? rng->name : "none");
put_rng(rng);
@@ -426,8 +460,6 @@ static ssize_t rng_quality_show(struct device *dev,
struct hwrng *rng;
rng = get_current_rng();
- if (IS_ERR(rng))
- return PTR_ERR(rng);
if (!rng) /* no need to put_rng */
return -ENODEV;
@@ -442,6 +474,7 @@ static ssize_t rng_quality_store(struct device *dev,
struct device_attribute *attr,
const char *buf, size_t len)
{
+ struct hwrng *rng;
u16 quality;
int ret = -EINVAL;
@@ -458,12 +491,13 @@ static ssize_t rng_quality_store(struct device *dev,
goto out;
}
- if (!current_rng) {
+ rng = rcu_dereference_protected(current_rng, lockdep_is_held(&rng_mutex));
+ if (!rng) {
ret = -ENODEV;
goto out;
}
- current_rng->quality = quality;
+ rng->quality = quality;
current_quality = quality; /* obsolete */
/* the best available RNG may have changed */
@@ -509,8 +543,20 @@ static int hwrng_fillfn(void *unused)
struct hwrng *rng;
rng = get_current_rng();
- if (IS_ERR(rng) || !rng)
+ if (!rng) {
+ /*
+ * Keep the task_struct alive until kthread_stop()
+ * is called to avoid UAF in drop_current_rng().
+ */
+ while (!kthread_should_stop()) {
+ set_current_state(TASK_INTERRUPTIBLE);
+ if (!kthread_should_stop())
+ schedule();
+ }
+ set_current_state(TASK_RUNNING);
break;
+ }
+
mutex_lock(&reading_mutex);
rc = rng_get_data(rng, rng_fillbuf,
rng_buffer_size(), 1);
@@ -538,14 +584,13 @@ static int hwrng_fillfn(void *unused)
add_hwgenerator_randomness((void *)rng_fillbuf, rc,
entropy >> 10);
}
- hwrng_fill = NULL;
return 0;
}
int hwrng_register(struct hwrng *rng)
{
int err = -EINVAL;
- struct hwrng *tmp;
+ struct hwrng *cur_rng, *tmp;
bool is_new_current = false;
if (!rng->name || (!rng->data_read && !rng->read))
@@ -561,12 +606,15 @@ int hwrng_register(struct hwrng *rng)
}
list_add_tail(&rng->list, &rng_list);
+ INIT_WORK(&rng->cleanup_work, cleanup_rng_work);
init_completion(&rng->cleanup_done);
complete(&rng->cleanup_done);
init_completion(&rng->dying);
- if (!current_rng ||
- (!cur_rng_set_by_user && rng->quality > current_rng->quality)) {
+ cur_rng = rcu_dereference_protected(current_rng,
+ lockdep_is_held(&rng_mutex));
+ if (!cur_rng ||
+ (!cur_rng_set_by_user && rng->quality > cur_rng->quality)) {
/*
* Set new rng as current as the new rng source
* provides better entropy quality and was not
@@ -574,7 +622,7 @@ int hwrng_register(struct hwrng *rng)
*/
err = set_current_rng(rng);
if (err)
- goto out_unlock;
+ goto out_list_del;
/* to use current_rng in add_early_randomness() we need
* to take a ref
*/
@@ -595,6 +643,8 @@ int hwrng_register(struct hwrng *rng)
if (is_new_current)
put_rng(rng);
return 0;
+out_list_del:
+ list_del_init(&rng->list);
out_unlock:
mutex_unlock(&rng_mutex);
out:
@@ -604,15 +654,19 @@ EXPORT_SYMBOL_GPL(hwrng_register);
void hwrng_unregister(struct hwrng *rng)
{
- struct hwrng *old_rng, *new_rng;
+ struct hwrng *cur_rng, *old_rng, *new_rng;
int err;
mutex_lock(&rng_mutex);
- old_rng = current_rng;
+ old_rng = rcu_dereference_protected(current_rng,
+ lockdep_is_held(&rng_mutex));
list_del(&rng->list);
complete_all(&rng->dying);
- if (current_rng == rng) {
+
+ cur_rng = rcu_dereference_protected(current_rng,
+ lockdep_is_held(&rng_mutex));
+ if (cur_rng == rng) {
err = enable_best_rng();
if (err) {
drop_current_rng();
@@ -621,12 +675,7 @@ void hwrng_unregister(struct hwrng *rng)
}
new_rng = get_current_rng_nolock();
- if (list_empty(&rng_list)) {
- mutex_unlock(&rng_mutex);
- if (hwrng_fill)
- kthread_stop(hwrng_fill);
- } else
- mutex_unlock(&rng_mutex);
+ mutex_unlock(&rng_mutex);
if (new_rng) {
if (old_rng != new_rng)
@@ -721,7 +770,7 @@ static int __init hwrng_modinit(void)
static void __exit hwrng_modexit(void)
{
mutex_lock(&rng_mutex);
- BUG_ON(current_rng);
+ WARN_ON(rcu_access_pointer(current_rng));
kfree(rng_buffer);
kfree(rng_fillbuf);
mutex_unlock(&rng_mutex);
diff --git a/drivers/char/hw_random/stm32-rng.c b/drivers/char/hw_random/stm32-rng.c
index bc22178f83e8..440c001c746b 100644
--- a/drivers/char/hw_random/stm32-rng.c
+++ b/drivers/char/hw_random/stm32-rng.c
@@ -156,18 +156,13 @@ static int stm32_rng_probe(struct platform_device *ofdev)
pm_runtime_set_autosuspend_delay(dev, 100);
pm_runtime_use_autosuspend(dev);
- pm_runtime_enable(dev);
+ err = devm_pm_runtime_enable(dev);
+ if (err)
+ return err;
return devm_hwrng_register(dev, &priv->rng);
}
-static int stm32_rng_remove(struct platform_device *ofdev)
-{
- pm_runtime_disable(&ofdev->dev);
-
- return 0;
-}
-
#ifdef CONFIG_PM
static int stm32_rng_runtime_suspend(struct device *dev)
{
@@ -209,7 +204,6 @@ static struct platform_driver stm32_rng_driver = {
.of_match_table = stm32_rng_match,
},
.probe = stm32_rng_probe,
- .remove = stm32_rng_remove,
};
module_platform_driver(stm32_rng_driver);
diff --git a/drivers/clk/sunxi-ng/ccu-sun50i-h616.c b/drivers/clk/sunxi-ng/ccu-sun50i-h616.c
index 21e918582aa5..5831bcfbe500 100644
--- a/drivers/clk/sunxi-ng/ccu-sun50i-h616.c
+++ b/drivers/clk/sunxi-ng/ccu-sun50i-h616.c
@@ -318,10 +318,16 @@ static SUNXI_CCU_M_WITH_MUX_GATE(gpu0_clk, "gpu0", gpu0_parents, 0x670,
24, 1, /* mux */
BIT(31), /* gate */
CLK_SET_RATE_PARENT);
+
+/*
+ * This clk is needed as a temporary fall back during GPU PLL freq changes.
+ * Set CLK_IS_CRITICAL flag to prevent from being disabled.
+ */
+#define SUN50I_H616_GPU_CLK1_REG 0x674
static SUNXI_CCU_M_WITH_GATE(gpu1_clk, "gpu1", "pll-periph0-2x", 0x674,
0, 2, /* M */
BIT(31),/* gate */
- 0);
+ CLK_IS_CRITICAL);
static SUNXI_CCU_GATE(bus_gpu_clk, "bus-gpu", "psi-ahb1-ahb2",
0x67c, BIT(0), 0);
@@ -1090,11 +1096,37 @@ static const u32 usb2_clk_regs[] = {
SUN50I_H616_USB3_CLK_REG,
};
+static struct ccu_mux_nb sun50i_h616_cpu_nb = {
+ .common = &cpux_clk.common,
+ .cm = &cpux_clk.mux,
+ .delay_us = 1, /* manual doesn't really say */
+ .bypass_index = 4, /* PLL_PERI0@600MHz, as recommended by manual */
+};
+
+static struct ccu_pll_nb sun50i_h616_pll_cpu_nb = {
+ .common = &pll_cpux_clk.common,
+ .enable = BIT(29), /* LOCK_ENABLE */
+ .lock = BIT(28),
+};
+
+static struct ccu_mux_nb sun50i_h616_gpu_nb = {
+ .common = &gpu0_clk.common,
+ .cm = &gpu0_clk.mux,
+ .delay_us = 1, /* manual doesn't really say */
+ .bypass_index = 1, /* GPU_CLK1@400MHz */
+};
+
+static struct ccu_pll_nb sun50i_h616_pll_gpu_nb = {
+ .common = &pll_gpu_clk.common,
+ .enable = BIT(29), /* LOCK_ENABLE */
+ .lock = BIT(28),
+};
+
static int sun50i_h616_ccu_probe(struct platform_device *pdev)
{
void __iomem *reg;
u32 val;
- int i;
+ int ret, i;
reg = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(reg))
@@ -1138,6 +1170,14 @@ static int sun50i_h616_ccu_probe(struct platform_device *pdev)
val &= ~(GENMASK(21, 16) | BIT(0));
writel(val | (11 << 16) | BIT(0), reg + SUN50I_H616_PLL_AUDIO_REG);
+ /*
+ * Set the input-divider for the gpu1 clock to 3, to reach a safe 400 MHz.
+ */
+ val = readl(reg + SUN50I_H616_GPU_CLK1_REG);
+ val &= ~GENMASK(1, 0);
+ val |= 2;
+ writel(val, reg + SUN50I_H616_GPU_CLK1_REG);
+
/*
* First clock parent (osc32K) is unusable for CEC. But since there
* is no good way to force parent switch (both run with same frequency),
@@ -1147,7 +1187,25 @@ static int sun50i_h616_ccu_probe(struct platform_device *pdev)
val |= BIT(24);
writel(val, reg + SUN50I_H616_HDMI_CEC_CLK_REG);
- return devm_sunxi_ccu_probe(&pdev->dev, reg, &sun50i_h616_ccu_desc);
+ ret = devm_sunxi_ccu_probe(&pdev->dev, reg, &sun50i_h616_ccu_desc);
+ if (ret)
+ return ret;
+
+ /* Reparent CPU during CPU PLL rate changes */
+ ccu_mux_notifier_register(pll_cpux_clk.common.hw.clk,
+ &sun50i_h616_cpu_nb);
+
+ /* Re-lock the CPU PLL after any rate changes */
+ ccu_pll_notifier_register(&sun50i_h616_pll_cpu_nb);
+
+ /* Reparent GPU during GPU PLL rate changes */
+ ccu_mux_notifier_register(pll_gpu_clk.common.hw.clk,
+ &sun50i_h616_gpu_nb);
+
+ /* Re-lock the GPU PLL after any rate changes */
+ ccu_pll_notifier_register(&sun50i_h616_pll_gpu_nb);
+
+ return 0;
}
static const struct of_device_id sun50i_h616_ccu_ids[] = {
diff --git a/drivers/clocksource/samsung_pwm_timer.c b/drivers/clocksource/samsung_pwm_timer.c
index 6e46781bc9ac..896ba9fe5e75 100644
--- a/drivers/clocksource/samsung_pwm_timer.c
+++ b/drivers/clocksource/samsung_pwm_timer.c
@@ -56,7 +56,7 @@
#define TCON_AUTORELOAD(chan) \
((chan < 5) ? _TCON_AUTORELOAD(chan) : _TCON_AUTORELOAD4(chan))
-DEFINE_SPINLOCK(samsung_pwm_lock);
+DEFINE_RAW_SPINLOCK(samsung_pwm_lock);
EXPORT_SYMBOL(samsung_pwm_lock);
struct samsung_pwm_clocksource {
@@ -87,14 +87,14 @@ static void samsung_timer_set_prescale(unsigned int channel, u16 prescale)
if (channel >= 2)
shift = TCFG0_PRESCALER1_SHIFT;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(pwm.base + REG_TCFG0);
reg &= ~(TCFG0_PRESCALER_MASK << shift);
reg |= (prescale - 1) << shift;
writel(reg, pwm.base + REG_TCFG0);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_timer_set_divisor(unsigned int channel, u8 divisor)
@@ -106,14 +106,14 @@ static void samsung_timer_set_divisor(unsigned int channel, u8 divisor)
bits = (fls(divisor) - 1) - pwm.variant.div_base;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(pwm.base + REG_TCFG1);
reg &= ~(TCFG1_MUX_MASK << shift);
reg |= bits << shift;
writel(reg, pwm.base + REG_TCFG1);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_stop(unsigned int channel)
@@ -124,13 +124,13 @@ static void samsung_time_stop(unsigned int channel)
if (channel > 0)
++channel;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
tcon &= ~TCON_START(channel);
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
@@ -142,7 +142,7 @@ static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
if (tcon_chan > 0)
++tcon_chan;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
@@ -153,7 +153,7 @@ static void samsung_time_setup(unsigned int channel, unsigned long tcnt)
writel_relaxed(tcnt, pwm.base + REG_TCMPB(channel));
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void samsung_time_start(unsigned int channel, bool periodic)
@@ -164,7 +164,7 @@ static void samsung_time_start(unsigned int channel, bool periodic)
if (channel > 0)
++channel;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl_relaxed(pwm.base + REG_TCON);
@@ -178,7 +178,7 @@ static void samsung_time_start(unsigned int channel, bool periodic)
writel_relaxed(tcon, pwm.base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int samsung_set_next_event(unsigned long cycles,
diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c
index 90dcf26f0973..106aef210003 100644
--- a/drivers/cpufreq/amd-pstate.c
+++ b/drivers/cpufreq/amd-pstate.c
@@ -309,9 +309,14 @@ static void amd_pstate_adjust_perf(unsigned int cpu,
unsigned long max_perf, min_perf, des_perf,
cap_perf, lowest_nonlinear_perf, max_freq;
struct cpufreq_policy *policy = cpufreq_cpu_get(cpu);
- struct amd_cpudata *cpudata = policy->driver_data;
+ struct amd_cpudata *cpudata;
unsigned int target_freq;
+ if (!policy)
+ return;
+
+ cpudata = policy->driver_data;
+
cap_perf = READ_ONCE(cpudata->highest_perf);
lowest_nonlinear_perf = READ_ONCE(cpudata->lowest_nonlinear_perf);
max_freq = READ_ONCE(cpudata->max_freq);
diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c
index 9ebd7b3d97be..216d8ea775e9 100644
--- a/drivers/crypto/hisilicon/qm.c
+++ b/drivers/crypto/hisilicon/qm.c
@@ -4557,6 +4557,15 @@ void hisi_qm_reset_done(struct pci_dev *pdev)
}
EXPORT_SYMBOL_GPL(hisi_qm_reset_done);
+static irqreturn_t qm_rsvd_irq(int irq, void *data)
+{
+ struct hisi_qm *qm = data;
+
+ dev_info(&qm->pdev->dev, "Reserved interrupt, ignore!\n");
+
+ return IRQ_HANDLED;
+}
+
static irqreturn_t qm_abnormal_irq(int irq, void *data)
{
struct hisi_qm *qm = data;
@@ -4856,7 +4865,7 @@ static void qm_unregister_abnormal_irq(struct hisi_qm *qm)
struct pci_dev *pdev = qm->pdev;
u32 irq_vector, val;
- if (qm->fun_type == QM_HW_VF)
+ if (qm->fun_type == QM_HW_VF && qm->ver < QM_HW_V3)
return;
val = qm->cap_tables.qm_cap_table[QM_ABN_IRQ_TYPE_CAP_IDX].cap_val;
@@ -4873,17 +4882,28 @@ static int qm_register_abnormal_irq(struct hisi_qm *qm)
u32 irq_vector, val;
int ret;
- if (qm->fun_type == QM_HW_VF)
- return 0;
-
val = qm->cap_tables.qm_cap_table[QM_ABN_IRQ_TYPE_CAP_IDX].cap_val;
if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_ABN_IRQ_TYPE_MASK))
return 0;
-
irq_vector = val & QM_IRQ_VECTOR_MASK;
+
+ /* For VF, this is a reserved interrupt in V3 version. */
+ if (qm->fun_type == QM_HW_VF) {
+ if (qm->ver < QM_HW_V3)
+ return 0;
+
+ ret = request_irq(pci_irq_vector(pdev, irq_vector), qm_rsvd_irq,
+ IRQF_NO_AUTOEN, qm->dev_name, qm);
+ if (ret) {
+ dev_err(&pdev->dev, "failed to request reserved irq, ret = %d!\n", ret);
+ return ret;
+ }
+ return 0;
+ }
+
ret = request_irq(pci_irq_vector(pdev, irq_vector), qm_abnormal_irq, 0, qm->dev_name, qm);
if (ret)
- dev_err(&qm->pdev->dev, "failed to request abnormal irq, ret = %d", ret);
+ dev_err(&qm->pdev->dev, "failed to request abnormal irq, ret = %d!\n", ret);
return ret;
}
@@ -5222,6 +5242,12 @@ static int hisi_qm_pci_init(struct hisi_qm *qm)
pci_set_master(pdev);
num_vec = qm_get_irq_num(qm);
+ if (!num_vec) {
+ dev_err(dev, "Device irq num is zero!\n");
+ ret = -EINVAL;
+ goto err_get_pci_res;
+ }
+ num_vec = roundup_pow_of_two(num_vec);
ret = pci_alloc_irq_vectors(pdev, num_vec, num_vec, PCI_IRQ_MSI);
if (ret < 0) {
dev_err(dev, "Failed to enable MSI vectors!\n");
diff --git a/drivers/crypto/keembay/keembay-ocs-aes-core.c b/drivers/crypto/keembay/keembay-ocs-aes-core.c
index 2a96fedc8d63..83c2bf9d2596 100644
--- a/drivers/crypto/keembay/keembay-ocs-aes-core.c
+++ b/drivers/crypto/keembay/keembay-ocs-aes-core.c
@@ -1554,7 +1554,7 @@ static int register_aes_algs(struct ocs_aes_dev *aes_dev)
/*
* If any algorithm fails to register, all preceding algorithms that
- * were successfully registered will be automatically unregistered.
+ * were registered in the same call are automatically unregistered.
*/
ret = crypto_register_aeads(algs_aead, ARRAY_SIZE(algs_aead));
if (ret)
@@ -1562,7 +1562,7 @@ static int register_aes_algs(struct ocs_aes_dev *aes_dev)
ret = crypto_register_skciphers(algs, ARRAY_SIZE(algs));
if (ret)
- crypto_unregister_aeads(algs_aead, ARRAY_SIZE(algs));
+ crypto_unregister_aeads(algs_aead, ARRAY_SIZE(algs_aead));
return ret;
}
diff --git a/drivers/crypto/qat/qat_common/adf_init.c b/drivers/crypto/qat/qat_common/adf_init.c
index 9d54dc3beafd..c397c58bd4a9 100644
--- a/drivers/crypto/qat/qat_common/adf_init.c
+++ b/drivers/crypto/qat/qat_common/adf_init.c
@@ -136,16 +136,19 @@ int adf_dev_init(struct adf_accel_dev *accel_dev)
* This is to facilitate any ordering dependencies between services
* prior to starting any of the accelerators.
*/
+ mutex_lock(&service_lock);
list_for_each(list_itr, &service_table) {
service = list_entry(list_itr, struct service_hndl, list);
if (service->event_hld(accel_dev, ADF_EVENT_INIT)) {
dev_err(&GET_DEV(accel_dev),
"Failed to initialise service %s\n",
service->name);
+ mutex_unlock(&service_lock);
return -EFAULT;
}
set_bit(accel_dev->accel_id, service->init_status);
}
+ mutex_unlock(&service_lock);
return 0;
}
@@ -190,16 +193,19 @@ int adf_dev_start(struct adf_accel_dev *accel_dev)
return -EFAULT;
}
+ mutex_lock(&service_lock);
list_for_each(list_itr, &service_table) {
service = list_entry(list_itr, struct service_hndl, list);
if (service->event_hld(accel_dev, ADF_EVENT_START)) {
dev_err(&GET_DEV(accel_dev),
"Failed to start service %s\n",
service->name);
+ mutex_unlock(&service_lock);
return -EFAULT;
}
set_bit(accel_dev->accel_id, service->start_status);
}
+ mutex_unlock(&service_lock);
clear_bit(ADF_STATUS_STARTING, &accel_dev->status);
set_bit(ADF_STATUS_STARTED, &accel_dev->status);
@@ -249,6 +255,7 @@ void adf_dev_stop(struct adf_accel_dev *accel_dev)
}
clear_bit(ADF_STATUS_CRYPTO_ALGS_REGISTERED, &accel_dev->status);
+ mutex_lock(&service_lock);
list_for_each(list_itr, &service_table) {
service = list_entry(list_itr, struct service_hndl, list);
if (!test_bit(accel_dev->accel_id, service->start_status))
@@ -261,6 +268,7 @@ void adf_dev_stop(struct adf_accel_dev *accel_dev)
clear_bit(accel_dev->accel_id, service->start_status);
}
}
+ mutex_unlock(&service_lock);
if (wait)
msleep(100);
@@ -307,6 +315,7 @@ void adf_dev_shutdown(struct adf_accel_dev *accel_dev)
&accel_dev->status);
}
+ mutex_lock(&service_lock);
list_for_each(list_itr, &service_table) {
service = list_entry(list_itr, struct service_hndl, list);
if (!test_bit(accel_dev->accel_id, service->init_status))
@@ -318,6 +327,7 @@ void adf_dev_shutdown(struct adf_accel_dev *accel_dev)
else
clear_bit(accel_dev->accel_id, service->init_status);
}
+ mutex_unlock(&service_lock);
hw_data->disable_iov(accel_dev);
@@ -347,6 +357,7 @@ int adf_dev_restarting_notify(struct adf_accel_dev *accel_dev)
struct service_hndl *service;
struct list_head *list_itr;
+ mutex_lock(&service_lock);
list_for_each(list_itr, &service_table) {
service = list_entry(list_itr, struct service_hndl, list);
if (service->event_hld(accel_dev, ADF_EVENT_RESTARTING))
@@ -354,6 +365,7 @@ int adf_dev_restarting_notify(struct adf_accel_dev *accel_dev)
"Failed to restart service %s.\n",
service->name);
}
+ mutex_unlock(&service_lock);
return 0;
}
@@ -362,6 +374,7 @@ int adf_dev_restarted_notify(struct adf_accel_dev *accel_dev)
struct service_hndl *service;
struct list_head *list_itr;
+ mutex_lock(&service_lock);
list_for_each(list_itr, &service_table) {
service = list_entry(list_itr, struct service_hndl, list);
if (service->event_hld(accel_dev, ADF_EVENT_RESTARTED))
@@ -369,6 +382,7 @@ int adf_dev_restarted_notify(struct adf_accel_dev *accel_dev)
"Failed to restart service %s.\n",
service->name);
}
+ mutex_unlock(&service_lock);
return 0;
}
diff --git a/drivers/crypto/stm32/stm32-cryp.c b/drivers/crypto/stm32/stm32-cryp.c
index 59ef541123ae..9e7e4bdd1ac5 100644
--- a/drivers/crypto/stm32/stm32-cryp.c
+++ b/drivers/crypto/stm32/stm32-cryp.c
@@ -4,6 +4,7 @@
* Author: Fabien Dessenne <fabien.dessenne@st.com>
*/
+#include <linux/bottom_half.h>
#include <linux/clk.h>
#include <linux/delay.h>
#include <linux/interrupt.h>
@@ -249,7 +250,7 @@ static inline int stm32_cryp_wait_output(struct stm32_cryp *cryp)
{
u32 status;
- return readl_relaxed_poll_timeout(cryp->regs + CRYP_SR, status,
+ return readl_relaxed_poll_timeout_atomic(cryp->regs + CRYP_SR, status,
status & SR_OFNE, 10, 100000);
}
@@ -1538,8 +1539,11 @@ static irqreturn_t stm32_cryp_irq_thread(int irq, void *arg)
it_mask &= ~IMSCR_OUT;
stm32_cryp_write(cryp, CRYP_IMSCR, it_mask);
- if (!cryp->payload_in && !cryp->header_in && !cryp->payload_out)
+ if (!cryp->payload_in && !cryp->header_in && !cryp->payload_out) {
+ local_bh_disable();
stm32_cryp_finish_req(cryp, 0);
+ local_bh_enable();
+ }
return IRQ_HANDLED;
}
diff --git a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
index d2df97cfcb29..c19920d3397b 100644
--- a/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
+++ b/drivers/crypto/virtio/virtio_crypto_akcipher_algs.c
@@ -36,8 +36,6 @@ struct virtio_crypto_akcipher_ctx {
struct virtio_crypto_akcipher_request {
struct virtio_crypto_request base;
- struct virtio_crypto_akcipher_ctx *akcipher_ctx;
- struct akcipher_request *akcipher_req;
void *src_buf;
void *dst_buf;
uint32_t opcode;
@@ -69,7 +67,9 @@ static void virtio_crypto_dataq_akcipher_callback(struct virtio_crypto_request *
{
struct virtio_crypto_akcipher_request *vc_akcipher_req =
container_of(vc_req, struct virtio_crypto_akcipher_request, base);
- struct akcipher_request *akcipher_req;
+ struct akcipher_request *akcipher_req =
+ container_of((void *)vc_akcipher_req, struct akcipher_request,
+ __ctx);
int error;
switch (vc_req->status) {
@@ -83,23 +83,16 @@ static void virtio_crypto_dataq_akcipher_callback(struct virtio_crypto_request *
case VIRTIO_CRYPTO_BADMSG:
error = -EBADMSG;
break;
-
- case VIRTIO_CRYPTO_KEY_REJECTED:
- error = -EKEYREJECTED;
- break;
-
default:
error = -EIO;
break;
}
- akcipher_req = vc_akcipher_req->akcipher_req;
- if (vc_akcipher_req->opcode != VIRTIO_CRYPTO_AKCIPHER_VERIFY) {
- /* actuall length maybe less than dst buffer */
- akcipher_req->dst_len = len - sizeof(vc_req->status);
- sg_copy_from_buffer(akcipher_req->dst, sg_nents(akcipher_req->dst),
- vc_akcipher_req->dst_buf, akcipher_req->dst_len);
- }
+ /* actual length may be less than dst buffer */
+ akcipher_req->dst_len = min_t(unsigned int, len - sizeof(vc_req->status),
+ akcipher_req->dst_len);
+ sg_copy_from_buffer(akcipher_req->dst, sg_nents(akcipher_req->dst),
+ vc_akcipher_req->dst_buf, akcipher_req->dst_len);
virtio_crypto_akcipher_finalize_req(vc_akcipher_req, akcipher_req, error);
}
@@ -220,46 +213,38 @@ static int virtio_crypto_alg_akcipher_close_session(struct virtio_crypto_akciphe
static int __virtio_crypto_akcipher_do_req(struct virtio_crypto_akcipher_request *vc_akcipher_req,
struct akcipher_request *req, struct data_queue *data_vq)
{
- struct virtio_crypto_akcipher_ctx *ctx = vc_akcipher_req->akcipher_ctx;
+ struct crypto_akcipher *atfm = crypto_akcipher_reqtfm(req);
+ struct virtio_crypto_akcipher_ctx *ctx = akcipher_tfm_ctx(atfm);
struct virtio_crypto_request *vc_req = &vc_akcipher_req->base;
struct virtio_crypto *vcrypto = ctx->vcrypto;
struct virtio_crypto_op_data_req *req_data = vc_req->req_data;
struct scatterlist *sgs[4], outhdr_sg, inhdr_sg, srcdata_sg, dstdata_sg;
- void *src_buf = NULL, *dst_buf = NULL;
+ void *src_buf, *dst_buf = NULL;
unsigned int num_out = 0, num_in = 0;
int node = dev_to_node(&vcrypto->vdev->dev);
unsigned long flags;
- int ret = -ENOMEM;
- bool verify = vc_akcipher_req->opcode == VIRTIO_CRYPTO_AKCIPHER_VERIFY;
- unsigned int src_len = verify ? req->src_len + req->dst_len : req->src_len;
+ int ret;
/* out header */
sg_init_one(&outhdr_sg, req_data, sizeof(*req_data));
sgs[num_out++] = &outhdr_sg;
/* src data */
- src_buf = kcalloc_node(src_len, 1, GFP_KERNEL, node);
+ src_buf = kcalloc_node(req->src_len, 1, GFP_KERNEL, node);
if (!src_buf)
- goto err;
+ return -ENOMEM;
- if (verify) {
- /* for verify operation, both src and dst data work as OUT direction */
- sg_copy_to_buffer(req->src, sg_nents(req->src), src_buf, src_len);
- sg_init_one(&srcdata_sg, src_buf, src_len);
- sgs[num_out++] = &srcdata_sg;
- } else {
- sg_copy_to_buffer(req->src, sg_nents(req->src), src_buf, src_len);
- sg_init_one(&srcdata_sg, src_buf, src_len);
- sgs[num_out++] = &srcdata_sg;
+ sg_copy_to_buffer(req->src, sg_nents(req->src), src_buf, req->src_len);
+ sg_init_one(&srcdata_sg, src_buf, req->src_len);
+ sgs[num_out++] = &srcdata_sg;
- /* dst data */
- dst_buf = kcalloc_node(req->dst_len, 1, GFP_KERNEL, node);
- if (!dst_buf)
- goto err;
+ /* dst data */
+ dst_buf = kcalloc_node(req->dst_len, 1, GFP_KERNEL, node);
+ if (!dst_buf)
+ goto free_src;
- sg_init_one(&dstdata_sg, dst_buf, req->dst_len);
- sgs[num_out + num_in++] = &dstdata_sg;
- }
+ sg_init_one(&dstdata_sg, dst_buf, req->dst_len);
+ sgs[num_out + num_in++] = &dstdata_sg;
vc_akcipher_req->src_buf = src_buf;
vc_akcipher_req->dst_buf = dst_buf;
@@ -278,9 +263,9 @@ static int __virtio_crypto_akcipher_do_req(struct virtio_crypto_akcipher_request
return 0;
err:
- kfree(src_buf);
kfree(dst_buf);
-
+free_src:
+ kfree(src_buf);
return -ENOMEM;
}
@@ -289,7 +274,8 @@ static int virtio_crypto_rsa_do_req(struct crypto_engine *engine, void *vreq)
struct akcipher_request *req = container_of(vreq, struct akcipher_request, base);
struct virtio_crypto_akcipher_request *vc_akcipher_req = akcipher_request_ctx(req);
struct virtio_crypto_request *vc_req = &vc_akcipher_req->base;
- struct virtio_crypto_akcipher_ctx *ctx = vc_akcipher_req->akcipher_ctx;
+ struct crypto_akcipher *atfm = crypto_akcipher_reqtfm(req);
+ struct virtio_crypto_akcipher_ctx *ctx = akcipher_tfm_ctx(atfm);
struct virtio_crypto *vcrypto = ctx->vcrypto;
struct data_queue *data_vq = vc_req->dataq;
struct virtio_crypto_op_header *header;
@@ -335,8 +321,6 @@ static int virtio_crypto_rsa_req(struct akcipher_request *req, uint32_t opcode)
vc_req->dataq = data_vq;
vc_req->alg_cb = virtio_crypto_dataq_akcipher_callback;
- vc_akcipher_req->akcipher_ctx = ctx;
- vc_akcipher_req->akcipher_req = req;
vc_akcipher_req->opcode = opcode;
return crypto_transfer_akcipher_request_to_engine(data_vq->engine, req);
@@ -352,16 +336,6 @@ static int virtio_crypto_rsa_decrypt(struct akcipher_request *req)
return virtio_crypto_rsa_req(req, VIRTIO_CRYPTO_AKCIPHER_DECRYPT);
}
-static int virtio_crypto_rsa_sign(struct akcipher_request *req)
-{
- return virtio_crypto_rsa_req(req, VIRTIO_CRYPTO_AKCIPHER_SIGN);
-}
-
-static int virtio_crypto_rsa_verify(struct akcipher_request *req)
-{
- return virtio_crypto_rsa_req(req, VIRTIO_CRYPTO_AKCIPHER_VERIFY);
-}
-
static int virtio_crypto_rsa_set_key(struct crypto_akcipher *tfm,
const void *key,
unsigned int keylen,
@@ -522,8 +496,11 @@ static struct virtio_crypto_akcipher_algo virtio_crypto_akcipher_algs[] = {
.algo = {
.encrypt = virtio_crypto_rsa_encrypt,
.decrypt = virtio_crypto_rsa_decrypt,
- .sign = virtio_crypto_rsa_sign,
- .verify = virtio_crypto_rsa_verify,
+ /*
+ * Must specify an arbitrary hash algorithm upon
+ * set_{pub,priv}_key (even though it's not used
+ * by encrypt/decrypt) because qemu checks for it.
+ */
.set_pub_key = virtio_crypto_p1pad_rsa_sha1_set_pub_key,
.set_priv_key = virtio_crypto_p1pad_rsa_sha1_set_priv_key,
.max_size = virtio_crypto_rsa_max_size,
@@ -531,8 +508,8 @@ static struct virtio_crypto_akcipher_algo virtio_crypto_akcipher_algs[] = {
.exit = virtio_crypto_rsa_exit_tfm,
.reqsize = sizeof(struct virtio_crypto_akcipher_request),
.base = {
- .cra_name = "pkcs1pad(rsa,sha1)",
- .cra_driver_name = "virtio-pkcs1-rsa-with-sha1",
+ .cra_name = "pkcs1pad(rsa)",
+ .cra_driver_name = "virtio-pkcs1-rsa",
.cra_priority = 150,
.cra_module = THIS_MODULE,
.cra_ctxsize = sizeof(struct virtio_crypto_akcipher_ctx),
diff --git a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c
index f7b8715b5074..40d8d95a0f7a 100644
--- a/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c
+++ b/drivers/crypto/virtio/virtio_crypto_skcipher_algs.c
@@ -31,8 +31,6 @@ struct virtio_crypto_sym_request {
/* Cipher or aead */
uint32_t type;
- struct virtio_crypto_skcipher_ctx *skcipher_ctx;
- struct skcipher_request *skcipher_req;
uint8_t *iv;
/* Encryption? */
bool encrypt;
@@ -60,7 +58,9 @@ static void virtio_crypto_dataq_sym_callback
{
struct virtio_crypto_sym_request *vc_sym_req =
container_of(vc_req, struct virtio_crypto_sym_request, base);
- struct skcipher_request *ablk_req;
+ struct skcipher_request *ablk_req =
+ container_of((void *)vc_sym_req, struct skcipher_request,
+ __ctx);
int error;
/* Finish the encrypt or decrypt process */
@@ -80,7 +80,6 @@ static void virtio_crypto_dataq_sym_callback
error = -EIO;
break;
}
- ablk_req = vc_sym_req->skcipher_req;
virtio_crypto_skcipher_finalize_req(vc_sym_req,
ablk_req, error);
}
@@ -328,7 +327,7 @@ __virtio_crypto_skcipher_do_req(struct virtio_crypto_sym_request *vc_sym_req,
struct data_queue *data_vq)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
- struct virtio_crypto_skcipher_ctx *ctx = vc_sym_req->skcipher_ctx;
+ struct virtio_crypto_skcipher_ctx *ctx = crypto_skcipher_ctx(tfm);
struct virtio_crypto_request *vc_req = &vc_sym_req->base;
unsigned int ivsize = crypto_skcipher_ivsize(tfm);
struct virtio_crypto *vcrypto = ctx->vcrypto;
@@ -484,8 +483,6 @@ static int virtio_crypto_skcipher_encrypt(struct skcipher_request *req)
vc_req->dataq = data_vq;
vc_req->alg_cb = virtio_crypto_dataq_sym_callback;
- vc_sym_req->skcipher_ctx = ctx;
- vc_sym_req->skcipher_req = req;
vc_sym_req->encrypt = true;
return crypto_transfer_skcipher_request_to_engine(data_vq->engine, req);
@@ -509,8 +506,6 @@ static int virtio_crypto_skcipher_decrypt(struct skcipher_request *req)
vc_req->dataq = data_vq;
vc_req->alg_cb = virtio_crypto_dataq_sym_callback;
- vc_sym_req->skcipher_ctx = ctx;
- vc_sym_req->skcipher_req = req;
vc_sym_req->encrypt = false;
return crypto_transfer_skcipher_request_to_engine(data_vq->engine, req);
diff --git a/drivers/cxl/core/hdm.c b/drivers/cxl/core/hdm.c
index 7d48f5a575ce..580cf7834cf1 100644
--- a/drivers/cxl/core/hdm.c
+++ b/drivers/cxl/core/hdm.c
@@ -641,7 +641,42 @@ static int cxl_decoder_commit(struct cxl_decoder *cxld)
return 0;
}
-static int cxl_decoder_reset(struct cxl_decoder *cxld)
+static int commit_reap(struct device *dev, const void *data)
+{
+ struct cxl_port *port = to_cxl_port(dev->parent);
+ struct cxl_decoder *cxld;
+
+ if (!is_switch_decoder(dev) && !is_endpoint_decoder(dev))
+ return 0;
+
+ cxld = to_cxl_decoder(dev);
+ if (port->commit_end == cxld->id &&
+ ((cxld->flags & CXL_DECODER_F_ENABLE) == 0)) {
+ port->commit_end--;
+ dev_dbg(&port->dev, "reap: %s commit_end: %d\n",
+ dev_name(&cxld->dev), port->commit_end);
+ }
+
+ return 0;
+}
+
+void cxl_port_commit_reap(struct cxl_decoder *cxld)
+{
+ struct cxl_port *port = to_cxl_port(cxld->dev.parent);
+
+ /*
+ * Once the highest committed decoder is disabled, free any other
+ * decoders that were pinned allocated by out-of-order release.
+ */
+ port->commit_end--;
+ dev_dbg(&port->dev, "reap: %s commit_end: %d\n", dev_name(&cxld->dev),
+ port->commit_end);
+ device_for_each_child_reverse_from(&port->dev, &cxld->dev, NULL,
+ commit_reap);
+}
+EXPORT_SYMBOL_NS_GPL(cxl_port_commit_reap, CXL);
+
+static void cxl_decoder_reset(struct cxl_decoder *cxld)
{
struct cxl_port *port = to_cxl_port(cxld->dev.parent);
struct cxl_hdm *cxlhdm = dev_get_drvdata(&port->dev);
@@ -650,14 +685,14 @@ static int cxl_decoder_reset(struct cxl_decoder *cxld)
u32 ctrl;
if ((cxld->flags & CXL_DECODER_F_ENABLE) == 0)
- return 0;
+ return;
- if (port->commit_end != id) {
+ if (port->commit_end == id)
+ cxl_port_commit_reap(cxld);
+ else
dev_dbg(&port->dev,
"%s: out of order reset, expected decoder%d.%d\n",
dev_name(&cxld->dev), port->id, port->commit_end);
- return -EBUSY;
- }
down_read(&cxl_dpa_rwsem);
ctrl = readl(hdm + CXL_HDM_DECODER0_CTRL_OFFSET(id));
@@ -670,10 +705,7 @@ static int cxl_decoder_reset(struct cxl_decoder *cxld)
writel(0, hdm + CXL_HDM_DECODER0_BASE_LOW_OFFSET(id));
up_read(&cxl_dpa_rwsem);
- port->commit_end--;
cxld->flags &= ~CXL_DECODER_F_ENABLE;
-
- return 0;
}
static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c
index 0c90f13870a4..43df0a3d20f1 100644
--- a/drivers/cxl/core/mbox.c
+++ b/drivers/cxl/core/mbox.c
@@ -223,6 +223,7 @@ static bool cxl_mem_raw_command_allowed(u16 opcode)
* cxl_payload_from_user_allowed() - Check contents of in_payload.
* @opcode: The mailbox command opcode.
* @payload_in: Pointer to the input payload passed in from user space.
+ * @in_size: Size of @payload_in in bytes.
*
* Return:
* * true - payload_in passes check for @opcode.
@@ -237,12 +238,15 @@ static bool cxl_mem_raw_command_allowed(u16 opcode)
*
* The specific checks are determined by the opcode.
*/
-static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in)
+static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in,
+ size_t in_size)
{
switch (opcode) {
case CXL_MBOX_OP_SET_PARTITION_INFO: {
struct cxl_mbox_set_partition_info *pi = payload_in;
+ if (in_size < sizeof(*pi))
+ return false;
if (pi->flags & CXL_SET_PARTITION_IMMEDIATE_FLAG)
return false;
break;
@@ -268,7 +272,8 @@ static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox,
if (IS_ERR(mbox->payload_in))
return PTR_ERR(mbox->payload_in);
- if (!cxl_payload_from_user_allowed(opcode, mbox->payload_in)) {
+ if (!cxl_payload_from_user_allowed(opcode, mbox->payload_in,
+ in_size)) {
dev_dbg(cxlds->dev, "%s: input payload not allowed\n",
cxl_mem_opcode_to_name(opcode));
kvfree(mbox->payload_in);
@@ -276,11 +281,7 @@ static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox,
}
}
- /* Prepare to handle a full payload for variable sized output */
- if (out_size == CXL_VARIABLE_PAYLOAD)
- mbox->size_out = cxlds->payload_size;
- else
- mbox->size_out = out_size;
+ mbox->size_out = min_t(size_t, out_size, cxlds->payload_size);
if (mbox->size_out) {
mbox->payload_out = kvzalloc(mbox->size_out, GFP_KERNEL);
diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c
index 20f052d3759e..15b468b6aec3 100644
--- a/drivers/cxl/core/port.c
+++ b/drivers/cxl/core/port.c
@@ -1094,29 +1094,24 @@ static void delete_endpoint(void *data)
{
struct cxl_memdev *cxlmd = data;
struct cxl_port *endpoint = dev_get_drvdata(&cxlmd->dev);
- struct cxl_port *parent_port;
- struct device *parent;
+ struct device *host = endpoint->dev.parent;
- parent_port = cxl_mem_find_port(cxlmd, NULL);
- if (!parent_port)
- goto out;
- parent = &parent_port->dev;
-
- device_lock(parent);
- if (parent->driver && !endpoint->dead) {
- devm_release_action(parent, cxl_unlink_uport, endpoint);
- devm_release_action(parent, unregister_port, endpoint);
+ device_lock(host);
+ if (host->driver && !endpoint->dead) {
+ devm_release_action(host, cxl_unlink_uport, endpoint);
+ devm_release_action(host, unregister_port, endpoint);
}
- device_unlock(parent);
- put_device(parent);
-out:
+ device_unlock(host);
put_device(&endpoint->dev);
+ put_device(host);
}
int cxl_endpoint_autoremove(struct cxl_memdev *cxlmd, struct cxl_port *endpoint)
{
+ struct device *host = endpoint->dev.parent;
struct device *dev = &cxlmd->dev;
+ get_device(host);
get_device(&endpoint->dev);
dev_set_drvdata(dev, endpoint);
return devm_add_action_or_reset(dev, delete_endpoint, cxlmd);
diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c
index 0e03c4908050..cd53738a628d 100644
--- a/drivers/cxl/core/region.c
+++ b/drivers/cxl/core/region.c
@@ -121,7 +121,7 @@ static struct cxl_region_ref *cxl_rr_load(struct cxl_port *port,
return xa_load(&port->regions, (unsigned long)cxlr);
}
-static int cxl_region_decode_reset(struct cxl_region *cxlr, int count)
+static void cxl_region_decode_reset(struct cxl_region *cxlr, int count)
{
struct cxl_region_params *p = &cxlr->params;
int i;
@@ -131,7 +131,6 @@ static int cxl_region_decode_reset(struct cxl_region *cxlr, int count)
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
struct cxl_port *iter = cxled_to_port(cxled);
struct cxl_ep *ep;
- int rc = 0;
while (!is_cxl_root(to_cxl_port(iter->dev.parent)))
iter = to_cxl_port(iter->dev.parent);
@@ -144,17 +143,11 @@ static int cxl_region_decode_reset(struct cxl_region *cxlr, int count)
cxl_rr = cxl_rr_load(iter, cxlr);
cxld = cxl_rr->decoder;
if (cxld->reset)
- rc = cxld->reset(cxld);
- if (rc)
- return rc;
+ cxld->reset(cxld);
}
- rc = cxled->cxld.reset(&cxled->cxld);
- if (rc)
- return rc;
+ cxled->cxld.reset(&cxled->cxld);
}
-
- return 0;
}
static int cxl_region_decode_commit(struct cxl_region *cxlr)
@@ -244,8 +237,10 @@ static ssize_t commit_store(struct device *dev, struct device_attribute *attr,
* The lock was dropped, so need to revalidate that the reset is
* still pending.
*/
- if (p->state == CXL_CONFIG_RESET_PENDING)
- rc = cxl_region_decode_reset(cxlr, p->interleave_ways);
+ if (p->state == CXL_CONFIG_RESET_PENDING) {
+ cxl_region_decode_reset(cxlr, p->interleave_ways);
+ p->state = CXL_CONFIG_ACTIVE;
+ }
}
if (rc)
@@ -253,8 +248,6 @@ static ssize_t commit_store(struct device *dev, struct device_attribute *attr,
if (commit)
p->state = CXL_CONFIG_COMMIT;
- else if (p->state == CXL_CONFIG_RESET_PENDING)
- p->state = CXL_CONFIG_ACTIVE;
out:
up_write(&cxl_region_rwsem);
@@ -1393,13 +1386,7 @@ static int cxl_region_detach(struct cxl_endpoint_decoder *cxled)
get_device(&cxlr->dev);
if (p->state > CXL_CONFIG_ACTIVE) {
- /*
- * TODO: tear down all impacted regions if a device is
- * removed out of order
- */
- rc = cxl_region_decode_reset(cxlr, p->interleave_ways);
- if (rc)
- goto out;
+ cxl_region_decode_reset(cxlr, p->interleave_ways);
p->state = CXL_CONFIG_ACTIVE;
}
diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h
index 827fa94cddda..0f3568071966 100644
--- a/drivers/cxl/cxl.h
+++ b/drivers/cxl/cxl.h
@@ -273,7 +273,7 @@ struct cxl_decoder {
struct cxl_region *region;
unsigned long flags;
int (*commit)(struct cxl_decoder *cxld);
- int (*reset)(struct cxl_decoder *cxld);
+ void (*reset)(struct cxl_decoder *cxld);
};
/*
@@ -555,6 +555,7 @@ static inline bool is_cxl_root(struct cxl_port *port)
bool is_cxl_port(struct device *dev);
struct cxl_port *to_cxl_port(struct device *dev);
+void cxl_port_commit_reap(struct cxl_decoder *cxld);
struct pci_bus;
int devm_cxl_register_pci_bus(struct device *host, struct device *uport,
struct pci_bus *bus);
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index 3d58514d0482..36177d14a724 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -715,6 +715,21 @@ int dma_buf_fd(struct dma_buf *dmabuf, int flags)
}
EXPORT_SYMBOL_NS_GPL(dma_buf_fd, DMA_BUF);
+/**
+ * dma_buf_fd_install - install a reserved fd for a dma-buf
+ * @dmabuf: [in] pointer to dma_buf
+ * @fd: [in] fd reserved with get_unused_fd_flags()
+ *
+ * Publishes a previously reserved fd into the caller's fd table.
+ * Must only be called after all fallible work (e.g. copy_to_user)
+ * has succeeded, as it cannot be undone safely once called.
+ */
+void dma_buf_fd_install(struct dma_buf *dmabuf, int fd)
+{
+ fd_install(fd, dmabuf->file);
+}
+EXPORT_SYMBOL_NS_GPL(dma_buf_fd_install, DMA_BUF);
+
/**
* dma_buf_get - returns the struct dma_buf related to an fd
* @fd: [in] fd associated with the struct dma_buf to be returned
diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
index 59d158873f4c..38b425c8e111 100644
--- a/drivers/dma-buf/dma-heap.c
+++ b/drivers/dma-buf/dma-heap.c
@@ -49,33 +49,6 @@ static dev_t dma_heap_devt;
static struct class *dma_heap_class;
static DEFINE_XARRAY_ALLOC(dma_heap_minors);
-static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
- unsigned int fd_flags,
- unsigned int heap_flags)
-{
- struct dma_buf *dmabuf;
- int fd;
-
- /*
- * Allocations from all heaps have to begin
- * and end on page boundaries.
- */
- len = PAGE_ALIGN(len);
- if (!len)
- return -EINVAL;
-
- dmabuf = heap->ops->allocate(heap, len, fd_flags, heap_flags);
- if (IS_ERR(dmabuf))
- return PTR_ERR(dmabuf);
-
- fd = dma_buf_fd(dmabuf, fd_flags);
- if (fd < 0) {
- dma_buf_put(dmabuf);
- /* just return, as put will call release and that will free */
- }
- return fd;
-}
-
static int dma_heap_open(struct inode *inode, struct file *file)
{
struct dma_heap *heap;
@@ -93,30 +66,42 @@ static int dma_heap_open(struct inode *inode, struct file *file)
return 0;
}
-static long dma_heap_ioctl_allocate(struct file *file, void *data)
+static struct dma_buf *dma_heap_ioctl_allocate(struct file *file, void *data)
{
struct dma_heap_allocation_data *heap_allocation = data;
struct dma_heap *heap = file->private_data;
+ struct dma_buf *dmabuf;
int fd;
+ size_t len;
if (heap_allocation->fd)
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
if (heap_allocation->fd_flags & ~DMA_HEAP_VALID_FD_FLAGS)
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS)
- return -EINVAL;
+ return ERR_PTR(-EINVAL);
+
+ len = PAGE_ALIGN(heap_allocation->len);
+ if (!len)
+ return ERR_PTR(-EINVAL);
+
+ dmabuf = heap->ops->allocate(heap, len, heap_allocation->fd_flags,
+ heap_allocation->heap_flags);
- fd = dma_heap_buffer_alloc(heap, heap_allocation->len,
- heap_allocation->fd_flags,
- heap_allocation->heap_flags);
- if (fd < 0)
- return fd;
+ if (IS_ERR(dmabuf))
+ return dmabuf;
+
+ fd = get_unused_fd_flags(heap_allocation->fd_flags);
+ if (fd < 0) {
+ dma_buf_put(dmabuf);
+ return ERR_PTR(fd);
+ }
heap_allocation->fd = fd;
- return 0;
+ return dmabuf;
}
static unsigned int dma_heap_ioctl_cmds[] = {
@@ -132,6 +117,8 @@ static long dma_heap_ioctl(struct file *file, unsigned int ucmd,
unsigned int in_size, out_size, drv_size, ksize;
int nr = _IOC_NR(ucmd);
int ret = 0;
+ int fd;
+ struct dma_buf *dmabuf;
if (nr >= ARRAY_SIZE(dma_heap_ioctl_cmds))
return -EINVAL;
@@ -168,15 +155,28 @@ static long dma_heap_ioctl(struct file *file, unsigned int ucmd,
switch (kcmd) {
case DMA_HEAP_IOCTL_ALLOC:
- ret = dma_heap_ioctl_allocate(file, kdata);
+ dmabuf = dma_heap_ioctl_allocate(file, kdata);
+
+ if (IS_ERR(dmabuf)) {
+ ret = PTR_ERR(dmabuf);
+ break;
+ }
+
+ fd = ((struct dma_heap_allocation_data *)kdata)->fd;
+ if (copy_to_user((void __user *)arg, kdata, out_size) != 0) {
+ put_unused_fd(fd);
+ dma_buf_put(dmabuf);
+ ret = -EFAULT;
+ } else {
+ dma_buf_fd_install(dmabuf, fd);
+ }
+
break;
default:
ret = -ENOTTY;
goto err;
}
- if (copy_to_user((void __user *)arg, kdata, out_size) != 0)
- ret = -EFAULT;
err:
if (kdata != stack_kdata)
kfree(kdata);
diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c
index 6277c5b8d125..3ba357451074 100644
--- a/drivers/dma-buf/udmabuf.c
+++ b/drivers/dma-buf/udmabuf.c
@@ -289,7 +289,16 @@ static long udmabuf_create(struct miscdevice *device,
flags = 0;
if (head->flags & UDMABUF_FLAGS_CLOEXEC)
flags |= O_CLOEXEC;
- return dma_buf_fd(buf, flags);
+
+ /*
+ * Ownership of ubuf is held by the dmabuf from here.
+ * If the following dma_buf_fd() fails, dma_buf_put() cleans up both the
+ * dmabuf and the ubuf (through udmabuf_ops.release).
+ */
+ ret = dma_buf_fd(buf, flags);
+ if (ret < 0)
+ dma_buf_put(buf);
+ return ret;
err:
while (pgbuf > 0)
diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index 53e755c03f41..96c3f455b7bd 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -933,6 +933,36 @@ void dma_release_channel(struct dma_chan *chan)
}
EXPORT_SYMBOL_GPL(dma_release_channel);
+static void dmaenginem_release_channel(void *chan)
+{
+ dma_release_channel(chan);
+}
+
+/**
+ * devm_dma_request_chan - try to allocate an exclusive slave channel
+ * @dev: pointer to client device structure
+ * @name: slave channel name
+ *
+ * Returns pointer to appropriate DMA channel on success or an error pointer.
+ *
+ * The operation is managed and will be undone on driver detach.
+ */
+
+struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name)
+{
+ struct dma_chan *chan = dma_request_chan(dev, name);
+ int ret = 0;
+
+ if (!IS_ERR(chan))
+ ret = devm_add_action_or_reset(dev, dmaenginem_release_channel, chan);
+
+ if (ret)
+ return ERR_PTR(ret);
+
+ return chan;
+}
+EXPORT_SYMBOL_GPL(devm_dma_request_chan);
+
/**
* dmaengine_get - register interest in dma_channels
*/
diff --git a/drivers/dma/dw-edma/dw-edma-core.c b/drivers/dma/dw-edma/dw-edma-core.c
index e68cb0ea7697..d46d3f89f996 100644
--- a/drivers/dma/dw-edma/dw-edma-core.c
+++ b/drivers/dma/dw-edma/dw-edma-core.c
@@ -7,6 +7,7 @@
*/
#include <linux/module.h>
+#include <linux/delay.h>
#include <linux/device.h>
#include <linux/kernel.h>
#include <linux/dmaengine.h>
@@ -199,6 +200,35 @@ static int dw_edma_start_transfer(struct dw_edma_chan *chan)
return 1;
}
+static void dw_edma_terminate_vdesc(struct virt_dma_desc *vd)
+{
+ list_del(&vd->node);
+ dma_cookie_complete(&vd->tx);
+ vchan_terminate_vdesc(vd);
+}
+
+static void dw_edma_terminate_vdesc_list(struct list_head *head)
+{
+ struct virt_dma_desc *vd, *_vd;
+
+ list_for_each_entry_safe(vd, _vd, head, node)
+ dw_edma_terminate_vdesc(vd);
+}
+
+/* Must be called with vc.lock held. */
+static void dw_edma_terminate_all_descs(struct dw_edma_chan *chan)
+{
+ /*
+ * This order must not be reversed. Cookies are assigned when
+ * descriptors are submitted, so desc_issued contains older cookies
+ * than desc_submitted. Completing desc_submitted first could move
+ * chan->vc.chan.completed_cookie backwards when desc_issued is
+ * terminated afterwards.
+ */
+ dw_edma_terminate_vdesc_list(&chan->vc.desc_issued);
+ dw_edma_terminate_vdesc_list(&chan->vc.desc_submitted);
+}
+
static int dw_edma_device_config(struct dma_chan *dchan,
struct dma_slave_config *config)
{
@@ -256,20 +286,22 @@ static int dw_edma_device_terminate_all(struct dma_chan *dchan)
struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
int err = 0;
+ guard(spinlock_irqsave)(&chan->vc.lock);
+
if (!chan->configured) {
- /* Do nothing */
+ dw_edma_terminate_all_descs(chan);
} else if (chan->status == EDMA_ST_PAUSE) {
+ dw_edma_terminate_all_descs(chan);
chan->status = EDMA_ST_IDLE;
- chan->configured = false;
} else if (chan->status == EDMA_ST_IDLE) {
- chan->configured = false;
+ dw_edma_terminate_all_descs(chan);
} else if (dw_edma_v0_core_ch_status(chan) == DMA_COMPLETE) {
/*
* The channel is in a false BUSY state, probably didn't
* receive or lost an interrupt
*/
+ dw_edma_terminate_all_descs(chan);
chan->status = EDMA_ST_IDLE;
- chan->configured = false;
} else if (chan->request > EDMA_REQ_PAUSE) {
err = -EPERM;
} else {
@@ -626,8 +658,7 @@ static void dw_edma_done_interrupt(struct dw_edma_chan *chan)
break;
case EDMA_REQ_STOP:
- list_del(&vd->node);
- vchan_cookie_complete(vd);
+ dw_edma_terminate_all_descs(chan);
chan->request = EDMA_REQ_NONE;
chan->status = EDMA_ST_IDLE;
break;
@@ -648,7 +679,9 @@ static void dw_edma_abort_interrupt(struct dw_edma_chan *chan)
spin_lock_irqsave(&chan->vc.lock, flags);
vd = vchan_next_desc(&chan->vc);
- if (vd) {
+ if (vd && chan->request == EDMA_REQ_STOP) {
+ dw_edma_terminate_all_descs(chan);
+ } else if (vd) {
dw_hdma_set_callback_result(vd, DMA_TRANS_ABORTED);
list_del(&vd->node);
vchan_cookie_complete(vd);
@@ -727,21 +760,49 @@ static int dw_edma_alloc_chan_resources(struct dma_chan *dchan)
return 0;
}
-static void dw_edma_free_chan_resources(struct dma_chan *dchan)
+static void dw_edma_wait_termination(struct dma_chan *dchan)
{
+ struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
unsigned long timeout = jiffies + msecs_to_jiffies(5000);
- int ret;
+ bool stopping;
+ /*
+ * A STOP may be deferred to a later interrupt while the channel is still
+ * running. Wait until that handler completes the termination.
+ */
while (time_before(jiffies, timeout)) {
- ret = dw_edma_device_terminate_all(dchan);
- if (!ret)
- break;
+ scoped_guard(spinlock_irqsave, &chan->vc.lock)
+ stopping = chan->request == EDMA_REQ_STOP;
- if (time_after_eq(jiffies, timeout))
+ if (!stopping)
return;
- cpu_relax();
+ fsleep(1000);
}
+
+ dev_warn(chan->dw->chip->dev,
+ "timeout waiting for channel termination\n");
+}
+
+static void dw_edma_device_synchronize(struct dma_chan *dchan)
+{
+ struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+
+ dw_edma_wait_termination(dchan);
+ vchan_synchronize(&chan->vc);
+}
+
+static void dw_edma_free_chan_resources(struct dma_chan *dchan)
+{
+ struct dw_edma_chan *chan = dchan2dw_edma_chan(dchan);
+
+ dw_edma_device_terminate_all(dchan);
+ dw_edma_device_synchronize(dchan);
+
+ scoped_guard(spinlock_irqsave, &chan->vc.lock)
+ chan->configured = false;
+
+ vchan_free_chan_resources(&chan->vc);
}
static int dw_edma_channel_setup(struct dw_edma *dw, bool write,
@@ -853,6 +914,7 @@ static int dw_edma_channel_setup(struct dw_edma *dw, bool write,
dma->device_pause = dw_edma_device_pause;
dma->device_resume = dw_edma_device_resume;
dma->device_terminate_all = dw_edma_device_terminate_all;
+ dma->device_synchronize = dw_edma_device_synchronize;
dma->device_issue_pending = dw_edma_device_issue_pending;
dma->device_tx_status = dw_edma_device_tx_status;
dma->device_prep_slave_sg = dw_edma_device_prep_slave_sg;
diff --git a/drivers/dma/dw-edma/dw-edma-pcie.c b/drivers/dma/dw-edma/dw-edma-pcie.c
index d6b5e2463884..e149575a6127 100644
--- a/drivers/dma/dw-edma/dw-edma-pcie.c
+++ b/drivers/dma/dw-edma/dw-edma-pcie.c
@@ -151,6 +151,9 @@ static int dw_edma_pcie_probe(struct pci_dev *pdev,
int err, nr_irqs;
int i, mask;
+ if (!pdata)
+ return -ENODEV;
+
/* Enable PCI device */
err = pcim_enable_device(pdev);
if (err) {
diff --git a/drivers/dma/dw-edma/dw-edma-v0-core.c b/drivers/dma/dw-edma/dw-edma-v0-core.c
index ace8980c40d5..1fb1b4d18e26 100644
--- a/drivers/dma/dw-edma/dw-edma-v0-core.c
+++ b/drivers/dma/dw-edma/dw-edma-v0-core.c
@@ -357,6 +357,20 @@ static void dw_edma_v0_core_write_chunk(struct dw_edma_chunk *chunk)
#endif /* CONFIG_64BIT */
}
+static void dw_edma_v0_sync_ll_data(struct dw_edma_chunk *chunk)
+{
+ /*
+ * In case of remote eDMA engine setup, the DW PCIe RP/EP internal
+ * configuration registers and application memory are normally accessed
+ * over different buses. Ensure LL-data reaches the memory before the
+ * doorbell register is toggled by issuing the dummy-read from the remote
+ * LL memory in a hope that the MRd TLP will return only after the
+ * last MWr TLP is completed
+ */
+ if (!(chunk->chan->dw->chip->flags & DW_EDMA_CHIP_LOCAL))
+ readl(chunk->ll_region.vaddr);
+}
+
void dw_edma_v0_core_start(struct dw_edma_chunk *chunk, bool first)
{
struct dw_edma_chan *chan = chunk->chan;
@@ -429,6 +443,9 @@ void dw_edma_v0_core_start(struct dw_edma_chunk *chunk, bool first)
SET_CH_32(dw, chan->dir, chan->id, llp.msb,
upper_32_bits(chunk->ll_region.paddr));
}
+
+ dw_edma_v0_sync_ll_data(chunk);
+
/* Doorbell */
SET_RW_32(dw, chan->dir, doorbell,
FIELD_PREP(EDMA_V0_DOORBELL_CH_MASK, chan->id));
diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c
index 2880a0b0f5e6..b0f32a798282 100644
--- a/drivers/dma/idxd/init.c
+++ b/drivers/dma/idxd/init.c
@@ -143,18 +143,12 @@ static void idxd_cleanup_interrupts(struct idxd_device *idxd)
static void idxd_clean_wqs(struct idxd_device *idxd)
{
- struct idxd_wq *wq;
struct device *conf_dev;
int i;
for (i = 0; i < idxd->max_wqs; i++) {
- wq = idxd->wqs[i];
- if (idxd->hw.wq_cap.op_config)
- bitmap_free(wq->opcap_bmap);
- kfree(wq->wqcfg);
- conf_dev = wq_confdev(wq);
+ conf_dev = wq_confdev(idxd->wqs[i]);
put_device(conf_dev);
- kfree(wq);
}
bitmap_free(idxd->wq_enable_map);
kfree(idxd->wqs);
@@ -196,7 +190,6 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "wq%d.%d", idxd->id, wq->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(wq);
goto err_unwind;
}
@@ -210,7 +203,6 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
wq->wqcfg = kzalloc_node(idxd->wqcfg_size, GFP_KERNEL, dev_to_node(dev));
if (!wq->wqcfg) {
put_device(conf_dev);
- kfree(wq);
rc = -ENOMEM;
goto err_unwind;
}
@@ -218,9 +210,7 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
if (idxd->hw.wq_cap.op_config) {
wq->opcap_bmap = bitmap_zalloc(IDXD_MAX_OPCAP_BITS, GFP_KERNEL);
if (!wq->opcap_bmap) {
- kfree(wq->wqcfg);
put_device(conf_dev);
- kfree(wq);
rc = -ENOMEM;
goto err_unwind;
}
@@ -235,13 +225,8 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
err_unwind:
while (--i >= 0) {
- wq = idxd->wqs[i];
- if (idxd->hw.wq_cap.op_config)
- bitmap_free(wq->opcap_bmap);
- kfree(wq->wqcfg);
- conf_dev = wq_confdev(wq);
+ conf_dev = wq_confdev(idxd->wqs[i]);
put_device(conf_dev);
- kfree(wq);
}
bitmap_free(idxd->wq_enable_map);
@@ -253,15 +238,12 @@ static int idxd_setup_wqs(struct idxd_device *idxd)
static void idxd_clean_engines(struct idxd_device *idxd)
{
- struct idxd_engine *engine;
struct device *conf_dev;
int i;
for (i = 0; i < idxd->max_engines; i++) {
- engine = idxd->engines[i];
- conf_dev = engine_confdev(engine);
+ conf_dev = engine_confdev(idxd->engines[i]);
put_device(conf_dev);
- kfree(engine);
}
kfree(idxd->engines);
}
@@ -296,7 +278,6 @@ static int idxd_setup_engines(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "engine%d.%d", idxd->id, engine->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(engine);
goto err;
}
@@ -307,10 +288,8 @@ static int idxd_setup_engines(struct idxd_device *idxd)
err:
while (--i >= 0) {
- engine = idxd->engines[i];
- conf_dev = engine_confdev(engine);
+ conf_dev = engine_confdev(idxd->engines[i]);
put_device(conf_dev);
- kfree(engine);
}
kfree(idxd->engines);
@@ -319,13 +298,10 @@ static int idxd_setup_engines(struct idxd_device *idxd)
static void idxd_clean_groups(struct idxd_device *idxd)
{
- struct idxd_group *group;
int i;
for (i = 0; i < idxd->max_groups; i++) {
- group = idxd->groups[i];
- put_device(group_confdev(group));
- kfree(group);
+ put_device(group_confdev(idxd->groups[i]));
}
kfree(idxd->groups);
}
@@ -360,7 +336,6 @@ static int idxd_setup_groups(struct idxd_device *idxd)
rc = dev_set_name(conf_dev, "group%d.%d", idxd->id, group->id);
if (rc < 0) {
put_device(conf_dev);
- kfree(group);
goto err;
}
@@ -380,7 +355,6 @@ static int idxd_setup_groups(struct idxd_device *idxd)
while (--i >= 0) {
group = idxd->groups[i];
put_device(group_confdev(group));
- kfree(group);
}
kfree(idxd->groups);
diff --git a/drivers/dma/ti/k3-udma-glue.c b/drivers/dma/ti/k3-udma-glue.c
index a739e60b0764..2d18463a6a58 100644
--- a/drivers/dma/ti/k3-udma-glue.c
+++ b/drivers/dma/ti/k3-udma-glue.c
@@ -83,6 +83,7 @@ struct k3_udma_glue_rx_channel {
struct k3_udma_glue_rx_flow *flows;
u32 flow_num;
u32 flows_ready;
+ bool single_fdq; /* one FDQ for all flows */
};
static void k3_udma_chan_dev_release(struct device *dev)
@@ -896,10 +897,13 @@ k3_udma_glue_request_rx_chn_priv(struct device *dev, const char *name,
ep_cfg = rx_chn->common.ep_config;
- if (xudma_is_pktdma(rx_chn->common.udmax))
+ if (xudma_is_pktdma(rx_chn->common.udmax)) {
rx_chn->udma_rchan_id = ep_cfg->mapped_channel_id;
- else
+ rx_chn->single_fdq = false;
+ } else {
rx_chn->udma_rchan_id = -1;
+ rx_chn->single_fdq = true;
+ }
/* request and cfg UDMAP RX channel */
rx_chn->udma_rchanx = xudma_rchan_get(rx_chn->common.udmax,
@@ -1060,6 +1064,9 @@ k3_udma_glue_request_remote_rx_chn(struct device *dev, const char *name,
rx_chn->common.chan_dev.dma_coherent = true;
dma_coerce_mask_and_coherent(&rx_chn->common.chan_dev,
DMA_BIT_MASK(48));
+ rx_chn->single_fdq = false;
+ } else {
+ rx_chn->single_fdq = true;
}
ret = k3_udma_glue_allocate_rx_flows(rx_chn, cfg);
@@ -1321,7 +1328,7 @@ EXPORT_SYMBOL_GPL(k3_udma_glue_tdown_rx_chn);
void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn,
u32 flow_num, void *data,
- void (*cleanup)(void *data, dma_addr_t desc_dma), bool skip_fdq)
+ void (*cleanup)(void *data, dma_addr_t desc_dma))
{
struct k3_udma_glue_rx_flow *flow = &rx_chn->flows[flow_num];
struct device *dev = rx_chn->common.dev;
@@ -1333,7 +1340,7 @@ void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn,
dev_dbg(dev, "RX reset flow %u occ_rx %u\n", flow_num, occ_rx);
/* Skip RX FDQ in case one FDQ is used for the set of flows */
- if (skip_fdq)
+ if (rx_chn->single_fdq && flow_num)
goto do_reset;
/*
diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c
index 5fb423261c3f..d1f83e315be7 100644
--- a/drivers/dma/xilinx/xilinx_dma.c
+++ b/drivers/dma/xilinx/xilinx_dma.c
@@ -164,6 +164,8 @@
/* Delay loop counter to prevent hardware failure */
#define XILINX_DMA_LOOP_COUNT 1000000
+/* Delay between polls (avoid a delay of 0 to prevent CPU stalls) */
+#define XILINX_DMA_POLL_DELAY_US 10
/* AXI DMA Specific Registers/Offsets */
#define XILINX_DMA_REG_SRCDSTADDR 0x18
@@ -1287,7 +1289,8 @@ static int xilinx_dma_stop_transfer(struct xilinx_dma_chan *chan)
/* Wait for the hardware to halt */
return xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- val & XILINX_DMA_DMASR_HALTED, 0,
+ val & XILINX_DMA_DMASR_HALTED,
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
}
@@ -1302,7 +1305,8 @@ static int xilinx_cdma_stop_transfer(struct xilinx_dma_chan *chan)
u32 val;
return xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- val & XILINX_DMA_DMASR_IDLE, 0,
+ val & XILINX_DMA_DMASR_IDLE,
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
}
@@ -1319,7 +1323,8 @@ static void xilinx_dma_start(struct xilinx_dma_chan *chan)
/* Wait for the hardware to start */
err = xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val,
- !(val & XILINX_DMA_DMASR_HALTED), 0,
+ !(val & XILINX_DMA_DMASR_HALTED),
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
if (err) {
@@ -1742,7 +1747,8 @@ static int xilinx_dma_reset(struct xilinx_dma_chan *chan)
/* Wait for the hardware to finish reset */
err = xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMACR, tmp,
- !(tmp & XILINX_DMA_DMACR_RESET), 0,
+ !(tmp & XILINX_DMA_DMACR_RESET),
+ XILINX_DMA_POLL_DELAY_US,
XILINX_DMA_LOOP_COUNT);
if (err) {
diff --git a/drivers/edac/amd64_edac.c b/drivers/edac/amd64_edac.c
index 2f854feeeb23..7b3d3615de5c 100644
--- a/drivers/edac/amd64_edac.c
+++ b/drivers/edac/amd64_edac.c
@@ -3300,7 +3300,7 @@ static void decode_umc_error(int node_id, struct mce *m)
err.err_code = ERR_CHANNEL;
}
- err.csrow = m->synd & 0x7;
+ err.csrow = m->synd & 0x3;
if (umc_normaddr_to_sysaddr(m->addr, pvt->mc_node_id, err.channel, &sys_addr)) {
err.err_code = ERR_NORM_ADDR;
diff --git a/drivers/edac/igen6_edac.c b/drivers/edac/igen6_edac.c
index 97468fbf17b7..98c56254d79b 100644
--- a/drivers/edac/igen6_edac.c
+++ b/drivers/edac/igen6_edac.c
@@ -1007,6 +1007,11 @@ static int igen6_pci_setup(struct pci_dev *pdev, u64 *mchbar)
return -ENODEV;
}
+static void imc_release(struct device *dev)
+{
+ /* Nothing to do, the 'imc' owns the 'dev' and will also release it. */
+}
+
static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev)
{
struct edac_mc_layer layers[2];
@@ -1051,6 +1056,7 @@ static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev)
mci->pvt_info = &igen6_pvt->imc[mc];
imc = mci->pvt_info;
+ imc->dev.release = imc_release;
device_initialize(&imc->dev);
/*
* EDAC core uses mci->pdev(pointer of structure device) as
diff --git a/drivers/firewire/core-topology.c b/drivers/firewire/core-topology.c
index f40c81534381..ddcc3fb6cfbd 100644
--- a/drivers/firewire/core-topology.c
+++ b/drivers/firewire/core-topology.c
@@ -161,6 +161,17 @@ static inline struct fw_node *fw_node(struct list_head *l)
return list_entry(l, struct fw_node, link);
}
+typedef void (*fw_node_callback_t)(struct fw_card *card, struct fw_node *node,
+ struct fw_node *parent);
+
+static void for_each_fw_node(struct fw_card *card, struct fw_node *root,
+ fw_node_callback_t callback);
+
+static void free_fw_node(struct fw_card *card, struct fw_node *node, struct fw_node *parent)
+{
+ fw_node_put(node);
+}
+
/*
* This function builds the tree representation of the topology given
* by the self IDs from the latest bus reset. During the construction
@@ -193,19 +204,19 @@ static struct fw_node *build_tree(struct fw_card *card,
if (next_sid == NULL) {
fw_err(card, "inconsistent extended self IDs\n");
- return NULL;
+ goto error;
}
q = *sid;
if (phy_id != SELF_ID_PHY_ID(q)) {
fw_err(card, "PHY ID mismatch in self ID: %d != %d\n",
phy_id, SELF_ID_PHY_ID(q));
- return NULL;
+ goto error;
}
if (child_port_count > stack_depth) {
fw_err(card, "topology stack underflow\n");
- return NULL;
+ goto error;
}
/*
@@ -223,7 +234,7 @@ static struct fw_node *build_tree(struct fw_card *card,
node = fw_node_create(q, port_count, card->color);
if (node == NULL) {
fw_err(card, "out of memory while building topology\n");
- return NULL;
+ goto error;
}
if (phy_id == (card->node_id & 0x3f))
@@ -264,6 +275,11 @@ static struct fw_node *build_tree(struct fw_card *card,
}
}
+ /* Pop the child nodes off the stack and push the new node. */
+ __list_del(h->prev, &stack);
+ list_add_tail(&node->link, &stack);
+ stack_depth += 1 - child_port_count;
+
/*
* Check that the node reports exactly one parent
* port, except for the root, which of course should
@@ -273,14 +289,9 @@ static struct fw_node *build_tree(struct fw_card *card,
(next_sid < end && parent_count != 1)) {
fw_err(card, "parent port inconsistency for node %d: "
"parent_count=%d\n", phy_id, parent_count);
- return NULL;
+ goto error;
}
- /* Pop the child nodes off the stack and push the new node. */
- __list_del(h->prev, &stack);
- list_add_tail(&node->link, &stack);
- stack_depth += 1 - child_port_count;
-
if (node->phy_speed == SCODE_BETA &&
parent_count + child_port_count > 1)
beta_repeaters_present = true;
@@ -304,12 +315,13 @@ static struct fw_node *build_tree(struct fw_card *card,
card->beta_repeaters_present = beta_repeaters_present;
return local_node;
+error:
+ ++card->color;
+ list_for_each_entry_safe(node, child, &stack, link)
+ for_each_fw_node(card, node, free_fw_node);
+ return NULL;
}
-typedef void (*fw_node_callback_t)(struct fw_card * card,
- struct fw_node * node,
- struct fw_node * parent);
-
static void for_each_fw_node(struct fw_card *card, struct fw_node *root,
fw_node_callback_t callback)
{
diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c
index 67ab5698a142..5f0eb6442e9d 100644
--- a/drivers/firmware/arm_ffa/driver.c
+++ b/drivers/firmware/arm_ffa/driver.c
@@ -554,7 +554,7 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
constituents = buffer;
}
- if ((void *)constituents - buffer > max_fragsize) {
+ if ((void *)constituents + sizeof(*constituents) - buffer > max_fragsize) {
pr_err("Memory Region Fragment > Tx Buffer size\n");
return -EFAULT;
}
@@ -563,7 +563,7 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize,
constituents->pg_cnt = args->sg->length / FFA_PAGE_SIZE;
constituents->reserved = 0;
constituents++;
- frag_len += sizeof(struct ffa_mem_region_addr_range);
+ frag_len += sizeof(*constituents);
} while ((args->sg = sg_next(args->sg)));
return ffa_transmit_fragment(func_id, addr, buf_sz, frag_len,
diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c
index 8c6f99d15f22..ef7023251187 100644
--- a/drivers/firmware/arm_scmi/bus.c
+++ b/drivers/firmware/arm_scmi/bus.c
@@ -7,17 +7,197 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#include <linux/atomic.h>
#include <linux/types.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/kernel.h>
#include <linux/slab.h>
#include <linux/device.h>
#include "common.h"
+BLOCKING_NOTIFIER_HEAD(scmi_requested_devices_nh);
+EXPORT_SYMBOL_GPL(scmi_requested_devices_nh);
+
static DEFINE_IDA(scmi_bus_id);
-static DEFINE_IDR(scmi_protocols);
-static DEFINE_SPINLOCK(protocol_lock);
+
+static DEFINE_IDR(scmi_requested_devices);
+/* Protect access to scmi_requested_devices */
+static DEFINE_MUTEX(scmi_requested_devices_mtx);
+
+struct scmi_requested_dev {
+ const struct scmi_device_id *id_table;
+ struct list_head node;
+};
+
+/* Track globally the creation of SCMI SystemPower related devices */
+static atomic_t scmi_syspower_registered = ATOMIC_INIT(0);
+
+/**
+ * scmi_protocol_device_request - Helper to request a device
+ *
+ * @id_table: A protocol/name pair descriptor for the device to be created.
+ *
+ * This helper let an SCMI driver request specific devices identified by the
+ * @id_table to be created for each active SCMI instance.
+ *
+ * The requested device name MUST NOT be already existent for any protocol;
+ * at first the freshly requested @id_table is annotated in the IDR table
+ * @scmi_requested_devices and then the requested device is advertised to any
+ * registered party via the @scmi_requested_devices_nh notification chain.
+ *
+ * Return: 0 on Success
+ */
+static int scmi_protocol_device_request(const struct scmi_device_id *id_table)
+{
+ int ret = 0;
+ unsigned int id = 0;
+ struct list_head *head, *phead = NULL;
+ struct scmi_requested_dev *rdev;
+
+ pr_debug("Requesting SCMI device (%s) for protocol %x\n",
+ id_table->name, id_table->protocol_id);
+
+ /*
+ * Search for the matching protocol rdev list and then search
+ * of any existent equally named device...fails if any duplicate found.
+ */
+ mutex_lock(&scmi_requested_devices_mtx);
+ idr_for_each_entry(&scmi_requested_devices, head, id) {
+ if (!phead) {
+ /* A list found registered in the IDR is never empty */
+ rdev = list_first_entry(head, struct scmi_requested_dev,
+ node);
+ if (rdev->id_table->protocol_id ==
+ id_table->protocol_id)
+ phead = head;
+ }
+ list_for_each_entry(rdev, head, node) {
+ if (!strcmp(rdev->id_table->name, id_table->name)) {
+ pr_err("Ignoring duplicate request [%d] %s\n",
+ rdev->id_table->protocol_id,
+ rdev->id_table->name);
+ ret = -EINVAL;
+ goto out;
+ }
+ }
+ }
+
+ /*
+ * No duplicate found for requested id_table, so let's create a new
+ * requested device entry for this new valid request.
+ */
+ rdev = kzalloc(sizeof(*rdev), GFP_KERNEL);
+ if (!rdev) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ rdev->id_table = id_table;
+
+ /*
+ * Append the new requested device table descriptor to the head of the
+ * related protocol list, eventually creating such head if not already
+ * there.
+ */
+ if (!phead) {
+ phead = kzalloc(sizeof(*phead), GFP_KERNEL);
+ if (!phead) {
+ kfree(rdev);
+ ret = -ENOMEM;
+ goto out;
+ }
+ INIT_LIST_HEAD(phead);
+
+ ret = idr_alloc(&scmi_requested_devices, (void *)phead,
+ id_table->protocol_id,
+ id_table->protocol_id + 1, GFP_KERNEL);
+ if (ret != id_table->protocol_id) {
+ pr_err("Failed to save SCMI device - ret:%d\n", ret);
+ kfree(rdev);
+ kfree(phead);
+ ret = -EINVAL;
+ goto out;
+ }
+ ret = 0;
+ }
+ list_add(&rdev->node, phead);
+
+out:
+ mutex_unlock(&scmi_requested_devices_mtx);
+
+ if (!ret)
+ blocking_notifier_call_chain(&scmi_requested_devices_nh,
+ SCMI_BUS_NOTIFY_DEVICE_REQUEST,
+ (void *)rdev->id_table);
+
+ return ret;
+}
+
+static int scmi_protocol_table_register(const struct scmi_device_id *id_table)
+{
+ int ret = 0;
+ const struct scmi_device_id *entry;
+
+ for (entry = id_table; entry->name && ret == 0; entry++)
+ ret = scmi_protocol_device_request(entry);
+
+ return ret;
+}
+
+/**
+ * scmi_protocol_device_unrequest - Helper to unrequest a device
+ *
+ * @id_table: A protocol/name pair descriptor for the device to be unrequested.
+ *
+ * The unrequested device, described by the provided id_table, is at first
+ * removed from the IDR @scmi_requested_devices and then the removal is
+ * advertised to any registered party via the @scmi_requested_devices_nh
+ * notification chain.
+ */
+static void scmi_protocol_device_unrequest(const struct scmi_device_id *id_table)
+{
+ struct scmi_requested_dev *rdev, *victim = NULL;
+ struct list_head *phead;
+
+ pr_debug("Unrequesting SCMI device (%s) for protocol %x\n",
+ id_table->name, id_table->protocol_id);
+
+ mutex_lock(&scmi_requested_devices_mtx);
+ phead = idr_find(&scmi_requested_devices, id_table->protocol_id);
+ if (phead) {
+ list_for_each_entry(rdev, phead, node) {
+ if (!strcmp(rdev->id_table->name, id_table->name)) {
+ victim = rdev;
+ list_del(&rdev->node);
+ break;
+ }
+ }
+
+ if (victim && list_empty(phead)) {
+ idr_remove(&scmi_requested_devices,
+ id_table->protocol_id);
+ kfree(phead);
+ }
+ }
+ mutex_unlock(&scmi_requested_devices_mtx);
+
+ if (victim) {
+ blocking_notifier_call_chain(&scmi_requested_devices_nh,
+ SCMI_BUS_NOTIFY_DEVICE_UNREQUEST,
+ (void *)victim->id_table);
+ kfree(victim);
+ }
+}
+
+static void
+scmi_protocol_table_unregister(const struct scmi_device_id *id_table)
+{
+ const struct scmi_device_id *entry;
+
+ for (entry = id_table; entry->name; entry++)
+ scmi_protocol_device_unrequest(entry);
+}
static const struct scmi_device_id *
scmi_dev_match_id(struct scmi_device *scmi_dev, struct scmi_driver *scmi_drv)
@@ -57,11 +237,12 @@ static int scmi_match_by_id_table(struct device *dev, void *data)
struct scmi_device_id *id_table = data;
return sdev->protocol_id == id_table->protocol_id &&
- !strcmp(sdev->name, id_table->name);
+ (id_table->name && !strcmp(sdev->name, id_table->name));
}
-struct scmi_device *scmi_child_dev_find(struct device *parent,
- int prot_id, const char *name)
+/* Returns a device_find_child() reference which must be dropped by caller. */
+static struct scmi_device *
+scmi_child_dev_find_get(struct device *parent, int prot_id, const char *name)
{
struct scmi_device_id id_table;
struct device *dev;
@@ -73,36 +254,9 @@ struct scmi_device *scmi_child_dev_find(struct device *parent,
if (!dev)
return NULL;
- /* Drop the refcnt bumped implicitly by device_find_child */
- put_device(dev);
-
return to_scmi_dev(dev);
}
-const struct scmi_protocol *scmi_protocol_get(int protocol_id)
-{
- const struct scmi_protocol *proto;
-
- proto = idr_find(&scmi_protocols, protocol_id);
- if (!proto || !try_module_get(proto->owner)) {
- pr_warn("SCMI Protocol 0x%x not found!\n", protocol_id);
- return NULL;
- }
-
- pr_debug("Found SCMI Protocol 0x%x\n", protocol_id);
-
- return proto;
-}
-
-void scmi_protocol_put(int protocol_id)
-{
- const struct scmi_protocol *proto;
-
- proto = idr_find(&scmi_protocols, protocol_id);
- if (proto)
- module_put(proto->owner);
-}
-
static int scmi_dev_probe(struct device *dev)
{
struct scmi_driver *scmi_drv = to_scmi_driver(dev->driver);
@@ -123,12 +277,13 @@ static void scmi_dev_remove(struct device *dev)
scmi_drv->remove(scmi_dev);
}
-static struct bus_type scmi_bus_type = {
+struct bus_type scmi_bus_type = {
.name = "scmi_protocol",
.match = scmi_dev_match,
.probe = scmi_dev_probe,
.remove = scmi_dev_remove,
};
+EXPORT_SYMBOL_GPL(scmi_bus_type);
int scmi_driver_register(struct scmi_driver *driver, struct module *owner,
const char *mod_name)
@@ -138,7 +293,7 @@ int scmi_driver_register(struct scmi_driver *driver, struct module *owner,
if (!driver->probe)
return -EINVAL;
- retval = scmi_protocol_device_request(driver->id_table);
+ retval = scmi_protocol_table_register(driver->id_table);
if (retval)
return retval;
@@ -148,32 +303,90 @@ int scmi_driver_register(struct scmi_driver *driver, struct module *owner,
driver->driver.mod_name = mod_name;
retval = driver_register(&driver->driver);
- if (!retval)
- pr_debug("registered new scmi driver %s\n", driver->name);
+ if (retval) {
+ scmi_protocol_table_unregister(driver->id_table);
+ return retval;
+ }
- return retval;
+ pr_debug("Registered new scmi driver %s\n", driver->name);
+
+ return 0;
}
EXPORT_SYMBOL_GPL(scmi_driver_register);
void scmi_driver_unregister(struct scmi_driver *driver)
{
driver_unregister(&driver->driver);
- scmi_protocol_device_unrequest(driver->id_table);
+ scmi_protocol_table_unregister(driver->id_table);
}
EXPORT_SYMBOL_GPL(scmi_driver_unregister);
+static void scmi_device_release_resources(struct scmi_device *scmi_dev)
+{
+ if (scmi_dev->id) {
+ ida_free(&scmi_bus_id, scmi_dev->id);
+ scmi_dev->id = 0;
+ }
+}
+
static void scmi_device_release(struct device *dev)
{
- kfree(to_scmi_dev(dev));
+ struct scmi_device *scmi_dev = to_scmi_dev(dev);
+
+ scmi_device_release_resources(scmi_dev);
+ kfree_const(scmi_dev->name);
+ kfree(scmi_dev);
+}
+
+static void __scmi_device_destroy(struct scmi_device *scmi_dev)
+{
+ pr_debug("(%s) Destroying SCMI device '%s' for protocol 0x%x (%s)\n",
+ of_node_full_name(scmi_dev->dev.parent->of_node),
+ dev_name(&scmi_dev->dev), scmi_dev->protocol_id,
+ scmi_dev->name);
+
+ if (scmi_dev->protocol_id == SCMI_PROTOCOL_SYSTEM)
+ atomic_set(&scmi_syspower_registered, 0);
+
+ device_del(&scmi_dev->dev);
+ scmi_device_release_resources(scmi_dev);
+ put_device(&scmi_dev->dev);
}
-struct scmi_device *
-scmi_device_create(struct device_node *np, struct device *parent, int protocol,
- const char *name)
+static struct scmi_device *
+__scmi_device_create(struct device_node *np, struct device *parent,
+ int protocol, const char *name)
{
int id, retval;
struct scmi_device *scmi_dev;
+ /*
+ * If the same protocol/name device already exist under the same parent
+ * (i.e. SCMI instance) just return the existent device.
+ * This avoids any race between the SCMI driver, creating devices for
+ * each DT defined protocol at probe time, and the concurrent
+ * registration of SCMI drivers.
+ */
+ scmi_dev = scmi_child_dev_find_get(parent, protocol, name);
+ if (scmi_dev) {
+ put_device(&scmi_dev->dev);
+ return scmi_dev;
+ }
+
+ /*
+ * Ignore any possible subsequent failures while creating the device
+ * since we are doomed anyway at that point; not using a mutex which
+ * spans across this whole function to keep things simple and to avoid
+ * to serialize all the __scmi_device_create calls across possibly
+ * different SCMI server instances (parent)
+ */
+ if (protocol == SCMI_PROTOCOL_SYSTEM &&
+ atomic_cmpxchg(&scmi_syspower_registered, 0, 1)) {
+ dev_warn(parent,
+ "SCMI SystemPower protocol device must be unique !\n");
+ return NULL;
+ }
+
scmi_dev = kzalloc(sizeof(*scmi_dev), GFP_KERNEL);
if (!scmi_dev)
return NULL;
@@ -203,85 +416,98 @@ scmi_device_create(struct device_node *np, struct device *parent, int protocol,
if (retval)
goto put_dev;
+ pr_debug("(%s) Created SCMI device '%s' for protocol 0x%x (%s)\n",
+ of_node_full_name(parent->of_node),
+ dev_name(&scmi_dev->dev), protocol, name);
+
return scmi_dev;
put_dev:
- kfree_const(scmi_dev->name);
+ scmi_device_release_resources(scmi_dev);
put_device(&scmi_dev->dev);
- ida_free(&scmi_bus_id, id);
return NULL;
}
-void scmi_device_destroy(struct scmi_device *scmi_dev)
-{
- kfree_const(scmi_dev->name);
- scmi_handle_put(scmi_dev->handle);
- ida_free(&scmi_bus_id, scmi_dev->id);
- device_unregister(&scmi_dev->dev);
-}
-
-void scmi_device_link_add(struct device *consumer, struct device *supplier)
-{
- struct device_link *link;
-
- link = device_link_add(consumer, supplier, DL_FLAG_AUTOREMOVE_CONSUMER);
-
- WARN_ON(!link);
-}
-
-void scmi_set_handle(struct scmi_device *scmi_dev)
-{
- scmi_dev->handle = scmi_handle_get(&scmi_dev->dev);
- if (scmi_dev->handle)
- scmi_device_link_add(&scmi_dev->dev, scmi_dev->handle->dev);
-}
-
-int scmi_protocol_register(const struct scmi_protocol *proto)
+/**
+ * scmi_device_create - A method to create one or more SCMI devices
+ *
+ * @np: A reference to the device node to use for the new device(s)
+ * @parent: The parent device to use identifying a specific SCMI instance
+ * @protocol: The SCMI protocol to be associated with this device
+ * @name: The requested-name of the device to be created; this is optional
+ * and if no @name is provided, all the devices currently known to
+ * be requested on the SCMI bus for @protocol will be created.
+ *
+ * This method can be invoked to create a single well-defined device (like
+ * a transport device or a device requested by an SCMI driver loaded after
+ * the core SCMI stack has been probed), or to create all the devices currently
+ * known to have been requested by the loaded SCMI drivers for a specific
+ * protocol (typically during SCMI core protocol enumeration at probe time).
+ *
+ * Return: The created device (or one of them if @name was NOT provided and
+ * multiple devices were created) or NULL if no device was created;
+ * note that NULL indicates an error ONLY in case a specific @name
+ * was provided: when @name param was not provided, a number of devices
+ * could have been potentially created for a whole protocol, unless no
+ * device was found to have been requested for that specific protocol.
+ */
+struct scmi_device *scmi_device_create(struct device_node *np,
+ struct device *parent, int protocol,
+ const char *name)
{
- int ret;
-
- if (!proto) {
- pr_err("invalid protocol\n");
- return -EINVAL;
- }
-
- if (!proto->instance_init) {
- pr_err("missing init for protocol 0x%x\n", proto->id);
- return -EINVAL;
+ struct list_head *phead;
+ struct scmi_requested_dev *rdev;
+ struct scmi_device *scmi_dev = NULL;
+
+ if (name)
+ return __scmi_device_create(np, parent, protocol, name);
+
+ mutex_lock(&scmi_requested_devices_mtx);
+ phead = idr_find(&scmi_requested_devices, protocol);
+ /* Nothing to do. */
+ if (!phead) {
+ mutex_unlock(&scmi_requested_devices_mtx);
+ return scmi_dev;
}
- spin_lock(&protocol_lock);
- ret = idr_alloc(&scmi_protocols, (void *)proto,
- proto->id, proto->id + 1, GFP_ATOMIC);
- spin_unlock(&protocol_lock);
- if (ret != proto->id) {
- pr_err("unable to allocate SCMI idr slot for 0x%x - err %d\n",
- proto->id, ret);
- return ret;
+ /* Walk the list of requested devices for protocol and create them */
+ list_for_each_entry(rdev, phead, node) {
+ struct scmi_device *sdev;
+
+ sdev = __scmi_device_create(np, parent,
+ rdev->id_table->protocol_id,
+ rdev->id_table->name);
+ /* Report errors and carry on... */
+ if (sdev)
+ scmi_dev = sdev;
+ else
+ pr_err("(%s) Failed to create device for protocol 0x%x (%s)\n",
+ of_node_full_name(parent->of_node),
+ rdev->id_table->protocol_id,
+ rdev->id_table->name);
}
+ mutex_unlock(&scmi_requested_devices_mtx);
- pr_debug("Registered SCMI Protocol 0x%x\n", proto->id);
-
- return 0;
+ return scmi_dev;
}
-EXPORT_SYMBOL_GPL(scmi_protocol_register);
+EXPORT_SYMBOL_GPL(scmi_device_create);
-void scmi_protocol_unregister(const struct scmi_protocol *proto)
+void scmi_device_destroy(struct device *parent, int protocol, const char *name)
{
- spin_lock(&protocol_lock);
- idr_remove(&scmi_protocols, proto->id);
- spin_unlock(&protocol_lock);
-
- pr_debug("Unregistered SCMI Protocol 0x%x\n", proto->id);
+ struct scmi_device *scmi_dev;
- return;
+ scmi_dev = scmi_child_dev_find_get(parent, protocol, name);
+ if (scmi_dev) {
+ __scmi_device_destroy(scmi_dev);
+ put_device(&scmi_dev->dev);
+ }
}
-EXPORT_SYMBOL_GPL(scmi_protocol_unregister);
+EXPORT_SYMBOL_GPL(scmi_device_destroy);
static int __scmi_devices_unregister(struct device *dev, void *data)
{
struct scmi_device *scmi_dev = to_scmi_dev(dev);
- scmi_device_destroy(scmi_dev);
+ __scmi_device_destroy(scmi_dev);
return 0;
}
@@ -303,6 +529,10 @@ int __init scmi_bus_init(void)
void __exit scmi_bus_exit(void)
{
+ /*
+ * Destroy all remaining devices: just in case the drivers were
+ * manually unbound and at first and then the modules unloaded.
+ */
scmi_devices_unregister();
bus_unregister(&scmi_bus_type);
ida_destroy(&scmi_bus_id);
diff --git a/drivers/firmware/arm_scmi/common.h b/drivers/firmware/arm_scmi/common.h
index 4ac72754c255..10409b43fb29 100644
--- a/drivers/firmware/arm_scmi/common.h
+++ b/drivers/firmware/arm_scmi/common.h
@@ -27,6 +27,46 @@
#include "protocols.h"
#include "notify.h"
+#define SCMI_MAX_RESPONSE_TIMEOUT (2 * MSEC_PER_SEC)
+
+enum scmi_error_codes {
+ SCMI_SUCCESS = 0, /* Success */
+ SCMI_ERR_SUPPORT = -1, /* Not supported */
+ SCMI_ERR_PARAMS = -2, /* Invalid Parameters */
+ SCMI_ERR_ACCESS = -3, /* Invalid access/permission denied */
+ SCMI_ERR_ENTRY = -4, /* Not found */
+ SCMI_ERR_RANGE = -5, /* Value out of range */
+ SCMI_ERR_BUSY = -6, /* Device busy */
+ SCMI_ERR_COMMS = -7, /* Communication Error */
+ SCMI_ERR_GENERIC = -8, /* Generic Error */
+ SCMI_ERR_HARDWARE = -9, /* Hardware Error */
+ SCMI_ERR_PROTOCOL = -10,/* Protocol Error */
+};
+
+static const int scmi_linux_errmap[] = {
+ /* better than switch case as long as return value is continuous */
+ 0, /* SCMI_SUCCESS */
+ -EOPNOTSUPP, /* SCMI_ERR_SUPPORT */
+ -EINVAL, /* SCMI_ERR_PARAM */
+ -EACCES, /* SCMI_ERR_ACCESS */
+ -ENOENT, /* SCMI_ERR_ENTRY */
+ -ERANGE, /* SCMI_ERR_RANGE */
+ -EBUSY, /* SCMI_ERR_BUSY */
+ -ECOMM, /* SCMI_ERR_COMMS */
+ -EIO, /* SCMI_ERR_GENERIC */
+ -EREMOTEIO, /* SCMI_ERR_HARDWARE */
+ -EPROTO, /* SCMI_ERR_PROTOCOL */
+};
+
+static inline int scmi_to_linux_errno(int errno)
+{
+ int err_idx = -errno;
+
+ if (err_idx >= SCMI_SUCCESS && err_idx < ARRAY_SIZE(scmi_linux_errmap))
+ return scmi_linux_errmap[err_idx];
+ return -EIO;
+}
+
#define MSG_ID_MASK GENMASK(7, 0)
#define MSG_XTRACT_ID(hdr) FIELD_GET(MSG_ID_MASK, (hdr))
#define MSG_TYPE_MASK GENMASK(9, 8)
@@ -96,18 +136,21 @@ static inline void unpack_scmi_header(u32 msg_hdr, struct scmi_msg_hdr *hdr)
struct scmi_revision_info *
scmi_revision_area_get(const struct scmi_protocol_handle *ph);
-int scmi_handle_put(const struct scmi_handle *handle);
-void scmi_device_link_add(struct device *consumer, struct device *supplier);
-struct scmi_handle *scmi_handle_get(struct device *dev);
-void scmi_set_handle(struct scmi_device *scmi_dev);
void scmi_setup_protocol_implemented(const struct scmi_protocol_handle *ph,
u8 *prot_imp);
int __init scmi_bus_init(void);
void __exit scmi_bus_exit(void);
+extern struct bus_type scmi_bus_type;
-const struct scmi_protocol *scmi_protocol_get(int protocol_id);
-void scmi_protocol_put(int protocol_id);
+#define SCMI_BUS_NOTIFY_DEVICE_REQUEST 0
+#define SCMI_BUS_NOTIFY_DEVICE_UNREQUEST 1
+extern struct blocking_notifier_head scmi_requested_devices_nh;
+
+struct scmi_device *scmi_device_create(struct device_node *np,
+ struct device *parent, int protocol,
+ const char *name);
+void scmi_device_destroy(struct device *parent, int protocol, const char *name);
int scmi_protocol_acquire(const struct scmi_handle *handle, u8 protocol_id);
void scmi_protocol_release(const struct scmi_handle *handle, u8 protocol_id);
@@ -116,6 +159,8 @@ void scmi_protocol_release(const struct scmi_handle *handle, u8 protocol_id);
/**
* struct scmi_chan_info - Structure representing a SCMI channel information
*
+ * @id: An identifier for this channel: this matches the protocol number
+ * used to initialize this channel
* @dev: Reference to device in the SCMI hierarchy corresponding to this
* channel
* @rx_timeout_ms: The configured RX timeout in milliseconds.
@@ -127,6 +172,7 @@ void scmi_protocol_release(const struct scmi_handle *handle, u8 protocol_id);
* @transport_info: Transport layer related information
*/
struct scmi_chan_info {
+ int id;
struct device *dev;
unsigned int rx_timeout_ms;
struct scmi_handle *handle;
@@ -153,7 +199,7 @@ struct scmi_chan_info {
*/
struct scmi_transport_ops {
int (*link_supplier)(struct device *dev);
- bool (*chan_available)(struct device *dev, int idx);
+ bool (*chan_available)(struct device_node *of_node, int idx);
int (*chan_setup)(struct scmi_chan_info *cinfo, struct device *dev,
bool tx);
int (*chan_free)(int id, void *p, void *data);
@@ -170,11 +216,6 @@ struct scmi_transport_ops {
bool (*poll_done)(struct scmi_chan_info *cinfo, struct scmi_xfer *xfer);
};
-int scmi_protocol_device_request(const struct scmi_device_id *id_table);
-void scmi_protocol_device_unrequest(const struct scmi_device_id *id_table);
-struct scmi_device *scmi_child_dev_find(struct device *parent,
- int prot_id, const char *name);
-
/**
* struct scmi_desc - Description of SoC integration
*
@@ -229,7 +270,6 @@ extern const struct scmi_desc scmi_optee_desc;
#endif
void scmi_rx_callback(struct scmi_chan_info *cinfo, u32 msg_hdr, void *priv);
-void scmi_free_channel(struct scmi_chan_info *cinfo, struct idr *idr, int id);
/* shmem related declarations */
struct scmi_shared_mem;
diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c
index 53fd9b17b9bd..cbc02edb20a9 100644
--- a/drivers/firmware/arm_scmi/driver.c
+++ b/drivers/firmware/arm_scmi/driver.c
@@ -14,6 +14,8 @@
* Copyright (C) 2018-2021 ARM Ltd.
*/
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
#include <linux/bitmap.h>
#include <linux/device.h>
#include <linux/export.h>
@@ -28,6 +30,7 @@
#include <linux/of_address.h>
#include <linux/of_device.h>
#include <linux/processor.h>
+#include <linux/rcupdate.h>
#include <linux/refcount.h>
#include <linux/slab.h>
@@ -37,19 +40,10 @@
#define CREATE_TRACE_POINTS
#include <trace/events/scmi.h>
-enum scmi_error_codes {
- SCMI_SUCCESS = 0, /* Success */
- SCMI_ERR_SUPPORT = -1, /* Not supported */
- SCMI_ERR_PARAMS = -2, /* Invalid Parameters */
- SCMI_ERR_ACCESS = -3, /* Invalid access/permission denied */
- SCMI_ERR_ENTRY = -4, /* Not found */
- SCMI_ERR_RANGE = -5, /* Value out of range */
- SCMI_ERR_BUSY = -6, /* Device busy */
- SCMI_ERR_COMMS = -7, /* Communication Error */
- SCMI_ERR_GENERIC = -8, /* Generic Error */
- SCMI_ERR_HARDWARE = -9, /* Hardware Error */
- SCMI_ERR_PROTOCOL = -10,/* Protocol Error */
-};
+static DEFINE_IDA(scmi_id);
+
+static DEFINE_IDR(scmi_protocols);
+static DEFINE_SPINLOCK(protocol_lock);
/* List of all SCMI devices active in system */
static LIST_HEAD(scmi_list);
@@ -58,19 +52,6 @@ static DEFINE_MUTEX(scmi_list_mutex);
/* Track the unique id for the transfers for debug & profiling purpose */
static atomic_t transfer_last_id;
-static DEFINE_IDR(scmi_requested_devices);
-static DEFINE_MUTEX(scmi_requested_devices_mtx);
-
-/* Track globally the creation of SCMI SystemPower related devices */
-static bool scmi_syspower_registered;
-/* Protect access to scmi_syspower_registered */
-static DEFINE_MUTEX(scmi_syspower_mtx);
-
-struct scmi_requested_dev {
- const struct scmi_device_id *id_table;
- struct list_head node;
-};
-
/**
* struct scmi_xfers_info - Structure to manage transfer information
*
@@ -120,6 +101,7 @@ struct scmi_protocol_instance {
/**
* struct scmi_info - Structure representing a SCMI instance
*
+ * @id: A sequence number starting from zero identifying this instance
* @dev: Device pointer
* @desc: SoC description for this instance
* @version: SCMI revision information containing protocol version,
@@ -147,8 +129,13 @@ struct scmi_protocol_instance {
* @notify_priv: Pointer to private data structure specific to notifications.
* @node: List head
* @users: Number of users of this instance
+ * @bus_nb: A notifier to listen for device bind/unbind on the scmi bus
+ * @dev_req_nb: A notifier to listen for device request/unrequest on the scmi
+ * bus
+ * @devreq_mtx: A mutex to serialize device creation for this SCMI instance
*/
struct scmi_info {
+ int id;
struct device *dev;
const struct scmi_desc *desc;
struct scmi_revision_info version;
@@ -166,32 +153,112 @@ struct scmi_info {
void *notify_priv;
struct list_head node;
int users;
+ struct notifier_block bus_nb;
+ struct notifier_block dev_req_nb;
+ /* Serialize device creation process for this instance */
+ struct mutex devreq_mtx;
};
#define handle_to_scmi_info(h) container_of(h, struct scmi_info, handle)
+#define bus_nb_to_scmi_info(nb) container_of(nb, struct scmi_info, bus_nb)
+#define req_nb_to_scmi_info(nb) container_of(nb, struct scmi_info, dev_req_nb)
-static const int scmi_linux_errmap[] = {
- /* better than switch case as long as return value is continuous */
- 0, /* SCMI_SUCCESS */
- -EOPNOTSUPP, /* SCMI_ERR_SUPPORT */
- -EINVAL, /* SCMI_ERR_PARAM */
- -EACCES, /* SCMI_ERR_ACCESS */
- -ENOENT, /* SCMI_ERR_ENTRY */
- -ERANGE, /* SCMI_ERR_RANGE */
- -EBUSY, /* SCMI_ERR_BUSY */
- -ECOMM, /* SCMI_ERR_COMMS */
- -EIO, /* SCMI_ERR_GENERIC */
- -EREMOTEIO, /* SCMI_ERR_HARDWARE */
- -EPROTO, /* SCMI_ERR_PROTOCOL */
-};
+static const struct scmi_protocol *scmi_protocol_get(int protocol_id)
+{
+ const struct scmi_protocol *proto;
+
+ proto = idr_find(&scmi_protocols, protocol_id);
+ if (!proto || !try_module_get(proto->owner)) {
+ pr_warn("SCMI Protocol 0x%x not found!\n", protocol_id);
+ return NULL;
+ }
+
+ pr_debug("Found SCMI Protocol 0x%x\n", protocol_id);
-static inline int scmi_to_linux_errno(int errno)
+ return proto;
+}
+
+static void scmi_protocol_put(int protocol_id)
{
- int err_idx = -errno;
+ const struct scmi_protocol *proto;
- if (err_idx >= SCMI_SUCCESS && err_idx < ARRAY_SIZE(scmi_linux_errmap))
- return scmi_linux_errmap[err_idx];
- return -EIO;
+ proto = idr_find(&scmi_protocols, protocol_id);
+ if (proto)
+ module_put(proto->owner);
+}
+
+int scmi_protocol_register(const struct scmi_protocol *proto)
+{
+ int ret;
+
+ if (!proto) {
+ pr_err("invalid protocol\n");
+ return -EINVAL;
+ }
+
+ if (!proto->instance_init) {
+ pr_err("missing init for protocol 0x%x\n", proto->id);
+ return -EINVAL;
+ }
+
+ spin_lock(&protocol_lock);
+ ret = idr_alloc(&scmi_protocols, (void *)proto,
+ proto->id, proto->id + 1, GFP_ATOMIC);
+ spin_unlock(&protocol_lock);
+ if (ret != proto->id) {
+ pr_err("unable to allocate SCMI idr slot for 0x%x - err %d\n",
+ proto->id, ret);
+ return ret;
+ }
+
+ pr_debug("Registered SCMI Protocol 0x%x\n", proto->id);
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(scmi_protocol_register);
+
+void scmi_protocol_unregister(const struct scmi_protocol *proto)
+{
+ spin_lock(&protocol_lock);
+ idr_remove(&scmi_protocols, proto->id);
+ spin_unlock(&protocol_lock);
+
+ pr_debug("Unregistered SCMI Protocol 0x%x\n", proto->id);
+}
+EXPORT_SYMBOL_GPL(scmi_protocol_unregister);
+
+/**
+ * scmi_create_protocol_devices - Create devices for all pending requests for
+ * this SCMI instance.
+ *
+ * @np: The device node describing the protocol
+ * @info: The SCMI instance descriptor
+ * @prot_id: The protocol ID
+ * @name: The optional name of the device to be created: if not provided this
+ * call will lead to the creation of all the devices currently requested
+ * for the specified protocol.
+ */
+static void scmi_create_protocol_devices(struct device_node *np,
+ struct scmi_info *info,
+ int prot_id, const char *name)
+{
+ struct scmi_device *sdev;
+
+ mutex_lock(&info->devreq_mtx);
+ sdev = scmi_device_create(np, info->dev, prot_id, name);
+ if (name && !sdev)
+ dev_err(info->dev,
+ "failed to create device for protocol 0x%X (%s)\n",
+ prot_id, name);
+ mutex_unlock(&info->devreq_mtx);
+}
+
+static void scmi_destroy_protocol_devices(struct scmi_info *info,
+ int prot_id, const char *name)
+{
+ mutex_lock(&info->devreq_mtx);
+ scmi_device_destroy(info->dev, prot_id, name);
+ mutex_unlock(&info->devreq_mtx);
}
void scmi_notification_instance_data_set(const struct scmi_handle *handle,
@@ -954,8 +1021,6 @@ static void reset_rx_to_maxsz(const struct scmi_protocol_handle *ph,
xfer->rx.len = info->desc->max_msg_size;
}
-#define SCMI_MAX_RESPONSE_TIMEOUT (2 * MSEC_PER_SEC)
-
/**
* do_xfer_with_response() - Do one transfer and wait until the delayed
* response is received
@@ -1873,13 +1938,6 @@ static bool scmi_is_transport_atomic(const struct scmi_handle *handle,
return ret;
}
-static inline
-struct scmi_handle *scmi_handle_get_from_info_unlocked(struct scmi_info *info)
-{
- info->users++;
- return &info->handle;
-}
-
/**
* scmi_handle_get() - Get the SCMI handle for a device
*
@@ -1891,7 +1949,7 @@ struct scmi_handle *scmi_handle_get_from_info_unlocked(struct scmi_info *info)
*
* Return: pointer to handle if successful, NULL on error
*/
-struct scmi_handle *scmi_handle_get(struct device *dev)
+static struct scmi_handle *scmi_handle_get(struct device *dev)
{
struct list_head *p;
struct scmi_info *info;
@@ -1901,7 +1959,8 @@ struct scmi_handle *scmi_handle_get(struct device *dev)
list_for_each(p, &scmi_list) {
info = list_entry(p, struct scmi_info, node);
if (dev->parent == info->dev) {
- handle = scmi_handle_get_from_info_unlocked(info);
+ info->users++;
+ handle = &info->handle;
break;
}
}
@@ -1922,7 +1981,7 @@ struct scmi_handle *scmi_handle_get(struct device *dev)
* Return: 0 is successfully released
* if null was passed, it returns -EINVAL;
*/
-int scmi_handle_put(const struct scmi_handle *handle)
+static int scmi_handle_put(const struct scmi_handle *handle)
{
struct scmi_info *info;
@@ -1938,6 +1997,33 @@ int scmi_handle_put(const struct scmi_handle *handle)
return 0;
}
+static bool scmi_device_link_add(struct device *consumer,
+ struct device *supplier)
+{
+ struct device_link *link;
+
+ link = device_link_add(consumer, supplier, DL_FLAG_AUTOREMOVE_CONSUMER);
+
+ return !WARN_ON(!link);
+}
+
+static void scmi_clear_handle(struct scmi_device *scmi_dev)
+{
+ if (!scmi_dev->handle)
+ return;
+
+ scmi_handle_put(scmi_dev->handle);
+ scmi_dev->handle = NULL;
+}
+
+static void scmi_set_handle(struct scmi_device *scmi_dev)
+{
+ scmi_dev->handle = scmi_handle_get(&scmi_dev->dev);
+ if (scmi_dev->handle &&
+ !scmi_device_link_add(&scmi_dev->dev, scmi_dev->handle->dev))
+ scmi_clear_handle(scmi_dev);
+}
+
static int __scmi_xfer_info_init(struct scmi_info *sinfo,
struct scmi_xfers_info *info)
{
@@ -2031,23 +2117,20 @@ static int scmi_xfer_info_init(struct scmi_info *sinfo)
return ret;
}
-static int scmi_chan_setup(struct scmi_info *info, struct device *dev,
+static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node,
int prot_id, bool tx)
{
int ret, idx;
+ char name[32];
struct scmi_chan_info *cinfo;
struct idr *idr;
+ struct scmi_device *tdev = NULL;
/* Transmit channel is first entry i.e. index 0 */
idx = tx ? 0 : 1;
idr = tx ? &info->tx_idr : &info->rx_idr;
- /* check if already allocated, used for multiple device per protocol */
- cinfo = idr_find(idr, prot_id);
- if (cinfo)
- return 0;
-
- if (!info->desc->ops->chan_available(dev, idx)) {
+ if (!info->desc->ops->chan_available(of_node, idx)) {
cinfo = idr_find(idr, SCMI_PROTOCOL_BASE);
if (unlikely(!cinfo)) /* Possible only if platform has no Rx */
return -EINVAL;
@@ -2058,42 +2141,69 @@ static int scmi_chan_setup(struct scmi_info *info, struct device *dev,
if (!cinfo)
return -ENOMEM;
- cinfo->dev = dev;
cinfo->rx_timeout_ms = info->desc->max_rx_timeout_ms;
+ /* Create a unique name for this transport device */
+ snprintf(name, 32, "__scmi_transport_device_%s_%02X",
+ idx ? "rx" : "tx", prot_id);
+ /* Create a uniquely named, dedicated transport device for this chan */
+ tdev = scmi_device_create(of_node, info->dev, prot_id, name);
+ if (!tdev) {
+ dev_err(info->dev,
+ "failed to create transport device (%s)\n", name);
+ devm_kfree(info->dev, cinfo);
+ return -EINVAL;
+ }
+ of_node_get(of_node);
+
+ cinfo->id = prot_id;
+ cinfo->dev = &tdev->dev;
+ cinfo->handle = &info->handle;
ret = info->desc->ops->chan_setup(cinfo, info->dev, tx);
- if (ret)
+ if (ret) {
+ of_node_put(of_node);
+ scmi_device_destroy(info->dev, prot_id, name);
+ devm_kfree(info->dev, cinfo);
return ret;
+ }
if (tx && is_polling_required(cinfo, info)) {
if (is_transport_polling_capable(info))
- dev_info(dev,
+ dev_info(&tdev->dev,
"Enabled polling mode TX channel - prot_id:%d\n",
prot_id);
else
- dev_warn(dev,
+ dev_warn(&tdev->dev,
"Polling mode NOT supported by transport.\n");
}
idr_alloc:
ret = idr_alloc(idr, cinfo, prot_id, prot_id + 1, GFP_KERNEL);
if (ret != prot_id) {
- dev_err(dev, "unable to allocate SCMI idr slot err %d\n", ret);
+ dev_err(info->dev,
+ "unable to allocate SCMI idr slot err %d\n", ret);
+ /* Destroy channel and device only if created by this call. */
+ if (tdev) {
+ info->desc->ops->chan_free(prot_id, cinfo, idr);
+ of_node_put(of_node);
+ scmi_device_destroy(info->dev, prot_id, name);
+ devm_kfree(info->dev, cinfo);
+ }
return ret;
}
- cinfo->handle = &info->handle;
return 0;
}
static inline int
-scmi_txrx_setup(struct scmi_info *info, struct device *dev, int prot_id)
+scmi_txrx_setup(struct scmi_info *info, struct device_node *of_node,
+ int prot_id)
{
- int ret = scmi_chan_setup(info, dev, prot_id, true);
+ int ret = scmi_chan_setup(info, of_node, prot_id, true);
if (!ret) {
/* Rx is optional, report only memory errors */
- ret = scmi_chan_setup(info, dev, prot_id, false);
+ ret = scmi_chan_setup(info, of_node, prot_id, false);
if (ret && ret != -ENOMEM)
ret = 0;
}
@@ -2102,308 +2212,156 @@ scmi_txrx_setup(struct scmi_info *info, struct device *dev, int prot_id)
}
/**
- * scmi_get_protocol_device - Helper to get/create an SCMI device.
- *
- * @np: A device node representing a valid active protocols for the referred
- * SCMI instance.
- * @info: The referred SCMI instance for which we are getting/creating this
- * device.
- * @prot_id: The protocol ID.
- * @name: The device name.
- *
- * Referring to the specific SCMI instance identified by @info, this helper
- * takes care to return a properly initialized device matching the requested
- * @proto_id and @name: if device was still not existent it is created as a
- * child of the specified SCMI instance @info and its transport properly
- * initialized as usual.
- *
- * Return: A properly initialized scmi device, NULL otherwise.
+ * scmi_channels_setup - Helper to initialize all required channels
+ *
+ * @info: The SCMI instance descriptor.
+ *
+ * Initialize all the channels found described in the DT against the underlying
+ * configured transport using custom defined dedicated devices instead of
+ * borrowing devices from the SCMI drivers; this way channels are initialized
+ * upfront during core SCMI stack probing and are no more coupled with SCMI
+ * devices used by SCMI drivers.
+ *
+ * Note that, even though a pair of TX/RX channels is associated to each
+ * protocol defined in the DT, a distinct freshly initialized channel is
+ * created only if the DT node for the protocol at hand describes a dedicated
+ * channel: in all the other cases the common BASE protocol channel is reused.
+ *
+ * Return: 0 on Success
*/
-static inline struct scmi_device *
-scmi_get_protocol_device(struct device_node *np, struct scmi_info *info,
- int prot_id, const char *name)
+static int scmi_channels_setup(struct scmi_info *info)
{
- struct scmi_device *sdev;
-
- /* Already created for this parent SCMI instance ? */
- sdev = scmi_child_dev_find(info->dev, prot_id, name);
- if (sdev)
- return sdev;
+ int ret;
+ struct device_node *child, *top_np = info->dev->of_node;
- mutex_lock(&scmi_syspower_mtx);
- if (prot_id == SCMI_PROTOCOL_SYSTEM && scmi_syspower_registered) {
- dev_warn(info->dev,
- "SCMI SystemPower protocol device must be unique !\n");
- mutex_unlock(&scmi_syspower_mtx);
+ /* Initialize a common generic channel at first */
+ ret = scmi_txrx_setup(info, top_np, SCMI_PROTOCOL_BASE);
+ if (ret)
+ return ret;
- return NULL;
- }
+ for_each_available_child_of_node(top_np, child) {
+ u32 prot_id;
- pr_debug("Creating SCMI device (%s) for protocol %x\n", name, prot_id);
+ if (of_property_read_u32(child, "reg", &prot_id))
+ continue;
- sdev = scmi_device_create(np, info->dev, prot_id, name);
- if (!sdev) {
- dev_err(info->dev, "failed to create %d protocol device\n",
- prot_id);
- mutex_unlock(&scmi_syspower_mtx);
+ if (!FIELD_FIT(MSG_PROTOCOL_ID_MASK, prot_id)) {
+ dev_err(info->dev,
+ "Out of range protocol %d\n", prot_id);
+ continue;
+ }
- return NULL;
+ ret = scmi_txrx_setup(info, child, prot_id);
+ if (ret) {
+ of_node_put(child);
+ return ret;
+ }
}
- if (scmi_txrx_setup(info, &sdev->dev, prot_id)) {
- dev_err(&sdev->dev, "failed to setup transport\n");
- scmi_device_destroy(sdev);
- mutex_unlock(&scmi_syspower_mtx);
+ return 0;
+}
- return NULL;
- }
+static int scmi_chan_destroy(int id, void *p, void *data)
+{
+ struct scmi_chan_info *cinfo = p;
- if (prot_id == SCMI_PROTOCOL_SYSTEM)
- scmi_syspower_registered = true;
+ if (cinfo->dev) {
+ struct scmi_info *info = handle_to_scmi_info(cinfo->handle);
+ struct scmi_device *sdev = to_scmi_dev(cinfo->dev);
- mutex_unlock(&scmi_syspower_mtx);
+ of_node_put(cinfo->dev->of_node);
+ scmi_device_destroy(info->dev, cinfo->id, sdev->name);
+ cinfo->dev = NULL;
+ }
- return sdev;
+ return 0;
}
-static inline void
-scmi_create_protocol_device(struct device_node *np, struct scmi_info *info,
- int prot_id, const char *name)
+static void scmi_cleanup_channels(struct scmi_info *info, struct idr *idr)
{
- struct scmi_device *sdev;
+ /* At first free all channels at the transport layer ... */
+ idr_for_each(idr, info->desc->ops->chan_free, idr);
- sdev = scmi_get_protocol_device(np, info, prot_id, name);
- if (!sdev)
- return;
+ /* ...then destroy all underlying devices */
+ idr_for_each(idr, scmi_chan_destroy, idr);
- /* setup handle now as the transport is ready */
- scmi_set_handle(sdev);
+ idr_destroy(idr);
}
-/**
- * scmi_create_protocol_devices - Create devices for all pending requests for
- * this SCMI instance.
- *
- * @np: The device node describing the protocol
- * @info: The SCMI instance descriptor
- * @prot_id: The protocol ID
- *
- * All devices previously requested for this instance (if any) are found and
- * created by scanning the proper @&scmi_requested_devices entry.
- */
-static void scmi_create_protocol_devices(struct device_node *np,
- struct scmi_info *info, int prot_id)
+static void scmi_cleanup_txrx_channels(struct scmi_info *info)
{
- struct list_head *phead;
-
- mutex_lock(&scmi_requested_devices_mtx);
- phead = idr_find(&scmi_requested_devices, prot_id);
- if (phead) {
- struct scmi_requested_dev *rdev;
+ scmi_cleanup_channels(info, &info->tx_idr);
- list_for_each_entry(rdev, phead, node)
- scmi_create_protocol_device(np, info, prot_id,
- rdev->id_table->name);
- }
- mutex_unlock(&scmi_requested_devices_mtx);
+ scmi_cleanup_channels(info, &info->rx_idr);
}
-/**
- * scmi_protocol_device_request - Helper to request a device
- *
- * @id_table: A protocol/name pair descriptor for the device to be created.
- *
- * This helper let an SCMI driver request specific devices identified by the
- * @id_table to be created for each active SCMI instance.
- *
- * The requested device name MUST NOT be already existent for any protocol;
- * at first the freshly requested @id_table is annotated in the IDR table
- * @scmi_requested_devices, then a matching device is created for each already
- * active SCMI instance. (if any)
- *
- * This way the requested device is created straight-away for all the already
- * initialized(probed) SCMI instances (handles) and it remains also annotated
- * as pending creation if the requesting SCMI driver was loaded before some
- * SCMI instance and related transports were available: when such late instance
- * is probed, its probe will take care to scan the list of pending requested
- * devices and create those on its own (see @scmi_create_protocol_devices and
- * its enclosing loop)
- *
- * Return: 0 on Success
- */
-int scmi_protocol_device_request(const struct scmi_device_id *id_table)
+static int scmi_bus_notifier(struct notifier_block *nb,
+ unsigned long action, void *data)
{
- int ret = 0;
- unsigned int id = 0;
- struct list_head *head, *phead = NULL;
- struct scmi_requested_dev *rdev;
- struct scmi_info *info;
-
- pr_debug("Requesting SCMI device (%s) for protocol %x\n",
- id_table->name, id_table->protocol_id);
-
- /*
- * Search for the matching protocol rdev list and then search
- * of any existent equally named device...fails if any duplicate found.
- */
- mutex_lock(&scmi_requested_devices_mtx);
- idr_for_each_entry(&scmi_requested_devices, head, id) {
- if (!phead) {
- /* A list found registered in the IDR is never empty */
- rdev = list_first_entry(head, struct scmi_requested_dev,
- node);
- if (rdev->id_table->protocol_id ==
- id_table->protocol_id)
- phead = head;
- }
- list_for_each_entry(rdev, head, node) {
- if (!strcmp(rdev->id_table->name, id_table->name)) {
- pr_err("Ignoring duplicate request [%d] %s\n",
- rdev->id_table->protocol_id,
- rdev->id_table->name);
- ret = -EINVAL;
- goto out;
- }
- }
- }
+ struct scmi_info *info = bus_nb_to_scmi_info(nb);
+ struct scmi_device *sdev = to_scmi_dev(data);
+ const char *status;
- /*
- * No duplicate found for requested id_table, so let's create a new
- * requested device entry for this new valid request.
- */
- rdev = kzalloc(sizeof(*rdev), GFP_KERNEL);
- if (!rdev) {
- ret = -ENOMEM;
- goto out;
- }
- rdev->id_table = id_table;
-
- /*
- * Append the new requested device table descriptor to the head of the
- * related protocol list, eventually creating such head if not already
- * there.
- */
- if (!phead) {
- phead = kzalloc(sizeof(*phead), GFP_KERNEL);
- if (!phead) {
- kfree(rdev);
- ret = -ENOMEM;
- goto out;
- }
- INIT_LIST_HEAD(phead);
-
- ret = idr_alloc(&scmi_requested_devices, (void *)phead,
- id_table->protocol_id,
- id_table->protocol_id + 1, GFP_KERNEL);
- if (ret != id_table->protocol_id) {
- pr_err("Failed to save SCMI device - ret:%d\n", ret);
- kfree(rdev);
- kfree(phead);
- ret = -EINVAL;
- goto out;
- }
- ret = 0;
- }
- list_add(&rdev->node, phead);
+ /* Skip transport devices and devices of different SCMI instances */
+ if (!strncmp(sdev->name, "__scmi_transport_device", 23) ||
+ sdev->dev.parent != info->dev)
+ return NOTIFY_DONE;
- /*
- * Now effectively create and initialize the requested device for every
- * already initialized SCMI instance which has registered the requested
- * protocol as a valid active one: i.e. defined in DT and supported by
- * current platform FW.
- */
- mutex_lock(&scmi_list_mutex);
- list_for_each_entry(info, &scmi_list, node) {
- struct device_node *child;
-
- child = idr_find(&info->active_protocols,
- id_table->protocol_id);
- if (child) {
- struct scmi_device *sdev;
-
- sdev = scmi_get_protocol_device(child, info,
- id_table->protocol_id,
- id_table->name);
- if (sdev) {
- /* Set handle if not already set: device existed */
- if (!sdev->handle)
- sdev->handle =
- scmi_handle_get_from_info_unlocked(info);
- /* Relink consumer and suppliers */
- if (sdev->handle)
- scmi_device_link_add(&sdev->dev,
- sdev->handle->dev);
- }
- } else {
- dev_err(info->dev,
- "Failed. SCMI protocol %d not active.\n",
- id_table->protocol_id);
- }
+ switch (action) {
+ case BUS_NOTIFY_BIND_DRIVER:
+ /* setup handle now as the transport is ready */
+ scmi_set_handle(sdev);
+ status = "about to be BOUND.";
+ break;
+ case BUS_NOTIFY_DRIVER_NOT_BOUND:
+ scmi_clear_handle(sdev);
+ status = "NOT BOUND.";
+ break;
+ case BUS_NOTIFY_UNBOUND_DRIVER:
+ scmi_clear_handle(sdev);
+ status = "UNBOUND.";
+ break;
+ default:
+ return NOTIFY_DONE;
}
- mutex_unlock(&scmi_list_mutex);
-out:
- mutex_unlock(&scmi_requested_devices_mtx);
+ dev_dbg(info->dev, "Device %s (%s) is now %s\n", dev_name(&sdev->dev),
+ sdev->name, status);
- return ret;
+ return NOTIFY_OK;
}
-/**
- * scmi_protocol_device_unrequest - Helper to unrequest a device
- *
- * @id_table: A protocol/name pair descriptor for the device to be unrequested.
- *
- * An helper to let an SCMI driver release its request about devices; note that
- * devices are created and initialized once the first SCMI driver request them
- * but they destroyed only on SCMI core unloading/unbinding.
- *
- * The current SCMI transport layer uses such devices as internal references and
- * as such they could be shared as same transport between multiple drivers so
- * that cannot be safely destroyed till the whole SCMI stack is removed.
- * (unless adding further burden of refcounting.)
- */
-void scmi_protocol_device_unrequest(const struct scmi_device_id *id_table)
+static int scmi_device_request_notifier(struct notifier_block *nb,
+ unsigned long action, void *data)
{
- struct list_head *phead;
+ struct device_node *np;
+ struct scmi_device_id *id_table = data;
+ struct scmi_info *info = req_nb_to_scmi_info(nb);
- pr_debug("Unrequesting SCMI device (%s) for protocol %x\n",
- id_table->name, id_table->protocol_id);
+ rcu_read_lock();
+ np = idr_find(&info->active_protocols, id_table->protocol_id);
+ rcu_read_unlock();
+ if (!np)
+ return NOTIFY_DONE;
- mutex_lock(&scmi_requested_devices_mtx);
- phead = idr_find(&scmi_requested_devices, id_table->protocol_id);
- if (phead) {
- struct scmi_requested_dev *victim, *tmp;
+ dev_dbg(info->dev, "%sRequested device (%s) for protocol 0x%x\n",
+ action == SCMI_BUS_NOTIFY_DEVICE_REQUEST ? "" : "UN-",
+ id_table->name, id_table->protocol_id);
- list_for_each_entry_safe(victim, tmp, phead, node) {
- if (!strcmp(victim->id_table->name, id_table->name)) {
- list_del(&victim->node);
- kfree(victim);
- break;
- }
- }
-
- if (list_empty(phead)) {
- idr_remove(&scmi_requested_devices,
- id_table->protocol_id);
- kfree(phead);
- }
+ switch (action) {
+ case SCMI_BUS_NOTIFY_DEVICE_REQUEST:
+ scmi_create_protocol_devices(np, info, id_table->protocol_id,
+ id_table->name);
+ break;
+ case SCMI_BUS_NOTIFY_DEVICE_UNREQUEST:
+ scmi_destroy_protocol_devices(info, id_table->protocol_id,
+ id_table->name);
+ break;
+ default:
+ return NOTIFY_DONE;
}
- mutex_unlock(&scmi_requested_devices_mtx);
-}
-
-static int scmi_cleanup_txrx_channels(struct scmi_info *info)
-{
- int ret;
- struct idr *idr = &info->tx_idr;
- ret = idr_for_each(idr, info->desc->ops->chan_free, idr);
- idr_destroy(&info->tx_idr);
-
- idr = &info->rx_idr;
- ret = idr_for_each(idr, info->desc->ops->chan_free, idr);
- idr_destroy(&info->rx_idr);
-
- return ret;
+ return NOTIFY_OK;
}
static int scmi_probe(struct platform_device *pdev)
@@ -2423,12 +2381,19 @@ static int scmi_probe(struct platform_device *pdev)
if (!info)
return -ENOMEM;
+ info->id = ida_alloc_min(&scmi_id, 0, GFP_KERNEL);
+ if (info->id < 0)
+ return info->id;
+
info->dev = dev;
info->desc = desc;
+ info->bus_nb.notifier_call = scmi_bus_notifier;
+ info->dev_req_nb.notifier_call = scmi_device_request_notifier;
INIT_LIST_HEAD(&info->node);
idr_init(&info->protocols);
mutex_init(&info->protocols_mtx);
idr_init(&info->active_protocols);
+ mutex_init(&info->devreq_mtx);
platform_set_drvdata(pdev, info);
idr_init(&info->tx_idr);
@@ -2452,17 +2417,27 @@ static int scmi_probe(struct platform_device *pdev)
if (desc->ops->link_supplier) {
ret = desc->ops->link_supplier(dev);
if (ret)
- return ret;
+ goto clear_ida;
}
- ret = scmi_txrx_setup(info, dev, SCMI_PROTOCOL_BASE);
+ /* Setup all channels described in the DT at first */
+ ret = scmi_channels_setup(info);
if (ret)
- return ret;
+ goto clear_ida;
- ret = scmi_xfer_info_init(info);
+ ret = bus_register_notifier(&scmi_bus_type, &info->bus_nb);
if (ret)
goto clear_txrx_setup;
+ ret = blocking_notifier_chain_register(&scmi_requested_devices_nh,
+ &info->dev_req_nb);
+ if (ret)
+ goto clear_bus_notifier;
+
+ ret = scmi_xfer_info_init(info);
+ if (ret)
+ goto clear_dev_req_notifier;
+
if (scmi_notification_init(handle))
dev_err(dev, "SCMI Notifications NOT available.\n");
@@ -2491,8 +2466,10 @@ static int scmi_probe(struct platform_device *pdev)
if (of_property_read_u32(child, "reg", &prot_id))
continue;
- if (!FIELD_FIT(MSG_PROTOCOL_ID_MASK, prot_id))
+ if (!FIELD_FIT(MSG_PROTOCOL_ID_MASK, prot_id)) {
dev_err(dev, "Out of range protocol %d\n", prot_id);
+ continue;
+ }
if (!scmi_is_protocol_implemented(handle, prot_id)) {
dev_err(dev, "SCMI protocol %d not implemented\n",
@@ -2513,26 +2490,28 @@ static int scmi_probe(struct platform_device *pdev)
}
of_node_get(child);
- scmi_create_protocol_devices(child, info, prot_id);
+ scmi_create_protocol_devices(child, info, prot_id, NULL);
}
return 0;
notification_exit:
scmi_notification_exit(&info->handle);
+clear_dev_req_notifier:
+ blocking_notifier_chain_unregister(&scmi_requested_devices_nh,
+ &info->dev_req_nb);
+clear_bus_notifier:
+ bus_unregister_notifier(&scmi_bus_type, &info->bus_nb);
clear_txrx_setup:
scmi_cleanup_txrx_channels(info);
+clear_ida:
+ ida_free(&scmi_id, info->id);
return ret;
}
-void scmi_free_channel(struct scmi_chan_info *cinfo, struct idr *idr, int id)
-{
- idr_remove(idr, id);
-}
-
static int scmi_remove(struct platform_device *pdev)
{
- int ret, id;
+ int id;
struct scmi_info *info = platform_get_drvdata(pdev);
struct device_node *child;
@@ -2543,6 +2522,9 @@ static int scmi_remove(struct platform_device *pdev)
list_del(&info->node);
mutex_unlock(&scmi_list_mutex);
+ blocking_notifier_chain_unregister(&scmi_requested_devices_nh,
+ &info->dev_req_nb);
+
scmi_notification_exit(&info->handle);
mutex_lock(&info->protocols_mtx);
@@ -2553,10 +2535,12 @@ static int scmi_remove(struct platform_device *pdev)
of_node_put(child);
idr_destroy(&info->active_protocols);
+ bus_unregister_notifier(&scmi_bus_type, &info->bus_nb);
+
/* Safe to free channels since no more users */
- ret = scmi_cleanup_txrx_channels(info);
- if (ret)
- dev_warn(&pdev->dev, "Failed to cleanup SCMI channels.\n");
+ scmi_cleanup_txrx_channels(info);
+
+ ida_free(&scmi_id, info->id);
return 0;
}
diff --git a/drivers/firmware/arm_scmi/mailbox.c b/drivers/firmware/arm_scmi/mailbox.c
index 81c902672a30..41edb639cf92 100644
--- a/drivers/firmware/arm_scmi/mailbox.c
+++ b/drivers/firmware/arm_scmi/mailbox.c
@@ -60,9 +60,9 @@ static void rx_callback(struct mbox_client *cl, void *m)
scmi_rx_callback(smbox->cinfo, shmem_read_header(smbox->shmem), NULL);
}
-static bool mailbox_chan_available(struct device *dev, int idx)
+static bool mailbox_chan_available(struct device_node *of_node, int idx)
{
- return !of_parse_phandle_with_args(dev->of_node, "mboxes",
+ return !of_parse_phandle_with_args(of_node, "mboxes",
"#mbox-cells", idx, NULL);
}
@@ -146,19 +146,27 @@ static int mailbox_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
cl->tx_block = false;
cl->knows_txdone = tx;
+ cinfo->transport_info = smbox;
+ smbox->cinfo = cinfo;
+
smbox->chan = mbox_request_channel(cl, tx ? 0 : 1);
if (IS_ERR(smbox->chan)) {
ret = PTR_ERR(smbox->chan);
+ smbox->chan = NULL;
if (ret != -EPROBE_DEFER)
dev_err(cdev, "failed to request SCMI %s mailbox\n",
tx ? "Tx" : "Rx");
- return ret;
+ goto err_clear_cinfo;
}
- cinfo->transport_info = smbox;
- smbox->cinfo = cinfo;
-
return 0;
+
+err_clear_cinfo:
+ cinfo->transport_info = NULL;
+ smbox->cinfo = NULL;
+ devm_iounmap(dev, smbox->shmem);
+ devm_kfree(dev, smbox);
+ return ret;
}
static int mailbox_chan_free(int id, void *p, void *data)
@@ -173,8 +181,6 @@ static int mailbox_chan_free(int id, void *p, void *data)
smbox->cinfo = NULL;
}
- scmi_free_channel(cinfo, data, id);
-
return 0;
}
diff --git a/drivers/firmware/arm_scmi/optee.c b/drivers/firmware/arm_scmi/optee.c
index f5f6ec83d3e1..dd3ddda136e0 100644
--- a/drivers/firmware/arm_scmi/optee.c
+++ b/drivers/firmware/arm_scmi/optee.c
@@ -328,11 +328,11 @@ static int scmi_optee_link_supplier(struct device *dev)
return 0;
}
-static bool scmi_optee_chan_available(struct device *dev, int idx)
+static bool scmi_optee_chan_available(struct device_node *of_node, int idx)
{
u32 channel_id;
- return !of_property_read_u32_index(dev->of_node, "linaro,optee-channel-id",
+ return !of_property_read_u32_index(of_node, "linaro,optee-channel-id",
idx, &channel_id);
}
@@ -488,8 +488,6 @@ static int scmi_optee_chan_free(int id, void *p, void *data)
cinfo->transport_info = NULL;
channel->cinfo = NULL;
- scmi_free_channel(cinfo, data, id);
-
return 0;
}
diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c
index 42ea308a2c1d..d2abe71b9883 100644
--- a/drivers/firmware/arm_scmi/smc.c
+++ b/drivers/firmware/arm_scmi/smc.c
@@ -54,9 +54,9 @@ static irqreturn_t smc_msg_done_isr(int irq, void *data)
return IRQ_HANDLED;
}
-static bool smc_chan_available(struct device *dev, int idx)
+static bool smc_chan_available(struct device_node *of_node, int idx)
{
- struct device_node *np = of_parse_phandle(dev->of_node, "shmem", 0);
+ struct device_node *np = of_parse_phandle(of_node, "shmem", 0);
if (!np)
return false;
@@ -141,6 +141,11 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
if (ret < 0)
return ret;
+ scmi_info->func_id = func_id;
+ scmi_info->cinfo = cinfo;
+ smc_channel_lock_init(scmi_info);
+ cinfo->transport_info = scmi_info;
+
/*
* If there is an interrupt named "a2p", then the service and
* completion of a message is signaled by an interrupt rather than by
@@ -152,17 +157,14 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev,
IRQF_NO_SUSPEND, dev_name(dev), scmi_info);
if (ret) {
dev_err(dev, "failed to setup SCMI smc irq\n");
+ cinfo->transport_info = NULL;
+ scmi_info->cinfo = NULL;
return ret;
}
} else {
cinfo->no_completion_irq = true;
}
- scmi_info->func_id = func_id;
- scmi_info->cinfo = cinfo;
- smc_channel_lock_init(scmi_info);
- cinfo->transport_info = scmi_info;
-
return 0;
}
@@ -185,8 +187,6 @@ static int smc_chan_free(int id, void *p, void *data)
cinfo->transport_info = NULL;
scmi_info->cinfo = NULL;
- scmi_free_channel(cinfo, data, id);
-
return 0;
}
diff --git a/drivers/firmware/arm_scmi/virtio.c b/drivers/firmware/arm_scmi/virtio.c
index 1db975c08896..d68c01cb7aa0 100644
--- a/drivers/firmware/arm_scmi/virtio.c
+++ b/drivers/firmware/arm_scmi/virtio.c
@@ -385,7 +385,7 @@ static int virtio_link_supplier(struct device *dev)
return 0;
}
-static bool virtio_chan_available(struct device *dev, int idx)
+static bool virtio_chan_available(struct device_node *of_node, int idx)
{
struct scmi_vio_channel *channels, *vioch = NULL;
@@ -489,8 +489,6 @@ static int virtio_chan_free(int id, void *p, void *data)
virtio_break_device(vioch->vqueue->vdev);
scmi_vio_channel_cleanup_sync(vioch);
- scmi_free_channel(cinfo, data, id);
-
return 0;
}
diff --git a/drivers/firmware/efi/mokvar-table.c b/drivers/firmware/efi/mokvar-table.c
index 37e1fe7a2f2b..bd84a22805b5 100644
--- a/drivers/firmware/efi/mokvar-table.c
+++ b/drivers/firmware/efi/mokvar-table.c
@@ -103,9 +103,7 @@ void __init efi_mokvar_table_init(void)
void *va = NULL;
unsigned long cur_offset = 0;
unsigned long offset_limit;
- unsigned long map_size = 0;
unsigned long map_size_needed = 0;
- unsigned long size;
struct efi_mokvar_table_entry *mokvar_entry;
int err;
@@ -134,48 +132,34 @@ void __init efi_mokvar_table_init(void)
*/
err = -EINVAL;
while (cur_offset + sizeof(*mokvar_entry) <= offset_limit) {
- mokvar_entry = va + cur_offset;
- map_size_needed = cur_offset + sizeof(*mokvar_entry);
- if (map_size_needed > map_size) {
- if (va)
- early_memunmap(va, map_size);
- /*
- * Map a little more than the fixed size entry
- * header, anticipating some data. It's safe to
- * do so as long as we stay within current memory
- * descriptor.
- */
- map_size = min(map_size_needed + 2*EFI_PAGE_SIZE,
- offset_limit);
- va = early_memremap(efi.mokvar_table, map_size);
- if (!va) {
- pr_err("Failed to map EFI MOKvar config table pa=0x%lx, size=%lu.\n",
- efi.mokvar_table, map_size);
- return;
- }
- mokvar_entry = va + cur_offset;
+ if (va)
+ early_memunmap(va, sizeof(*mokvar_entry));
+ va = early_memremap(efi.mokvar_table + cur_offset, sizeof(*mokvar_entry));
+ if (!va) {
+ pr_err("Failed to map EFI MOKvar config table pa=0x%lx, size=%zu.\n",
+ efi.mokvar_table + cur_offset, sizeof(*mokvar_entry));
+ return;
}
+ mokvar_entry = va;
/* Check for last sentinel entry */
if (mokvar_entry->name[0] == '\0') {
if (mokvar_entry->data_size != 0)
break;
err = 0;
+ map_size_needed = cur_offset + sizeof(*mokvar_entry);
break;
}
- /* Sanity check that the name is null terminated */
- size = strnlen(mokvar_entry->name,
- sizeof(mokvar_entry->name));
- if (size >= sizeof(mokvar_entry->name))
- break;
+ /* Enforce that the name is NUL terminated */
+ mokvar_entry->name[sizeof(mokvar_entry->name) - 1] = '\0';
/* Advance to the next entry */
- cur_offset = map_size_needed + mokvar_entry->data_size;
+ cur_offset += sizeof(*mokvar_entry) + mokvar_entry->data_size;
}
if (va)
- early_memunmap(va, map_size);
+ early_memunmap(va, sizeof(*mokvar_entry));
if (err) {
pr_err("EFI MOKvar config table is not valid\n");
return;
diff --git a/drivers/gpio/gpio-exar.c b/drivers/gpio/gpio-exar.c
index e641bc96c4e7..0096f8a9377b 100644
--- a/drivers/gpio/gpio-exar.c
+++ b/drivers/gpio/gpio-exar.c
@@ -196,6 +196,7 @@ static int gpio_exar_probe(struct platform_device *pdev)
index = ida_alloc(&ida_index, GFP_KERNEL);
if (index < 0)
return index;
+ exar_gpio->index = index;
ret = devm_add_action_or_reset(dev, exar_devm_ida_free, exar_gpio);
if (ret)
@@ -211,7 +212,6 @@ static int gpio_exar_probe(struct platform_device *pdev)
exar_gpio->gpio_chip.set = exar_set_value;
exar_gpio->gpio_chip.base = -1;
exar_gpio->gpio_chip.ngpio = ngpios;
- exar_gpio->index = index;
exar_gpio->first_pin = first_pin;
ret = devm_gpiochip_add_data(dev, &exar_gpio->gpio_chip, exar_gpio);
diff --git a/drivers/gpio/gpio-mvebu.c b/drivers/gpio/gpio-mvebu.c
index 3ead4c95fa3d..f58f835d0acb 100644
--- a/drivers/gpio/gpio-mvebu.c
+++ b/drivers/gpio/gpio-mvebu.c
@@ -1002,7 +1002,7 @@ static int mvebu_gpio_suspend(struct platform_device *pdev, pm_message_t state)
BUG();
}
- if (IS_ENABLED(CONFIG_PWM))
+ if (IS_REACHABLE(CONFIG_PWM) && mvchip->mvpwm)
mvebu_pwm_suspend(mvchip);
return 0;
@@ -1054,7 +1054,7 @@ static int mvebu_gpio_resume(struct platform_device *pdev)
BUG();
}
- if (IS_ENABLED(CONFIG_PWM))
+ if (IS_REACHABLE(CONFIG_PWM) && mvchip->mvpwm)
mvebu_pwm_resume(mvchip);
return 0;
@@ -1238,7 +1238,7 @@ static int mvebu_gpio_probe(struct platform_device *pdev)
"failed to register gpiochip\n");
/* Some MVEBU SoCs have simple PWM support for GPIO lines */
- if (IS_ENABLED(CONFIG_PWM)) {
+ if (IS_REACHABLE(CONFIG_PWM)) {
err = mvebu_pwm_probe(pdev, mvchip, id);
if (err)
return err;
diff --git a/drivers/gpio/gpio-mxc.c b/drivers/gpio/gpio-mxc.c
index d456077c74f2..97186bce32bb 100644
--- a/drivers/gpio/gpio-mxc.c
+++ b/drivers/gpio/gpio-mxc.c
@@ -7,6 +7,8 @@
// Authors: Daniel Mack, Juergen Beisert.
// Copyright (C) 2004-2010 Freescale Semiconductor, Inc. All Rights Reserved.
+#include <linux/bitops.h>
+#include <linux/cleanup.h>
#include <linux/clk.h>
#include <linux/err.h>
#include <linux/init.h>
@@ -25,6 +27,12 @@
#include <linux/of_device.h>
#include <linux/bug.h>
+#define IMX_SCU_WAKEUP_OFF 0
+#define IMX_SCU_WAKEUP_LOW_LVL 4
+#define IMX_SCU_WAKEUP_FALL_EDGE 5
+#define IMX_SCU_WAKEUP_RISE_EDGE 6
+#define IMX_SCU_WAKEUP_HIGH_LVL 7
+
/* device type dependent stuff */
struct mxc_gpio_hwdata {
unsigned dr_reg;
@@ -59,9 +67,12 @@ struct mxc_gpio_port {
struct irq_domain *domain;
struct gpio_chip gc;
struct device *dev;
- u32 both_edges;
+ unsigned long both_edges;
struct mxc_gpio_reg_saved gpio_saved_reg;
bool power_off;
+ bool is_pad_wakeup;
+ unsigned long wakeup_pads;
+ u32 pad_type[32];
const struct mxc_gpio_hwdata *hwdata;
};
@@ -131,6 +142,9 @@ static const struct of_device_id mxc_gpio_dt_ids[] = {
{ .compatible = "fsl,imx31-gpio", .data = &imx31_gpio_hwdata },
{ .compatible = "fsl,imx35-gpio", .data = &imx35_gpio_hwdata },
{ .compatible = "fsl,imx7d-gpio", .data = &imx35_gpio_hwdata },
+ { .compatible = "fsl,imx8dxl-gpio", .data = &imx35_gpio_hwdata },
+ { .compatible = "fsl,imx8qm-gpio", .data = &imx35_gpio_hwdata },
+ { .compatible = "fsl,imx8qxp-gpio", .data = &imx35_gpio_hwdata },
{ /* sentinel */ }
};
MODULE_DEVICE_TABLE(of, mxc_gpio_dt_ids);
@@ -148,13 +162,12 @@ static int gpio_set_irq_type(struct irq_data *d, u32 type)
{
struct irq_chip_generic *gc = irq_data_get_irq_chip_data(d);
struct mxc_gpio_port *port = gc->private;
- unsigned long flags;
u32 bit, val;
u32 gpio_idx = d->hwirq;
int edge;
void __iomem *reg = port->base;
- port->both_edges &= ~(1 << gpio_idx);
+ clear_bit(gpio_idx, &port->both_edges);
switch (type) {
case IRQ_TYPE_EDGE_RISING:
edge = GPIO_INT_RISE_EDGE;
@@ -174,7 +187,7 @@ static int gpio_set_irq_type(struct irq_data *d, u32 type)
edge = GPIO_INT_HIGH_LEV;
pr_debug("mxc: set GPIO %d to high trigger\n", gpio_idx);
}
- port->both_edges |= 1 << gpio_idx;
+ set_bit(gpio_idx, &port->both_edges);
}
break;
case IRQ_TYPE_LEVEL_LOW:
@@ -187,40 +200,38 @@ static int gpio_set_irq_type(struct irq_data *d, u32 type)
return -EINVAL;
}
- raw_spin_lock_irqsave(&port->gc.bgpio_lock, flags);
+ scoped_guard(raw_spinlock_irqsave, &port->gc.bgpio_lock) {
+ if (GPIO_EDGE_SEL >= 0) {
+ val = readl(port->base + GPIO_EDGE_SEL);
+ if (edge == GPIO_INT_BOTH_EDGES)
+ writel(val | (1 << gpio_idx),
+ port->base + GPIO_EDGE_SEL);
+ else
+ writel(val & ~(1 << gpio_idx),
+ port->base + GPIO_EDGE_SEL);
+ }
- if (GPIO_EDGE_SEL >= 0) {
- val = readl(port->base + GPIO_EDGE_SEL);
- if (edge == GPIO_INT_BOTH_EDGES)
- writel(val | (1 << gpio_idx),
- port->base + GPIO_EDGE_SEL);
- else
- writel(val & ~(1 << gpio_idx),
- port->base + GPIO_EDGE_SEL);
- }
+ if (edge != GPIO_INT_BOTH_EDGES) {
+ reg += GPIO_ICR1 + ((gpio_idx & 0x10) >> 2); /* lower or upper register */
+ bit = gpio_idx & 0xf;
+ val = readl(reg) & ~(0x3 << (bit << 1));
+ writel(val | (edge << (bit << 1)), reg);
+ }
- if (edge != GPIO_INT_BOTH_EDGES) {
- reg += GPIO_ICR1 + ((gpio_idx & 0x10) >> 2); /* lower or upper register */
- bit = gpio_idx & 0xf;
- val = readl(reg) & ~(0x3 << (bit << 1));
- writel(val | (edge << (bit << 1)), reg);
+ writel(1 << gpio_idx, port->base + GPIO_ISR);
+ port->pad_type[gpio_idx] = type;
}
- writel(1 << gpio_idx, port->base + GPIO_ISR);
-
- raw_spin_unlock_irqrestore(&port->gc.bgpio_lock, flags);
-
return port->gc.direction_input(&port->gc, gpio_idx);
}
static void mxc_flip_edge(struct mxc_gpio_port *port, u32 gpio)
{
void __iomem *reg = port->base;
- unsigned long flags;
u32 bit, val;
int edge;
- raw_spin_lock_irqsave(&port->gc.bgpio_lock, flags);
+ guard(raw_spinlock_irqsave)(&port->gc.bgpio_lock);
reg += GPIO_ICR1 + ((gpio & 0x10) >> 2); /* lower or upper register */
bit = gpio & 0xf;
@@ -236,12 +247,9 @@ static void mxc_flip_edge(struct mxc_gpio_port *port, u32 gpio)
} else {
pr_err("mxc: invalid configuration for GPIO %d: %x\n",
gpio, edge);
- goto unlock;
+ return;
}
writel(val | (edge << (bit << 1)), reg);
-
-unlock:
- raw_spin_unlock_irqrestore(&port->gc.bgpio_lock, flags);
}
/* handle 32 interrupts in one status register */
@@ -250,7 +258,7 @@ static void mxc_gpio_irq_handler(struct mxc_gpio_port *port, u32 irq_stat)
while (irq_stat != 0) {
int irqoffset = fls(irq_stat) - 1;
- if (port->both_edges & (1 << irqoffset))
+ if (test_bit(irqoffset, &port->both_edges))
mxc_flip_edge(port, irqoffset);
generic_handle_domain_irq(port->domain, irqoffset);
@@ -266,6 +274,9 @@ static void mx3_gpio_irq_handler(struct irq_desc *desc)
struct mxc_gpio_port *port = irq_desc_get_handler_data(desc);
struct irq_chip *chip = irq_desc_get_chip(desc);
+ if (port->is_pad_wakeup)
+ return;
+
chained_irq_enter(chip, desc);
irq_stat = readl(port->base + GPIO_ISR) & readl(port->base + GPIO_IMR);
@@ -324,8 +335,12 @@ static int gpio_set_wake_irq(struct irq_data *d, u32 enable)
else
ret = disable_irq_wake(port->irq);
}
+ if (ret)
+ return ret;
- return ret;
+ assign_bit(gpio_idx, &port->wakeup_pads, enable);
+
+ return 0;
}
static int mxc_gpio_init_gc(struct mxc_gpio_port *port, int irq_base)
@@ -377,7 +392,6 @@ static int mxc_gpio_probe(struct platform_device *pdev)
return -ENOMEM;
port->dev = &pdev->dev;
-
port->hwdata = device_get_match_data(&pdev->dev);
port->base = devm_platform_ioremap_resource(pdev, 0);
@@ -509,6 +523,86 @@ static void mxc_gpio_restore_regs(struct mxc_gpio_port *port)
writel(port->gpio_saved_reg.dr, port->base + GPIO_DR);
}
+static bool mxc_gpio_generic_config(struct mxc_gpio_port *port,
+ unsigned int offset, unsigned long conf)
+{
+ struct device_node *np = port->dev->of_node;
+
+ if (of_device_is_compatible(np, "fsl,imx8dxl-gpio") ||
+ of_device_is_compatible(np, "fsl,imx8qxp-gpio") ||
+ of_device_is_compatible(np, "fsl,imx8qm-gpio"))
+ return (gpiochip_generic_config(&port->gc, offset, conf) == 0);
+
+ return false;
+}
+
+static bool mxc_gpio_set_pad_wakeup(struct mxc_gpio_port *port, bool enable)
+{
+ unsigned long config;
+ bool ret = false;
+ int i, type;
+ bool is_imx8qm = of_device_is_compatible(port->dev->of_node, "fsl,imx8qm-gpio");
+
+ static const u32 pad_type_map[] = {
+ IMX_SCU_WAKEUP_OFF, /* 0 */
+ IMX_SCU_WAKEUP_RISE_EDGE, /* IRQ_TYPE_EDGE_RISING */
+ IMX_SCU_WAKEUP_FALL_EDGE, /* IRQ_TYPE_EDGE_FALLING */
+ IMX_SCU_WAKEUP_RISE_EDGE, /* IRQ_TYPE_EDGE_BOTH */
+ IMX_SCU_WAKEUP_HIGH_LVL, /* IRQ_TYPE_LEVEL_HIGH */
+ IMX_SCU_WAKEUP_OFF, /* 5 */
+ IMX_SCU_WAKEUP_OFF, /* 6 */
+ IMX_SCU_WAKEUP_OFF, /* 7 */
+ IMX_SCU_WAKEUP_LOW_LVL, /* IRQ_TYPE_LEVEL_LOW */
+ };
+
+ for (i = 0; i < 32; i++) {
+ if ((port->wakeup_pads & (1 << i))) {
+ type = port->pad_type[i];
+ if (enable)
+ config = pad_type_map[type];
+ else
+ config = IMX_SCU_WAKEUP_OFF;
+
+ if (is_imx8qm && config == IMX_SCU_WAKEUP_FALL_EDGE) {
+ dev_warn_once(port->dev,
+ "No falling-edge support for wakeup on i.MX8QM\n");
+ config = IMX_SCU_WAKEUP_OFF;
+ }
+
+ ret |= mxc_gpio_generic_config(port, i, config);
+ }
+ }
+
+ return ret;
+}
+
+static int __maybe_unused mxc_gpio_noirq_suspend(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct mxc_gpio_port *port = platform_get_drvdata(pdev);
+
+ if (port->wakeup_pads > 0)
+ port->is_pad_wakeup = mxc_gpio_set_pad_wakeup(port, true);
+
+ return 0;
+}
+
+static int __maybe_unused mxc_gpio_noirq_resume(struct device *dev)
+{
+ struct platform_device *pdev = to_platform_device(dev);
+ struct mxc_gpio_port *port = platform_get_drvdata(pdev);
+
+ if (port->wakeup_pads > 0)
+ mxc_gpio_set_pad_wakeup(port, false);
+ port->is_pad_wakeup = false;
+
+ return 0;
+}
+
+static const struct dev_pm_ops mxc_gpio_dev_pm_ops = {
+ SET_NOIRQ_SYSTEM_SLEEP_PM_OPS(mxc_gpio_noirq_suspend, mxc_gpio_noirq_resume)
+};
+
static int mxc_gpio_syscore_suspend(void)
{
struct mxc_gpio_port *port;
@@ -548,6 +642,7 @@ static struct platform_driver mxc_gpio_driver = {
.name = "gpio-mxc",
.of_match_table = mxc_gpio_dt_ids,
.suppress_bind_attrs = true,
+ .pm = &mxc_gpio_dev_pm_ops,
},
.probe = mxc_gpio_probe,
};
diff --git a/drivers/gpio/gpio-pca953x.c b/drivers/gpio/gpio-pca953x.c
index 7b683553adf5..2af6027a260f 100644
--- a/drivers/gpio/gpio-pca953x.c
+++ b/drivers/gpio/gpio-pca953x.c
@@ -519,15 +519,23 @@ static int pca953x_read_regs(struct pca953x_chip *chip, int reg, unsigned long *
return 0;
}
-static int pca953x_gpio_direction_input(struct gpio_chip *gc, unsigned off)
+static int pca953x_gpio_direction_input_unlocked(struct gpio_chip *gc,
+ unsigned int off)
{
struct pca953x_chip *chip = gpiochip_get_data(gc);
u8 dirreg = chip->recalc_addr(chip, chip->regs->direction, off);
u8 bit = BIT(off % BANK_SZ);
+ return regmap_write_bits(chip->regmap, dirreg, bit, bit);
+}
+
+static int pca953x_gpio_direction_input(struct gpio_chip *gc, unsigned int off)
+{
+ struct pca953x_chip *chip = gpiochip_get_data(gc);
+
guard(mutex)(&chip->i2c_lock);
- return regmap_write_bits(chip->regmap, dirreg, bit, bit);
+ return pca953x_gpio_direction_input_unlocked(gc, off);
}
static int pca953x_gpio_direction_output(struct gpio_chip *gc,
@@ -756,9 +764,9 @@ static void pca953x_irq_bus_sync_unlock(struct irq_data *d)
DECLARE_BITMAP(reg_direction, MAX_LINE);
int level;
- if (chip->driver_data & PCA_PCAL) {
- guard(mutex)(&chip->i2c_lock);
+ guard(mutex)(&chip->i2c_lock);
+ if (chip->driver_data & PCA_PCAL) {
/* Enable latch on interrupt-enabled inputs */
pca953x_write_regs(chip, PCAL953X_IN_LATCH, chip->irq_mask);
@@ -777,7 +785,7 @@ static void pca953x_irq_bus_sync_unlock(struct irq_data *d)
/* Look for any newly setup interrupt */
for_each_set_bit(level, irq_mask, gc->ngpio)
- pca953x_gpio_direction_input(&chip->gpio_chip, level);
+ pca953x_gpio_direction_input_unlocked(&chip->gpio_chip, level);
mutex_unlock(&chip->irq_lock);
}
diff --git a/drivers/gpio/gpio-xilinx.c b/drivers/gpio/gpio-xilinx.c
index 8f094c898002..9a4396662b95 100644
--- a/drivers/gpio/gpio-xilinx.c
+++ b/drivers/gpio/gpio-xilinx.c
@@ -292,14 +292,7 @@ static void xgpio_save_regs(struct xgpio_instance *chip)
static int xgpio_request(struct gpio_chip *chip, unsigned int offset)
{
- int ret;
-
- ret = pm_runtime_get_sync(chip->parent);
- /*
- * If the device is already active pm_runtime_get() will return 1 on
- * success, but gpio_request still needs to return 0.
- */
- return ret < 0 ? ret : 0;
+ return pm_runtime_resume_and_get(chip->parent);
}
static void xgpio_free(struct gpio_chip *chip, unsigned int offset)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
index c4287e09658f..c8e619ae33f4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h
@@ -384,6 +384,7 @@ bool amdgpu_get_bios(struct amdgpu_device *adev);
bool amdgpu_read_bios(struct amdgpu_device *adev);
bool amdgpu_soc15_read_bios_from_rom(struct amdgpu_device *adev,
u8 *bios, u32 length_bytes);
+void amdgpu_bios_release(struct amdgpu_device *adev);
/*
* Clocks
*/
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c
index 31c3aaeb3bb4..4453d32323e6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c
@@ -324,7 +324,7 @@ static int acp_hw_init(void *handle)
r = device_for_each_child(adev->acp.parent, &adev->acp.acp_genpd->gpd,
acp_genpd_add_device);
if (r)
- goto failure;
+ goto failure_remove_mfd;
break;
}
default:
@@ -446,7 +446,7 @@ static int acp_hw_init(void *handle)
r = device_for_each_child(adev->acp.parent, &adev->acp.acp_genpd->gpd,
acp_genpd_add_device);
if (r)
- goto failure;
+ goto failure_remove_mfd;
}
/* Assert Soft reset of ACP */
@@ -464,7 +464,7 @@ static int acp_hw_init(void *handle)
if (--count == 0) {
dev_err(&adev->pdev->dev, "Failed to reset ACP\n");
r = -ETIMEDOUT;
- goto failure;
+ goto failure_remove_mfd;
}
udelay(100);
}
@@ -482,7 +482,7 @@ static int acp_hw_init(void *handle)
if (--count == 0) {
dev_err(&adev->pdev->dev, "Failed to reset ACP\n");
r = -ETIMEDOUT;
- goto failure;
+ goto failure_remove_mfd;
}
udelay(100);
}
@@ -492,6 +492,11 @@ static int acp_hw_init(void *handle)
cgs_write_register(adev->acp.cgs_device, mmACP_SOFT_RESET, val);
return 0;
+failure_remove_mfd:
+ device_for_each_child(adev->acp.parent, NULL,
+ acp_genpd_remove_device);
+ mfd_remove_devices(adev->acp.parent);
+
failure:
kfree(i2s_pdata);
kfree(adev->acp.acp_res);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
index d58c8e452ca3..355173a6ac3d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
@@ -495,7 +495,7 @@ void amdgpu_amdkfd_get_cu_info(struct amdgpu_device *adev, struct kfd_cu_info *c
int amdgpu_amdkfd_get_dmabuf_info(struct amdgpu_device *adev, int dma_buf_fd,
struct amdgpu_device **dmabuf_adev,
- uint64_t *bo_size, void *metadata_buffer,
+ uint64_t *bo_size, void **metadata_buffer,
size_t buffer_size, uint32_t *metadata_size,
uint32_t *flags)
{
@@ -530,9 +530,28 @@ int amdgpu_amdkfd_get_dmabuf_info(struct amdgpu_device *adev, int dma_buf_fd,
*dmabuf_adev = adev;
if (bo_size)
*bo_size = amdgpu_bo_size(bo);
- if (metadata_buffer)
- r = amdgpu_bo_get_metadata(bo, metadata_buffer, buffer_size,
- metadata_size, &metadata_flags);
+ if (metadata_buffer) {
+ /* first get metadata_size by buffer = NULL */
+ r = amdgpu_bo_get_metadata(bo, NULL, 0,
+ metadata_size, NULL);
+ if (r)
+ goto out_put;
+
+ /* user buf_size is bigger than bo metadata_size
+ * allocate a buf at kernel space and copy */
+ if (*metadata_size <= buffer_size) {
+ *metadata_buffer = kzalloc(*metadata_size, GFP_KERNEL);
+
+ if (!*metadata_buffer) {
+ r = -ENOMEM;
+ goto out_put;
+ }
+
+ r = amdgpu_bo_get_metadata(bo, *metadata_buffer, *metadata_size,
+ NULL, &metadata_flags);
+ } else
+ r = -EINVAL;
+ }
if (flags) {
*flags = (bo->preferred_domains & AMDGPU_GEM_DOMAIN_VRAM) ?
KFD_IOC_ALLOC_MEM_FLAGS_VRAM
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
index c7672a1d1560..86a3315098a2 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
@@ -227,7 +227,7 @@ void amdgpu_amdkfd_get_cu_info(struct amdgpu_device *adev,
struct kfd_cu_info *cu_info);
int amdgpu_amdkfd_get_dmabuf_info(struct amdgpu_device *adev, int dma_buf_fd,
struct amdgpu_device **dmabuf_adev,
- uint64_t *bo_size, void *metadata_buffer,
+ uint64_t *bo_size, void **metadata_buffer,
size_t buffer_size, uint32_t *metadata_size,
uint32_t *flags);
uint8_t amdgpu_amdkfd_get_xgmi_hops_count(struct amdgpu_device *dst,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
index aaeb915a776d..eabba7037b8a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
@@ -1665,7 +1665,8 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu(
*/
if (flags & KFD_IOC_ALLOC_MEM_FLAGS_VRAM) {
domain = alloc_domain = AMDGPU_GEM_DOMAIN_VRAM;
- alloc_flags = AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE;
+ alloc_flags = AMDGPU_GEM_CREATE_VRAM_WIPE_ON_RELEASE |
+ AMDGPU_GEM_CREATE_VRAM_CLEARED;
alloc_flags |= (flags & KFD_IOC_ALLOC_MEM_FLAGS_PUBLIC) ?
AMDGPU_GEM_CREATE_CPU_ACCESS_REQUIRED : 0;
} else if (flags & KFD_IOC_ALLOC_MEM_FLAGS_GTT) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c
index 816014ea5381..679473a48d99 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c
@@ -43,6 +43,21 @@ static void amdgpu_atombios_lookup_i2c_gpio_quirks(struct amdgpu_device *adev,
}
+/* VBIOS-reported table size is unchecked against the image; cap iterations and
+ * adev->i2c_bus[] indexing to AMDGPU_MAX_I2C_BUS.
+ */
+static int amdgpu_atombios_gpio_i2c_num_entries(uint16_t size)
+{
+ u32 bytes;
+
+ if (size < sizeof(ATOM_COMMON_TABLE_HEADER))
+ return 0;
+
+ bytes = size - sizeof(ATOM_COMMON_TABLE_HEADER);
+ return (int)min_t(u32, bytes / sizeof(ATOM_GPIO_I2C_ASSIGMENT),
+ AMDGPU_MAX_I2C_BUS);
+}
+
static struct amdgpu_i2c_bus_rec amdgpu_atombios_get_bus_rec_for_i2c_gpio(ATOM_GPIO_I2C_ASSIGMENT *gpio)
{
struct amdgpu_i2c_bus_rec i2c;
@@ -103,8 +118,7 @@ struct amdgpu_i2c_bus_rec amdgpu_atombios_lookup_i2c_gpio(struct amdgpu_device *
if (amdgpu_atom_parse_data_header(ctx, index, &size, NULL, NULL, &data_offset)) {
i2c_info = (struct _ATOM_GPIO_I2C_INFO *)(ctx->bios + data_offset);
- num_indices = (size - sizeof(ATOM_COMMON_TABLE_HEADER)) /
- sizeof(ATOM_GPIO_I2C_ASSIGMENT);
+ num_indices = amdgpu_atombios_gpio_i2c_num_entries(size);
gpio = &i2c_info->asGPIO_Info[0];
for (i = 0; i < num_indices; i++) {
@@ -137,8 +151,7 @@ void amdgpu_atombios_i2c_init(struct amdgpu_device *adev)
if (amdgpu_atom_parse_data_header(ctx, index, &size, NULL, NULL, &data_offset)) {
i2c_info = (struct _ATOM_GPIO_I2C_INFO *)(ctx->bios + data_offset);
- num_indices = (size - sizeof(ATOM_COMMON_TABLE_HEADER)) /
- sizeof(ATOM_GPIO_I2C_ASSIGMENT);
+ num_indices = amdgpu_atombios_gpio_i2c_num_entries(size);
gpio = &i2c_info->asGPIO_Info[0];
for (i = 0; i < num_indices; i++) {
@@ -1793,6 +1806,15 @@ const struct attribute_group amdgpu_vbios_version_attr_group = {
.attrs = amdgpu_vbios_version_attrs
};
+int amdgpu_atombios_sysfs_init(struct amdgpu_device *adev)
+{
+ if (adev->mode_info.atom_context)
+ return devm_device_add_group(adev->dev,
+ &amdgpu_vbios_version_attr_group);
+
+ return 0;
+}
+
/**
* amdgpu_atombios_fini - free the driver info and callbacks for atombios
*
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.h
index 4153d520e2a3..b639a80ee3fc 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.h
@@ -217,5 +217,6 @@ int amdgpu_atombios_get_data_table(struct amdgpu_device *adev,
void amdgpu_atombios_fini(struct amdgpu_device *adev);
int amdgpu_atombios_init(struct amdgpu_device *adev);
+int amdgpu_atombios_sysfs_init(struct amdgpu_device *adev);
#endif
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
index d57fa0aee358..1b783bb58ea0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c
@@ -47,41 +47,50 @@
/* Check if current bios is an ATOM BIOS.
* Return true if it is ATOM BIOS. Otherwise, return false.
*/
-static bool check_atom_bios(uint8_t *bios, size_t size)
+static bool check_atom_bios(struct amdgpu_device *adev, size_t size)
{
uint16_t tmp, bios_header_start;
+ uint8_t *bios = adev->bios;
if (!bios || size < 0x49) {
- DRM_INFO("vbios mem is null or mem size is wrong\n");
+ dev_dbg(adev->dev, "VBIOS mem is null or mem size is wrong\n");
return false;
}
if (!AMD_IS_VALID_VBIOS(bios)) {
- DRM_INFO("BIOS signature incorrect %x %x\n", bios[0], bios[1]);
+ dev_dbg(adev->dev, "VBIOS signature incorrect %x %x\n", bios[0],
+ bios[1]);
return false;
}
bios_header_start = bios[0x48] | (bios[0x49] << 8);
if (!bios_header_start) {
- DRM_INFO("Can't locate bios header\n");
+ dev_dbg(adev->dev, "Can't locate VBIOS header\n");
return false;
}
tmp = bios_header_start + 4;
if (size < tmp) {
- DRM_INFO("BIOS header is broken\n");
+ dev_dbg(adev->dev, "VBIOS header is broken\n");
return false;
}
if (!memcmp(bios + tmp, "ATOM", 4) ||
!memcmp(bios + tmp, "MOTA", 4)) {
- DRM_DEBUG("ATOMBIOS detected\n");
+ dev_dbg(adev->dev, "ATOMBIOS detected\n");
return true;
}
return false;
}
+void amdgpu_bios_release(struct amdgpu_device *adev)
+{
+ kfree(adev->bios);
+ adev->bios = NULL;
+ adev->bios_size = 0;
+}
+
/* If you boot an IGP board with a discrete card as the primary,
* the IGP rom is not accessible via the rom bar as the IGP rom is
* part of the system bios. On boot, the system bios puts a
@@ -105,9 +114,8 @@ static bool igp_read_bios_from_vram(struct amdgpu_device *adev)
adev->bios = NULL;
vram_base = pci_resource_start(adev->pdev, 0);
bios = ioremap_wc(vram_base, size);
- if (!bios) {
+ if (!bios)
return false;
- }
adev->bios = kmalloc(size, GFP_KERNEL);
if (!adev->bios) {
@@ -118,8 +126,8 @@ static bool igp_read_bios_from_vram(struct amdgpu_device *adev)
memcpy_fromio(adev->bios, bios, size);
iounmap(bios);
- if (!check_atom_bios(adev->bios, size)) {
- kfree(adev->bios);
+ if (!check_atom_bios(adev, size)) {
+ amdgpu_bios_release(adev);
return false;
}
@@ -134,9 +142,8 @@ bool amdgpu_read_bios(struct amdgpu_device *adev)
adev->bios = NULL;
/* XXX: some cards may return 0 for rom size? ddx has a workaround */
bios = pci_map_rom(adev->pdev, &size);
- if (!bios) {
+ if (!bios)
return false;
- }
adev->bios = kzalloc(size, GFP_KERNEL);
if (adev->bios == NULL) {
@@ -147,8 +154,8 @@ bool amdgpu_read_bios(struct amdgpu_device *adev)
memcpy_fromio(adev->bios, bios, size);
pci_unmap_rom(adev->pdev, bios);
- if (!check_atom_bios(adev->bios, size)) {
- kfree(adev->bios);
+ if (!check_atom_bios(adev, size)) {
+ amdgpu_bios_release(adev);
return false;
}
@@ -169,9 +176,9 @@ static bool amdgpu_read_bios_from_rom(struct amdgpu_device *adev)
header[AMD_VBIOS_SIGNATURE_END] = 0;
if ((!AMD_IS_VALID_VBIOS(header)) ||
- 0 != memcmp((char *)&header[AMD_VBIOS_SIGNATURE_OFFSET],
- AMD_VBIOS_SIGNATURE,
- strlen(AMD_VBIOS_SIGNATURE)))
+ memcmp((char *)&header[AMD_VBIOS_SIGNATURE_OFFSET],
+ AMD_VBIOS_SIGNATURE,
+ strlen(AMD_VBIOS_SIGNATURE)) != 0)
return false;
/* valid vbios, go on */
@@ -187,8 +194,8 @@ static bool amdgpu_read_bios_from_rom(struct amdgpu_device *adev)
/* read complete BIOS */
amdgpu_asic_read_bios_from_rom(adev, adev->bios, len);
- if (!check_atom_bios(adev->bios, len)) {
- kfree(adev->bios);
+ if (!check_atom_bios(adev, len)) {
+ amdgpu_bios_release(adev);
return false;
}
@@ -217,14 +224,15 @@ static bool amdgpu_read_platform_bios(struct amdgpu_device *adev)
memcpy_fromio(adev->bios, bios, romlen);
iounmap(bios);
- if (!check_atom_bios(adev->bios, romlen))
+ if (!check_atom_bios(adev, romlen))
goto free_bios;
adev->bios_size = romlen;
return true;
free_bios:
- kfree(adev->bios);
+ amdgpu_bios_release(adev);
+
return false;
}
@@ -265,7 +273,7 @@ static int amdgpu_atrm_call(acpi_handle atrm_handle, uint8_t *bios,
status = acpi_evaluate_object(atrm_handle, NULL, &atrm_arg, &buffer);
if (ACPI_FAILURE(status)) {
- printk("failed to evaluate ATRM got %s\n", acpi_format_exception(status));
+ DRM_ERROR("failed to evaluate ATRM got %s\n", acpi_format_exception(status));
return -ENODEV;
}
@@ -339,8 +347,8 @@ static bool amdgpu_atrm_get_bios(struct amdgpu_device *adev)
break;
}
- if (!check_atom_bios(adev->bios, size)) {
- kfree(adev->bios);
+ if (!check_atom_bios(adev, size)) {
+ amdgpu_bios_release(adev);
return false;
}
adev->bios_size = size;
@@ -407,14 +415,15 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
struct acpi_table_header *hdr;
acpi_size tbl_size;
UEFI_ACPI_VFCT *vfct;
- unsigned offset;
+ unsigned int offset;
+ bool r = false;
if (!ACPI_SUCCESS(acpi_get_table("VFCT", 1, &hdr)))
return false;
tbl_size = hdr->length;
if (tbl_size < sizeof(UEFI_ACPI_VFCT)) {
dev_info(adev->dev, "ACPI VFCT table present but broken (too short #1),skipping\n");
- return false;
+ goto out;
}
vfct = (UEFI_ACPI_VFCT *)hdr;
@@ -427,13 +436,13 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
offset += sizeof(VFCT_IMAGE_HEADER);
if (offset > tbl_size) {
dev_info(adev->dev, "ACPI VFCT image header truncated,skipping\n");
- return false;
+ goto out;
}
offset += vhdr->ImageLength;
if (offset > tbl_size) {
dev_info(adev->dev, "ACPI VFCT image truncated,skipping\n");
- return false;
+ goto out;
}
if (vhdr->ImageLength &&
@@ -442,17 +451,21 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
vhdr->ImageLength,
GFP_KERNEL);
- if (!check_atom_bios(adev->bios, vhdr->ImageLength)) {
- kfree(adev->bios);
- return false;
+ if (!check_atom_bios(adev, vhdr->ImageLength)) {
+ amdgpu_bios_release(adev);
+ goto out;
}
adev->bios_size = vhdr->ImageLength;
- return true;
+ r = true;
+ goto out;
}
}
dev_info(adev->dev, "ACPI VFCT table present but broken (too short #2),skipping\n");
- return false;
+
+out:
+ acpi_put_table(hdr);
+ return r;
}
#else
static inline bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
index e9d51002001a..a9e046576f7e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c
@@ -39,6 +39,25 @@
#include "amdgpu_gem.h"
#include "amdgpu_ras.h"
+/*
+ * Maximum IB length (dwords) for rings whose emit_ib packet format
+ * documents a 20-bit size field.
+ */
+#define AMDGPU_GFX_SDMA_IB_PACKET_SIZE_MAX_DW 0xFFFFF
+#define AMDGPU_MM_IB_PACKET_SIZE_MAX_DW 0x7FFFF0
+
+static u32 amdgpu_cs_ib_packet_size_max_dw(enum amdgpu_ring_type type)
+{
+ switch (type) {
+ case AMDGPU_RING_TYPE_GFX:
+ case AMDGPU_RING_TYPE_COMPUTE:
+ case AMDGPU_RING_TYPE_SDMA:
+ return AMDGPU_GFX_SDMA_IB_PACKET_SIZE_MAX_DW;
+ default:
+ return AMDGPU_MM_IB_PACKET_SIZE_MAX_DW;
+ }
+}
+
static int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p,
struct amdgpu_device *adev,
struct drm_file *filp,
@@ -343,7 +362,6 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p,
job = p->jobs[r];
ring = amdgpu_job_ring(job);
- ib = &job->ibs[job->num_ibs++];
/* MM engine doesn't support user fences */
if (p->uf_entry.tv.bo && ring->funcs->no_user_fence)
@@ -362,6 +380,12 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p,
return -EINVAL;
}
+ if (chunk_ib->ib_bytes / 4 >
+ amdgpu_cs_ib_packet_size_max_dw(ring->funcs->type))
+ return -EINVAL;
+
+ ib = &job->ibs[job->num_ibs++];
+
if (chunk_ib->flags & AMDGPU_IB_FLAG_PREAMBLE)
job->preamble_status |= AMDGPU_PREAMBLE_IB_PRESENT;
@@ -384,7 +408,7 @@ static int amdgpu_cs_p2_dependencies(struct amdgpu_cs_parser *p,
{
struct drm_amdgpu_cs_chunk_dep *deps = chunk->kdata;
struct amdgpu_fpriv *fpriv = p->filp->driver_priv;
- unsigned num_deps;
+ unsigned int num_deps;
int i, r;
num_deps = chunk->length_dw * 4 /
@@ -456,7 +480,7 @@ static int amdgpu_cs_p2_syncobj_in(struct amdgpu_cs_parser *p,
struct amdgpu_cs_chunk *chunk)
{
struct drm_amdgpu_cs_chunk_sem *deps = chunk->kdata;
- unsigned num_deps;
+ unsigned int num_deps;
int i, r;
num_deps = chunk->length_dw * 4 /
@@ -474,7 +498,7 @@ static int amdgpu_cs_p2_syncobj_timeline_wait(struct amdgpu_cs_parser *p,
struct amdgpu_cs_chunk *chunk)
{
struct drm_amdgpu_cs_chunk_syncobj *syncobj_deps = chunk->kdata;
- unsigned num_deps;
+ unsigned int num_deps;
int i, r;
num_deps = chunk->length_dw * 4 /
@@ -494,7 +518,7 @@ static int amdgpu_cs_p2_syncobj_out(struct amdgpu_cs_parser *p,
struct amdgpu_cs_chunk *chunk)
{
struct drm_amdgpu_cs_chunk_sem *deps = chunk->kdata;
- unsigned num_deps;
+ unsigned int num_deps;
int i;
num_deps = chunk->length_dw * 4 /
@@ -528,7 +552,7 @@ static int amdgpu_cs_p2_syncobj_timeline_signal(struct amdgpu_cs_parser *p,
struct amdgpu_cs_chunk *chunk)
{
struct drm_amdgpu_cs_chunk_syncobj *syncobj_deps = chunk->kdata;
- unsigned num_deps;
+ unsigned int num_deps;
int i;
num_deps = chunk->length_dw * 4 /
@@ -721,6 +745,7 @@ static void amdgpu_cs_get_threshold_for_moves(struct amdgpu_device *adev,
if (used_vis_vram < total_vis_vram) {
u64 free_vis_vram = total_vis_vram - used_vis_vram;
+
adev->mm_stats.accum_us_vis = min(adev->mm_stats.accum_us_vis +
increment_us, us_upper_bound);
@@ -1043,9 +1068,8 @@ static int amdgpu_cs_patch_ibs(struct amdgpu_cs_parser *p,
/* the IB should be reserved at this point */
r = amdgpu_bo_kmap(aobj, (void **)&kptr);
- if (r) {
+ if (r)
return r;
- }
kptr += va_start - (m->start * AMDGPU_GPU_PAGE_SIZE);
@@ -1318,7 +1342,7 @@ static void amdgpu_cs_parser_fini(struct amdgpu_cs_parser *parser)
{
struct amdgpu_device *adev = parser->adev;
struct amdgpu_bo_list_entry *e;
- unsigned i;
+ unsigned int i;
for (i = 0; i < parser->num_post_deps; i++) {
drm_syncobj_put(parser->post_deps[i].syncobj);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
index 3898b67c35bc..9663da2e3ca9 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c
@@ -304,7 +304,7 @@ static int amdgpu_ctx_get_stable_pstate(struct amdgpu_ctx *ctx,
static int amdgpu_ctx_init(struct amdgpu_ctx_mgr *mgr, int32_t priority,
struct drm_file *filp, struct amdgpu_ctx *ctx)
{
- u32 current_stable_pstate;
+ struct amdgpu_fpriv *fpriv = filp->driver_priv;
int r;
r = amdgpu_ctx_priority_permit(filp, priority);
@@ -322,36 +322,22 @@ static int amdgpu_ctx_init(struct amdgpu_ctx_mgr *mgr, int32_t priority,
ctx->vram_lost_counter = atomic_read(&mgr->adev->vram_lost_counter);
ctx->init_priority = priority;
ctx->override_priority = AMDGPU_CTX_PRIORITY_UNSET;
+ ctx->stable_pstate = AMDGPU_CTX_STABLE_PSTATE_NONE;
- r = amdgpu_ctx_get_stable_pstate(ctx, ¤t_stable_pstate);
- if (r)
- return r;
-
- if (mgr->adev->pm.stable_pstate_ctx)
- ctx->stable_pstate = mgr->adev->pm.stable_pstate_ctx->stable_pstate;
- else
- ctx->stable_pstate = current_stable_pstate;
-
+ ctx->ctx_mgr = &(fpriv->ctx_mgr);
return 0;
}
-static int amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx,
- u32 stable_pstate)
+static int __amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx,
+ u32 stable_pstate)
{
struct amdgpu_device *adev = ctx->mgr->adev;
enum amd_dpm_forced_level level;
+ struct amdgpu_ctx *current_ctx;
u32 current_stable_pstate;
- int r;
+ int r = 0;
- mutex_lock(&adev->pm.stable_pstate_ctx_lock);
- if (adev->pm.stable_pstate_ctx && adev->pm.stable_pstate_ctx != ctx) {
- r = -EBUSY;
- goto done;
- }
-
- r = amdgpu_ctx_get_stable_pstate(ctx, ¤t_stable_pstate);
- if (r || (stable_pstate == current_stable_pstate))
- goto done;
+ lockdep_assert_held(&adev->pm.stable_pstate_ctx_lock);
switch (stable_pstate) {
case AMDGPU_CTX_STABLE_PSTATE_NONE:
@@ -370,17 +356,41 @@ static int amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx,
level = AMD_DPM_FORCED_LEVEL_PROFILE_PEAK;
break;
default:
- r = -EINVAL;
- goto done;
+ return -EINVAL;
}
+ current_ctx = adev->pm.stable_pstate_ctx;
+ if (current_ctx && current_ctx != ctx)
+ return -EBUSY;
+
+ r = amdgpu_ctx_get_stable_pstate(ctx, ¤t_stable_pstate);
+ if (r || current_stable_pstate == stable_pstate)
+ return r;
+
r = amdgpu_dpm_force_performance_level(adev, level);
+ if (r)
+ return r;
- if (level == AMD_DPM_FORCED_LEVEL_AUTO)
- adev->pm.stable_pstate_ctx = NULL;
- else
+ if (!current_ctx) {
adev->pm.stable_pstate_ctx = ctx;
-done:
+ /*
+ * Serialized by context taking ownership for the first time
+ * while holding adev->pm.stable_pstate_ctx_lock).
+ */
+ WRITE_ONCE(ctx->stable_pstate, current_stable_pstate);
+ }
+
+ return 0;
+}
+
+static int amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx,
+ u32 stable_pstate)
+{
+ struct amdgpu_device *adev = ctx->mgr->adev;
+ int r;
+
+ mutex_lock(&adev->pm.stable_pstate_ctx_lock);
+ r = __amdgpu_ctx_set_stable_pstate(ctx, stable_pstate);
mutex_unlock(&adev->pm.stable_pstate_ctx_lock);
return r;
@@ -406,7 +416,12 @@ static void amdgpu_ctx_fini(struct kref *ref)
}
if (drm_dev_enter(adev_to_drm(adev), &idx)) {
- amdgpu_ctx_set_stable_pstate(ctx, ctx->stable_pstate);
+ mutex_lock(&adev->pm.stable_pstate_ctx_lock);
+ if (adev->pm.stable_pstate_ctx == ctx) {
+ __amdgpu_ctx_set_stable_pstate(ctx, ctx->stable_pstate);
+ adev->pm.stable_pstate_ctx = NULL;
+ }
+ mutex_unlock(&adev->pm.stable_pstate_ctx_lock);
drm_dev_exit(idx);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h
index 0fa0e56daf67..f1e27b6e16f4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.h
@@ -57,6 +57,7 @@ struct amdgpu_ctx {
unsigned long ras_counter_ce;
unsigned long ras_counter_ue;
uint32_t stable_pstate;
+ struct amdgpu_ctx_mgr *ctx_mgr;
};
struct amdgpu_ctx_mgr {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
index 2300752e2fa8..952a3fb18c2d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c
@@ -114,6 +114,20 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f,
*pos &= (1UL << 22) - 1;
+ if (size > PAGE_SIZE)
+ return -EINVAL;
+
+ /*
+ * Access the user buffer only while none of the locks below are held:
+ * a fault takes mmap_lock, and grbm_idx_mutex and srbm_mutex nest
+ * inside it.
+ */
+ u32 *data __free(kfree) = kmalloc(size, GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+ if (!read && copy_from_user(data, buf, size))
+ return -EFAULT;
+
r = pm_runtime_get_sync(adev_to_drm(adev)->dev);
if (r < 0) {
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
@@ -146,28 +160,16 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f,
mutex_lock(&adev->pm.mutex);
while (size) {
- uint32_t value;
-
- if (read) {
- value = RREG32(*pos >> 2);
- r = put_user(value, (uint32_t *)buf);
- } else {
- r = get_user(value, (uint32_t *)buf);
- if (!r)
- amdgpu_mm_wreg_mmio_rlc(adev, *pos >> 2, value);
- }
- if (r) {
- result = r;
- goto end;
- }
+ if (read)
+ data[result >> 2] = RREG32(*pos >> 2);
+ else
+ amdgpu_mm_wreg_mmio_rlc(adev, *pos >> 2, data[result >> 2]);
result += 4;
- buf += 4;
*pos += 4;
size -= 4;
}
-end:
if (use_bank) {
amdgpu_gfx_select_se_sh(adev, 0xffffffff, 0xffffffff, 0xffffffff);
mutex_unlock(&adev->grbm_idx_mutex);
@@ -183,6 +185,10 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f,
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
amdgpu_virt_disable_access_debugfs(adev);
+
+ if (read && copy_to_user(buf, data, result))
+ return -EFAULT;
+
return result;
}
@@ -232,11 +238,24 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off
struct amdgpu_device *adev = rd->adev;
ssize_t result = 0;
int r;
- uint32_t value;
if (size & 0x3 || offset & 0x3)
return -EINVAL;
+ if (size > PAGE_SIZE)
+ return -EINVAL;
+
+ /*
+ * Access the user buffer only while none of the locks below are held:
+ * a fault takes mmap_lock, and grbm_idx_mutex and srbm_mutex nest
+ * inside it.
+ */
+ u32 *data __free(kfree) = kmalloc(size, GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+ if (write_en && copy_from_user(data, buf, size))
+ return -EFAULT;
+
r = pm_runtime_get_sync(adev_to_drm(adev)->dev);
if (r < 0) {
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
@@ -276,24 +295,15 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off
mutex_lock(&adev->pm.mutex);
while (size) {
- if (!write_en) {
- value = RREG32(offset >> 2);
- r = put_user(value, (uint32_t *)buf);
- } else {
- r = get_user(value, (uint32_t *)buf);
- if (!r)
- amdgpu_mm_wreg_mmio_rlc(adev, offset >> 2, value);
- }
- if (r) {
- result = r;
- goto end;
- }
+ if (!write_en)
+ data[result >> 2] = RREG32(offset >> 2);
+ else
+ amdgpu_mm_wreg_mmio_rlc(adev, offset >> 2, data[result >> 2]);
offset += 4;
size -= 4;
result += 4;
- buf += 4;
}
-end:
+
if (rd->id.use_grbm) {
amdgpu_gfx_select_se_sh(adev, 0xffffffff, 0xffffffff, 0xffffffff);
mutex_unlock(&adev->grbm_idx_mutex);
@@ -313,6 +323,10 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
amdgpu_virt_disable_access_debugfs(adev);
+
+ if (!write_en && copy_to_user(buf, data, result))
+ return -EFAULT;
+
return result;
}
@@ -983,7 +997,6 @@ static ssize_t amdgpu_debugfs_gpr_read(struct file *f, char __user *buf,
{
struct amdgpu_device *adev = f->f_inode->i_private;
int r;
- ssize_t result = 0;
uint32_t offset, se, sh, cu, wave, simd, thread, bank, *data;
if (size > 4096 || size & 3 || *pos & 3)
@@ -1028,25 +1041,11 @@ static ssize_t amdgpu_debugfs_gpr_read(struct file *f, char __user *buf,
pm_runtime_mark_last_busy(adev_to_drm(adev)->dev);
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
+ amdgpu_virt_disable_access_debugfs(adev);
- while (size) {
- uint32_t value;
-
- value = data[result >> 2];
- r = put_user(value, (uint32_t *)buf);
- if (r) {
- amdgpu_virt_disable_access_debugfs(adev);
- goto err;
- }
-
- result += 4;
- buf += 4;
- size -= 4;
- }
-
+ r = copy_to_user(buf, data, size) ? -EFAULT : size;
kfree(data);
- amdgpu_virt_disable_access_debugfs(adev);
- return result;
+ return r;
err:
pm_runtime_put_autosuspend(adev_to_drm(adev)->dev);
@@ -1607,9 +1606,14 @@ static int amdgpu_debugfs_vm_info_show(struct seq_file *m, void *unused)
list_for_each_entry(file, &dev->filelist, lhead) {
struct amdgpu_fpriv *fpriv = file->driver_priv;
struct amdgpu_vm *vm = &fpriv->vm;
+ struct amdgpu_task_info *ti;
+
+ ti = amdgpu_vm_get_task_info_vm(vm);
+ if (ti) {
+ seq_printf(m, "pid:%d\tProcess:%s ----------\n", ti->pid, ti->process_name);
+ amdgpu_vm_put_task_info(ti);
+ }
- seq_printf(m, "pid:%d\tProcess:%s ----------\n",
- vm->task_info.pid, vm->task_info.process_name);
r = amdgpu_bo_reserve(vm->root.bo, true);
if (r)
break;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
index f17577ed6252..e34e2cd8ac18 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c
@@ -2403,6 +2403,12 @@ static int amdgpu_device_fw_loading(struct amdgpu_device *adev)
static int amdgpu_device_init_schedulers(struct amdgpu_device *adev)
{
+ struct drm_sched_init_args args = {
+ .ops = &amdgpu_sched_ops,
+ .hang_limit = amdgpu_job_hang_limit,
+ .timeout_wq = adev->reset_domain->wq,
+ .dev = adev->dev,
+ };
long timeout;
int r, i;
@@ -2428,11 +2434,12 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev)
break;
}
- r = drm_sched_init(&ring->sched, &amdgpu_sched_ops,
- ring->num_hw_submission, amdgpu_job_hang_limit,
- timeout, adev->reset_domain->wq,
- ring->sched_score, ring->name,
- adev->dev);
+ args.timeout = timeout;
+ args.credit_limit = ring->num_hw_submission;
+ args.score = ring->sched_score;
+ args.name = ring->name;
+
+ r = drm_sched_init(&ring->sched, &args);
if (r) {
DRM_ERROR("Failed to create scheduler on ring %s.\n",
ring->name);
@@ -3977,6 +3984,11 @@ int amdgpu_device_init(struct amdgpu_device *adev,
/* Get a log2 for easy divisions. */
adev->mm_stats.log2_max_MBps = ilog2(max(1u, max_MBps));
+ r = amdgpu_atombios_sysfs_init(adev);
+ if (r)
+ drm_err(&adev->ddev,
+ "registering atombios sysfs failed (%d).\n", r);
+
r = amdgpu_pm_sysfs_init(adev);
if (r) {
adev->pm_sysfs_en = false;
@@ -4102,8 +4114,6 @@ static void amdgpu_device_unmap_mmio(struct amdgpu_device *adev)
iounmap(adev->rmmio);
adev->rmmio = NULL;
- if (adev->mman.aper_base_kaddr)
- iounmap(adev->mman.aper_base_kaddr);
adev->mman.aper_base_kaddr = NULL;
/* Memory manager related */
@@ -4198,8 +4208,7 @@ void amdgpu_device_fini_sw(struct amdgpu_device *adev)
if (amdgpu_emu_mode != 1)
amdgpu_atombios_fini(adev);
- kfree(adev->bios);
- adev->bios = NULL;
+ amdgpu_bios_release(adev);
px = amdgpu_device_supports_px(adev_to_drm(adev));
@@ -4907,6 +4916,13 @@ int amdgpu_device_mode1_reset(struct amdgpu_device *adev)
if (ret)
dev_err(adev->dev, "GPU mode1 reset failed\n");
+ /* enable mmio access after mode 1 reset completed */
+ if (!ret)
+ adev->no_hw_access = false;
+
+ /* ensure no_hw_access is updated before we access hw */
+ smp_mb();
+
amdgpu_device_load_pci_state(adev->pdev);
/* wait for asic to come out of reset */
@@ -4955,6 +4971,14 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev,
amdgpu_fence_driver_force_completion(ring);
}
+ /*
+ * KIQ rings are polling-fence/no_scheduler like MES, so realign their
+ * fence too (one ring per XCC), otherwise the first post-reset KIQ
+ * submission polls forever on a stale seq.
+ */
+ if (adev->gfx.kiq.ring.fence_drv.initialized && adev->gfx.kiq.ring.sched.ready)
+ amdgpu_fence_driver_force_completion(&adev->gfx.kiq.ring);
+
amdgpu_fence_driver_isr_toggle(adev, false);
if (job && job->vm)
@@ -5170,9 +5194,16 @@ int amdgpu_do_asic_reset(struct list_head *device_list_handle,
tmp_adev->reset_vram_lost = vram_lost;
memset(&tmp_adev->reset_task_info, 0,
sizeof(tmp_adev->reset_task_info));
- if (reset_context->job && reset_context->job->vm)
- tmp_adev->reset_task_info =
- reset_context->job->vm->task_info;
+ if (reset_context->job && reset_context->job->vm) {
+ struct amdgpu_task_info *ti;
+ struct amdgpu_vm *vm = reset_context->job->vm;
+
+ ti = amdgpu_vm_get_task_info_vm(vm);
+ if (ti) {
+ tmp_adev->reset_task_info = *ti;
+ amdgpu_vm_put_task_info(ti);
+ }
+ }
amdgpu_reset_capture_coredumpm(tmp_adev);
#endif
if (vram_lost) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c
index 9b1c4d5be61f..4005f0505df0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c
@@ -903,6 +903,8 @@ static int amdgpu_discovery_sysfs_ips(struct amdgpu_device *adev,
ip_hw_instance->kobj.kset = &ip_hw_id->hw_id_kset;
res = kobject_add(&ip_hw_instance->kobj, NULL,
"%d", ip_hw_instance->num_instance);
+ if (res)
+ kobject_put(&ip_hw_instance->kobj);
next_ip:
ip_offset += struct_size(ip, base_address, ip->num_base_address);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
index 4dcc7de961d0..533a283c5e58 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_dma_buf.c
@@ -173,7 +173,7 @@ static struct sg_table *amdgpu_dma_buf_map(struct dma_buf_attachment *attach,
if (!bo->tbo.pin_count) {
/* move buffer into GTT or VRAM */
struct ttm_operation_ctx ctx = { false, false };
- unsigned domains = AMDGPU_GEM_DOMAIN_GTT;
+ unsigned int domains = AMDGPU_GEM_DOMAIN_GTT;
if (bo->preferred_domains & AMDGPU_GEM_DOMAIN_VRAM &&
attach->peer2peer) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
index 87871f5f199e..7a19c3e73209 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
@@ -2856,12 +2856,10 @@ static struct pci_error_handlers amdgpu_pci_err_handler = {
extern const struct attribute_group amdgpu_vram_mgr_attr_group;
extern const struct attribute_group amdgpu_gtt_mgr_attr_group;
-extern const struct attribute_group amdgpu_vbios_version_attr_group;
static const struct attribute_group *amdgpu_sysfs_groups[] = {
&amdgpu_vram_mgr_attr_group,
&amdgpu_gtt_mgr_attr_group,
- &amdgpu_vbios_version_attr_group,
NULL,
};
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
index 418e4c77ceb8..c16c7f6517e6 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fence.c
@@ -42,7 +42,6 @@
#include "amdgpu_reset.h"
/*
- * Fences
* Fences mark an event in the GPUs pipeline and are used
* for GPU/CPU synchronization. When the fence is written,
* it is expected that all buffers associated with that fence
@@ -139,7 +138,7 @@ static u32 amdgpu_fence_read(struct amdgpu_ring *ring)
* Returns 0 on success, -ENOMEM on failure.
*/
int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f, struct amdgpu_job *job,
- unsigned flags)
+ unsigned int flags)
{
struct amdgpu_device *adev = ring->adev;
struct dma_fence *fence;
@@ -173,11 +172,11 @@ int amdgpu_fence_emit(struct amdgpu_ring *ring, struct dma_fence **f, struct amd
adev->fence_context + ring->idx, seq);
/* Against remove in amdgpu_job_{free, free_cb} */
dma_fence_get(fence);
- }
- else
+ } else {
dma_fence_init(fence, &amdgpu_fence_ops,
&ring->fence_drv.lock,
adev->fence_context + ring->idx, seq);
+ }
}
amdgpu_ring_emit_fence(ring, ring->fence_drv.gpu_addr,
@@ -393,7 +392,7 @@ signed long amdgpu_fence_wait_polling(struct amdgpu_ring *ring,
* Returns the number of emitted fences on the ring. Used by the
* dynpm code to ring track activity.
*/
-unsigned amdgpu_fence_count_emitted(struct amdgpu_ring *ring)
+unsigned int amdgpu_fence_count_emitted(struct amdgpu_ring *ring)
{
uint64_t emitted;
@@ -421,7 +420,7 @@ unsigned amdgpu_fence_count_emitted(struct amdgpu_ring *ring)
*/
int amdgpu_fence_driver_start_ring(struct amdgpu_ring *ring,
struct amdgpu_irq_src *irq_src,
- unsigned irq_type)
+ unsigned int irq_type)
{
struct amdgpu_device *adev = ring->adev;
uint64_t index;
@@ -636,6 +635,7 @@ void amdgpu_fence_driver_hw_init(struct amdgpu_device *adev)
for (i = 0; i < AMDGPU_MAX_RINGS; i++) {
struct amdgpu_ring *ring = adev->rings[i];
+
if (!ring || !ring->fence_drv.initialized)
continue;
@@ -824,6 +824,7 @@ static int amdgpu_debugfs_fence_info_show(struct seq_file *m, void *unused)
for (i = 0; i < AMDGPU_MAX_RINGS; ++i) {
struct amdgpu_ring *ring = adev->rings[i];
+
if (!ring || !ring->fence_drv.initialized)
continue;
@@ -897,6 +898,7 @@ static void amdgpu_debugfs_reset_work(struct work_struct *work)
reset_work);
struct amdgpu_reset_context reset_context;
+
memset(&reset_context, 0, sizeof(reset_context));
reset_context.method = AMD_RESET_METHOD_NONE;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
index c76f1e1ee395..6bf31296869a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c
@@ -652,20 +652,17 @@ void amdgpu_gmc_get_vbios_allocations(struct amdgpu_device *adev)
case CHIP_RENOIR:
adev->mman.keep_stolen_vga_memory = true;
break;
- case CHIP_POLARIS10:
- case CHIP_POLARIS11:
- case CHIP_POLARIS12:
- /* MacBookPros with switchable graphics put VRAM at 0 when
- * the iGPU is enabled which results in cursor issues if
- * the cursor ends up at 0. Reserve vram at 0 in that case.
- */
- if (adev->gmc.vram_start == 0)
- adev->mman.keep_stolen_vga_memory = true;
- break;
default:
adev->mman.keep_stolen_vga_memory = false;
break;
}
+ /* MacBookPros with switchable graphics put VRAM at 0 when
+ * the iGPU is enabled which results in cursor issues if
+ * the cursor ends up at 0. Reserve vram at 0 in that case.
+ */
+ if ((adev->pdev->subsystem_vendor == PCI_VENDOR_ID_APPLE) &&
+ (adev->gmc.vram_start == 0))
+ adev->mman.keep_stolen_vga_memory = true;
if (amdgpu_sriov_vf(adev) ||
!amdgpu_device_has_display_hardware(adev)) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
index 03d115d2b5ed..bef38e03d6be 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c
@@ -22,7 +22,7 @@
*/
#include "amdgpu_ids.h"
-#include <linux/idr.h>
+#include <linux/xarray.h>
#include <linux/dma-fence-array.h>
@@ -35,10 +35,13 @@
* PASIDs are global address space identifiers that can be shared
* between the GPU, an IOMMU and the driver. VMs on different devices
* may use the same PASID if they share the same address
- * space. Therefore PASIDs are allocated using a global IDA. VMs are
- * looked up from the PASID per amdgpu_device.
+ * space. Therefore PASIDs are allocated using IDR cyclic allocator
+ * (similar to kernel PID allocation) which naturally delays reuse.
+ * VMs are looked up from the PASID per amdgpu_device.
*/
-static DEFINE_IDA(amdgpu_pasid_ida);
+
+static DEFINE_XARRAY_FLAGS(amdgpu_pasid_xa, XA_FLAGS_LOCK_IRQ | XA_FLAGS_ALLOC1);
+static u32 amdgpu_pasid_xa_next;
/* Helper to free pasid from a fence callback */
struct amdgpu_pasid_cb {
@@ -50,8 +53,8 @@ struct amdgpu_pasid_cb {
* amdgpu_pasid_alloc - Allocate a PASID
* @bits: Maximum width of the PASID in bits, must be at least 1
*
- * Allocates a PASID of the given width while keeping smaller PASIDs
- * available if possible.
+ * Uses kernel's IDR cyclic allocator (same as PID allocation).
+ * Allocates sequentially with automatic wrap-around.
*
* Returns a positive integer on success. Returns %-EINVAL if bits==0.
* Returns %-ENOSPC if no PASID was available. Returns %-ENOMEM on
@@ -59,30 +62,37 @@ struct amdgpu_pasid_cb {
*/
int amdgpu_pasid_alloc(unsigned int bits)
{
- int pasid = -EINVAL;
+ u32 pasid;
+ int r;
- for (bits = min(bits, 31U); bits > 0; bits--) {
- pasid = ida_simple_get(&amdgpu_pasid_ida,
- 1U << (bits - 1), 1U << bits,
- GFP_KERNEL);
- if (pasid != -ENOSPC)
- break;
- }
+ if (bits == 0)
+ return -EINVAL;
- if (pasid >= 0)
- trace_amdgpu_pasid_allocated(pasid);
+ r = xa_alloc_cyclic_irq(&amdgpu_pasid_xa, &pasid, xa_mk_value(0),
+ XA_LIMIT(1, (1U << bits) - 1),
+ &amdgpu_pasid_xa_next, GFP_KERNEL);
+ if (r < 0)
+ return r;
+ trace_amdgpu_pasid_allocated(pasid);
return pasid;
}
/**
* amdgpu_pasid_free - Free a PASID
* @pasid: PASID to free
+ *
+ * Called in IRQ context.
*/
void amdgpu_pasid_free(u32 pasid)
{
+ unsigned long flags;
+
trace_amdgpu_pasid_freed(pasid);
- ida_simple_remove(&amdgpu_pasid_ida, pasid);
+
+ xa_lock_irqsave(&amdgpu_pasid_xa, flags);
+ __xa_erase(&amdgpu_pasid_xa, pasid);
+ xa_unlock_irqrestore(&amdgpu_pasid_xa, flags);
}
static void amdgpu_pasid_free_cb(struct dma_fence *fence,
@@ -582,3 +592,13 @@ void amdgpu_vmid_mgr_fini(struct amdgpu_device *adev)
}
}
}
+
+/**
+ * amdgpu_pasid_mgr_cleanup - cleanup PASID manager
+ *
+ * Cleanup the IDR allocator.
+ */
+void amdgpu_pasid_mgr_cleanup(void)
+{
+ xa_destroy(&amdgpu_pasid_xa);
+}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
index 06c8a0034fa5..235e7561741a 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h
@@ -74,6 +74,7 @@ int amdgpu_pasid_alloc(unsigned int bits);
void amdgpu_pasid_free(u32 pasid);
void amdgpu_pasid_free_delayed(struct dma_resv *resv,
u32 pasid);
+void amdgpu_pasid_mgr_cleanup(void);
bool amdgpu_vmid_had_gpu_reset(struct amdgpu_device *adev,
struct amdgpu_vmid *id);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
index 5a14d51c32d6..d321db549a1e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c
@@ -35,7 +35,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
{
struct amdgpu_ring *ring = to_amdgpu_ring(s_job->sched);
struct amdgpu_job *job = to_amdgpu_job(s_job);
- struct amdgpu_task_info ti;
+ struct amdgpu_task_info *ti;
struct amdgpu_device *adev = ring->adev;
int idx;
int r;
@@ -48,7 +48,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
return DRM_GPU_SCHED_STAT_ENODEV;
}
- memset(&ti, 0, sizeof(struct amdgpu_task_info));
+
adev->job_hang = true;
if (amdgpu_gpu_recovery &&
@@ -58,12 +58,16 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job)
goto exit;
}
- amdgpu_vm_get_task_info(ring->adev, job->pasid, &ti);
DRM_ERROR("ring %s timeout, signaled seq=%u, emitted seq=%u\n",
- job->base.sched->name, atomic_read(&ring->fence_drv.last_seq),
- ring->fence_drv.sync_seq);
- DRM_ERROR("Process information: process %s pid %d thread %s pid %d\n",
- ti.process_name, ti.tgid, ti.task_name, ti.pid);
+ job->base.sched->name, atomic_read(&ring->fence_drv.last_seq),
+ ring->fence_drv.sync_seq);
+
+ ti = amdgpu_vm_get_task_info_pasid(ring->adev, job->pasid);
+ if (ti) {
+ DRM_ERROR("Process information: process %s pid %d thread %s pid %d\n",
+ ti->process_name, ti->tgid, ti->task_name, ti->pid);
+ amdgpu_vm_put_task_info(ti);
+ }
if (amdgpu_device_should_recover_gpu(ring->adev)) {
struct amdgpu_reset_context reset_context;
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c
index c85c5c159554..b489a97ba4ce 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c
@@ -737,7 +737,11 @@ static int psp_load_toc(struct psp_context *psp,
struct psp_gfx_cmd_resp *cmd = acquire_psp_cmd_buf(psp);
/* Copy toc to psp firmware private buffer */
- psp_copy_fw(psp, psp->toc.start_addr, psp->toc.size_bytes);
+ ret = psp_copy_fw(psp, psp->toc.start_addr, psp->toc.size_bytes);
+ if (ret) {
+ release_psp_cmd_buf(psp);
+ return ret;
+ }
psp_prep_load_toc_cmd_buf(cmd, psp->fw_pri_mc_addr, psp->toc.size_bytes);
@@ -958,8 +962,11 @@ static int psp_rl_load(struct amdgpu_device *adev)
cmd = acquire_psp_cmd_buf(psp);
- memset(psp->fw_pri_buf, 0, PSP_1_MEG);
- memcpy(psp->fw_pri_buf, psp->rl.start_addr, psp->rl.size_bytes);
+ ret = psp_copy_fw(psp, psp->rl.start_addr, psp->rl.size_bytes);
+ if (ret) {
+ release_psp_cmd_buf(psp);
+ return ret;
+ }
cmd->cmd_id = GFX_CMD_ID_LOAD_IP_FW;
cmd->cmd.cmd_load_ip_fw.fw_phy_addr_lo = lower_32_bits(psp->fw_pri_mc_addr);
@@ -1166,8 +1173,12 @@ int psp_ta_load(struct psp_context *psp, struct ta_context *context)
cmd = acquire_psp_cmd_buf(psp);
- psp_copy_fw(psp, context->bin_desc.start_addr,
- context->bin_desc.size_bytes);
+ ret = psp_copy_fw(psp, context->bin_desc.start_addr,
+ context->bin_desc.size_bytes);
+ if (ret) {
+ release_psp_cmd_buf(psp);
+ return ret;
+ }
psp_prep_ta_load_cmd_buf(cmd, psp->fw_pri_mc_addr, context);
@@ -3508,17 +3519,24 @@ static ssize_t psp_usbc_pd_fw_sysfs_write(struct device *dev,
return count;
}
-void psp_copy_fw(struct psp_context *psp, uint8_t *start_addr, uint32_t bin_size)
+int psp_copy_fw(struct psp_context *psp, uint8_t *start_addr, uint32_t bin_size)
{
int idx;
if (!drm_dev_enter(adev_to_drm(psp->adev), &idx))
- return;
+ return -ENODEV;
+
+ if (!bin_size || bin_size > PSP_1_MEG) {
+ dev_err(psp->adev->dev, "PSP firmware is invalid\n");
+ drm_dev_exit(idx);
+ return -EINVAL;
+ }
memset(psp->fw_pri_buf, 0, PSP_1_MEG);
memcpy(psp->fw_pri_buf, start_addr, bin_size);
drm_dev_exit(idx);
+ return 0;
}
static DEVICE_ATTR(usbc_pd_fw, 0644,
@@ -3614,6 +3632,13 @@ static ssize_t amdgpu_psp_vbflash_read(struct file *filp, struct kobject *kobj,
return 0;
}
+static const struct bin_attribute psp_vbflash_bin_attr = {
+ .attr = {.name = "psp_vbflash", .mode = 0660},
+ .size = AMD_VBIOS_FILE_MAX_SIZE_B,
+ .write = amdgpu_psp_vbflash_write,
+ .read = amdgpu_psp_vbflash_read,
+};
+
static ssize_t amdgpu_psp_vbflash_status(struct device *dev,
struct device_attribute *attr,
char *buf)
@@ -3631,13 +3656,6 @@ static ssize_t amdgpu_psp_vbflash_status(struct device *dev,
return sysfs_emit(buf, "0x%x\n", vbflash_status);
}
-static const struct bin_attribute psp_vbflash_bin_attr = {
- .attr = {.name = "psp_vbflash", .mode = 0660},
- .size = 0,
- .write = amdgpu_psp_vbflash_write,
- .read = amdgpu_psp_vbflash_read,
-};
-
static DEVICE_ATTR(psp_vbflash_status, 0440, amdgpu_psp_vbflash_status, NULL);
int amdgpu_psp_sysfs_init(struct amdgpu_device *adev)
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h
index 58ce3ebb446c..4b36f6bf97a4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h
@@ -515,7 +515,7 @@ int psp_get_fw_attestation_records_addr(struct psp_context *psp,
int psp_load_fw_list(struct psp_context *psp,
struct amdgpu_firmware_info **ucode_list, int ucode_count);
-void psp_copy_fw(struct psp_context *psp, uint8_t *start_addr, uint32_t bin_size);
+int psp_copy_fw(struct psp_context *psp, uint8_t *start_addr, uint32_t bin_size);
int is_psp_fw_valid(struct psp_bin_desc bin);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
index 2c8ee880cefa..2ea823fdcd9f 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c
@@ -1777,15 +1777,20 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
/* Change the size here instead of the init above so only lpfn is affected */
amdgpu_ttm_set_buffer_funcs_status(adev, false);
#ifdef CONFIG_64BIT
-#ifdef CONFIG_X86
- if (adev->gmc.xgmi.connected_to_cpu)
- adev->mman.aper_base_kaddr = ioremap_cache(adev->gmc.aper_base,
- adev->gmc.visible_vram_size);
-
- else
-#endif
- adev->mman.aper_base_kaddr = ioremap_wc(adev->gmc.aper_base,
- adev->gmc.visible_vram_size);
+ if (adev->gmc.xgmi.connected_to_cpu) {
+ void *kaddr = devm_memremap(adev->dev, adev->gmc.aper_base,
+ adev->gmc.visible_vram_size,
+ MEMREMAP_WB);
+ if (IS_ERR(kaddr))
+ return PTR_ERR(kaddr);
+ adev->mman.aper_base_kaddr = (__force void __iomem *)kaddr;
+ } else {
+ adev->mman.aper_base_kaddr = devm_ioremap_wc(adev->dev,
+ adev->gmc.aper_base,
+ adev->gmc.visible_vram_size);
+ if (!adev->mman.aper_base_kaddr)
+ return -ENOMEM;
+ }
#endif
/*
@@ -1923,7 +1928,6 @@ int amdgpu_ttm_init(struct amdgpu_device *adev)
*/
void amdgpu_ttm_fini(struct amdgpu_device *adev)
{
- int idx;
if (!adev->mman.initialized)
return;
@@ -1942,14 +1946,7 @@ void amdgpu_ttm_fini(struct amdgpu_device *adev)
amdgpu_ttm_fw_reserve_vram_fini(adev);
amdgpu_ttm_drv_reserve_vram_fini(adev);
- if (drm_dev_enter(adev_to_drm(adev), &idx)) {
-
- if (adev->mman.aper_base_kaddr)
- iounmap(adev->mman.aper_base_kaddr);
- adev->mman.aper_base_kaddr = NULL;
-
- drm_dev_exit(idx);
- }
+ adev->mman.aper_base_kaddr = NULL;
amdgpu_vram_mgr_fini(adev);
amdgpu_gtt_mgr_fini(adev);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c
index aad3c8b4c810..da40be587f1e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c
@@ -109,6 +109,7 @@ static int amdgpu_umc_do_page_retirement(struct amdgpu_device *adev,
}
kfree(err_data->err_addr);
+ err_data->err_addr = NULL;
return AMDGPU_RAS_SUCCESS;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c
index dff155cfb07f..f34199ac3e0b 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c
@@ -96,16 +96,16 @@
*/
struct amdgpu_uvd_cs_ctx {
struct amdgpu_cs_parser *parser;
- unsigned reg, count;
- unsigned data0, data1;
- unsigned idx;
+ unsigned int reg, count;
+ unsigned int data0, data1;
+ unsigned int idx;
struct amdgpu_ib *ib;
/* does the IB has a msg command */
bool has_msg_cmd;
/* minimum buffer sizes */
- unsigned *buf_sizes;
+ unsigned int *buf_sizes;
};
#ifdef CONFIG_DRM_AMDGPU_SI
@@ -186,7 +186,7 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
unsigned long bo_size;
const char *fw_name;
const struct common_firmware_header *hdr;
- unsigned family_id;
+ unsigned int family_id;
int i, j, r;
INIT_DELAYED_WORK(&adev->uvd.idle_work, amdgpu_uvd_idle_work_handler);
@@ -283,7 +283,7 @@ int amdgpu_uvd_sw_init(struct amdgpu_device *adev)
family_id = le32_to_cpu(hdr->ucode_version) & 0xff;
if (adev->asic_type < CHIP_VEGA20) {
- unsigned version_major, version_minor;
+ unsigned int version_major, version_minor;
version_major = (le32_to_cpu(hdr->ucode_version) >> 24) & 0xff;
version_minor = (le32_to_cpu(hdr->ucode_version) >> 8) & 0xff;
@@ -425,7 +425,7 @@ int amdgpu_uvd_entity_init(struct amdgpu_device *adev)
int amdgpu_uvd_suspend(struct amdgpu_device *adev)
{
- unsigned size;
+ unsigned int size;
void *ptr;
int i, j, idx;
bool in_ras_intr = amdgpu_ras_intr_triggered();
@@ -474,7 +474,7 @@ int amdgpu_uvd_suspend(struct amdgpu_device *adev)
int amdgpu_uvd_resume(struct amdgpu_device *adev)
{
- unsigned size;
+ unsigned int size;
void *ptr;
int i, idx;
@@ -496,7 +496,7 @@ int amdgpu_uvd_resume(struct amdgpu_device *adev)
adev->uvd.inst[i].saved_bo = NULL;
} else {
const struct common_firmware_header *hdr;
- unsigned offset;
+ unsigned int offset;
hdr = (const struct common_firmware_header *)adev->uvd.fw->data;
if (adev->firmware.load_type != AMDGPU_FW_LOAD_PSP) {
@@ -547,6 +547,7 @@ void amdgpu_uvd_free_handles(struct amdgpu_device *adev, struct drm_file *filp)
static void amdgpu_uvd_force_into_uvd_segment(struct amdgpu_bo *abo)
{
int i;
+
for (i = 0; i < abo->placement.num_placement; ++i) {
abo->placements[i].fpfn = 0 >> PAGE_SHIFT;
abo->placements[i].lpfn = (256 * 1024 * 1024) >> PAGE_SHIFT;
@@ -584,7 +585,7 @@ static int amdgpu_uvd_cs_pass1(struct amdgpu_uvd_cs_ctx *ctx)
r = amdgpu_cs_find_mapping(ctx->parser, addr, &bo, &mapping);
if (r) {
- DRM_ERROR("Can't find BO for addr 0x%08Lx\n", addr);
+ DRM_ERROR("Can't find BO for addr 0x%08llx\n", addr);
return r;
}
@@ -594,6 +595,7 @@ static int amdgpu_uvd_cs_pass1(struct amdgpu_uvd_cs_ctx *ctx)
if (cmd == 0x0 || cmd == 0x3) {
/* yes, force it into VRAM */
uint32_t domain = AMDGPU_GEM_DOMAIN_VRAM;
+
amdgpu_bo_placement_from_domain(bo, domain);
}
amdgpu_uvd_force_into_uvd_segment(bo);
@@ -614,21 +616,19 @@ static int amdgpu_uvd_cs_pass1(struct amdgpu_uvd_cs_ctx *ctx)
* Peek into the decode message and calculate the necessary buffer sizes.
*/
static int amdgpu_uvd_cs_msg_decode(struct amdgpu_device *adev, uint32_t *msg,
- unsigned buf_sizes[])
+ unsigned int buf_sizes[])
{
- unsigned stream_type = msg[4];
- unsigned width = msg[6];
- unsigned height = msg[7];
- unsigned dpb_size = msg[9];
- unsigned pitch = msg[28];
- unsigned level = msg[57];
+ unsigned int stream_type = msg[4];
+ unsigned int width = msg[6];
+ unsigned int height = msg[7];
+ unsigned int dpb_size = msg[9];
+ unsigned int pitch = msg[28];
- unsigned width_in_mb = width / 16;
- unsigned height_in_mb = ALIGN(height / 16, 2);
- unsigned fs_in_mb = width_in_mb * height_in_mb;
+ unsigned int width_in_mb = width / 16;
+ unsigned int height_in_mb = ALIGN(height / 16, 2);
- unsigned image_size, tmp, min_dpb_size, num_dpb_buffer;
- unsigned min_ctx_size = ~0;
+ unsigned int image_size, tmp, min_dpb_size, num_dpb_buffer;
+ unsigned int min_ctx_size = ~0;
/* Reject invalid dimensions */
if (width < 16 || height < 16 || width > 4096 || height > 4096) {
@@ -644,35 +644,9 @@ static int amdgpu_uvd_cs_msg_decode(struct amdgpu_device *adev, uint32_t *msg,
switch (stream_type) {
case 0: /* H264 */
- switch(level) {
- case 30:
- num_dpb_buffer = 8100 / fs_in_mb;
- break;
- case 31:
- num_dpb_buffer = 18000 / fs_in_mb;
- break;
- case 32:
- num_dpb_buffer = 20480 / fs_in_mb;
- break;
- case 41:
- num_dpb_buffer = 32768 / fs_in_mb;
- break;
- case 42:
- num_dpb_buffer = 34816 / fs_in_mb;
- break;
- case 50:
- num_dpb_buffer = 110400 / fs_in_mb;
- break;
- case 51:
- num_dpb_buffer = 184320 / fs_in_mb;
- break;
- default:
- num_dpb_buffer = 184320 / fs_in_mb;
- break;
- }
- num_dpb_buffer++;
+ num_dpb_buffer = ((msg[61] >> 16) & 0xff) + 1;
if (num_dpb_buffer > 17)
- num_dpb_buffer = 17;
+ return -EINVAL;
/* reference picture buffer */
min_dpb_size = image_size * num_dpb_buffer;
@@ -722,40 +696,14 @@ static int amdgpu_uvd_cs_msg_decode(struct amdgpu_device *adev, uint32_t *msg,
break;
case 7: /* H264 Perf */
- switch(level) {
- case 30:
- num_dpb_buffer = 8100 / fs_in_mb;
- break;
- case 31:
- num_dpb_buffer = 18000 / fs_in_mb;
- break;
- case 32:
- num_dpb_buffer = 20480 / fs_in_mb;
- break;
- case 41:
- num_dpb_buffer = 32768 / fs_in_mb;
- break;
- case 42:
- num_dpb_buffer = 34816 / fs_in_mb;
- break;
- case 50:
- num_dpb_buffer = 110400 / fs_in_mb;
- break;
- case 51:
- num_dpb_buffer = 184320 / fs_in_mb;
- break;
- default:
- num_dpb_buffer = 184320 / fs_in_mb;
- break;
- }
- num_dpb_buffer++;
+ num_dpb_buffer = ((msg[61] >> 16) & 0xff) + 1;
if (num_dpb_buffer > 17)
- num_dpb_buffer = 17;
+ return -EINVAL;
/* reference picture buffer */
min_dpb_size = image_size * num_dpb_buffer;
- if (!adev->uvd.use_ctx_buf){
+ if (!adev->uvd.use_ctx_buf) {
/* macroblock context buffer */
min_dpb_size +=
width_in_mb * height_in_mb * num_dpb_buffer * 192;
@@ -821,7 +769,7 @@ static int amdgpu_uvd_cs_msg_decode(struct amdgpu_device *adev, uint32_t *msg,
* Make sure that we don't open up to many sessions.
*/
static int amdgpu_uvd_cs_msg(struct amdgpu_uvd_cs_ctx *ctx,
- struct amdgpu_bo *bo, unsigned offset)
+ struct amdgpu_bo *bo, unsigned int offset)
{
struct amdgpu_device *adev = ctx->parser->adev;
int32_t *msg, msg_type, handle;
@@ -927,7 +875,7 @@ static int amdgpu_uvd_cs_pass2(struct amdgpu_uvd_cs_ctx *ctx)
r = amdgpu_cs_find_mapping(ctx->parser, addr, &bo, &mapping);
if (r) {
- DRM_ERROR("Can't find BO for addr 0x%08Lx\n", addr);
+ DRM_ERROR("Can't find BO for addr 0x%08llx\n", addr);
return r;
}
@@ -946,7 +894,7 @@ static int amdgpu_uvd_cs_pass2(struct amdgpu_uvd_cs_ctx *ctx)
if (cmd < 0x4) {
if ((end - start) < ctx->buf_sizes[cmd]) {
DRM_ERROR("buffer (%d) to small (%d / %d)!\n", cmd,
- (unsigned)(end - start),
+ (unsigned int)(end - start),
ctx->buf_sizes[cmd]);
return -EINVAL;
}
@@ -954,7 +902,7 @@ static int amdgpu_uvd_cs_pass2(struct amdgpu_uvd_cs_ctx *ctx)
} else if (cmd == 0x206) {
if ((end - start) < ctx->buf_sizes[4]) {
DRM_ERROR("buffer (%d) to small (%d / %d)!\n", cmd,
- (unsigned)(end - start),
+ (unsigned int)(end - start),
ctx->buf_sizes[4]);
return -EINVAL;
}
@@ -965,14 +913,14 @@ static int amdgpu_uvd_cs_pass2(struct amdgpu_uvd_cs_ctx *ctx)
if (!ctx->parser->adev->uvd.address_64_bit) {
if ((start >> 28) != ((end - 1) >> 28)) {
- DRM_ERROR("reloc %LX-%LX crossing 256MB boundary!\n",
+ DRM_ERROR("reloc %llx-%llx crossing 256MB boundary!\n",
start, end);
return -EINVAL;
}
if ((cmd == 0 || cmd == 0x3) &&
(start >> 28) != (ctx->parser->adev->uvd.inst->gpu_addr >> 28)) {
- DRM_ERROR("msg/fb buffer %LX-%LX out of 256MB segment!\n",
+ DRM_ERROR("msg/fb buffer %llx-%llx out of 256MB segment!\n",
start, end);
return -EINVAL;
}
@@ -1006,7 +954,7 @@ static int amdgpu_uvd_cs_reg(struct amdgpu_uvd_cs_ctx *ctx,
ctx->idx++;
for (i = 0; i <= ctx->count; ++i) {
- unsigned reg = ctx->reg + i;
+ unsigned int reg = ctx->reg + i;
if (ctx->idx >= ctx->ib->length_dw) {
DRM_ERROR("Register command after end of CS!\n");
@@ -1052,7 +1000,8 @@ static int amdgpu_uvd_cs_packets(struct amdgpu_uvd_cs_ctx *ctx,
for (ctx->idx = 0 ; ctx->idx < ctx->ib->length_dw; ) {
uint32_t cmd = amdgpu_ib_get_value(ctx->ib, ctx->idx);
- unsigned type = CP_PACKET_GET_TYPE(cmd);
+ unsigned int type = CP_PACKET_GET_TYPE(cmd);
+
switch (type) {
case PACKET_TYPE0:
ctx->reg = CP_PACKET0_GET_REG(cmd);
@@ -1086,7 +1035,7 @@ int amdgpu_uvd_ring_parse_cs(struct amdgpu_cs_parser *parser,
struct amdgpu_ib *ib)
{
struct amdgpu_uvd_cs_ctx ctx = {};
- unsigned buf_sizes[] = {
+ unsigned int buf_sizes[] = {
[0x00000000] = 2048,
[0x00000001] = 0xFFFFFFFF,
[0x00000002] = 0xFFFFFFFF,
@@ -1213,8 +1162,9 @@ static int amdgpu_uvd_send_msg(struct amdgpu_ring *ring, struct amdgpu_bo *bo,
}
/* multiple fence commands without any stream commands in between can
- crash the vcpu so just try to emmit a dummy create/destroy msg to
- avoid this */
+ * crash the vcpu so just try to emmit a dummy create/destroy msg to
+ * avoid this
+ */
int amdgpu_uvd_get_create_msg(struct amdgpu_ring *ring, uint32_t handle,
struct dma_fence **fence)
{
@@ -1280,15 +1230,14 @@ static void amdgpu_uvd_idle_work_handler(struct work_struct *work)
{
struct amdgpu_device *adev =
container_of(work, struct amdgpu_device, uvd.idle_work.work);
- unsigned fences = 0, i, j;
+ unsigned int fences = 0, i, j;
for (i = 0; i < adev->uvd.num_uvd_inst; ++i) {
if (adev->uvd.harvest_config & (1 << i))
continue;
fences += amdgpu_fence_count_emitted(&adev->uvd.inst[i].ring);
- for (j = 0; j < adev->uvd.num_enc_rings; ++j) {
+ for (j = 0; j < adev->uvd.num_enc_rings; ++j)
fences += amdgpu_fence_count_emitted(&adev->uvd.inst[i].ring_enc[j]);
- }
}
if (fences == 0) {
@@ -1384,7 +1333,7 @@ int amdgpu_uvd_ring_test_ib(struct amdgpu_ring *ring, long timeout)
*/
uint32_t amdgpu_uvd_used_handles(struct amdgpu_device *adev)
{
- unsigned i;
+ unsigned int i;
uint32_t used_handles = 0;
for (i = 0; i < adev->uvd.max_handles; ++i) {
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c
index 4161be20d766..01346b4c22e0 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vce.c
@@ -99,7 +99,7 @@ int amdgpu_vce_sw_init(struct amdgpu_device *adev, unsigned long size)
{
const char *fw_name;
const struct common_firmware_header *hdr;
- unsigned ucode_version, version_major, version_minor, binary_id;
+ unsigned int ucode_version, version_major, version_minor, binary_id;
int i, r;
switch (adev->asic_type) {
@@ -213,7 +213,7 @@ int amdgpu_vce_sw_init(struct amdgpu_device *adev, unsigned long size)
*/
int amdgpu_vce_sw_fini(struct amdgpu_device *adev)
{
- unsigned i;
+ unsigned int i;
if (adev->vce.vcpu_bo == NULL)
return 0;
@@ -292,7 +292,7 @@ int amdgpu_vce_resume(struct amdgpu_device *adev)
{
void *cpu_addr;
const struct common_firmware_header *hdr;
- unsigned offset;
+ unsigned int offset;
int r, idx;
if (adev->vce.vcpu_bo == NULL)
@@ -338,7 +338,7 @@ static void amdgpu_vce_idle_work_handler(struct work_struct *work)
{
struct amdgpu_device *adev =
container_of(work, struct amdgpu_device, vce.idle_work.work);
- unsigned i, count = 0;
+ unsigned int i, count = 0;
for (i = 0; i < adev->vce.num_rings; i++)
count += amdgpu_fence_count_emitted(&adev->vce.ring[i]);
@@ -415,6 +415,7 @@ void amdgpu_vce_free_handles(struct amdgpu_device *adev, struct drm_file *filp)
{
struct amdgpu_ring *ring = &adev->vce.ring[0];
int i, r;
+
for (i = 0; i < AMDGPU_MAX_VCE_HANDLES; ++i) {
uint32_t handle = atomic_read(&adev->vce.handles[i]);
@@ -442,7 +443,7 @@ void amdgpu_vce_free_handles(struct amdgpu_device *adev, struct drm_file *filp)
static int amdgpu_vce_get_create_msg(struct amdgpu_ring *ring, uint32_t handle,
struct dma_fence **fence)
{
- const unsigned ib_size_dw = 1024;
+ const unsigned int ib_size_dw = 1024;
struct amdgpu_job *job;
struct amdgpu_ib *ib;
struct amdgpu_ib ib_msg;
@@ -534,7 +535,7 @@ static int amdgpu_vce_get_create_msg(struct amdgpu_ring *ring, uint32_t handle,
static int amdgpu_vce_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,
bool direct, struct dma_fence **fence)
{
- const unsigned ib_size_dw = 1024;
+ const unsigned int ib_size_dw = 1024;
struct amdgpu_job *job;
struct amdgpu_ib *ib;
struct dma_fence *f = NULL;
@@ -601,12 +602,12 @@ static int amdgpu_vce_get_destroy_msg(struct amdgpu_ring *ring, uint32_t handle,
*/
static int amdgpu_vce_validate_bo(struct amdgpu_cs_parser *p,
struct amdgpu_ib *ib, int lo, int hi,
- unsigned size, int32_t index)
+ unsigned int size, int32_t index)
{
int64_t offset = ((uint64_t)size) * ((int64_t)index);
struct ttm_operation_ctx ctx = { false, false };
struct amdgpu_bo_va_mapping *mapping;
- unsigned i, fpfn, lpfn;
+ unsigned int i, fpfn, lpfn;
struct amdgpu_bo *bo;
uint64_t addr;
int r;
@@ -624,7 +625,7 @@ static int amdgpu_vce_validate_bo(struct amdgpu_cs_parser *p,
r = amdgpu_cs_find_mapping(p, addr, &bo, &mapping);
if (r) {
- DRM_ERROR("Can't find BO for addr 0x%010Lx %d %d %d %d\n",
+ DRM_ERROR("Can't find BO for addr 0x%010llx %d %d %d %d\n",
addr, lo, hi, size, index);
return r;
}
@@ -651,7 +652,7 @@ static int amdgpu_vce_validate_bo(struct amdgpu_cs_parser *p,
* Patch relocation inside command stream with real buffer address
*/
static int amdgpu_vce_cs_reloc(struct amdgpu_cs_parser *p, struct amdgpu_ib *ib,
- int lo, int hi, unsigned size, uint32_t index)
+ int lo, int hi, unsigned int size, uint32_t index)
{
struct amdgpu_bo_va_mapping *mapping;
struct amdgpu_bo *bo;
@@ -670,14 +671,14 @@ static int amdgpu_vce_cs_reloc(struct amdgpu_cs_parser *p, struct amdgpu_ib *ib,
r = amdgpu_cs_find_mapping(p, addr, &bo, &mapping);
if (r) {
- DRM_ERROR("Can't find BO for addr 0x%010Lx %d %d %d %d\n",
+ DRM_ERROR("Can't find BO for addr 0x%010llx %d %d %d %d\n",
addr, lo, hi, size, index);
return r;
}
if ((addr + (uint64_t)size) >
(mapping->last + 1) * AMDGPU_GPU_PAGE_SIZE) {
- DRM_ERROR("BO too small for addr 0x%010Lx %d %d\n",
+ DRM_ERROR("BO too small for addr 0x%010llx %d %d\n",
addr, lo, hi);
return -EINVAL;
}
@@ -700,12 +701,12 @@ static int amdgpu_vce_cs_reloc(struct amdgpu_cs_parser *p, struct amdgpu_ib *ib,
* @allocated: allocated a new handle?
*
* Validates the handle and return the found session index or -EINVAL
- * we we don't have another free session index.
+ * we don't have another free session index.
*/
static int amdgpu_vce_validate_handle(struct amdgpu_cs_parser *p,
uint32_t handle, uint32_t *allocated)
{
- unsigned i;
+ unsigned int i;
/* validate the handle */
for (i = 0; i < AMDGPU_MAX_VCE_HANDLES; ++i) {
@@ -743,7 +744,7 @@ int amdgpu_vce_ring_parse_cs(struct amdgpu_cs_parser *p,
struct amdgpu_job *job,
struct amdgpu_ib *ib)
{
- unsigned fb_idx = 0, bs_idx = 0;
+ unsigned int fb_idx = 0, bs_idx = 0;
int session_idx = -1;
uint32_t destroyed = 0;
uint32_t created = 0;
@@ -751,7 +752,7 @@ int amdgpu_vce_ring_parse_cs(struct amdgpu_cs_parser *p,
uint32_t tmp, handle = 0;
uint32_t dummy = 0xffffffff;
uint32_t *size = &dummy;
- unsigned idx;
+ unsigned int idx;
int i, r = 0;
job->vm = NULL;
@@ -1104,7 +1105,7 @@ void amdgpu_vce_ring_emit_ib(struct amdgpu_ring *ring,
*
*/
void amdgpu_vce_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 seq,
- unsigned flags)
+ unsigned int flags)
{
WARN_ON(flags & AMDGPU_FENCE_FLAG_64BIT);
@@ -1126,7 +1127,7 @@ int amdgpu_vce_ring_test_ring(struct amdgpu_ring *ring)
{
struct amdgpu_device *adev = ring->adev;
uint32_t rptr;
- unsigned i;
+ unsigned int i;
int r, timeout = adev->usec_timeout;
/* skip ring test for sriov*/
@@ -1191,7 +1192,7 @@ int amdgpu_vce_ring_test_ib(struct amdgpu_ring *ring, long timeout)
enum amdgpu_ring_priority_level amdgpu_vce_get_ring_prio(int ring)
{
- switch(ring) {
+ switch (ring) {
case 0:
return AMDGPU_RING_PRIO_0;
case 1:
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
index 3c07f956381a..3f2b0f192ef4 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c
@@ -2033,6 +2033,108 @@ long amdgpu_vm_wait_idle(struct amdgpu_vm *vm, long timeout)
return dma_fence_wait_timeout(vm->last_unlocked, true, timeout);
}
+static void amdgpu_vm_destroy_task_info(struct kref *kref)
+{
+ struct amdgpu_task_info *ti = container_of(kref, struct amdgpu_task_info, refcount);
+
+ kfree(ti);
+}
+
+static inline struct amdgpu_vm *
+amdgpu_vm_get_vm_from_pasid(struct amdgpu_device *adev, u32 pasid)
+{
+ struct amdgpu_vm *vm;
+ unsigned long flags;
+
+ xa_lock_irqsave(&adev->vm_manager.pasids, flags);
+ vm = xa_load(&adev->vm_manager.pasids, pasid);
+ xa_unlock_irqrestore(&adev->vm_manager.pasids, flags);
+
+ return vm;
+}
+
+/**
+ * amdgpu_vm_put_task_info - reference down the vm task_info ptr
+ *
+ * @task_info: task_info struct under discussion.
+ *
+ * frees the vm task_info ptr at the last put
+ */
+void amdgpu_vm_put_task_info(struct amdgpu_task_info *task_info)
+{
+ kref_put(&task_info->refcount, amdgpu_vm_destroy_task_info);
+}
+
+/**
+ * amdgpu_vm_get_task_info_vm - Extracts task info for a vm.
+ *
+ * @vm: VM to get info from
+ *
+ * Returns the reference counted task_info structure, which must be
+ * referenced down with amdgpu_vm_put_task_info.
+ */
+struct amdgpu_task_info *
+amdgpu_vm_get_task_info_vm(struct amdgpu_vm *vm)
+{
+ struct amdgpu_task_info *ti = NULL;
+
+ if (vm) {
+ ti = vm->task_info;
+ kref_get(&vm->task_info->refcount);
+ }
+
+ return ti;
+}
+
+/**
+ * amdgpu_vm_get_task_info_pasid - Extracts task info for a PASID.
+ *
+ * @adev: drm device pointer
+ * @pasid: PASID identifier for VM
+ *
+ * Returns the reference counted task_info structure, which must be
+ * referenced down with amdgpu_vm_put_task_info.
+ */
+struct amdgpu_task_info *
+amdgpu_vm_get_task_info_pasid(struct amdgpu_device *adev, u32 pasid)
+{
+ return amdgpu_vm_get_task_info_vm(
+ amdgpu_vm_get_vm_from_pasid(adev, pasid));
+}
+
+static int amdgpu_vm_create_task_info(struct amdgpu_vm *vm)
+{
+ vm->task_info = kzalloc(sizeof(struct amdgpu_task_info), GFP_KERNEL);
+ if (!vm->task_info)
+ return -ENOMEM;
+
+ kref_init(&vm->task_info->refcount);
+ return 0;
+}
+
+/**
+ * amdgpu_vm_set_task_info - Sets VMs task info.
+ *
+ * @vm: vm for which to set the info
+ */
+void amdgpu_vm_set_task_info(struct amdgpu_vm *vm)
+{
+ if (!vm->task_info)
+ return;
+
+ if (vm->task_info->pid == current->pid)
+ return;
+
+ vm->task_info->pid = current->pid;
+ get_task_comm(vm->task_info->task_name, current);
+
+ if (current->group_leader->mm != current->mm)
+ return;
+
+ vm->task_info->tgid = current->group_leader->pid;
+ get_task_comm(vm->task_info->process_name, current->group_leader);
+}
+
/**
* amdgpu_vm_init - initialize a vm instance
*
@@ -2120,6 +2222,10 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm)
if (r)
goto error_unreserve;
+ r = amdgpu_vm_create_task_info(vm);
+ if (r)
+ DRM_DEBUG("Failed to create task info for VM\n");
+
amdgpu_bo_unreserve(vm->root.bo);
INIT_KFIFO(vm->faults);
@@ -2270,6 +2376,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm)
root = amdgpu_bo_ref(vm->root.bo);
amdgpu_bo_reserve(root, true);
+ amdgpu_vm_put_task_info(vm->task_info);
amdgpu_vm_set_pasid(adev, vm, 0);
dma_fence_wait(vm->last_unlocked, false);
dma_fence_put(vm->last_unlocked);
@@ -2377,6 +2484,7 @@ void amdgpu_vm_manager_fini(struct amdgpu_device *adev)
xa_destroy(&adev->vm_manager.pasids);
amdgpu_vmid_mgr_fini(adev);
+ amdgpu_pasid_mgr_cleanup();
}
/**
@@ -2434,48 +2542,6 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
return 0;
}
-/**
- * amdgpu_vm_get_task_info - Extracts task info for a PASID.
- *
- * @adev: drm device pointer
- * @pasid: PASID identifier for VM
- * @task_info: task_info to fill.
- */
-void amdgpu_vm_get_task_info(struct amdgpu_device *adev, u32 pasid,
- struct amdgpu_task_info *task_info)
-{
- struct amdgpu_vm *vm;
- unsigned long flags;
-
- xa_lock_irqsave(&adev->vm_manager.pasids, flags);
-
- vm = xa_load(&adev->vm_manager.pasids, pasid);
- if (vm)
- *task_info = vm->task_info;
-
- xa_unlock_irqrestore(&adev->vm_manager.pasids, flags);
-}
-
-/**
- * amdgpu_vm_set_task_info - Sets VMs task info.
- *
- * @vm: vm for which to set the info
- */
-void amdgpu_vm_set_task_info(struct amdgpu_vm *vm)
-{
- if (vm->task_info.pid)
- return;
-
- vm->task_info.pid = current->pid;
- get_task_comm(vm->task_info.task_name, current);
-
- if (current->group_leader->mm != current->mm)
- return;
-
- vm->task_info.tgid = current->group_leader->pid;
- get_task_comm(vm->task_info.process_name, current->group_leader);
-}
-
/**
* amdgpu_vm_handle_fault - graceful handling of VM faults.
* @adev: amdgpu device pointer
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
index 6546e786bf00..ac2f8a043bc1 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h
@@ -174,10 +174,11 @@ struct amdgpu_vm_pte_funcs {
};
struct amdgpu_task_info {
- char process_name[TASK_COMM_LEN];
- char task_name[TASK_COMM_LEN];
- pid_t pid;
- pid_t tgid;
+ char process_name[TASK_COMM_LEN];
+ char task_name[TASK_COMM_LEN];
+ pid_t pid;
+ pid_t tgid;
+ struct kref refcount;
};
/**
@@ -323,7 +324,7 @@ struct amdgpu_vm {
uint64_t pd_phys_addr;
/* Some basic info about the task */
- struct amdgpu_task_info task_info;
+ struct amdgpu_task_info *task_info;
/* Store positions of group of BOs */
struct ttm_lru_bulk_move lru_bulk_move;
@@ -452,8 +453,14 @@ bool amdgpu_vm_need_pipeline_sync(struct amdgpu_ring *ring,
struct amdgpu_job *job);
void amdgpu_vm_check_compute_bug(struct amdgpu_device *adev);
-void amdgpu_vm_get_task_info(struct amdgpu_device *adev, u32 pasid,
- struct amdgpu_task_info *task_info);
+struct amdgpu_task_info *
+amdgpu_vm_get_task_info_pasid(struct amdgpu_device *adev, u32 pasid);
+
+struct amdgpu_task_info *
+amdgpu_vm_get_task_info_vm(struct amdgpu_vm *vm);
+
+void amdgpu_vm_put_task_info(struct amdgpu_task_info *task_info);
+
bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid,
uint64_t addr, bool write_fault);
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
index 370d02bdde86..c9930abada8d 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c
@@ -978,7 +978,7 @@ int amdgpu_vm_ptes_update(struct amdgpu_vm_update_params *params,
trace_amdgpu_vm_update_ptes(params, frag_start, upd_end,
min(nptes, 32u), dst, incr,
upd_flags,
- vm->task_info.tgid,
+ vm->task_info ? vm->task_info->tgid : 0,
vm->immediate.fence_context);
amdgpu_vm_pte_update_flags(params, to_amdgpu_bo_vm(pt),
cursor.level, pe_start, dst,
diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
index 24867e96b520..4d8912345c38 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c
@@ -212,8 +212,26 @@ static struct attribute *amdgpu_vram_mgr_attributes[] = {
NULL
};
+static umode_t amdgpu_vram_attrs_is_visible(struct kobject *kobj,
+ struct attribute *attr, int i)
+{
+ struct device *dev = kobj_to_dev(kobj);
+ struct drm_device *ddev = dev_get_drvdata(dev);
+ struct amdgpu_device *adev = drm_to_adev(ddev);
+
+ if (attr == &dev_attr_mem_info_vram_vendor.attr &&
+ !adev->gmc.vram_vendor)
+ return 0;
+
+ if (!ttm_resource_manager_used(&adev->mman.vram_mgr.manager))
+ return 0;
+
+ return attr->mode;
+}
+
const struct attribute_group amdgpu_vram_mgr_attr_group = {
- .attrs = amdgpu_vram_mgr_attributes
+ .attrs = amdgpu_vram_mgr_attributes,
+ .is_visible = amdgpu_vram_attrs_is_visible
};
/**
diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c
index fe565dff703f..15cb1fca68b4 100644
--- a/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c
@@ -308,6 +308,14 @@ static const u32 verde_rlc_save_restore_register_list[] =
0x00000000
};
+static void gfx_v6_0_free_microcode(struct amdgpu_device *adev)
+{
+ amdgpu_ucode_release(&adev->gfx.pfp_fw);
+ amdgpu_ucode_release(&adev->gfx.me_fw);
+ amdgpu_ucode_release(&adev->gfx.ce_fw);
+ amdgpu_ucode_release(&adev->gfx.rlc_fw);
+}
+
static int gfx_v6_0_init_microcode(struct amdgpu_device *adev)
{
const char *chip_name;
@@ -3197,6 +3205,7 @@ static int gfx_v6_0_sw_fini(void *handle)
amdgpu_ring_fini(&adev->gfx.compute_ring[i]);
amdgpu_gfx_rlc_fini(adev);
+ gfx_v6_0_free_microcode(adev);
return 0;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
index 918cf169cb95..5cdecf52fda6 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c
@@ -100,7 +100,7 @@ static int gmc_v10_0_process_interrupt(struct amdgpu_device *adev,
bool retry_fault = !!(entry->src_data[1] & 0x80);
bool write_fault = !!(entry->src_data[1] & 0x20);
struct amdgpu_vmhub *hub = &adev->vmhub[entry->vmid_src];
- struct amdgpu_task_info task_info;
+ struct amdgpu_task_info *task_info;
uint32_t status = 0;
u64 addr;
@@ -148,19 +148,22 @@ static int gmc_v10_0_process_interrupt(struct amdgpu_device *adev,
if (!printk_ratelimit())
return 0;
- memset(&task_info, 0, sizeof(struct amdgpu_task_info));
- amdgpu_vm_get_task_info(adev, entry->pasid, &task_info);
-
dev_err(adev->dev,
- "[%s] page fault (src_id:%u ring:%u vmid:%u pasid:%u, "
- "for process %s pid %d thread %s pid %d)\n",
+ "[%s] page fault (src_id:%u ring:%u vmid:%u pasid:%u)\n",
entry->vmid_src ? "mmhub" : "gfxhub",
- entry->src_id, entry->ring_id, entry->vmid,
- entry->pasid, task_info.process_name, task_info.tgid,
- task_info.task_name, task_info.pid);
+ entry->src_id, entry->ring_id, entry->vmid, entry->pasid);
+ task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid);
+ if (task_info) {
+ dev_err(adev->dev,
+ " in process %s pid %d thread %s pid %d\n",
+ task_info->process_name, task_info->tgid,
+ task_info->task_name, task_info->pid);
+ amdgpu_vm_put_task_info(task_info);
+ }
+
dev_err(adev->dev, " in page starting at address 0x%016llx from client 0x%x (%s)\n",
- addr, entry->client_id,
- soc15_ih_clientid_name[entry->client_id]);
+ addr, entry->client_id,
+ soc15_ih_clientid_name[entry->client_id]);
if (!amdgpu_sriov_vf(adev))
hub->vmhub_funcs->print_l2_protection_fault_status(adev,
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
index 310a5607d83b..db50f2ab7ec4 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c
@@ -106,20 +106,24 @@ static int gmc_v11_0_process_interrupt(struct amdgpu_device *adev,
}
if (printk_ratelimit()) {
- struct amdgpu_task_info task_info;
-
- memset(&task_info, 0, sizeof(struct amdgpu_task_info));
- amdgpu_vm_get_task_info(adev, entry->pasid, &task_info);
+ struct amdgpu_task_info *task_info;
dev_err(adev->dev,
- "[%s] page fault (src_id:%u ring:%u vmid:%u pasid:%u, "
- "for process %s pid %d thread %s pid %d)\n",
+ "[%s] page fault (src_id:%u ring:%u vmid:%u pasid:%u)\n",
entry->vmid_src ? "mmhub" : "gfxhub",
- entry->src_id, entry->ring_id, entry->vmid,
- entry->pasid, task_info.process_name, task_info.tgid,
- task_info.task_name, task_info.pid);
+ entry->src_id, entry->ring_id, entry->vmid, entry->pasid);
+ task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid);
+ if (task_info) {
+ dev_err(adev->dev,
+ " in process %s pid %d thread %s pid %d)\n",
+ task_info->process_name, task_info->tgid,
+ task_info->task_name, task_info->pid);
+ amdgpu_vm_put_task_info(task_info);
+ }
+
dev_err(adev->dev, " in page starting at address 0x%016llx from client %d\n",
- addr, entry->client_id);
+ addr, entry->client_id);
+
if (!amdgpu_sriov_vf(adev))
hub->vmhub_funcs->print_l2_protection_fault_status(adev, status);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
index c17a36504d17..4cd8ee614753 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c
@@ -1461,18 +1461,24 @@ static int gmc_v8_0_process_interrupt(struct amdgpu_device *adev,
gmc_v8_0_set_fault_enable_default(adev, false);
if (printk_ratelimit()) {
- struct amdgpu_task_info task_info;
+ struct amdgpu_task_info *task_info;
- memset(&task_info, 0, sizeof(struct amdgpu_task_info));
- amdgpu_vm_get_task_info(adev, entry->pasid, &task_info);
+ dev_err(adev->dev, "GPU fault detected: %d 0x%08x\n",
+ entry->src_id, entry->src_data[0]);
+
+ task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid);
+ if (task_info) {
+ dev_err(adev->dev, " for process %s pid %d thread %s pid %d\n",
+ task_info->process_name, task_info->tgid,
+ task_info->task_name, task_info->pid);
+ amdgpu_vm_put_task_info(task_info);
+ }
- dev_err(adev->dev, "GPU fault detected: %d 0x%08x for process %s pid %d thread %s pid %d\n",
- entry->src_id, entry->src_data[0], task_info.process_name,
- task_info.tgid, task_info.task_name, task_info.pid);
dev_err(adev->dev, " VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x%08X\n",
- addr);
+ addr);
dev_err(adev->dev, " VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n",
status);
+
gmc_v8_0_vm_decode_fault(adev, status, addr, mc_client,
entry->pasid);
}
diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
index 3e631aefa791..5b26ef0df5e2 100644
--- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c
@@ -532,7 +532,7 @@ static int gmc_v9_0_process_interrupt(struct amdgpu_device *adev,
bool retry_fault = !!(entry->src_data[1] & 0x80);
bool write_fault = !!(entry->src_data[1] & 0x20);
uint32_t status = 0, cid = 0, rw = 0;
- struct amdgpu_task_info task_info;
+ struct amdgpu_task_info *task_info;
struct amdgpu_vmhub *hub;
const char *mmhub_cid;
const char *hub_name;
@@ -579,16 +579,20 @@ static int gmc_v9_0_process_interrupt(struct amdgpu_device *adev,
hub = &adev->vmhub[AMDGPU_GFXHUB_0];
}
- memset(&task_info, 0, sizeof(struct amdgpu_task_info));
- amdgpu_vm_get_task_info(adev, entry->pasid, &task_info);
-
dev_err(adev->dev,
- "[%s] %s page fault (src_id:%u ring:%u vmid:%u "
- "pasid:%u, for process %s pid %d thread %s pid %d)\n",
- hub_name, retry_fault ? "retry" : "no-retry",
- entry->src_id, entry->ring_id, entry->vmid,
- entry->pasid, task_info.process_name, task_info.tgid,
- task_info.task_name, task_info.pid);
+ "[%s] %s page fault (src_id:%u ring:%u vmid:%u pasid:%u)\n", hub_name,
+ retry_fault ? "retry" : "no-retry",
+ entry->src_id, entry->ring_id, entry->vmid, entry->pasid);
+
+ task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid);
+ if (task_info) {
+ dev_err(adev->dev,
+ " for process %s pid %d thread %s pid %d)\n",
+ task_info->process_name, task_info->tgid,
+ task_info->task_name, task_info->pid);
+ amdgpu_vm_put_task_info(task_info);
+ }
+
dev_err(adev->dev, " in page starting at address 0x%016llx from IH client 0x%x (%s)\n",
addr, entry->client_id,
soc15_ih_clientid_name[entry->client_id]);
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
index 3301ad980f28..af8f244e9c94 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_0.c
@@ -762,6 +762,7 @@ static const struct amd_ip_funcs jpeg_v2_0_ip_funcs = {
static const struct amdgpu_ring_funcs jpeg_v2_0_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_JPEG,
.align_mask = 0xf,
+ .no_user_fence = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = jpeg_v2_0_dec_ring_get_rptr,
.get_wptr = jpeg_v2_0_dec_ring_get_wptr,
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
index f87d0f6ffc93..a094dd138c0d 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v2_5.c
@@ -641,6 +641,7 @@ static const struct amd_ip_funcs jpeg_v2_6_ip_funcs = {
static const struct amdgpu_ring_funcs jpeg_v2_5_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_JPEG,
.align_mask = 0xf,
+ .no_user_fence = true,
.vmhub = AMDGPU_MMHUB_1,
.get_rptr = jpeg_v2_5_dec_ring_get_rptr,
.get_wptr = jpeg_v2_5_dec_ring_get_wptr,
@@ -671,6 +672,7 @@ static const struct amdgpu_ring_funcs jpeg_v2_5_dec_ring_vm_funcs = {
static const struct amdgpu_ring_funcs jpeg_v2_6_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_JPEG,
.align_mask = 0xf,
+ .no_user_fence = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = jpeg_v2_5_dec_ring_get_rptr,
.get_wptr = jpeg_v2_5_dec_ring_get_wptr,
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
index 323d68b2124f..54cbf62a8dea 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v3_0.c
@@ -561,6 +561,7 @@ static const struct amdgpu_ring_funcs jpeg_v3_0_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_JPEG,
.align_mask = 0xf,
.vmhub = AMDGPU_MMHUB_0,
+ .no_user_fence = true,
.get_rptr = jpeg_v3_0_dec_ring_get_rptr,
.get_wptr = jpeg_v3_0_dec_ring_get_wptr,
.set_wptr = jpeg_v3_0_dec_ring_set_wptr,
diff --git a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
index 63b0d0b810ec..ae9e390c8bf6 100644
--- a/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/jpeg_v4_0.c
@@ -557,6 +557,7 @@ static const struct amdgpu_ring_funcs jpeg_v4_0_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_JPEG,
.align_mask = 0xf,
.vmhub = AMDGPU_MMHUB_0,
+ .no_user_fence = true,
.get_rptr = jpeg_v4_0_dec_ring_get_rptr,
.get_wptr = jpeg_v4_0_dec_ring_get_wptr,
.set_wptr = jpeg_v4_0_dec_ring_set_wptr,
diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v11_0.c b/drivers/gpu/drm/amd/amdgpu/psp_v11_0.c
index 9518b4394a6e..589052f9be13 100644
--- a/drivers/gpu/drm/amd/amdgpu/psp_v11_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/psp_v11_0.c
@@ -296,7 +296,9 @@ static int psp_v11_0_bootloader_load_component(struct psp_context *psp,
return ret;
/* Copy PSP System Driver binary to memory */
- psp_copy_fw(psp, bin_desc->start_addr, bin_desc->size_bytes);
+ ret = psp_copy_fw(psp, bin_desc->start_addr, bin_desc->size_bytes);
+ if (ret)
+ return ret;
/* Provide the sys driver to bootloader */
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36,
@@ -342,7 +344,9 @@ static int psp_v11_0_bootloader_load_sos(struct psp_context *psp)
return ret;
/* Copy Secure OS binary to PSP memory */
- psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes);
+ ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes);
+ if (ret)
+ return ret;
/* Provide the PSP secure OS to bootloader */
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36,
diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v12_0.c b/drivers/gpu/drm/amd/amdgpu/psp_v12_0.c
index 0b2ac418e4ac..b829f7ef67c4 100644
--- a/drivers/gpu/drm/amd/amdgpu/psp_v12_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/psp_v12_0.c
@@ -148,7 +148,9 @@ static int psp_v12_0_bootloader_load_sysdrv(struct psp_context *psp)
return ret;
/* Copy PSP System Driver binary to memory */
- psp_copy_fw(psp, psp->sys.start_addr, psp->sys.size_bytes);
+ ret = psp_copy_fw(psp, psp->sys.start_addr, psp->sys.size_bytes);
+ if (ret)
+ return ret;
/* Provide the sys driver to bootloader */
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36,
@@ -187,7 +189,9 @@ static int psp_v12_0_bootloader_load_sos(struct psp_context *psp)
return ret;
/* Copy Secure OS binary to PSP memory */
- psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes);
+ ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes);
+ if (ret)
+ return ret;
/* Provide the PSP secure OS to bootloader */
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36,
diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c b/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c
index c7cb30efe43d..c44d79b90b5d 100644
--- a/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c
@@ -182,10 +182,9 @@ static int psp_v13_0_bootloader_load_component(struct psp_context *psp,
if (ret)
return ret;
- memset(psp->fw_pri_buf, 0, PSP_1_MEG);
-
- /* Copy PSP KDB binary to memory */
- memcpy(psp->fw_pri_buf, bin_desc->start_addr, bin_desc->size_bytes);
+ ret = psp_copy_fw(psp, bin_desc->start_addr, bin_desc->size_bytes);
+ if (ret)
+ return ret;
/* Provide the PSP KDB to bootloader */
WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_36,
@@ -251,10 +250,9 @@ static int psp_v13_0_bootloader_load_sos(struct psp_context *psp)
if (ret)
return ret;
- memset(psp->fw_pri_buf, 0, PSP_1_MEG);
-
- /* Copy Secure OS binary to PSP memory */
- memcpy(psp->fw_pri_buf, psp->sos.start_addr, psp->sos.size_bytes);
+ ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes);
+ if (ret)
+ return ret;
/* Provide the PSP secure OS to bootloader */
WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_36,
diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.c b/drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.c
index 321089dfa7db..6764ceded08d 100644
--- a/drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.c
+++ b/drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.c
@@ -115,10 +115,9 @@ static int psp_v13_0_4_bootloader_load_component(struct psp_context *psp,
if (ret)
return ret;
- memset(psp->fw_pri_buf, 0, PSP_1_MEG);
-
- /* Copy PSP KDB binary to memory */
- memcpy(psp->fw_pri_buf, bin_desc->start_addr, bin_desc->size_bytes);
+ ret = psp_copy_fw(psp, bin_desc->start_addr, bin_desc->size_bytes);
+ if (ret)
+ return ret;
/* Provide the PSP KDB to bootloader */
WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_36,
@@ -178,10 +177,9 @@ static int psp_v13_0_4_bootloader_load_sos(struct psp_context *psp)
if (ret)
return ret;
- memset(psp->fw_pri_buf, 0, PSP_1_MEG);
-
- /* Copy Secure OS binary to PSP memory */
- memcpy(psp->fw_pri_buf, psp->sos.start_addr, psp->sos.size_bytes);
+ ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes);
+ if (ret)
+ return ret;
/* Provide the PSP secure OS to bootloader */
WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_36,
diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c b/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c
index 01f3bcc62a6c..953a59df5387 100644
--- a/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c
+++ b/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c
@@ -104,7 +104,9 @@ static int psp_v3_1_bootloader_load_sysdrv(struct psp_context *psp)
return ret;
/* Copy PSP System Driver binary to memory */
- psp_copy_fw(psp, psp->sys.start_addr, psp->sys.size_bytes);
+ ret = psp_copy_fw(psp, psp->sys.start_addr, psp->sys.size_bytes);
+ if (ret)
+ return ret;
/* Provide the sys driver to bootloader */
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36,
@@ -143,7 +145,9 @@ static int psp_v3_1_bootloader_load_sos(struct psp_context *psp)
return ret;
/* Copy Secure OS binary to PSP memory */
- psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes);
+ ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes);
+ if (ret)
+ return ret;
/* Provide the PSP secure OS to bootloader */
WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36,
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c
index 486d9b5c1b9e..00c7d38784a1 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c
@@ -460,6 +460,9 @@ static void sdma_v3_0_ring_emit_hdp_flush(struct amdgpu_ring *ring)
else
ref_and_mask = REG_SET_FIELD(ref_and_mask, GPU_HDP_FLUSH_DONE, SDMA1, 1);
+ /* Hang workaround */
+ amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP));
+
amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) |
SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(1) |
SDMA_PKT_POLL_REGMEM_HEADER_FUNC(3)); /* == */
@@ -487,6 +490,10 @@ static void sdma_v3_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se
unsigned flags)
{
bool write64bit = flags & AMDGPU_FENCE_FLAG_64BIT;
+
+ /* Hang workaround */
+ amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP));
+
/* write the fence */
amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_FENCE));
amdgpu_ring_write(ring, lower_32_bits(addr));
@@ -495,7 +502,11 @@ static void sdma_v3_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se
/* optionally write high bits as well */
if (write64bit) {
+ /* Hang workaround */
+ amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP));
+
addr += 4;
+
amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_FENCE));
amdgpu_ring_write(ring, lower_32_bits(addr));
amdgpu_ring_write(ring, upper_32_bits(addr));
@@ -1043,6 +1054,9 @@ static void sdma_v3_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring)
uint32_t seq = ring->fence_drv.sync_seq;
uint64_t addr = ring->fence_drv.gpu_addr;
+ /* Hang workaround */
+ amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP));
+
/* wait for idle */
amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) |
SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(0) |
@@ -1071,6 +1085,9 @@ static void sdma_v3_0_ring_emit_vm_flush(struct amdgpu_ring *ring,
{
amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr);
+ /* Hang workaround */
+ amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP));
+
/* wait for flush */
amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) |
SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(0) |
@@ -1579,11 +1596,11 @@ static const struct amdgpu_ring_funcs sdma_v3_0_ring_funcs = {
.get_wptr = sdma_v3_0_ring_get_wptr,
.set_wptr = sdma_v3_0_ring_set_wptr,
.emit_frame_size =
- 6 + /* sdma_v3_0_ring_emit_hdp_flush */
+ 6 + 1 + /* sdma_v3_0_ring_emit_hdp_flush */
3 + /* hdp invalidate */
- 6 + /* sdma_v3_0_ring_emit_pipeline_sync */
- VI_FLUSH_GPU_TLB_NUM_WREG * 3 + 6 + /* sdma_v3_0_ring_emit_vm_flush */
- 10 + 10 + 10, /* sdma_v3_0_ring_emit_fence x3 for user fence, vm fence */
+ 6 + 1 + /* sdma_v3_0_ring_emit_pipeline_sync */
+ VI_FLUSH_GPU_TLB_NUM_WREG * 3 + 6 + 1 + /* sdma_v3_0_ring_emit_vm_flush */
+ (10 + 1) * 3, /* sdma_v3_0_ring_emit_fence x3 for user fence, vm fence */
.emit_ib_size = 7 + 6, /* sdma_v3_0_ring_emit_ib */
.emit_ib = sdma_v3_0_ring_emit_ib,
.emit_fence = sdma_v3_0_ring_emit_fence,
diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
index d029984eb9a1..16fb6f8af872 100644
--- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c
@@ -2133,7 +2133,7 @@ static int sdma_v4_0_print_iv_entry(struct amdgpu_device *adev,
struct amdgpu_iv_entry *entry)
{
int instance;
- struct amdgpu_task_info task_info;
+ struct amdgpu_task_info *task_info;
u64 addr;
instance = sdma_v4_0_irq_id_to_seq(entry->client_id);
@@ -2145,15 +2145,20 @@ static int sdma_v4_0_print_iv_entry(struct amdgpu_device *adev,
addr = (u64)entry->src_data[0] << 12;
addr |= ((u64)entry->src_data[1] & 0xf) << 44;
- memset(&task_info, 0, sizeof(struct amdgpu_task_info));
- amdgpu_vm_get_task_info(adev, entry->pasid, &task_info);
-
dev_dbg_ratelimited(adev->dev,
- "[sdma%d] address:0x%016llx src_id:%u ring:%u vmid:%u "
- "pasid:%u, for process %s pid %d thread %s pid %d\n",
- instance, addr, entry->src_id, entry->ring_id, entry->vmid,
- entry->pasid, task_info.process_name, task_info.tgid,
- task_info.task_name, task_info.pid);
+ "[sdma%d] address:0x%016llx src_id:%u ring:%u vmid:%u pasid:%u\n",
+ instance, addr, entry->src_id, entry->ring_id, entry->vmid,
+ entry->pasid);
+
+ task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid);
+ if (task_info) {
+ dev_dbg_ratelimited(adev->dev,
+ " for process %s pid %d thread %s pid %d\n",
+ task_info->process_name, task_info->tgid,
+ task_info->task_name, task_info->pid);
+ amdgpu_vm_put_task_info(task_info);
+ }
+
return 0;
}
diff --git a/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c b/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c
index eff6e2f1c482..27b2c4c7c63e 100644
--- a/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c
@@ -959,7 +959,7 @@ static const struct amdgpu_ring_funcs vce_v3_0_ring_phys_funcs = {
static const struct amdgpu_ring_funcs vce_v3_0_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCE,
- .align_mask = 0xf,
+ .align_mask = 0x1f,
.nop = VCE_CMD_NO_OP,
.support_64bit_ptrs = false,
.no_user_fence = true,
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c
index 08bdacf8690b..c271f3f0c70a 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v2_0.c
@@ -2013,6 +2013,7 @@ static const struct amd_ip_funcs vcn_v2_0_ip_funcs = {
static const struct amdgpu_ring_funcs vcn_v2_0_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_DEC,
.align_mask = 0xf,
+ .no_user_fence = true,
.secure_submission_supported = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = vcn_v2_0_dec_ring_get_rptr,
@@ -2045,6 +2046,7 @@ static const struct amdgpu_ring_funcs vcn_v2_0_enc_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_ENC,
.align_mask = 0x3f,
.nop = VCN_ENC_CMD_NO_OP,
+ .no_user_fence = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = vcn_v2_0_enc_ring_get_rptr,
.get_wptr = vcn_v2_0_enc_ring_get_wptr,
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c b/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c
index 8a7006d62a87..f2a07794a286 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c
@@ -1527,6 +1527,7 @@ static void vcn_v2_5_dec_ring_set_wptr(struct amdgpu_ring *ring)
static const struct amdgpu_ring_funcs vcn_v2_5_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_DEC,
.align_mask = 0xf,
+ .no_user_fence = true,
.secure_submission_supported = true,
.vmhub = AMDGPU_MMHUB_1,
.get_rptr = vcn_v2_5_dec_ring_get_rptr,
@@ -1558,6 +1559,7 @@ static const struct amdgpu_ring_funcs vcn_v2_5_dec_ring_vm_funcs = {
static const struct amdgpu_ring_funcs vcn_v2_6_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_DEC,
.align_mask = 0xf,
+ .no_user_fence = true,
.secure_submission_supported = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = vcn_v2_5_dec_ring_get_rptr,
@@ -1659,6 +1661,7 @@ static const struct amdgpu_ring_funcs vcn_v2_5_enc_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_ENC,
.align_mask = 0x3f,
.nop = VCN_ENC_CMD_NO_OP,
+ .no_user_fence = true,
.vmhub = AMDGPU_MMHUB_1,
.get_rptr = vcn_v2_5_enc_ring_get_rptr,
.get_wptr = vcn_v2_5_enc_ring_get_wptr,
@@ -1689,6 +1692,7 @@ static const struct amdgpu_ring_funcs vcn_v2_6_enc_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_ENC,
.align_mask = 0x3f,
.nop = VCN_ENC_CMD_NO_OP,
+ .no_user_fence = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = vcn_v2_5_enc_ring_get_rptr,
.get_wptr = vcn_v2_5_enc_ring_get_wptr,
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c
index 478d75b268b1..b73c642b7dc6 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v3_0.c
@@ -1736,6 +1736,7 @@ static const struct amdgpu_ring_funcs vcn_v3_0_dec_sw_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_DEC,
.align_mask = 0x3f,
.nop = VCN_DEC_SW_CMD_NO_OP,
+ .no_user_fence = true,
.secure_submission_supported = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = vcn_v3_0_dec_ring_get_rptr,
@@ -1914,6 +1915,7 @@ static int vcn_v3_0_ring_patch_cs_in_place(struct amdgpu_cs_parser *p,
static const struct amdgpu_ring_funcs vcn_v3_0_dec_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_DEC,
.align_mask = 0xf,
+ .no_user_fence = true,
.secure_submission_supported = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = vcn_v3_0_dec_ring_get_rptr,
@@ -2016,6 +2018,7 @@ static const struct amdgpu_ring_funcs vcn_v3_0_enc_ring_vm_funcs = {
.type = AMDGPU_RING_TYPE_VCN_ENC,
.align_mask = 0x3f,
.nop = VCN_ENC_CMD_NO_OP,
+ .no_user_fence = true,
.vmhub = AMDGPU_MMHUB_0,
.get_rptr = vcn_v3_0_enc_ring_get_rptr,
.get_wptr = vcn_v3_0_enc_ring_get_wptr,
diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
index 2ae426f5e133..608ee2a4d627 100644
--- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
+++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c
@@ -1696,7 +1696,7 @@ static int vcn_v4_0_dec_msg(struct amdgpu_cs_parser *p, struct amdgpu_job *job,
create = ptr + addr + offset - start;
- /* H246, HEVC and VP9 can run on any instance */
+ /* H264, HEVC and VP9 can run on any instance */
if (create[0] == 0x7 || create[0] == 0x10 || create[0] == 0x11)
continue;
@@ -1710,37 +1710,58 @@ static int vcn_v4_0_dec_msg(struct amdgpu_cs_parser *p, struct amdgpu_job *job,
return r;
}
-#define RADEON_VCN_ENGINE_TYPE_DECODE (0x00000003)
+#define RADEON_VCN_ENGINE_TYPE_ENCODE (0x00000002)
+#define RADEON_VCN_ENGINE_TYPE_DECODE (0x00000003)
+#define RADEON_VCN_ENGINE_INFO (0x30000001)
+#define RENCODE_ENCODE_STANDARD_AV1 2
+#define RENCODE_IB_PARAM_SESSION_INIT 0x00000003
+
+/* return the offset in ib if id is found, -1 otherwise */
+static int vcn_v4_0_enc_find_ib_param(struct amdgpu_ib *ib, uint32_t id, int start)
+{
+ int i;
+ uint32_t len;
+
+ for (i = start; (len = amdgpu_ib_get_value(ib, i)) >= 8; i += len / 4) {
+ if (amdgpu_ib_get_value(ib, i + 1) == id)
+ return i;
+ }
+ return -1;
+}
static int vcn_v4_0_ring_patch_cs_in_place(struct amdgpu_cs_parser *p,
struct amdgpu_job *job,
struct amdgpu_ib *ib)
{
struct amdgpu_ring *ring = amdgpu_job_ring(job);
- struct amdgpu_vcn_decode_buffer *decode_buffer;
- uint64_t addr;
uint32_t val;
+ int idx = 0, sidx;
/* The first instance can decode anything */
if (!ring->me)
return 0;
- /* unified queue ib header has 8 double words. */
- if (ib->length_dw < 8)
- return 0;
-
- val = amdgpu_ib_get_value(ib, 6); //RADEON_VCN_ENGINE_TYPE
- if (val != RADEON_VCN_ENGINE_TYPE_DECODE)
- return 0;
-
- decode_buffer = (struct amdgpu_vcn_decode_buffer *)&ib->ptr[10];
-
- if (!(decode_buffer->valid_buf_flag & 0x1))
- return 0;
-
- addr = ((u64)decode_buffer->msg_buffer_address_hi) << 32 |
- decode_buffer->msg_buffer_address_lo;
- return vcn_v4_0_dec_msg(p, job, addr);
+ while ((idx = vcn_v4_0_enc_find_ib_param(ib, RADEON_VCN_ENGINE_INFO, idx)) >= 0) {
+ val = amdgpu_ib_get_value(ib, idx + 2); /* RADEON_VCN_ENGINE_TYPE */
+ if (val == RADEON_VCN_ENGINE_TYPE_DECODE) {
+ uint32_t valid_buf_flag = amdgpu_ib_get_value(ib, idx + 6);
+ uint64_t msg_buffer_addr;
+
+ if (!(valid_buf_flag & 0x1))
+ return 0;
+
+ msg_buffer_addr = ((u64)amdgpu_ib_get_value(ib, idx + 7)) << 32 |
+ amdgpu_ib_get_value(ib, idx + 8);
+ return vcn_v4_0_dec_msg(p, job, msg_buffer_addr);
+ } else if (val == RADEON_VCN_ENGINE_TYPE_ENCODE) {
+ sidx = vcn_v4_0_enc_find_ib_param(ib, RENCODE_IB_PARAM_SESSION_INIT, idx);
+ if (sidx >= 0 &&
+ amdgpu_ib_get_value(ib, sidx + 2) == RENCODE_ENCODE_STANDARD_AV1)
+ return vcn_v4_0_limit_sched(p, job);
+ }
+ idx += amdgpu_ib_get_value(ib, idx) / 4;
+ }
+ return 0;
}
static const struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {
@@ -1748,6 +1769,7 @@ static const struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = {
.align_mask = 0x3f,
.nop = VCN_ENC_CMD_NO_OP,
.vmhub = AMDGPU_MMHUB_0,
+ .no_user_fence = true,
.get_rptr = vcn_v4_0_unified_ring_get_rptr,
.get_wptr = vcn_v4_0_unified_ring_get_wptr,
.set_wptr = vcn_v4_0_unified_ring_set_wptr,
diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c
index 1bfd7b49fe9c..4ad7db4e414d 100644
--- a/drivers/gpu/drm/amd/amdgpu/vi.c
+++ b/drivers/gpu/drm/amd/amdgpu/vi.c
@@ -1432,10 +1432,31 @@ static uint64_t vi_get_pcie_replay_count(struct amdgpu_device *adev)
return (nak_r + nak_g);
}
+struct vi_reset_quirk {
+ u16 device;
+ u16 subsystem_vendor;
+ u16 subsystem_device;
+ u8 revision;
+};
+
+static const struct vi_reset_quirk vi_reset_quirks[] = {
+ { 0x67ef, PCI_VENDOR_ID_APPLE, 0x0190, 0xe3 }, /* Radeon Pro 555X */
+ { 0x67ef, PCI_VENDOR_ID_APPLE, 0x018f, 0xc2 }, /* Radeon Pro 560X */
+ { 0x67ef, PCI_VENDOR_ID_APPLE, 0x017a, 0xc7 }, /* Radeon Pro 555 */
+};
+
static bool vi_need_reset_on_init(struct amdgpu_device *adev)
{
+ unsigned int i;
u32 clock_cntl, pc;
+ for (i = 0; i < ARRAY_SIZE(vi_reset_quirks); i++)
+ if (adev->pdev->device == vi_reset_quirks[i].device &&
+ adev->pdev->subsystem_vendor == vi_reset_quirks[i].subsystem_vendor &&
+ adev->pdev->subsystem_device == vi_reset_quirks[i].subsystem_device &&
+ adev->pdev->revision == vi_reset_quirks[i].revision)
+ return true;
+
if (adev->flags & AMD_IS_APU)
return false;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
index 4853a8b43f08..0826f8a1da65 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c
@@ -772,6 +772,9 @@ static int kfd_ioctl_get_process_apertures_new(struct file *filp,
goto out_unlock;
}
+ if (args->num_of_nodes > kfd_topology_get_num_devices())
+ return -EINVAL;
+
/* Fill in process-aperture information for all available
* nodes, but not more than args->num_of_nodes as that is
* the amount of memory allocated by user
@@ -1504,16 +1507,10 @@ static int kfd_ioctl_get_dmabuf_info(struct file *filep,
if (!dev)
return -EINVAL;
- if (args->metadata_ptr) {
- metadata_buffer = kzalloc(args->metadata_size, GFP_KERNEL);
- if (!metadata_buffer)
- return -ENOMEM;
- }
-
/* Get dmabuf info from KGD */
r = amdgpu_amdkfd_get_dmabuf_info(dev->adev, args->dmabuf_fd,
&dmabuf_adev, &args->size,
- metadata_buffer, args->metadata_size,
+ &metadata_buffer, args->metadata_size,
&args->metadata_size, &flags);
if (r)
goto exit;
@@ -1528,7 +1525,7 @@ static int kfd_ioctl_get_dmabuf_info(struct file *filep,
args->flags = flags;
/* Copy metadata buffer to user mode */
- if (metadata_buffer) {
+ if (metadata_buffer && args->metadata_ptr) {
r = copy_to_user((void __user *)args->metadata_ptr,
metadata_buffer, args->metadata_size);
if (r != 0)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
index e45c6bc8d10b..78a5119c86ec 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c
@@ -1316,6 +1316,23 @@ int kfd_parse_crat_table(void *crat_image, struct list_head *device_list,
sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1);
while ((char *)sub_type_hdr + sizeof(struct crat_subtype_generic) <
((char *)crat_image) + image_len) {
+ if (!sub_type_hdr->length ||
+ sub_type_hdr->length < sizeof(struct crat_subtype_generic)) {
+ pr_warn("Invalid CRAT subtype length %u\n",
+ sub_type_hdr->length);
+ ret = -EINVAL;
+ break;
+ }
+
+ /* Validate subtype fits within remaining image */
+ if ((char *)sub_type_hdr + sub_type_hdr->length >
+ (char *)crat_image + image_len) {
+ pr_warn("CRAT subtype length %u exceeds image bounds\n",
+ sub_type_hdr->length);
+ ret = -EINVAL;
+ break;
+ }
+
if (sub_type_hdr->flags & CRAT_SUBTYPE_FLAGS_ENABLED) {
ret = kfd_parse_subtype(sub_type_hdr, device_list);
if (ret)
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device.c b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
index e2c055abfea9..a4eadd6416f8 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device.c
@@ -1177,6 +1177,11 @@ int kfd_debugfs_hang_hws(struct kfd_dev *dev)
return -EINVAL;
}
+ if (dev->shared_resources.enable_mes) {
+ dev_err(dev->adev->dev, "Inducing MES hang is not supported\n");
+ return -EINVAL;
+ }
+
return dqm_debugfs_hang_hws(dev->dqm);
}
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
index 689abc3d3a39..8d48fc01ad35 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c
@@ -1005,6 +1005,15 @@ static int evict_process_queues_cpsch(struct device_queue_manager *dqm,
}
}
}
+
+ /*
+ * Heavy-weight TLB flush after MES removes queues to ensure
+ * in-flight memory accesses complete before memory is freed/migrated.
+ * HWS does this automatically, MES does not.
+ */
+ if (dqm->dev->shared_resources.enable_mes && pdd->drm_file)
+ kfd_flush_tlb(pdd, TLB_FLUSH_HEAVYWEIGHT);
+
pdd->last_evict_timestamp = get_jiffies_64();
if (!dqm->dev->shared_resources.enable_mes)
retval = execute_queues_cpsch(dqm,
@@ -2437,6 +2446,11 @@ static void kfd_process_hw_exception(struct work_struct *work)
amdgpu_amdkfd_gpu_reset(dqm->dev->adev);
}
+size_t mqd_size_from_queue_type(struct device_queue_manager *dqm, enum kfd_queue_type type)
+{
+ return dqm->mqd_mgrs[get_mqd_type_from_queue_type(type)]->mqd_size;
+}
+
#if defined(CONFIG_DEBUG_FS)
static void seq_reg_dump(struct seq_file *m,
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
index a537b9ef3e16..593f5d273aa9 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h
@@ -279,6 +279,8 @@ unsigned int get_queues_per_pipe(struct device_queue_manager *dqm);
unsigned int get_pipes_per_mec(struct device_queue_manager *dqm);
unsigned int get_num_sdma_queues(struct device_queue_manager *dqm);
unsigned int get_num_xgmi_sdma_queues(struct device_queue_manager *dqm);
+size_t mqd_size_from_queue_type(struct device_queue_manager *dqm,
+ enum kfd_queue_type type);
static inline unsigned int get_sh_mem_bases_32(struct kfd_process_device *pdd)
{
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
index 72e84b26370e..c964850b667f 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c
@@ -132,7 +132,7 @@ svm_migrate_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys,
uint64_t gart_s, gart_d;
struct dma_fence *next;
uint64_t size;
- int r;
+ int r = 0;
mutex_lock(&adev->mman.gtt_window_lock);
@@ -252,15 +252,6 @@ svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long addr)
return page;
}
-static void svm_migrate_put_sys_page(unsigned long addr)
-{
- struct page *page;
-
- page = pfn_to_page(addr >> PAGE_SHIFT);
- unlock_page(page);
- put_page(page);
-}
-
static unsigned long svm_migrate_successful_pages(struct migrate_vma *migrate)
{
unsigned long cpages = 0;
@@ -298,7 +289,7 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
dma_addr_t *src;
uint64_t *dst;
uint64_t i, j;
- int r;
+ int r = 0;
pr_debug("svms 0x%p [0x%lx 0x%lx 0x%llx]\n", prange->svms, prange->start,
prange->last, ttm_res_offset);
@@ -322,6 +313,7 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
DMA_BIDIRECTIONAL);
r = dma_mapping_error(dev, src[i]);
if (r) {
+ src[i] = 0;
dev_err(adev->dev, "%s: fail %d dma_map_page\n",
__func__, r);
goto out_free_vram_pages;
@@ -346,7 +338,8 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
pr_debug_ratelimited("dma mapping src to 0x%llx, pfn 0x%lx\n",
src[i] >> PAGE_SHIFT, page_to_pfn(spage));
- if (j >= (cursor.size >> PAGE_SHIFT) - 1 && i < npages - 1) {
+ /* accumulated j + 1 pages reach end of current drm_buddy_block */
+ if (j + 1 >= (cursor.size >> PAGE_SHIFT)) {
r = svm_migrate_copy_memory_gart(adev, src + i - j,
dst + i - j, j + 1,
FROM_RAM_TO_VRAM,
@@ -360,7 +353,8 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange,
}
}
- r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j,
+ if (j > 0)
+ r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j,
FROM_RAM_TO_VRAM, mfence);
out_free_vram_pages:
@@ -566,9 +560,10 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
dma_addr_t *scratch, uint64_t npages)
{
struct device *dev = adev->dev;
- uint64_t *src;
+ struct page *dpage = NULL;
dma_addr_t *dst;
- struct page *dpage;
+ uint64_t *src;
+
uint64_t i = 0, j;
uint64_t addr;
int r = 0;
@@ -622,6 +617,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
r = dma_mapping_error(dev, dst[i]);
if (r) {
dev_err(adev->dev, "%s: fail %d dma_map_page\n", __func__, r);
+ dst[i] = 0;
goto out_oom;
}
@@ -629,17 +625,39 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange,
dst[i] >> PAGE_SHIFT, page_to_pfn(dpage));
migrate->dst[i] = migrate_pfn(page_to_pfn(dpage));
+
+ dpage = NULL;
j++;
}
- r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j,
- FROM_VRAM_TO_RAM, mfence);
-
+ if (j > 0)
+ r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j,
+ FROM_VRAM_TO_RAM, mfence);
out_oom:
if (r) {
pr_debug("failed %d copy to ram\n", r);
+
+ /* first release current dpage when dma_map_page fail */
+ if (dpage) {
+ unlock_page(dpage);
+ put_page(dpage);
+ }
+
+ /* release previous allocated sys pages and unmap dma address */
while (i--) {
- svm_migrate_put_sys_page(dst[i]);
+
+ if (dst[i]) {
+ dma_unmap_page(dev, dst[i], PAGE_SIZE,
+ DMA_BIDIRECTIONAL);
+ dst[i] = 0;
+ }
+
+ dpage = migrate_pfn_to_page(migrate->dst[i]);
+ if (!dpage)
+ continue;
+
+ unlock_page(dpage);
+ put_page(dpage);
migrate->dst[i] = 0;
}
}
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
index ac4a344bacd8..cdcaba185052 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h
@@ -404,7 +404,8 @@ enum kfd_queue_type {
KFD_QUEUE_TYPE_SDMA,
KFD_QUEUE_TYPE_HIQ,
KFD_QUEUE_TYPE_DIQ,
- KFD_QUEUE_TYPE_SDMA_XGMI
+ KFD_QUEUE_TYPE_SDMA_XGMI,
+ KFD_QUEUE_TYPE_MAX,
};
enum kfd_queue_format {
@@ -1048,6 +1049,7 @@ struct kfd_dev *kfd_device_by_id(uint32_t gpu_id);
struct kfd_dev *kfd_device_by_pci_dev(const struct pci_dev *pdev);
struct kfd_dev *kfd_device_by_adev(const struct amdgpu_device *adev);
int kfd_topology_enum_kfd_devices(uint8_t idx, struct kfd_dev **kdev);
+uint32_t kfd_topology_get_num_devices(void);
int kfd_numa_node_to_apic_id(int numa_node_id);
void kfd_double_confirm_iommu_support(struct kfd_dev *gpu);
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
index a15bf1e38276..7c85bb9e270e 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c
@@ -829,6 +829,23 @@ int kfd_criu_restore_queue(struct kfd_process *p,
goto exit;
}
+ pdd = kfd_process_device_data_by_id(p, q_data->gpu_id);
+ if (!pdd) {
+ pr_err("Failed to get pdd\n");
+ ret = -EINVAL;
+ goto exit;
+ }
+
+ if (q_data->type >= KFD_QUEUE_TYPE_MAX) {
+ ret = -EINVAL;
+ goto exit;
+ }
+
+ if (q_data->mqd_size != mqd_size_from_queue_type(pdd->dev->dqm, q_data->type)) {
+ ret = -EINVAL;
+ goto exit;
+ }
+
*priv_data_offset += sizeof(*q_data);
q_extra_data_size = (uint64_t)q_data->ctl_stack_size + q_data->mqd_size;
@@ -851,13 +868,6 @@ int kfd_criu_restore_queue(struct kfd_process *p,
*priv_data_offset += q_extra_data_size;
- pdd = kfd_process_device_data_by_id(p, q_data->gpu_id);
- if (!pdd) {
- pr_err("Failed to get pdd\n");
- ret = -EINVAL;
- goto exit;
- }
-
if (!pdd->doorbell_index &&
kfd_alloc_process_doorbells(pdd->dev, &pdd->doorbell_index) < 0) {
ret = -ENOMEM;
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
index 0472b56de245..60007dc2c276 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c
@@ -238,16 +238,17 @@ void kfd_smi_event_update_thermal_throttling(struct kfd_dev *dev,
void kfd_smi_event_update_vmfault(struct kfd_dev *dev, uint16_t pasid)
{
- struct amdgpu_task_info task_info;
-
- memset(&task_info, 0, sizeof(struct amdgpu_task_info));
- amdgpu_vm_get_task_info(dev->adev, pasid, &task_info);
- /* Report VM faults from user applications, not retry from kernel */
- if (!task_info.pid)
- return;
-
- kfd_smi_event_add(0, dev, KFD_SMI_EVENT_VMFAULT, "%x:%s\n",
- task_info.pid, task_info.task_name);
+ struct amdgpu_task_info *task_info;
+
+ task_info = amdgpu_vm_get_task_info_pasid(dev->adev, pasid);
+ if (task_info) {
+ /* Report VM faults from user applications, not retry from kernel */
+ if (task_info->pid)
+ kfd_smi_event_add(task_info->tgid, dev,
+ KFD_SMI_EVENT_VMFAULT, "%x:%s\n",
+ task_info->pid, task_info->task_name);
+ amdgpu_vm_put_task_info(task_info);
+ }
}
void kfd_smi_event_page_fault_start(struct kfd_dev *dev, pid_t pid,
diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
index d841200a405b..1960792da066 100644
--- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
+++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c
@@ -2109,6 +2109,17 @@ int kfd_topology_remove_device(struct kfd_dev *gpu)
return res;
}
+uint32_t kfd_topology_get_num_devices(void)
+{
+ uint32_t num_devices;
+
+ down_read(&topology_lock);
+ num_devices = sys_props.num_devices;
+ up_read(&topology_lock);
+
+ return num_devices;
+}
+
/* kfd_topology_enum_kfd_devices - Enumerate through all devices in KFD
* topology. If GPU device is found @idx, then valid kfd_dev pointer is
* returned through @kdev
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
index deaf98957cb7..14f2cfcba5f6 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c
@@ -1881,19 +1881,19 @@ static void amdgpu_dm_fini(struct amdgpu_device *adev)
hdcp_destroy(&adev->dev->kobj, adev->dm.hdcp_workqueue);
adev->dm.hdcp_workqueue = NULL;
}
+#endif
- if (adev->dm.dc)
+ if (adev->dm.dc) {
+#ifdef CONFIG_DRM_AMD_DC_HDCP
dc_deinit_callbacks(adev->dm.dc);
#endif
-
- if (adev->dm.dc)
dc_dmub_srv_destroy(&adev->dm.dc->ctx->dmub_srv);
-
- if (dc_enable_dmub_notifications(adev->dm.dc)) {
- kfree(adev->dm.dmub_notify);
- adev->dm.dmub_notify = NULL;
- destroy_workqueue(adev->dm.delayed_hpd_wq);
- adev->dm.delayed_hpd_wq = NULL;
+ if (dc_enable_dmub_notifications(adev->dm.dc)) {
+ kfree(adev->dm.dmub_notify);
+ adev->dm.dmub_notify = NULL;
+ destroy_workqueue(adev->dm.delayed_hpd_wq);
+ adev->dm.delayed_hpd_wq = NULL;
+ }
}
if (adev->dm.dmub_bo)
@@ -2645,9 +2645,14 @@ static int dm_suspend(void *handle)
dc_allow_idle_optimizations(adev->dm.dc, false);
dm->cached_dc_state = dc_copy_state(dm->dc->current_state);
+ if (!dm->cached_dc_state) {
+ drm_err(adev_to_drm(adev),
+ "Failed to allocate cached DC state during suspend\n");
+ mutex_unlock(&dm->dc_lock);
+ return -ENOMEM;
+ }
- if (dm->cached_dc_state)
- dm_gpureset_toggle_interrupts(adev, dm->cached_dc_state, false);
+ dm_gpureset_toggle_interrupts(adev, dm->cached_dc_state, false);
amdgpu_dm_commit_zero_streams(dm->dc);
@@ -5602,11 +5607,16 @@ decide_crtc_timing_for_drm_display_mode(struct drm_display_mode *drm_mode,
const struct drm_display_mode *native_mode,
bool scale_enabled)
{
+ /* Inserted modes copy the native sync; a sink mode with equal totals may not */
if (scale_enabled) {
copy_crtc_timing_for_drm_display_mode(native_mode, drm_mode);
} else if (native_mode->clock == drm_mode->clock &&
native_mode->htotal == drm_mode->htotal &&
- native_mode->vtotal == drm_mode->vtotal) {
+ native_mode->vtotal == drm_mode->vtotal &&
+ native_mode->hsync_start == drm_mode->hsync_start &&
+ native_mode->hsync_end == drm_mode->hsync_end &&
+ native_mode->vsync_start == drm_mode->vsync_start &&
+ native_mode->vsync_end == drm_mode->vsync_end) {
copy_crtc_timing_for_drm_display_mode(native_mode, drm_mode);
} else {
/* no scaling nor amdgpu inserted, no need to patch */
@@ -8618,9 +8628,10 @@ static void amdgpu_dm_atomic_commit_tail(struct drm_atomic_state *state)
DRM_INFO("[HDCP_DM] hdcp_update_display enable_encryption = %x\n", enable_encryption);
- hdcp_update_display(
- adev->dm.hdcp_workqueue, aconnector->dc_link->link_index, aconnector,
- new_con_state->hdcp_content_type, enable_encryption);
+ if (aconnector->dc_link)
+ hdcp_update_display(
+ adev->dm.hdcp_workqueue, aconnector->dc_link->link_index, aconnector,
+ new_con_state->hdcp_content_type, enable_encryption);
}
}
#endif
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
index 12385b6f8443..6036b138245b 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h
@@ -43,7 +43,7 @@
* in amdgpu_dm_kms.h file
*/
-#define AMDGPU_DM_MAX_DISPLAY_INDEX 31
+#define AMDGPU_DM_MAX_DISPLAY_COUNT 32
#define AMDGPU_DM_MAX_CRTC 6
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
index 1ec643a0d00d..ca2edba4c0f7 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c
@@ -278,13 +278,13 @@ static void dm_crtc_reset_state(struct drm_crtc *crtc)
{
struct dm_crtc_state *state;
- if (crtc->state)
- dm_crtc_destroy_state(crtc, crtc->state);
-
state = kzalloc(sizeof(*state), GFP_KERNEL);
- if (WARN_ON(!state))
+ if (!state)
return;
+ if (crtc->state)
+ dm_crtc_destroy_state(crtc, crtc->state);
+
__drm_atomic_helper_crtc_reset(crtc, &state->base);
}
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
index 6110d88efdbb..678ee48075e9 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c
@@ -256,7 +256,7 @@ void hdcp_reset_display(struct hdcp_workqueue *hdcp_work, unsigned int link_inde
cancel_delayed_work(&hdcp_w->property_validate_dwork);
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; conn_index++) {
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) {
hdcp_w->encryption_status[conn_index] =
MOD_HDCP_ENCRYPTION_STATUS_HDCP_OFF;
if (hdcp_w->aconnector[conn_index]) {
@@ -303,7 +303,7 @@ static void event_property_update(struct work_struct *work)
struct drm_connector *connector;
struct drm_connector_state *conn_state;
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; conn_index++) {
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) {
aconnector = hdcp_work->aconnector[conn_index];
if (!aconnector)
@@ -375,7 +375,7 @@ static void event_property_validate(struct work_struct *work)
guard(mutex)(&hdcp_work->mutex);
- for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX;
+ for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT;
conn_index++) {
aconnector = hdcp_work->aconnector[conn_index];
@@ -753,10 +753,10 @@ struct hdcp_workqueue *hdcp_create_workqueue(struct amdgpu_device *adev,
memset(hdcp_work[i].aconnector, 0,
sizeof(struct amdgpu_dm_connector *) *
- AMDGPU_DM_MAX_DISPLAY_INDEX);
+ AMDGPU_DM_MAX_DISPLAY_COUNT);
memset(hdcp_work[i].encryption_status, 0,
sizeof(enum mod_hdcp_encryption_status) *
- AMDGPU_DM_MAX_DISPLAY_INDEX);
+ AMDGPU_DM_MAX_DISPLAY_COUNT);
}
cp_psp->funcs.update_stream_config = update_config;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
index 69b445b011c8..d18c4cc7d69f 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h
@@ -43,7 +43,7 @@ struct hdcp_workqueue {
struct delayed_work callback_dwork;
struct delayed_work watchdog_timer_dwork;
struct delayed_work property_validate_dwork;
- struct amdgpu_dm_connector *aconnector[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ struct amdgpu_dm_connector *aconnector[AMDGPU_DM_MAX_DISPLAY_COUNT];
struct mutex mutex;
struct mod_hdcp hdcp;
@@ -51,7 +51,7 @@ struct hdcp_workqueue {
struct mod_hdcp_display display;
struct mod_hdcp_link link;
- enum mod_hdcp_encryption_status encryption_status[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ enum mod_hdcp_encryption_status encryption_status[AMDGPU_DM_MAX_DISPLAY_COUNT];
/* when display is unplugged from mst hub, connctor will be
* destroyed within dm_dp_mst_connector_destroy. connector
* hdcp perperties, like type, undesired, desired, enabled,
@@ -61,9 +61,9 @@ struct hdcp_workqueue {
* will be retrieved from hdcp_work within dm_dp_mst_get_modes
*/
/* un-desired, desired, enabled */
- unsigned int content_protection[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ unsigned int content_protection[AMDGPU_DM_MAX_DISPLAY_COUNT];
/* hdcp1.x, hdcp2.x */
- unsigned int hdcp_content_type[AMDGPU_DM_MAX_DISPLAY_INDEX];
+ unsigned int hdcp_content_type[AMDGPU_DM_MAX_DISPLAY_COUNT];
uint8_t max_link;
diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
index b3a1dbeac483..bf8084619264 100644
--- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
+++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c
@@ -1336,19 +1336,18 @@ static const struct drm_plane_helper_funcs dm_plane_helper_funcs = {
static void dm_drm_plane_reset(struct drm_plane *plane)
{
- struct dm_plane_state *amdgpu_state = NULL;
+ struct dm_plane_state *amdgpu_state;
+
+ amdgpu_state = kzalloc(sizeof(*amdgpu_state), GFP_KERNEL);
+ if (!amdgpu_state)
+ return;
if (plane->state)
plane->funcs->atomic_destroy_state(plane, plane->state);
- amdgpu_state = kzalloc(sizeof(*amdgpu_state), GFP_KERNEL);
- WARN_ON(amdgpu_state == NULL);
-
- if (amdgpu_state)
- __drm_atomic_helper_plane_reset(plane, &amdgpu_state->base);
+ __drm_atomic_helper_plane_reset(plane, &amdgpu_state->base);
#ifdef CONFIG_DRM_AMD_DC_HDR
- if (amdgpu_state)
- amdgpu_state->sdr_boost = DEFAULT_SDR_BOOST;
+ amdgpu_state->sdr_boost = DEFAULT_SDR_BOOST;
#endif
}
diff --git a/drivers/gpu/drm/amd/display/dc/clk_mgr/dce110/dce110_clk_mgr.c b/drivers/gpu/drm/amd/display/dc/clk_mgr/dce110/dce110_clk_mgr.c
index bce53ab36f3e..d50b9440210e 100644
--- a/drivers/gpu/drm/amd/display/dc/clk_mgr/dce110/dce110_clk_mgr.c
+++ b/drivers/gpu/drm/amd/display/dc/clk_mgr/dce110/dce110_clk_mgr.c
@@ -218,7 +218,7 @@ void dce11_pplib_apply_display_requirements(
* , then change minimum memory clock based on real-time bandwidth
* limitation.
*/
- if ((dc->ctx->asic_id.chip_family == FAMILY_AI) &&
+ if (dc->bw_vbios && (dc->ctx->asic_id.chip_family == FAMILY_AI) &&
ASICREV_IS_VEGA20_P(dc->ctx->asic_id.hw_internal_rev) && (context->stream_count >= 2)) {
pp_display_cfg->min_memory_clock_khz = max(pp_display_cfg->min_memory_clock_khz,
(uint32_t) div64_s64(
diff --git a/drivers/gpu/drm/amd/display/dc/core/dc.c b/drivers/gpu/drm/amd/display/dc/core/dc.c
index 6a415e2e6b8e..cfcd4ed1666d 100644
--- a/drivers/gpu/drm/amd/display/dc/core/dc.c
+++ b/drivers/gpu/drm/amd/display/dc/core/dc.c
@@ -831,16 +831,18 @@ static void dc_destruct(struct dc *dc)
dc_destroy_resource_pool(dc);
- if (dc->ctx->gpio_service)
- dal_gpio_service_destroy(&dc->ctx->gpio_service);
+ if (dc->ctx) {
+ if (dc->ctx->gpio_service)
+ dal_gpio_service_destroy(&dc->ctx->gpio_service);
- if (dc->ctx->created_bios)
- dal_bios_parser_destroy(&dc->ctx->dc_bios);
+ if (dc->ctx->created_bios)
+ dal_bios_parser_destroy(&dc->ctx->dc_bios);
- dc_perf_trace_destroy(&dc->ctx->perf_trace);
+ dc_perf_trace_destroy(&dc->ctx->perf_trace);
- kfree(dc->ctx);
- dc->ctx = NULL;
+ kfree(dc->ctx);
+ dc->ctx = NULL;
+ }
kfree(dc->bw_vbios);
dc->bw_vbios = NULL;
@@ -2283,7 +2285,8 @@ static void dc_state_free(struct kref *kref)
void dc_release_state(struct dc_state *context)
{
- kref_put(&context->refcount, dc_state_free);
+ if (context != NULL)
+ kref_put(&context->refcount, dc_state_free);
}
bool dc_set_generic_gpio_for_stereo(bool enable,
diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_link_dp.c b/drivers/gpu/drm/amd/display/dc/core/dc_link_dp.c
index 82b747c0ed69..fa0ae49fa22b 100644
--- a/drivers/gpu/drm/amd/display/dc/core/dc_link_dp.c
+++ b/drivers/gpu/drm/amd/display/dc/core/dc_link_dp.c
@@ -1676,10 +1676,10 @@ static enum dc_status configure_lttpr_mode_non_transparent(
/* Driver does not need to train the first hop. Skip DPCD read and clear
* AUX_RD_INTERVAL for DPTX-to-DPIA hop.
*/
- if (link->ep_type == DISPLAY_ENDPOINT_USB4_DPIA)
+ if (link->ep_type == DISPLAY_ENDPOINT_USB4_DPIA && repeater_cnt > 0 && repeater_cnt < MAX_REPEATER_CNT)
link->dpcd_caps.lttpr_caps.aux_rd_interval[--repeater_cnt] = 0;
- for (repeater_id = repeater_cnt; repeater_id > 0; repeater_id--) {
+ for (repeater_id = repeater_cnt; repeater_id > 0 && repeater_id < MAX_REPEATER_CNT; repeater_id--) {
aux_interval_address = DP_TRAINING_AUX_RD_INTERVAL_PHY_REPEATER1 +
((DP_REPEATER_CONFIGURATION_AND_STATUS_SIZE) * (repeater_id - 1));
core_link_read_dpcd(
@@ -2808,8 +2808,7 @@ bool perform_link_training_with_retries(
if (status == LINK_TRAINING_ABORT) {
enum dc_connection_type type = dc_connection_none;
- dc_link_detect_sink(link, &type);
- if (type == dc_connection_none) {
+ if (dc_link_detect_sink(link, &type) && type == dc_connection_none) {
DC_LOG_HW_LINK_TRAINING("%s: Aborting training because sink unplugged\n", __func__);
break;
}
@@ -3331,7 +3330,7 @@ bool dp_verify_link_cap_with_retries(
memset(&link->verified_link_cap, 0,
sizeof(struct dc_link_settings));
- if (!dc_link_detect_sink(link, &type) || type == dc_connection_none) {
+ if (link->link_enc && (!dc_link_detect_sink(link, &type) || type == dc_connection_none)) {
link->verified_link_cap = fail_safe_link_settings;
break;
} else if (dp_verify_link_cap(link, known_limit_link_setting,
@@ -6188,8 +6187,11 @@ bool dc_link_dp_set_test_pattern(
break;
}
+ if (!pipe_ctx->stream)
+ return false;
+
if (pipe_ctx->stream_res.tg->funcs->lock_doublebuffer_enable) {
- if (pipe_ctx->stream && should_use_dmub_lock(pipe_ctx->stream->link)) {
+ if (should_use_dmub_lock(pipe_ctx->stream->link)) {
union dmub_hw_lock_flags hw_locks = { 0 };
struct dmub_hw_lock_inst_flags inst_flags = { 0 };
diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_link_dpcd.c b/drivers/gpu/drm/amd/display/dc/core/dc_link_dpcd.c
index aefca9756dbe..f3838286c172 100644
--- a/drivers/gpu/drm/amd/display/dc/core/dc_link_dpcd.c
+++ b/drivers/gpu/drm/amd/display/dc/core/dc_link_dpcd.c
@@ -161,7 +161,8 @@ static void dpcd_extend_address_range(
if (new_addr_range.start != in_address || new_addr_range.end != end_address) {
*out_address = new_addr_range.start;
*out_size = ADDRESS_RANGE_SIZE(new_addr_range.start, new_addr_range.end);
- *out_data = kzalloc(*out_size * sizeof(**out_data), GFP_KERNEL);
+ *out_data = kcalloc(*out_size, sizeof(**out_data), GFP_KERNEL);
+ ASSERT(*out_data);
}
}
diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_resource.c b/drivers/gpu/drm/amd/display/dc/core/dc_resource.c
index 876b70157faa..5adae27fb460 100644
--- a/drivers/gpu/drm/amd/display/dc/core/dc_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/core/dc_resource.c
@@ -2115,7 +2115,7 @@ static int acquire_first_free_pipe(
pipe_ctx->plane_res.mpcc_inst = pool->dpps[i]->inst;
pipe_ctx->pipe_idx = i;
- if (i >= pool->timing_generator_count) {
+ if (i >= pool->timing_generator_count && pool->timing_generator_count != 0) {
int tg_inst = pool->timing_generator_count - 1;
pipe_ctx->stream_res.tg = pool->timing_generators[tg_inst];
diff --git a/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c b/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c
index 31bb7e782c6b..51e2debcf1c8 100644
--- a/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c
+++ b/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c
@@ -396,7 +396,8 @@ void dc_dmub_srv_get_visual_confirm_color_cmd(struct dc *dc, struct pipe_ctx *pi
enum dmub_status status;
unsigned int panel_inst = 0;
- dc_get_edp_link_panel_inst(dc, pipe_ctx->stream->link, &panel_inst);
+ if (!dc_get_edp_link_panel_inst(dc, pipe_ctx->stream->link, &panel_inst))
+ return;
memset(&cmd, 0, sizeof(cmd));
@@ -803,7 +804,10 @@ void dc_dmub_srv_log_diagnostic_data(struct dc_dmub_srv *dc_dmub_srv)
{
struct dmub_diagnostic_data diag_data = {0};
- if (!dc_dmub_srv || !dc_dmub_srv->dmub) {
+ if (!dc_dmub_srv)
+ return;
+
+ if (!dc_dmub_srv->dmub) {
DC_LOG_ERROR("%s: invalid parameters.", __func__);
return;
}
@@ -1033,3 +1037,35 @@ bool dc_dmub_check_min_version(struct dmub_srv *srv)
return true;
return srv->hw_funcs.is_psrsu_supported(srv);
}
+
+void dc_dmub_srv_enable_dpia_trace(const struct dc *dc)
+{
+ struct dc_dmub_srv *dc_dmub_srv = dc->ctx->dmub_srv;
+ struct dmub_srv *dmub;
+ enum dmub_status status;
+ static const uint32_t timeout_us = 30;
+
+ if (!dc_dmub_srv)
+ return;
+
+ if (!dc_dmub_srv->dmub) {
+ DC_LOG_ERROR("%s: invalid parameters.", __func__);
+ return;
+ }
+
+ dmub = dc_dmub_srv->dmub;
+
+ status = dmub_srv_send_gpint_command(dmub, DMUB_GPINT__SET_TRACE_BUFFER_MASK_WORD1, 0x0010, timeout_us);
+ if (status != DMUB_STATUS_OK) {
+ DC_LOG_ERROR("timeout updating trace buffer mask word\n");
+ return;
+ }
+
+ status = dmub_srv_send_gpint_command(dmub, DMUB_GPINT__UPDATE_TRACE_BUFFER_MASK, 0x0000, timeout_us);
+ if (status != DMUB_STATUS_OK) {
+ DC_LOG_ERROR("timeout updating trace buffer mask word\n");
+ return;
+ }
+
+ DC_LOG_DEBUG("Enabled DPIA trace\n");
+}
\ No newline at end of file
diff --git a/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h b/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h
index 9a248ced03b9..452fb1c2f5a6 100644
--- a/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h
+++ b/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.h
@@ -90,4 +90,7 @@ void dc_dmub_srv_log_diagnostic_data(struct dc_dmub_srv *dc_dmub_srv);
void dc_send_update_cursor_info_to_dmu(struct pipe_ctx *pCtx, uint8_t pipe_idx);
bool dc_dmub_check_min_version(struct dmub_srv *srv);
+
+void dc_dmub_srv_enable_dpia_trace(const struct dc *dc);
+
#endif /* _DMUB_DC_SRV_H_ */
diff --git a/drivers/gpu/drm/amd/display/dc/dc_link.h b/drivers/gpu/drm/amd/display/dc/dc_link.h
index 17f080f8af6c..7a9e8e63b1da 100644
--- a/drivers/gpu/drm/amd/display/dc/dc_link.h
+++ b/drivers/gpu/drm/amd/display/dc/dc_link.h
@@ -285,6 +285,9 @@ const struct dc_link_status *dc_link_get_status(const struct dc_link *dc_link);
*/
static inline struct dc_link *dc_get_link_at_index(struct dc *dc, uint32_t link_index)
{
+ if (link_index >= ARRAY_SIZE(dc->links))
+ return NULL;
+
return dc->links[link_index];
}
diff --git a/drivers/gpu/drm/amd/display/dc/dc_types.h b/drivers/gpu/drm/amd/display/dc/dc_types.h
index 6050a3469a57..579a38478d0f 100644
--- a/drivers/gpu/drm/amd/display/dc/dc_types.h
+++ b/drivers/gpu/drm/amd/display/dc/dc_types.h
@@ -403,7 +403,7 @@ struct dc_lttpr_caps {
uint8_t max_ext_timeout;
union dp_main_link_channel_coding_lttpr_cap main_link_channel_coding;
union dp_128b_132b_supported_lttpr_link_rates supported_128b_132b_rates;
- uint8_t aux_rd_interval[MAX_REPEATER_CNT - 1];
+ uint8_t aux_rd_interval[MAX_REPEATER_CNT];
};
struct dc_dongle_dfp_cap_ext {
diff --git a/drivers/gpu/drm/amd/display/dc/dce/Makefile b/drivers/gpu/drm/amd/display/dc/dce/Makefile
index 0d7db132a20f..01490c9ba958 100644
--- a/drivers/gpu/drm/amd/display/dc/dce/Makefile
+++ b/drivers/gpu/drm/amd/display/dc/dce/Makefile
@@ -29,7 +29,7 @@
DCE = dce_audio.o dce_stream_encoder.o dce_link_encoder.o dce_hwseq.o \
dce_mem_input.o dce_clock_source.o dce_scl_filters.o dce_transform.o \
dce_opp.o dce_dmcu.o dce_abm.o dce_ipp.o dce_aux.o \
-dce_i2c.o dce_i2c_hw.o dce_i2c_sw.o dmub_psr.o dmub_abm.o dce_panel_cntl.o \
+dce_i2c.o dce_i2c_hw.o dce_i2c_sw.o dmub_psr.o dmub_abm.o dmub_abm_lcd.o dce_panel_cntl.o \
dmub_hw_lock_mgr.o dmub_outbox.o
AMD_DAL_DCE = $(addprefix $(AMDDALPATH)/dc/dce/,$(DCE))
diff --git a/drivers/gpu/drm/amd/display/dc/dce/dmub_abm.c b/drivers/gpu/drm/amd/display/dc/dce/dmub_abm.c
index fb0dec4ed3a6..3b85fc665f41 100644
--- a/drivers/gpu/drm/amd/display/dc/dce/dmub_abm.c
+++ b/drivers/gpu/drm/amd/display/dc/dce/dmub_abm.c
@@ -24,212 +24,139 @@
*/
#include "dmub_abm.h"
-#include "dce_abm.h"
+#include "dmub_abm_lcd.h"
#include "dc.h"
-#include "dc_dmub_srv.h"
-#include "dmub/dmub_srv.h"
#include "core_types.h"
-#include "dm_services.h"
-#include "reg_helper.h"
-#include "fixed31_32.h"
-
-#include "atom.h"
#define TO_DMUB_ABM(abm)\
container_of(abm, struct dce_abm, base)
-#define REG(reg) \
- (dce_abm->regs->reg)
-
-#undef FN
-#define FN(reg_name, field_name) \
- dce_abm->abm_shift->field_name, dce_abm->abm_mask->field_name
-
-#define CTX \
- dce_abm->base.ctx
-
-#define DISABLE_ABM_IMMEDIATELY 255
-
-
+#define ABM_FEATURE_NO_SUPPORT 0
+#define ABM_LCD_SUPPORT 1
-static void dmub_abm_enable_fractional_pwm(struct dc_context *dc)
+static unsigned int abm_feature_support(struct abm *abm, unsigned int panel_inst)
{
- union dmub_rb_cmd cmd;
- uint32_t fractional_pwm = (dc->dc->config.disable_fractional_pwm == false) ? 1 : 0;
- uint32_t edp_id_count = dc->dc_edp_id_count;
+ struct dc_context *dc = abm->ctx;
+ struct dc_link *edp_links[MAX_NUM_EDP];
int i;
- uint8_t panel_mask = 0;
-
- for (i = 0; i < edp_id_count; i++)
- panel_mask |= 0x01 << i;
-
- memset(&cmd, 0, sizeof(cmd));
- cmd.abm_set_pwm_frac.header.type = DMUB_CMD__ABM;
- cmd.abm_set_pwm_frac.header.sub_type = DMUB_CMD__ABM_SET_PWM_FRAC;
- cmd.abm_set_pwm_frac.abm_set_pwm_frac_data.fractional_pwm = fractional_pwm;
- cmd.abm_set_pwm_frac.abm_set_pwm_frac_data.version = DMUB_CMD_ABM_CONTROL_VERSION_1;
- cmd.abm_set_pwm_frac.abm_set_pwm_frac_data.panel_mask = panel_mask;
- cmd.abm_set_pwm_frac.header.payload_bytes = sizeof(struct dmub_cmd_abm_set_pwm_frac_data);
-
- dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
- dc_dmub_srv_cmd_execute(dc->dmub_srv);
- dc_dmub_srv_wait_idle(dc->dmub_srv);
-}
-
-static void dmub_abm_init(struct abm *abm, uint32_t backlight)
-{
- struct dce_abm *dce_abm = TO_DMUB_ABM(abm);
-
- REG_WRITE(DC_ABM1_HG_SAMPLE_RATE, 0x3);
- REG_WRITE(DC_ABM1_HG_SAMPLE_RATE, 0x1);
- REG_WRITE(DC_ABM1_LS_SAMPLE_RATE, 0x3);
- REG_WRITE(DC_ABM1_LS_SAMPLE_RATE, 0x1);
- REG_WRITE(BL1_PWM_BL_UPDATE_SAMPLE_RATE, 0x1);
-
- REG_SET_3(DC_ABM1_HG_MISC_CTRL, 0,
- ABM1_HG_NUM_OF_BINS_SEL, 0,
- ABM1_HG_VMAX_SEL, 1,
- ABM1_HG_BIN_BITWIDTH_SIZE_SEL, 0);
-
- REG_SET_3(DC_ABM1_IPCSC_COEFF_SEL, 0,
- ABM1_IPCSC_COEFF_SEL_R, 2,
- ABM1_IPCSC_COEFF_SEL_G, 4,
- ABM1_IPCSC_COEFF_SEL_B, 2);
-
- REG_UPDATE(BL1_PWM_CURRENT_ABM_LEVEL,
- BL1_PWM_CURRENT_ABM_LEVEL, backlight);
-
- REG_UPDATE(BL1_PWM_TARGET_ABM_LEVEL,
- BL1_PWM_TARGET_ABM_LEVEL, backlight);
+ int edp_num;
+ unsigned int ret = ABM_FEATURE_NO_SUPPORT;
- REG_UPDATE(BL1_PWM_USER_LEVEL,
- BL1_PWM_USER_LEVEL, backlight);
+ get_edp_links(dc->dc, edp_links, &edp_num);
- REG_UPDATE_2(DC_ABM1_LS_MIN_MAX_PIXEL_VALUE_THRES,
- ABM1_LS_MIN_PIXEL_VALUE_THRES, 0,
- ABM1_LS_MAX_PIXEL_VALUE_THRES, 1000);
+ for (i = 0; i < edp_num; i++) {
+ if (panel_inst == i)
+ break;
+ }
- REG_SET_3(DC_ABM1_HGLS_REG_READ_PROGRESS, 0,
- ABM1_HG_REG_READ_MISSED_FRAME_CLEAR, 1,
- ABM1_LS_REG_READ_MISSED_FRAME_CLEAR, 1,
- ABM1_BL_REG_READ_MISSED_FRAME_CLEAR, 1);
+ if (i < edp_num) {
+ ret = ABM_LCD_SUPPORT;
+ }
- dmub_abm_enable_fractional_pwm(abm->ctx);
+ return ret;
}
-static unsigned int dmub_abm_get_current_backlight(struct abm *abm)
+static void dmub_abm_init_ex(struct abm *abm, uint32_t backlight)
{
- struct dce_abm *dce_abm = TO_DMUB_ABM(abm);
- unsigned int backlight = REG_READ(BL1_PWM_CURRENT_ABM_LEVEL);
-
- /* return backlight in hardware format which is unsigned 17 bits, with
- * 1 bit integer and 16 bit fractional
- */
- return backlight;
+ dmub_abm_init(abm, backlight);
}
-static unsigned int dmub_abm_get_target_backlight(struct abm *abm)
+static unsigned int dmub_abm_get_current_backlight_ex(struct abm *abm)
{
- struct dce_abm *dce_abm = TO_DMUB_ABM(abm);
- unsigned int backlight = REG_READ(BL1_PWM_TARGET_ABM_LEVEL);
+ return dmub_abm_get_current_backlight(abm);
+}
- /* return backlight in hardware format which is unsigned 17 bits, with
- * 1 bit integer and 16 bit fractional
- */
- return backlight;
+static unsigned int dmub_abm_get_target_backlight_ex(struct abm *abm)
+{
+ return dmub_abm_get_target_backlight(abm);
}
-static bool dmub_abm_set_level(struct abm *abm, uint32_t level)
+static bool dmub_abm_set_level_ex(struct abm *abm, uint32_t level)
{
- union dmub_rb_cmd cmd;
- struct dc_context *dc = abm->ctx;
- struct dc_link *edp_links[MAX_NUM_EDP];
- int i;
- int edp_num;
- uint8_t panel_mask = 0;
+ bool ret = false;
+ unsigned int feature_support, i;
+ uint8_t panel_mask0 = 0;
- get_edp_links(dc->dc, edp_links, &edp_num);
+ for (i = 0; i < MAX_NUM_EDP; i++) {
+ feature_support = abm_feature_support(abm, i);
- for (i = 0; i < edp_num; i++) {
- if (edp_links[i]->link_status.link_active)
- panel_mask |= (0x01 << i);
+ if (feature_support == ABM_LCD_SUPPORT)
+ panel_mask0 |= (0x01 << i);
}
- memset(&cmd, 0, sizeof(cmd));
- cmd.abm_set_level.header.type = DMUB_CMD__ABM;
- cmd.abm_set_level.header.sub_type = DMUB_CMD__ABM_SET_LEVEL;
- cmd.abm_set_level.abm_set_level_data.level = level;
- cmd.abm_set_level.abm_set_level_data.version = DMUB_CMD_ABM_CONTROL_VERSION_1;
- cmd.abm_set_level.abm_set_level_data.panel_mask = panel_mask;
- cmd.abm_set_level.header.payload_bytes = sizeof(struct dmub_cmd_abm_set_level_data);
+ if (panel_mask0)
+ ret = dmub_abm_set_level(abm, level, panel_mask0);
- dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
- dc_dmub_srv_cmd_execute(dc->dmub_srv);
- dc_dmub_srv_wait_idle(dc->dmub_srv);
-
- return true;
+ return ret;
}
-static bool dmub_abm_init_config(struct abm *abm,
+static bool dmub_abm_init_config_ex(struct abm *abm,
const char *src,
unsigned int bytes,
unsigned int inst)
{
- union dmub_rb_cmd cmd;
- struct dc_context *dc = abm->ctx;
- uint8_t panel_mask = 0x01 << inst;
+ unsigned int feature_support;
+
+ feature_support = abm_feature_support(abm, inst);
+
+ if (feature_support == ABM_LCD_SUPPORT)
+ dmub_abm_init_config(abm, src, bytes, inst);
+
+ return true;
+}
- // TODO: Optimize by only reading back final 4 bytes
- dmub_flush_buffer_mem(&dc->dmub_srv->dmub->scratch_mem_fb);
+static bool dmub_abm_set_pause_ex(struct abm *abm, bool pause, unsigned int panel_inst, unsigned int stream_inst)
+{
+ bool ret = false;
+ unsigned int feature_support;
- // Copy iramtable into cw7
- memcpy(dc->dmub_srv->dmub->scratch_mem_fb.cpu_addr, (void *)src, bytes);
+ feature_support = abm_feature_support(abm, panel_inst);
- memset(&cmd, 0, sizeof(cmd));
- // Fw will copy from cw7 to fw_state
- cmd.abm_init_config.header.type = DMUB_CMD__ABM;
- cmd.abm_init_config.header.sub_type = DMUB_CMD__ABM_INIT_CONFIG;
- cmd.abm_init_config.abm_init_config_data.src.quad_part = dc->dmub_srv->dmub->scratch_mem_fb.gpu_addr;
- cmd.abm_init_config.abm_init_config_data.bytes = bytes;
- cmd.abm_init_config.abm_init_config_data.version = DMUB_CMD_ABM_CONTROL_VERSION_1;
- cmd.abm_init_config.abm_init_config_data.panel_mask = panel_mask;
+ if (feature_support == ABM_LCD_SUPPORT)
+ ret = dmub_abm_set_pause(abm, pause, panel_inst, stream_inst);
- cmd.abm_init_config.header.payload_bytes = sizeof(struct dmub_cmd_abm_init_config_data);
+ return ret;
+}
- dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
- dc_dmub_srv_cmd_execute(dc->dmub_srv);
- dc_dmub_srv_wait_idle(dc->dmub_srv);
+static bool dmub_abm_set_pipe_ex(struct abm *abm, uint32_t otg_inst, uint32_t option, uint32_t panel_inst)
+{
+ bool ret = false;
+ unsigned int feature_support;
- return true;
+ feature_support = abm_feature_support(abm, panel_inst);
+
+ if (feature_support == ABM_LCD_SUPPORT)
+ ret = dmub_abm_set_pipe(abm, otg_inst, option, panel_inst);
+
+ return ret;
}
-static bool dmub_abm_set_pause(struct abm *abm, bool pause, unsigned int panel_inst, unsigned int stream_inst)
+static bool dmub_abm_set_backlight_level_pwm_ex(struct abm *abm,
+ unsigned int backlight_pwm_u16_16,
+ unsigned int frame_ramp,
+ unsigned int controller_id,
+ unsigned int panel_inst)
{
- union dmub_rb_cmd cmd;
- struct dc_context *dc = abm->ctx;
- uint8_t panel_mask = 0x01 << panel_inst;
+ bool ret = false;
+ unsigned int feature_support;
- memset(&cmd, 0, sizeof(cmd));
- cmd.abm_pause.header.type = DMUB_CMD__ABM;
- cmd.abm_pause.header.sub_type = DMUB_CMD__ABM_PAUSE;
- cmd.abm_pause.abm_pause_data.enable = pause;
- cmd.abm_pause.abm_pause_data.panel_mask = panel_mask;
- cmd.abm_set_level.header.payload_bytes = sizeof(struct dmub_cmd_abm_pause_data);
+ feature_support = abm_feature_support(abm, panel_inst);
- dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
- dc_dmub_srv_cmd_execute(dc->dmub_srv);
- dc_dmub_srv_wait_idle(dc->dmub_srv);
+ if (feature_support == ABM_LCD_SUPPORT)
+ ret = dmub_abm_set_backlight_level(abm, backlight_pwm_u16_16, frame_ramp, panel_inst);
- return true;
+ return ret;
}
static const struct abm_funcs abm_funcs = {
- .abm_init = dmub_abm_init,
- .set_abm_level = dmub_abm_set_level,
- .get_current_backlight = dmub_abm_get_current_backlight,
- .get_target_backlight = dmub_abm_get_target_backlight,
- .init_abm_config = dmub_abm_init_config,
- .set_abm_pause = dmub_abm_set_pause,
+ .abm_init = dmub_abm_init_ex,
+ .set_abm_level = dmub_abm_set_level_ex,
+ .get_current_backlight = dmub_abm_get_current_backlight_ex,
+ .get_target_backlight = dmub_abm_get_target_backlight_ex,
+ .init_abm_config = dmub_abm_init_config_ex,
+ .set_abm_pause = dmub_abm_set_pause_ex,
+ .set_pipe_ex = dmub_abm_set_pipe_ex,
+ .set_backlight_level_pwm = dmub_abm_set_backlight_level_pwm_ex,
};
static void dmub_abm_construct(
@@ -256,16 +183,19 @@ struct abm *dmub_abm_create(
const struct dce_abm_shift *abm_shift,
const struct dce_abm_mask *abm_mask)
{
- struct dce_abm *abm_dce = kzalloc(sizeof(*abm_dce), GFP_KERNEL);
+ if (ctx->dc->caps.dmcub_support) {
+ struct dce_abm *abm_dce = kzalloc(sizeof(*abm_dce), GFP_KERNEL);
- if (abm_dce == NULL) {
- BREAK_TO_DEBUGGER();
- return NULL;
- }
+ if (abm_dce == NULL) {
+ BREAK_TO_DEBUGGER();
+ return NULL;
+ }
- dmub_abm_construct(abm_dce, ctx, regs, abm_shift, abm_mask);
+ dmub_abm_construct(abm_dce, ctx, regs, abm_shift, abm_mask);
- return &abm_dce->base;
+ return &abm_dce->base;
+ }
+ return NULL;
}
void dmub_abm_destroy(struct abm **abm)
diff --git a/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c b/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c
new file mode 100644
index 000000000000..19a97da77df8
--- /dev/null
+++ b/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c
@@ -0,0 +1,286 @@
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: AMD
+ *
+ */
+
+#include "dmub_abm.h"
+#include "dce_abm.h"
+#include "dc.h"
+#include "dc_dmub_srv.h"
+#include "dmub/dmub_srv.h"
+#include "core_types.h"
+#include "dm_services.h"
+#include "reg_helper.h"
+#include "fixed31_32.h"
+
+#ifdef _WIN32
+#include "atombios.h"
+#else
+#include "atom.h"
+#endif
+
+#define TO_DMUB_ABM(abm)\
+ container_of(abm, struct dce_abm, base)
+
+#define REG(reg) \
+ (dce_abm->regs->reg)
+
+#undef FN
+#define FN(reg_name, field_name) \
+ dce_abm->abm_shift->field_name, dce_abm->abm_mask->field_name
+
+#define CTX \
+ dce_abm->base.ctx
+
+#define DISABLE_ABM_IMMEDIATELY 255
+
+
+
+static void dmub_abm_enable_fractional_pwm(struct dc_context *dc)
+{
+ union dmub_rb_cmd cmd;
+ uint32_t fractional_pwm = (dc->dc->config.disable_fractional_pwm == false) ? 1 : 0;
+ uint32_t edp_id_count = dc->dc_edp_id_count;
+ int i;
+ uint8_t panel_mask = 0;
+
+ for (i = 0; i < edp_id_count; i++)
+ panel_mask |= 0x01 << i;
+
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.abm_set_pwm_frac.header.type = DMUB_CMD__ABM;
+ cmd.abm_set_pwm_frac.header.sub_type = DMUB_CMD__ABM_SET_PWM_FRAC;
+ cmd.abm_set_pwm_frac.abm_set_pwm_frac_data.fractional_pwm = fractional_pwm;
+ cmd.abm_set_pwm_frac.abm_set_pwm_frac_data.version = DMUB_CMD_ABM_CONTROL_VERSION_1;
+ cmd.abm_set_pwm_frac.abm_set_pwm_frac_data.panel_mask = panel_mask;
+ cmd.abm_set_pwm_frac.header.payload_bytes = sizeof(struct dmub_cmd_abm_set_pwm_frac_data);
+
+ dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
+ dc_dmub_srv_cmd_execute(dc->dmub_srv);
+ dc_dmub_srv_wait_idle(dc->dmub_srv);
+}
+
+void dmub_abm_init(struct abm *abm, uint32_t backlight)
+{
+ struct dce_abm *dce_abm = TO_DMUB_ABM(abm);
+
+ REG_WRITE(DC_ABM1_HG_SAMPLE_RATE, 0x3);
+ REG_WRITE(DC_ABM1_HG_SAMPLE_RATE, 0x1);
+ REG_WRITE(DC_ABM1_LS_SAMPLE_RATE, 0x3);
+ REG_WRITE(DC_ABM1_LS_SAMPLE_RATE, 0x1);
+ REG_WRITE(BL1_PWM_BL_UPDATE_SAMPLE_RATE, 0x1);
+
+ REG_SET_3(DC_ABM1_HG_MISC_CTRL, 0,
+ ABM1_HG_NUM_OF_BINS_SEL, 0,
+ ABM1_HG_VMAX_SEL, 1,
+ ABM1_HG_BIN_BITWIDTH_SIZE_SEL, 0);
+
+ REG_SET_3(DC_ABM1_IPCSC_COEFF_SEL, 0,
+ ABM1_IPCSC_COEFF_SEL_R, 2,
+ ABM1_IPCSC_COEFF_SEL_G, 4,
+ ABM1_IPCSC_COEFF_SEL_B, 2);
+
+ REG_UPDATE(BL1_PWM_CURRENT_ABM_LEVEL,
+ BL1_PWM_CURRENT_ABM_LEVEL, backlight);
+
+ REG_UPDATE(BL1_PWM_TARGET_ABM_LEVEL,
+ BL1_PWM_TARGET_ABM_LEVEL, backlight);
+
+ REG_UPDATE(BL1_PWM_USER_LEVEL,
+ BL1_PWM_USER_LEVEL, backlight);
+
+ REG_UPDATE_2(DC_ABM1_LS_MIN_MAX_PIXEL_VALUE_THRES,
+ ABM1_LS_MIN_PIXEL_VALUE_THRES, 0,
+ ABM1_LS_MAX_PIXEL_VALUE_THRES, 1000);
+
+ REG_SET_3(DC_ABM1_HGLS_REG_READ_PROGRESS, 0,
+ ABM1_HG_REG_READ_MISSED_FRAME_CLEAR, 1,
+ ABM1_LS_REG_READ_MISSED_FRAME_CLEAR, 1,
+ ABM1_BL_REG_READ_MISSED_FRAME_CLEAR, 1);
+
+ dmub_abm_enable_fractional_pwm(abm->ctx);
+}
+
+unsigned int dmub_abm_get_current_backlight(struct abm *abm)
+{
+ struct dce_abm *dce_abm = TO_DMUB_ABM(abm);
+ unsigned int backlight = REG_READ(BL1_PWM_CURRENT_ABM_LEVEL);
+
+ /* return backlight in hardware format which is unsigned 17 bits, with
+ * 1 bit integer and 16 bit fractional
+ */
+ return backlight;
+}
+
+unsigned int dmub_abm_get_target_backlight(struct abm *abm)
+{
+ struct dce_abm *dce_abm = TO_DMUB_ABM(abm);
+ unsigned int backlight = REG_READ(BL1_PWM_TARGET_ABM_LEVEL);
+
+ /* return backlight in hardware format which is unsigned 17 bits, with
+ * 1 bit integer and 16 bit fractional
+ */
+ return backlight;
+}
+
+bool dmub_abm_set_level(struct abm *abm, uint32_t level, uint8_t panel_mask)
+{
+ union dmub_rb_cmd cmd;
+ struct dc_context *dc = abm->ctx;
+
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.abm_set_level.header.type = DMUB_CMD__ABM;
+ cmd.abm_set_level.header.sub_type = DMUB_CMD__ABM_SET_LEVEL;
+ cmd.abm_set_level.abm_set_level_data.level = level;
+ cmd.abm_set_level.abm_set_level_data.version = DMUB_CMD_ABM_CONTROL_VERSION_1;
+ cmd.abm_set_level.abm_set_level_data.panel_mask = panel_mask;
+ cmd.abm_set_level.header.payload_bytes = sizeof(struct dmub_cmd_abm_set_level_data);
+
+ dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
+ dc_dmub_srv_cmd_execute(dc->dmub_srv);
+ dc_dmub_srv_wait_idle(dc->dmub_srv);
+
+ return true;
+}
+
+#ifndef TRIM_AMBIENT_GAMMA
+void dmub_abm_set_ambient_level(struct abm *abm, unsigned int ambient_lux, uint8_t panel_mask)
+{
+ union dmub_rb_cmd cmd;
+ struct dc_context *dc = abm->ctx;
+
+ if (ambient_lux > 0xFFFF)
+ ambient_lux = 0xFFFF;
+
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.abm_set_ambient_level.header.type = DMUB_CMD__ABM;
+ cmd.abm_set_ambient_level.header.sub_type = DMUB_CMD__ABM_SET_AMBIENT_LEVEL;
+ cmd.abm_set_ambient_level.abm_set_ambient_level_data.ambient_lux = ambient_lux;
+ cmd.abm_set_ambient_level.abm_set_ambient_level_data.version = DMUB_CMD_ABM_CONTROL_VERSION_1;
+ cmd.abm_set_ambient_level.abm_set_ambient_level_data.panel_mask = panel_mask;
+ cmd.abm_set_ambient_level.header.payload_bytes = sizeof(struct dmub_cmd_abm_set_ambient_level_data);
+
+ dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
+ dc_dmub_srv_cmd_execute(dc->dmub_srv);
+ dc_dmub_srv_wait_idle(dc->dmub_srv);
+}
+#endif
+
+void dmub_abm_init_config(struct abm *abm,
+ const char *src,
+ unsigned int bytes,
+ unsigned int inst)
+{
+ union dmub_rb_cmd cmd;
+ struct dc_context *dc = abm->ctx;
+ uint8_t panel_mask = 0x01 << inst;
+
+ // TODO: Optimize by only reading back final 4 bytes
+ dmub_flush_buffer_mem(&dc->dmub_srv->dmub->scratch_mem_fb);
+
+ // Copy iramtable into cw7
+ memcpy(dc->dmub_srv->dmub->scratch_mem_fb.cpu_addr, (void *)src, bytes);
+
+ memset(&cmd, 0, sizeof(cmd));
+ // Fw will copy from cw7 to fw_state
+ cmd.abm_init_config.header.type = DMUB_CMD__ABM;
+ cmd.abm_init_config.header.sub_type = DMUB_CMD__ABM_INIT_CONFIG;
+ cmd.abm_init_config.abm_init_config_data.src.quad_part = dc->dmub_srv->dmub->scratch_mem_fb.gpu_addr;
+ cmd.abm_init_config.abm_init_config_data.bytes = bytes;
+ cmd.abm_init_config.abm_init_config_data.version = DMUB_CMD_ABM_CONTROL_VERSION_1;
+ cmd.abm_init_config.abm_init_config_data.panel_mask = panel_mask;
+
+ cmd.abm_init_config.header.payload_bytes = sizeof(struct dmub_cmd_abm_init_config_data);
+
+ dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
+ dc_dmub_srv_cmd_execute(dc->dmub_srv);
+ dc_dmub_srv_wait_idle(dc->dmub_srv);
+
+}
+
+bool dmub_abm_set_pause(struct abm *abm, bool pause, unsigned int panel_inst, unsigned int stream_inst)
+{
+ union dmub_rb_cmd cmd;
+ struct dc_context *dc = abm->ctx;
+ uint8_t panel_mask = 0x01 << panel_inst;
+
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.abm_pause.header.type = DMUB_CMD__ABM;
+ cmd.abm_pause.header.sub_type = DMUB_CMD__ABM_PAUSE;
+ cmd.abm_pause.abm_pause_data.enable = pause;
+ cmd.abm_pause.abm_pause_data.panel_mask = panel_mask;
+ cmd.abm_set_level.header.payload_bytes = sizeof(struct dmub_cmd_abm_pause_data);
+
+ dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
+ dc_dmub_srv_cmd_execute(dc->dmub_srv);
+ dc_dmub_srv_wait_idle(dc->dmub_srv);
+
+ return true;
+}
+
+bool dmub_abm_set_pipe(struct abm *abm, uint32_t otg_inst, uint32_t option, uint32_t panel_inst)
+{
+ union dmub_rb_cmd cmd;
+ struct dc_context *dc = abm->ctx;
+ uint8_t ramping_boundary = 0xFF;
+
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.abm_set_pipe.header.type = DMUB_CMD__ABM;
+ cmd.abm_set_pipe.header.sub_type = DMUB_CMD__ABM_SET_PIPE;
+ cmd.abm_set_pipe.abm_set_pipe_data.otg_inst = otg_inst;
+ cmd.abm_set_pipe.abm_set_pipe_data.set_pipe_option = option;
+ cmd.abm_set_pipe.abm_set_pipe_data.panel_inst = panel_inst;
+ cmd.abm_set_pipe.abm_set_pipe_data.ramping_boundary = ramping_boundary;
+ cmd.abm_set_pipe.header.payload_bytes = sizeof(struct dmub_cmd_abm_set_pipe_data);
+
+ dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
+ dc_dmub_srv_cmd_execute(dc->dmub_srv);
+ dc_dmub_srv_wait_idle(dc->dmub_srv);
+
+ return true;
+}
+
+bool dmub_abm_set_backlight_level(struct abm *abm,
+ unsigned int backlight_pwm_u16_16,
+ unsigned int frame_ramp,
+ unsigned int panel_inst)
+{
+ union dmub_rb_cmd cmd;
+ struct dc_context *dc = abm->ctx;
+
+ memset(&cmd, 0, sizeof(cmd));
+ cmd.abm_set_backlight.header.type = DMUB_CMD__ABM;
+ cmd.abm_set_backlight.header.sub_type = DMUB_CMD__ABM_SET_BACKLIGHT;
+ cmd.abm_set_backlight.abm_set_backlight_data.frame_ramp = frame_ramp;
+ cmd.abm_set_backlight.abm_set_backlight_data.backlight_user_level = backlight_pwm_u16_16;
+ cmd.abm_set_backlight.abm_set_backlight_data.version = DMUB_CMD_ABM_CONTROL_VERSION_1;
+ cmd.abm_set_backlight.abm_set_backlight_data.panel_mask = (0x01 << panel_inst);
+ cmd.abm_set_backlight.header.payload_bytes = sizeof(struct dmub_cmd_abm_set_backlight_data);
+
+ dc_dmub_srv_cmd_queue(dc->dmub_srv, &cmd);
+ dc_dmub_srv_cmd_execute(dc->dmub_srv);
+ dc_dmub_srv_wait_idle(dc->dmub_srv);
+
+ return true;
+}
+
diff --git a/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.h b/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.h
new file mode 100644
index 000000000000..00b4e268768e
--- /dev/null
+++ b/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.h
@@ -0,0 +1,46 @@
+/*
+ * Copyright 2019 Advanced Micro Devices, Inc.
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
+ * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR
+ * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
+ * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
+ * OTHER DEALINGS IN THE SOFTWARE.
+ *
+ * Authors: AMD
+ *
+ */
+
+#ifndef __DMUB_ABM_LCD_H__
+#define __DMUB_ABM_LCD_H__
+
+#include "abm.h"
+
+void dmub_abm_init(struct abm *abm, uint32_t backlight);
+bool dmub_abm_set_level(struct abm *abm, uint32_t level, uint8_t panel_mask);
+unsigned int dmub_abm_get_current_backlight(struct abm *abm);
+unsigned int dmub_abm_get_target_backlight(struct abm *abm);
+void dmub_abm_init_config(struct abm *abm,
+ const char *src,
+ unsigned int bytes,
+ unsigned int inst);
+
+bool dmub_abm_set_pause(struct abm *abm, bool pause, unsigned int panel_inst, unsigned int stream_inst);
+bool dmub_abm_set_pipe(struct abm *abm, uint32_t otg_inst, uint32_t option, uint32_t panel_inst);
+bool dmub_abm_set_backlight_level(struct abm *abm,
+ unsigned int backlight_pwm_u16_16,
+ unsigned int frame_ramp,
+ unsigned int panel_inst);
+#endif
diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c
index c542d2ab9160..d2e71bf6b8f5 100644
--- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c
+++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c
@@ -963,7 +963,7 @@ void dce110_edp_backlight_control(
{
struct dc_context *ctx = link->ctx;
struct bp_transmitter_control cntl = { 0 };
- uint8_t panel_instance;
+ uint8_t panel_instance = 0;
unsigned int pre_T11_delay = OLED_PRE_T11_DELAY;
unsigned int post_T7_delay = OLED_POST_T7_DELAY;
@@ -1016,7 +1016,8 @@ void dce110_edp_backlight_control(
*/
/* dc_service_sleep_in_milliseconds(50); */
/*edp 1.2*/
- panel_instance = link->panel_cntl->inst;
+ if (link->panel_cntl)
+ panel_instance = link->panel_cntl->inst;
if (cntl.action == TRANSMITTER_CONTROL_BACKLIGHT_ON) {
if (!link->dc->config.edp_no_power_sequencing)
diff --git a/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c b/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c
index 19d7cfa53211..1f1f66d85769 100644
--- a/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c
@@ -1065,7 +1065,10 @@ static void bw_calcs_data_update_from_pplib(struct dc *dc)
struct dm_pp_clock_levels clks = {0};
int memory_type_multiplier = MEMORY_TYPE_MULTIPLIER_CZ;
- if (dc->bw_vbios && dc->bw_vbios->memory_type == bw_def_hbm)
+ if (!dc->bw_vbios)
+ return;
+
+ if (dc->bw_vbios->memory_type == bw_def_hbm)
memory_type_multiplier = MEMORY_TYPE_HBM;
/*do system clock TODO PPLIB: after PPLIB implement,
diff --git a/drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c b/drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c
index 8db9f7514466..556c453666a7 100644
--- a/drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c
@@ -1355,7 +1355,6 @@ static bool dce64_construct(
dc->caps.max_downscale_ratio = 200;
dc->caps.i2c_speed_in_khz = 40;
dc->caps.max_cursor_size = 64;
- dc->caps.is_apu = true;
/*************************************************
* Create resources *
diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c
index b120aa67d26c..e2e8823f1371 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c
@@ -1490,7 +1490,7 @@ void dcn10_init_hw(struct dc *dc)
dc->clk_mgr->funcs->init_clocks(dc->clk_mgr);
/* Align bw context with hw config when system resume. */
- if (dc->clk_mgr->clks.dispclk_khz != 0 && dc->clk_mgr->clks.dppclk_khz != 0) {
+ if (dc->clk_mgr && dc->clk_mgr->clks.dispclk_khz != 0 && dc->clk_mgr->clks.dppclk_khz != 0) {
dc->current_state->bw_ctx.bw.dcn.clk.dispclk_khz = dc->clk_mgr->clks.dispclk_khz;
dc->current_state->bw_ctx.bw.dcn.clk.dppclk_khz = dc->clk_mgr->clks.dppclk_khz;
}
@@ -1630,7 +1630,7 @@ void dcn10_init_hw(struct dc *dc)
REG_UPDATE(DCFCLK_CNTL, DCFCLK_GATE_DIS, 0);
}
- if (dc->clk_mgr->funcs->notify_wm_ranges)
+ if (dc->clk_mgr && dc->clk_mgr->funcs->notify_wm_ranges)
dc->clk_mgr->funcs->notify_wm_ranges(dc->clk_mgr);
}
diff --git a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubbub.c b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubbub.c
index aacb1fb5c73e..d2eebe39a55e 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubbub.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubbub.c
@@ -595,7 +595,8 @@ static bool hubbub2_program_watermarks(
hubbub1->base.ctx->dc->clk_mgr->clks.p_state_change_support == false)
safe_to_lower = true;
- hubbub1_program_pstate_watermarks(hubbub, watermarks, refclk_mhz, safe_to_lower);
+ if (hubbub1_program_pstate_watermarks(hubbub, watermarks, refclk_mhz, safe_to_lower))
+ wm_pending = true;
REG_SET(DCHUBBUB_ARB_SAT_LEVEL, 0,
DCHUBBUB_ARB_SAT_LEVEL, 60 * refclk_mhz);
diff --git a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubp.c b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubp.c
index aa252dc26326..4040ec51ac57 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubp.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubp.c
@@ -925,7 +925,8 @@ bool hubp2_is_flip_pending(struct hubp *hubp)
if (flip_pending)
return true;
- if (earliest_inuse_address.grph.addr.quad_part != hubp->request_address.grph.addr.quad_part)
+ if (hubp &&
+ earliest_inuse_address.grph.addr.quad_part != hubp->request_address.grph.addr.quad_part)
return true;
return false;
diff --git a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c
index d252d10c8134..8a32b86c34d3 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c
@@ -1693,16 +1693,18 @@ static void dcn20_program_pipe(
dc->res_pool->hubbub->funcs->program_det_size(
dc->res_pool->hubbub, pipe_ctx->plane_res.hubp->inst, pipe_ctx->det_buffer_size_kb);
- if (pipe_ctx->update_flags.raw || pipe_ctx->plane_state->update_flags.raw || pipe_ctx->stream->update_flags.raw)
+ if (pipe_ctx->plane_state && (pipe_ctx->update_flags.raw ||
+ pipe_ctx->plane_state->update_flags.raw ||
+ pipe_ctx->stream->update_flags.raw))
dcn20_update_dchubp_dpp(dc, pipe_ctx, context);
- if (pipe_ctx->update_flags.bits.enable
- || pipe_ctx->plane_state->update_flags.bits.hdr_mult)
+ if ((pipe_ctx->plane_state && pipe_ctx->plane_state->update_flags.bits.hdr_mult) ||
+ pipe_ctx->update_flags.bits.enable)
hws->funcs.set_hdr_multiplier(pipe_ctx);
- if (pipe_ctx->update_flags.bits.enable ||
- pipe_ctx->plane_state->update_flags.bits.in_transfer_func_change ||
- pipe_ctx->plane_state->update_flags.bits.gamma_change)
+ if ((pipe_ctx->plane_state && pipe_ctx->plane_state->update_flags.bits.in_transfer_func_change) ||
+ (pipe_ctx->plane_state && pipe_ctx->plane_state->update_flags.bits.gamma_change) ||
+ pipe_ctx->update_flags.bits.enable)
hws->funcs.set_input_transfer_func(dc, pipe_ctx, pipe_ctx->plane_state);
/* dcn10_translate_regamma_to_hw_format takes 750us to finish
@@ -2137,6 +2139,9 @@ bool dcn20_wait_for_blank_complete(
{
int counter;
+ if (!opp)
+ return false;
+
for (counter = 0; counter < 1000; counter++) {
if (opp->funcs->dpg_is_blanked(opp))
break;
diff --git a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_resource.c b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_resource.c
index 8224b9bf01d1..dbfb9552b52c 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_resource.c
@@ -2021,6 +2021,7 @@ bool dcn20_fast_validate_bw(
{
bool out = false;
int split[MAX_PIPES] = { 0 };
+ bool merge[MAX_PIPES] = { false };
int pipe_cnt, i, pipe_idx, vlevel;
ASSERT(pipes);
@@ -2045,7 +2046,7 @@ bool dcn20_fast_validate_bw(
if (vlevel > context->bw_ctx.dml.soc.num_states)
goto validate_fail;
- vlevel = dcn20_validate_apply_pipe_split_flags(dc, context, vlevel, split, NULL);
+ vlevel = dcn20_validate_apply_pipe_split_flags(dc, context, vlevel, split, merge);
/*initialize pipe_just_split_from to invalid idx*/
for (i = 0; i < MAX_PIPES; i++)
@@ -2173,10 +2174,11 @@ bool dcn20_get_dcc_compression_cap(const struct dc *dc,
const struct dc_dcc_surface_param *input,
struct dc_surface_dcc_cap *output)
{
- return dc->res_pool->hubbub->funcs->get_dcc_compression_cap(
- dc->res_pool->hubbub,
- input,
- output);
+ if (dc->res_pool->hubbub->funcs->get_dcc_compression_cap)
+ return dc->res_pool->hubbub->funcs->get_dcc_compression_cap(
+ dc->res_pool->hubbub, input, output);
+
+ return false;
}
static void dcn20_destroy_resource_pool(struct resource_pool **pool)
diff --git a/drivers/gpu/drm/amd/display/dc/dcn201/dcn201_resource.c b/drivers/gpu/drm/amd/display/dc/dcn201/dcn201_resource.c
index 407d995bfa99..fee527977c37 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn201/dcn201_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn201/dcn201_resource.c
@@ -1009,8 +1009,10 @@ static struct pipe_ctx *dcn201_acquire_idle_pipe_for_layer(
struct pipe_ctx *head_pipe = resource_get_head_pipe_for_stream(res_ctx, stream);
struct pipe_ctx *idle_pipe = find_idle_secondary_pipe(res_ctx, pool, head_pipe);
- if (!head_pipe)
+ if (!head_pipe) {
ASSERT(0);
+ return NULL;
+ }
if (!idle_pipe)
return NULL;
diff --git a/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hwseq.c
index 69cc192a7e71..10c63c47097a 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hwseq.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hwseq.c
@@ -141,7 +141,7 @@ static bool dmub_abm_set_pipe(struct abm *abm, uint32_t otg_inst, uint32_t optio
{
union dmub_rb_cmd cmd;
struct dc_context *dc = abm->ctx;
- uint32_t ramping_boundary = 0xFFFF;
+ uint8_t ramping_boundary = 0xFF;
memset(&cmd, 0, sizeof(cmd));
cmd.abm_set_pipe.header.type = DMUB_CMD__ABM;
@@ -182,17 +182,22 @@ void dcn21_set_abm_immediate_disable(struct pipe_ctx *pipe_ctx)
void dcn21_set_pipe(struct pipe_ctx *pipe_ctx)
{
struct abm *abm = pipe_ctx->stream_res.abm;
- uint32_t otg_inst = pipe_ctx->stream_res.tg->inst;
+ struct timing_generator *tg = pipe_ctx->stream_res.tg;
struct panel_cntl *panel_cntl = pipe_ctx->stream->link->panel_cntl;
struct dmcu *dmcu = pipe_ctx->stream->ctx->dc->res_pool->dmcu;
+ uint32_t otg_inst;
+
+ if (!abm || !tg || !panel_cntl)
+ return;
+
+ otg_inst = tg->inst;
if (dmcu) {
dce110_set_pipe(pipe_ctx);
return;
}
- if (abm && panel_cntl)
- dmub_abm_set_pipe(abm, otg_inst, SET_ABM_PIPE_NORMAL, panel_cntl->inst);
+ dmub_abm_set_pipe(abm, otg_inst, SET_ABM_PIPE_NORMAL, panel_cntl->inst);
}
bool dcn21_set_backlight_level(struct pipe_ctx *pipe_ctx,
@@ -202,16 +207,21 @@ bool dcn21_set_backlight_level(struct pipe_ctx *pipe_ctx,
union dmub_rb_cmd cmd;
struct dc_context *dc = pipe_ctx->stream->ctx;
struct abm *abm = pipe_ctx->stream_res.abm;
- uint32_t otg_inst = pipe_ctx->stream_res.tg->inst;
+ struct timing_generator *tg = pipe_ctx->stream_res.tg;
struct panel_cntl *panel_cntl = pipe_ctx->stream->link->panel_cntl;
+ uint32_t otg_inst;
+
+ if (!abm || !tg || !panel_cntl)
+ return false;
+
+ otg_inst = tg->inst;
if (dc->dc->res_pool->dmcu) {
dce110_set_backlight_level(pipe_ctx, backlight_pwm_u16_16, frame_ramp);
return true;
}
- if (abm && panel_cntl)
- dmub_abm_set_pipe(abm, otg_inst, SET_ABM_PIPE_NORMAL, panel_cntl->inst);
+ dmub_abm_set_pipe(abm, otg_inst, SET_ABM_PIPE_NORMAL, panel_cntl->inst);
memset(&cmd, 0, sizeof(cmd));
cmd.abm_set_backlight.header.type = DMUB_CMD__ABM;
diff --git a/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_resource.c b/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_resource.c
index dd71ea2568e2..61e72ccd4a67 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_resource.c
@@ -818,6 +818,7 @@ bool dcn21_fast_validate_bw(struct dc *dc,
{
bool out = false;
int split[MAX_PIPES] = { 0 };
+ bool merge[MAX_PIPES] = { false };
int pipe_cnt, i, pipe_idx, vlevel;
ASSERT(pipes);
@@ -860,7 +861,7 @@ bool dcn21_fast_validate_bw(struct dc *dc,
goto validate_fail;
}
- vlevel = dcn20_validate_apply_pipe_split_flags(dc, context, vlevel, split, NULL);
+ vlevel = dcn20_validate_apply_pipe_split_flags(dc, context, vlevel, split, merge);
for (i = 0, pipe_idx = 0; i < dc->res_pool->pipe_count; i++) {
struct pipe_ctx *pipe = &context->res_ctx.pipe_ctx[i];
diff --git a/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_hwseq.c
index d4ee533deff3..2ef1c227f105 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_hwseq.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_hwseq.c
@@ -281,10 +281,10 @@ void dcn31_init_hw(struct dc *dc)
if (!dcb->funcs->is_accelerated_mode(dcb) && dc->res_pool->hubbub->funcs->init_watermarks)
dc->res_pool->hubbub->funcs->init_watermarks(dc->res_pool->hubbub);
- if (dc->clk_mgr->funcs->notify_wm_ranges)
+ if (dc->clk_mgr && dc->clk_mgr->funcs->notify_wm_ranges)
dc->clk_mgr->funcs->notify_wm_ranges(dc->clk_mgr);
- if (dc->clk_mgr->funcs->set_hard_max_memclk && !dc->clk_mgr->dc_mode_softmax_enabled)
+ if (dc->clk_mgr && dc->clk_mgr->funcs->set_hard_max_memclk && !dc->clk_mgr->dc_mode_softmax_enabled)
dc->clk_mgr->funcs->set_hard_max_memclk(dc->clk_mgr);
if (dc->res_pool->hubbub->funcs->force_pstate_change_control)
diff --git a/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_optc.c b/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_optc.c
index 7dd36e402bac..56487ba4d00e 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_optc.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_optc.c
@@ -139,6 +139,13 @@ static bool optc314_disable_crtc(struct timing_generator *optc)
REG_UPDATE(OTG_CONTROL,
OTG_MASTER_EN, 0);
+ REG_UPDATE_5(OPTC_DATA_SOURCE_SELECT,
+ OPTC_SEG0_SRC_SEL, 0xf,
+ OPTC_SEG1_SRC_SEL, 0xf,
+ OPTC_SEG2_SRC_SEL, 0xf,
+ OPTC_SEG3_SRC_SEL, 0xf,
+ OPTC_NUM_OF_INPUT_SEGMENT, 0);
+
REG_UPDATE(CONTROL,
VTG0_ENABLE, 0);
diff --git a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_optc.c b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_optc.c
index 37c645a882dd..4a2fa9bb5393 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_optc.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_optc.c
@@ -145,6 +145,13 @@ static bool optc32_disable_crtc(struct timing_generator *optc)
REG_UPDATE(OTG_CONTROL,
OTG_MASTER_EN, 0);
+ REG_UPDATE_5(OPTC_DATA_SOURCE_SELECT,
+ OPTC_SEG0_SRC_SEL, 0xf,
+ OPTC_SEG1_SRC_SEL, 0xf,
+ OPTC_SEG2_SRC_SEL, 0xf,
+ OPTC_SEG3_SRC_SEL, 0xf,
+ OPTC_NUM_OF_INPUT_SEGMENT, 0);
+
REG_UPDATE(CONTROL,
VTG0_ENABLE, 0);
diff --git a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c
index 1b1534ffee9f..9f5425e6474f 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c
@@ -1664,6 +1664,9 @@ static void dcn32_enable_phantom_plane(struct dc *dc,
else
phantom_plane = dc_create_plane_state(dc);
+ if (!phantom_plane)
+ break;
+
memcpy(&phantom_plane->address, &curr_pipe->plane_state->address, sizeof(phantom_plane->address));
memcpy(&phantom_plane->scaling_quality, &curr_pipe->plane_state->scaling_quality,
sizeof(phantom_plane->scaling_quality));
@@ -2563,8 +2566,10 @@ struct pipe_ctx *dcn32_acquire_idle_pipe_for_head_pipe_in_layer(
struct resource_context *old_ctx = &stream->ctx->dc->current_state->res_ctx;
int head_index;
- if (!head_pipe)
+ if (!head_pipe) {
ASSERT(0);
+ return NULL;
+ }
/*
* Modified from dcn20_acquire_idle_pipe_for_layer
diff --git a/drivers/gpu/drm/amd/display/dc/dcn321/dcn321_resource.c b/drivers/gpu/drm/amd/display/dc/dcn321/dcn321_resource.c
index 85430e5baa45..4a1be0e37128 100644
--- a/drivers/gpu/drm/amd/display/dc/dcn321/dcn321_resource.c
+++ b/drivers/gpu/drm/amd/display/dc/dcn321/dcn321_resource.c
@@ -1208,7 +1208,7 @@ static struct stream_encoder *dcn321_stream_encoder_create(
vpg = dcn321_vpg_create(ctx, vpg_inst);
afmt = dcn321_afmt_create(ctx, afmt_inst);
- if (!enc1 || !vpg || !afmt) {
+ if (!enc1 || !vpg || !afmt || eng_id >= ARRAY_SIZE(stream_enc_regs)) {
kfree(enc1);
kfree(vpg);
kfree(afmt);
diff --git a/drivers/gpu/drm/amd/display/dc/dml/dcn302/dcn302_fpu.c b/drivers/gpu/drm/amd/display/dc/dml/dcn302/dcn302_fpu.c
index e2bcd205aa93..8da97a96b1ce 100644
--- a/drivers/gpu/drm/amd/display/dc/dml/dcn302/dcn302_fpu.c
+++ b/drivers/gpu/drm/amd/display/dc/dml/dcn302/dcn302_fpu.c
@@ -304,6 +304,16 @@ void dcn302_fpu_update_bw_bounding_box(struct dc *dc, struct clk_bw_params *bw_p
dram_speed_mts[num_states++] = bw_params->clk_table.entries[j++].memclk_mhz * 16;
}
+ /* bw_params->clk_table.entries[MAX_NUM_DPM_LVL].
+ * MAX_NUM_DPM_LVL is 8.
+ * dcn3_02_soc.clock_limits[DC__VOLTAGE_STATES].
+ * DC__VOLTAGE_STATES is 40.
+ */
+ if (num_states > MAX_NUM_DPM_LVL) {
+ ASSERT(0);
+ return;
+ }
+
dcn3_02_soc.num_states = num_states;
for (i = 0; i < dcn3_02_soc.num_states; i++) {
dcn3_02_soc.clock_limits[i].state = i;
diff --git a/drivers/gpu/drm/amd/display/dc/dml/dcn303/dcn303_fpu.c b/drivers/gpu/drm/amd/display/dc/dml/dcn303/dcn303_fpu.c
index 3eb3a021ab7d..c283780ad062 100644
--- a/drivers/gpu/drm/amd/display/dc/dml/dcn303/dcn303_fpu.c
+++ b/drivers/gpu/drm/amd/display/dc/dml/dcn303/dcn303_fpu.c
@@ -299,6 +299,16 @@ void dcn303_fpu_update_bw_bounding_box(struct dc *dc, struct clk_bw_params *bw_p
dram_speed_mts[num_states++] = bw_params->clk_table.entries[j++].memclk_mhz * 16;
}
+ /* bw_params->clk_table.entries[MAX_NUM_DPM_LVL].
+ * MAX_NUM_DPM_LVL is 8.
+ * dcn3_02_soc.clock_limits[DC__VOLTAGE_STATES].
+ * DC__VOLTAGE_STATES is 40.
+ */
+ if (num_states > MAX_NUM_DPM_LVL) {
+ ASSERT(0);
+ return;
+ }
+
dcn3_03_soc.num_states = num_states;
for (i = 0; i < dcn3_03_soc.num_states; i++) {
dcn3_03_soc.clock_limits[i].state = i;
diff --git a/drivers/gpu/drm/amd/display/dc/dml/dcn32/dcn32_fpu.c b/drivers/gpu/drm/amd/display/dc/dml/dcn32/dcn32_fpu.c
index 9d8917f72d18..12bc611de449 100644
--- a/drivers/gpu/drm/amd/display/dc/dml/dcn32/dcn32_fpu.c
+++ b/drivers/gpu/drm/amd/display/dc/dml/dcn32/dcn32_fpu.c
@@ -2508,6 +2508,16 @@ void dcn32_update_bw_bounding_box_fpu(struct dc *dc, struct clk_bw_params *bw_pa
dram_speed_mts[num_states++] = bw_params->clk_table.entries[j++].memclk_mhz * 16;
}
+ /* bw_params->clk_table.entries[MAX_NUM_DPM_LVL].
+ * MAX_NUM_DPM_LVL is 8.
+ * dcn3_02_soc.clock_limits[DC__VOLTAGE_STATES].
+ * DC__VOLTAGE_STATES is 40.
+ */
+ if (num_states > MAX_NUM_DPM_LVL) {
+ ASSERT(0);
+ return;
+ }
+
dcn3_2_soc.num_states = num_states;
for (i = 0; i < dcn3_2_soc.num_states; i++) {
dcn3_2_soc.clock_limits[i].state = i;
diff --git a/drivers/gpu/drm/amd/display/dc/dml/dcn321/dcn321_fpu.c b/drivers/gpu/drm/amd/display/dc/dml/dcn321/dcn321_fpu.c
index 383a409a3f54..2c7d58ac6008 100644
--- a/drivers/gpu/drm/amd/display/dc/dml/dcn321/dcn321_fpu.c
+++ b/drivers/gpu/drm/amd/display/dc/dml/dcn321/dcn321_fpu.c
@@ -652,6 +652,16 @@ void dcn321_update_bw_bounding_box_fpu(struct dc *dc, struct clk_bw_params *bw_p
dram_speed_mts[num_states++] = bw_params->clk_table.entries[j++].memclk_mhz * 16;
}
+ /* bw_params->clk_table.entries[MAX_NUM_DPM_LVL].
+ * MAX_NUM_DPM_LVL is 8.
+ * dcn3_02_soc.clock_limits[DC__VOLTAGE_STATES].
+ * DC__VOLTAGE_STATES is 40.
+ */
+ if (num_states > MAX_NUM_DPM_LVL) {
+ ASSERT(0);
+ return;
+ }
+
dcn3_21_soc.num_states = num_states;
for (i = 0; i < dcn3_21_soc.num_states; i++) {
dcn3_21_soc.clock_limits[i].state = i;
diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/abm.h b/drivers/gpu/drm/amd/display/dc/inc/hw/abm.h
index ecb4191b6e64..d2190a3320f6 100644
--- a/drivers/gpu/drm/amd/display/dc/inc/hw/abm.h
+++ b/drivers/gpu/drm/amd/display/dc/inc/hw/abm.h
@@ -55,6 +55,10 @@ struct abm_funcs {
unsigned int bytes,
unsigned int inst);
bool (*set_abm_pause)(struct abm *abm, bool pause, unsigned int panel_inst, unsigned int otg_inst);
+ bool (*set_pipe_ex)(struct abm *abm,
+ unsigned int otg_inst,
+ unsigned int option,
+ unsigned int panel_inst);
};
#endif
diff --git a/drivers/gpu/drm/amd/display/dc/link/link_hwss_dio.c b/drivers/gpu/drm/amd/display/dc/link/link_hwss_dio.c
index 4227adbc646a..5ba7749ca52b 100644
--- a/drivers/gpu/drm/amd/display/dc/link/link_hwss_dio.c
+++ b/drivers/gpu/drm/amd/display/dc/link/link_hwss_dio.c
@@ -56,7 +56,10 @@ void reset_dio_stream_encoder(struct pipe_ctx *pipe_ctx)
struct link_encoder *link_enc = link_enc_cfg_get_link_enc(pipe_ctx->stream->link);
struct stream_encoder *stream_enc = pipe_ctx->stream_res.stream_enc;
- if (stream_enc && stream_enc->funcs->disable_fifo)
+ if (!stream_enc)
+ return;
+
+ if (stream_enc->funcs->disable_fifo)
stream_enc->funcs->disable_fifo(stream_enc);
link_enc->funcs->connect_dig_be_to_fe(
diff --git a/drivers/gpu/drm/amd/display/dc/link/link_hwss_hpo_dp.c b/drivers/gpu/drm/amd/display/dc/link/link_hwss_hpo_dp.c
index fd9809b17882..7dbe7909f163 100644
--- a/drivers/gpu/drm/amd/display/dc/link/link_hwss_hpo_dp.c
+++ b/drivers/gpu/drm/amd/display/dc/link/link_hwss_hpo_dp.c
@@ -137,7 +137,8 @@ static void reset_hpo_dp_stream_encoder(struct pipe_ctx *pipe_ctx)
dto_params.timing = &pipe_ctx->stream->timing;
stream_enc->funcs->disable(stream_enc);
- dccg->funcs->set_dtbclk_dto(dccg, &dto_params);
+ if (dccg && dccg->funcs->set_dtbclk_dto)
+ dccg->funcs->set_dtbclk_dto(dccg, &dto_params);
dccg->funcs->disable_symclk32_se(dccg, stream_enc->inst);
dccg->funcs->set_dpstreamclk(dccg, REFCLK, tg->inst, stream_enc->inst);
}
diff --git a/drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h b/drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h
index d8c05bc45957..06717e9a28e9 100644
--- a/drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h
+++ b/drivers/gpu/drm/amd/display/dmub/inc/dmub_cmd.h
@@ -540,6 +540,24 @@ enum dmub_gpint_command {
* DESC: Notifies DMCUB detection is done so detection required can be cleared.
*/
DMUB_GPINT__NOTIFY_DETECTION_DONE = 12,
+ /**
+ * DESC: Updates the trace buffer lower 32-bit mask.
+ * ARGS: The new mask
+ * RETURN: Lower 32-bit mask.
+ */
+ DMUB_GPINT__UPDATE_TRACE_BUFFER_MASK = 101,
+ /**
+ * DESC: Updates the trace buffer lower 32-bit mask.
+ * ARGS: The new mask
+ * RETURN: Lower 32-bit mask.
+ */
+ DMUB_GPINT__SET_TRACE_BUFFER_MASK_WORD0 = 102,
+ /**
+ * DESC: Updates the trace buffer mask bi0~bit15.
+ * ARGS: The new mask
+ * RETURN: Lower 32-bit mask.
+ */
+ DMUB_GPINT__SET_TRACE_BUFFER_MASK_WORD1 = 103,
};
/**
diff --git a/drivers/gpu/drm/amd/pm/amdgpu_pm.c b/drivers/gpu/drm/amd/pm/amdgpu_pm.c
index 500cbd157237..1eaf20cad9fa 100644
--- a/drivers/gpu/drm/amd/pm/amdgpu_pm.c
+++ b/drivers/gpu/drm/amd/pm/amdgpu_pm.c
@@ -1436,6 +1436,8 @@ static ssize_t amdgpu_set_pp_power_profile_mode(struct device *dev,
if (ret)
return -EINVAL;
parameter_size++;
+ if (!tmp_str)
+ break;
while (isspace(*tmp_str))
tmp_str++;
}
diff --git a/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c b/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c
index be70e2b6376a..16721657507a 100644
--- a/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c
+++ b/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c
@@ -5239,7 +5239,8 @@ static int si_init_smc_table(struct amdgpu_device *adev)
break;
}
- if (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_HARDWAREDC)
+ if ((adev->flags & AMD_IS_MOBILITY) &&
+ (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_HARDWAREDC))
table->systemFlags |= PPSMC_SYSTEMFLAG_GPIO_DC;
if (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_REGULATOR_HOT) {
@@ -7210,6 +7211,19 @@ static void si_parse_pplib_non_clock_info(struct amdgpu_device *adev,
adev->pm.dpm.uvd_ps = rps;
}
+static void si_update_limits(struct amdgpu_clock_and_voltage_limits *limits,
+ struct rv7xx_pl *pl)
+{
+ if (pl->sclk > limits->sclk)
+ limits->sclk = pl->sclk;
+ if (pl->mclk > limits->mclk)
+ limits->mclk = pl->mclk;
+ if (pl->vddc > limits->vddc)
+ limits->vddc = pl->vddc;
+ if (pl->vddci > limits->vddci)
+ limits->vddci = pl->vddci;
+}
+
static void si_parse_pplib_clock_info(struct amdgpu_device *adev,
struct amdgpu_ps *rps, int index,
union pplib_clock_info *clock_info)
@@ -7218,7 +7232,6 @@ static void si_parse_pplib_clock_info(struct amdgpu_device *adev,
struct evergreen_power_info *eg_pi = evergreen_get_pi(adev);
struct si_power_info *si_pi = si_get_pi(adev);
struct si_ps *ps = si_get_ps(rps);
- struct amdgpu_clock_and_voltage_limits *limits;
u16 leakage_voltage;
struct rv7xx_pl *pl = &ps->performance_levels[index];
int ret;
@@ -7283,26 +7296,14 @@ static void si_parse_pplib_clock_info(struct amdgpu_device *adev,
* VBIOS can contain conflicting values between:
* - the maximum allowed clocks and voltages on AC or DC
* - the clocks and voltages in power states on AC or DC
+ *
+ * Assume that the AC limits are the maximum of all power states,
+ * and the DC limits are the maximum of battery power states.
*/
+ si_update_limits(&adev->pm.dpm.dyn_state.max_clock_voltage_on_ac, pl);
if ((rps->class & ATOM_PPLIB_CLASSIFICATION_UI_MASK) ==
- ATOM_PPLIB_CLASSIFICATION_UI_PERFORMANCE)
- limits = &adev->pm.dpm.dyn_state.max_clock_voltage_on_ac;
- else if ((rps->class & ATOM_PPLIB_CLASSIFICATION_UI_MASK) ==
ATOM_PPLIB_CLASSIFICATION_UI_BATTERY)
- limits = &adev->pm.dpm.dyn_state.max_clock_voltage_on_dc;
- else
- limits = NULL;
-
- if (limits) {
- if (pl->sclk > limits->sclk)
- limits->sclk = pl->sclk;
- if (pl->mclk > limits->mclk)
- limits->mclk = pl->mclk;
- if (pl->vddc > limits->vddc)
- limits->vddc = pl->vddc;
- if (pl->vddci > limits->vddci)
- limits->vddci = pl->vddci;
- }
+ si_update_limits(&adev->pm.dpm.dyn_state.max_clock_voltage_on_dc, pl);
}
union pplib_power_state {
diff --git a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c
index 7e37354a0341..640350bbd908 100644
--- a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c
+++ b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c
@@ -269,15 +269,21 @@ static const ATOM_VOLTAGE_OBJECT_V3 *atomctrl_lookup_voltage_type_v3(
unsigned int offset = offsetof(ATOM_VOLTAGE_OBJECT_INFO_V3_1, asVoltageObj[0]);
uint8_t *start = (uint8_t *)voltage_object_info_table;
- while (offset < size) {
+ while (offset + sizeof(ATOM_VOLTAGE_OBJECT_HEADER_V3) <= size) {
const ATOM_VOLTAGE_OBJECT_V3 *voltage_object =
(const ATOM_VOLTAGE_OBJECT_V3 *)(start + offset);
+ u16 obj_size;
+
+ obj_size = le16_to_cpu(voltage_object->asGpioVoltageObj.sHeader.usSize);
+ if (obj_size < sizeof(voltage_object->asGpioVoltageObj.sHeader) ||
+ offset + obj_size > size)
+ break;
if (voltage_type == voltage_object->asGpioVoltageObj.sHeader.ucVoltageType &&
voltage_mode == voltage_object->asGpioVoltageObj.sHeader.ucVoltageMode)
return voltage_object;
- offset += le16_to_cpu(voltage_object->asGpioVoltageObj.sHeader.usSize);
+ offset += obj_size;
}
return NULL;
diff --git a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c
index a47a47238e2b..e5aa78718202 100644
--- a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c
+++ b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c
@@ -36,16 +36,21 @@ static const union atom_voltage_object_v4 *pp_atomfwctrl_lookup_voltage_type_v4(
offsetof(struct atom_voltage_objects_info_v4_1, voltage_object[0]);
unsigned long start = (unsigned long)voltage_object_info_table;
- while (offset < size) {
+ while (offset + sizeof(struct atom_voltage_object_header_v4) <= size) {
const union atom_voltage_object_v4 *voltage_object =
(const union atom_voltage_object_v4 *)(start + offset);
+ u16 obj_size;
+
+ obj_size = le16_to_cpu(voltage_object->gpio_voltage_obj.header.object_size);
+ if (obj_size < sizeof(voltage_object->gpio_voltage_obj.header) ||
+ offset + obj_size > size)
+ break;
if (voltage_type == voltage_object->gpio_voltage_obj.header.voltage_type &&
voltage_mode == voltage_object->gpio_voltage_obj.header.voltage_mode)
return voltage_object;
- offset += le16_to_cpu(voltage_object->gpio_voltage_obj.header.object_size);
-
+ offset += obj_size;
}
return NULL;
diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c
index ae8854b90f37..cff7f2589d02 100644
--- a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c
+++ b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c
@@ -1894,8 +1894,13 @@ static int smu_v13_0_0_mode1_reset(struct smu_context *smu)
else
ret = smu_cmn_send_smc_msg(smu, SMU_MSG_Mode1Reset, NULL);
- if (!ret)
+ if (!ret) {
+ /* disable mmio access while doing mode 1 reset*/
+ smu->adev->no_hw_access = true;
+ /* ensure no_hw_access is globally visible before any MMIO */
+ smp_mb();
msleep(SMU13_MODE1_RESET_WAIT_TIME_IN_MS);
+ }
return ret;
}
diff --git a/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c b/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c
index a94f1a9e8f40..79f67413c117 100644
--- a/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c
+++ b/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c
@@ -194,13 +194,12 @@ static int aspeed_gfx_load(struct drm_device *drm)
}
reset_control_deassert(priv->rst);
- priv->clk = devm_clk_get(drm->dev, NULL);
+ priv->clk = devm_clk_get_enabled(drm->dev, NULL);
if (IS_ERR(priv->clk)) {
dev_err(&pdev->dev,
- "missing or invalid clk device tree entry");
+ "failed to get and enable clock\n");
return PTR_ERR(priv->clk);
}
- clk_prepare_enable(priv->clk);
/* Sanitize control registers */
writel(0, priv->base + CRT_CTRL1);
diff --git a/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c b/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c
index 3e6fe8604959..ac8bc356bcac 100644
--- a/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c
+++ b/drivers/gpu/drm/bridge/adv7511/adv7511_drv.c
@@ -1292,17 +1292,6 @@ static int adv7511_probe(struct i2c_client *i2c, const struct i2c_device_id *id)
INIT_WORK(&adv7511->hpd_work, adv7511_hpd_work);
- if (i2c->irq) {
- init_waitqueue_head(&adv7511->wq);
-
- ret = devm_request_threaded_irq(dev, i2c->irq, NULL,
- adv7511_irq_handler,
- IRQF_ONESHOT, dev_name(dev),
- adv7511);
- if (ret)
- goto err_unregister_cec;
- }
-
adv7511_power_off(adv7511);
i2c_set_clientdata(i2c, adv7511);
@@ -1326,6 +1315,17 @@ static int adv7511_probe(struct i2c_client *i2c, const struct i2c_device_id *id)
adv7511_audio_init(dev, adv7511);
+ if (i2c->irq) {
+ init_waitqueue_head(&adv7511->wq);
+
+ ret = devm_request_threaded_irq(dev, i2c->irq, NULL,
+ adv7511_irq_handler,
+ IRQF_ONESHOT, dev_name(dev),
+ adv7511);
+ if (ret)
+ goto err_unregister_audio;
+ }
+
if (adv7511->type == ADV7533 || adv7511->type == ADV7535) {
ret = adv7533_attach_dsi(adv7511);
if (ret)
diff --git a/drivers/gpu/drm/display/drm_dp_helper.c b/drivers/gpu/drm/display/drm_dp_helper.c
index 14dcaca0d8c1..74fd93180f70 100644
--- a/drivers/gpu/drm/display/drm_dp_helper.c
+++ b/drivers/gpu/drm/display/drm_dp_helper.c
@@ -3843,7 +3843,9 @@ static int dp_aux_backlight_update_status(struct backlight_device *bd)
if (!backlight_is_blank(bd)) {
if (!bl->enabled) {
- drm_edp_backlight_enable(bl->aux, &bl->info, brightness);
+ ret = drm_edp_backlight_enable(bl->aux, &bl->info, brightness);
+ if (ret)
+ return ret;
bl->enabled = true;
return 0;
}
diff --git a/drivers/gpu/drm/drm_fb_helper.c b/drivers/gpu/drm/drm_fb_helper.c
index a0abfb3f05f0..954ca9215fb8 100644
--- a/drivers/gpu/drm/drm_fb_helper.c
+++ b/drivers/gpu/drm/drm_fb_helper.c
@@ -1942,11 +1942,11 @@ __drm_fb_helper_initial_config_and_unlock(struct drm_fb_helper *fb_helper,
drm_client_modeset_probe(&fb_helper->client, width, height);
ret = drm_fb_helper_single_fb_probe(fb_helper, bpp_sel);
if (ret < 0) {
- if (ret == -EAGAIN) {
- fb_helper->preferred_bpp = bpp_sel;
- fb_helper->deferred_setup = true;
+ fb_helper->preferred_bpp = bpp_sel;
+ /* modesets are probed but fb_helper->fb isn't; defer restore too */
+ fb_helper->deferred_setup = true;
+ if (ret == -EAGAIN)
ret = 0;
- }
mutex_unlock(&fb_helper->lock);
return ret;
diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c
index 13a6812f7d71..7d9b5cf9bdfd 100644
--- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c
+++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c
@@ -133,12 +133,17 @@ int etnaviv_sched_push_job(struct etnaviv_gem_submit *submit)
int etnaviv_sched_init(struct etnaviv_gpu *gpu)
{
+ struct drm_sched_init_args args = {
+ .ops = &etnaviv_sched_ops,
+ .credit_limit = etnaviv_hw_jobs_limit,
+ .hang_limit = etnaviv_job_hang_limit,
+ .timeout = msecs_to_jiffies(500),
+ .name = dev_name(gpu->dev),
+ .dev = gpu->dev,
+ };
int ret;
- ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops,
- etnaviv_hw_jobs_limit, etnaviv_job_hang_limit,
- msecs_to_jiffies(500), NULL, NULL,
- dev_name(gpu->dev), gpu->dev);
+ ret = drm_sched_init(&gpu->sched, &args);
if (ret)
return ret;
diff --git a/drivers/gpu/drm/gud/gud_pipe.c b/drivers/gpu/drm/gud/gud_pipe.c
index 1204d7e9be57..529eb6318977 100644
--- a/drivers/gpu/drm/gud/gud_pipe.c
+++ b/drivers/gpu/drm/gud/gud_pipe.c
@@ -489,6 +489,7 @@ int gud_pipe_check(struct drm_simple_display_pipe *pipe,
for_each_new_connector_in_state(state, connector, connector_state, i) {
if (connector_state->crtc)
break;
+ connector_state = NULL;
}
/*
diff --git a/drivers/gpu/drm/i915/display/icl_dsi.c b/drivers/gpu/drm/i915/display/icl_dsi.c
index f7422f0cf579..1134d0162921 100644
--- a/drivers/gpu/drm/i915/display/icl_dsi.c
+++ b/drivers/gpu/drm/i915/display/icl_dsi.c
@@ -913,7 +913,7 @@ gen11_dsi_set_transcoder_timings(struct intel_encoder *encoder,
* non-compressed link speeds, and simplifies down to the ratio between
* compressed and non-compressed bpp.
*/
- if (crtc_state->dsc.compression_enable) {
+ if (is_vid_mode(intel_dsi) && crtc_state->dsc.compression_enable) {
mul = crtc_state->dsc.compressed_bpp;
div = mipi_dsi_pixel_format_to_bpp(intel_dsi->pixel_format);
}
@@ -1518,7 +1518,7 @@ static void gen11_dsi_get_timings(struct intel_encoder *encoder,
struct drm_display_mode *adjusted_mode =
&pipe_config->hw.adjusted_mode;
- if (pipe_config->dsc.compressed_bpp) {
+ if (is_vid_mode(intel_dsi) && pipe_config->dsc.compressed_bpp) {
int div = pipe_config->dsc.compressed_bpp;
int mul = mipi_dsi_pixel_format_to_bpp(intel_dsi->pixel_format);
diff --git a/drivers/gpu/drm/i915/display/intel_bios.c b/drivers/gpu/drm/i915/display/intel_bios.c
index efbb0cffd3bc..3ccbe6026946 100644
--- a/drivers/gpu/drm/i915/display/intel_bios.c
+++ b/drivers/gpu/drm/i915/display/intel_bios.c
@@ -618,6 +618,21 @@ static void dump_pnp_id(struct drm_i915_private *i915,
pnp_id->mfg_week, pnp_id->mfg_year + 1990);
}
+static bool is_panel_type_valid(int panel_type)
+{
+ return panel_type >= 0 && panel_type < 16;
+}
+
+static bool is_panel_type_pnp(int panel_type)
+{
+ return panel_type == 0xff;
+}
+
+static bool is_panel_type_valid_or_pnp(int panel_type)
+{
+ return is_panel_type_valid(panel_type) || is_panel_type_pnp(panel_type);
+}
+
static int opregion_get_panel_type(struct drm_i915_private *i915,
const struct intel_bios_encoder_data *devdata,
const struct edid *edid, bool use_fallback)
@@ -635,15 +650,21 @@ static int vbt_get_panel_type(struct drm_i915_private *i915,
if (!lvds_options)
return -1;
- if (lvds_options->panel_type > 0xf &&
- lvds_options->panel_type != 0xff) {
+ if (!is_panel_type_valid_or_pnp(lvds_options->panel_type)) {
drm_dbg_kms(&i915->drm, "Invalid VBT panel type 0x%x\n",
lvds_options->panel_type);
return -1;
}
- if (devdata && devdata->child.handle == DEVICE_HANDLE_LFP2)
+ if (devdata && devdata->child.handle == DEVICE_HANDLE_LFP2) {
+ if (!is_panel_type_valid_or_pnp(lvds_options->panel_type2)) {
+ drm_dbg_kms(&i915->drm, "Invalid VBT panel type 2 0x%x\n",
+ lvds_options->panel_type2);
+ return -1;
+ }
+
return lvds_options->panel_type2;
+ }
drm_WARN_ON(&i915->drm, devdata && devdata->child.handle != DEVICE_HANDLE_LFP1);
@@ -755,13 +776,12 @@ static int get_panel_type(struct drm_i915_private *i915,
panel_types[i].name, panel_types[i].panel_type);
}
- if (panel_types[PANEL_TYPE_OPREGION].panel_type >= 0)
+ if (is_panel_type_valid(panel_types[PANEL_TYPE_OPREGION].panel_type))
i = PANEL_TYPE_OPREGION;
- else if (panel_types[PANEL_TYPE_VBT].panel_type == 0xff &&
- panel_types[PANEL_TYPE_PNPID].panel_type >= 0)
+ else if (is_panel_type_pnp(panel_types[PANEL_TYPE_VBT].panel_type) &&
+ is_panel_type_valid(panel_types[PANEL_TYPE_PNPID].panel_type))
i = PANEL_TYPE_PNPID;
- else if (panel_types[PANEL_TYPE_VBT].panel_type != 0xff &&
- panel_types[PANEL_TYPE_VBT].panel_type >= 0)
+ else if (is_panel_type_valid(panel_types[PANEL_TYPE_VBT].panel_type))
i = PANEL_TYPE_VBT;
else
i = PANEL_TYPE_FALLBACK;
diff --git a/drivers/gpu/drm/i915/display/intel_dmc.c b/drivers/gpu/drm/i915/display/intel_dmc.c
index e52ecc0738a6..722fdffb09f9 100644
--- a/drivers/gpu/drm/i915/display/intel_dmc.c
+++ b/drivers/gpu/drm/i915/display/intel_dmc.c
@@ -645,7 +645,7 @@ static u32 parse_dmc_fw_header(struct intel_dmc *dmc,
}
/* Cache the dmc header info. */
- if (mmio_count > mmio_count_max) {
+ if (!mmio_count || mmio_count > mmio_count_max) {
drm_err(&i915->drm, "DMC firmware has wrong mmio count %u\n", mmio_count);
return 0;
}
@@ -660,13 +660,21 @@ static u32 parse_dmc_fw_header(struct intel_dmc *dmc,
dmc_info->mmioaddr[i] = _MMIO(mmioaddr[i]);
dmc_info->mmiodata[i] = mmiodata[i];
}
- dmc_info->mmio_count = mmio_count;
- dmc_info->start_mmioaddr = start_mmioaddr;
rem_size -= header_len_bytes;
/* fw_size is in dwords, so multiplied by 4 to convert into bytes. */
- payload_size = dmc_header->fw_size * 4;
+ if (check_mul_overflow(dmc_header->fw_size, 4u, &payload_size)) {
+ drm_err(&i915->drm, "DMC fw_size too large (%u dwords)\n",
+ dmc_header->fw_size);
+ return 0;
+ }
+
+ if (!payload_size) {
+ drm_err(&i915->drm, "DMC firmware has empty payload\n");
+ return 0;
+ }
+
if (rem_size < payload_size)
goto error_truncated;
@@ -674,12 +682,15 @@ static u32 parse_dmc_fw_header(struct intel_dmc *dmc,
drm_err(&i915->drm, "DMC FW too big (%u bytes)\n", payload_size);
return 0;
}
- dmc_info->dmc_fw_size = dmc_header->fw_size;
dmc_info->payload = kmalloc(payload_size, GFP_KERNEL);
if (!dmc_info->payload)
return 0;
+ dmc_info->mmio_count = mmio_count;
+ dmc_info->start_mmioaddr = start_mmioaddr;
+ dmc_info->dmc_fw_size = dmc_header->fw_size;
+
payload = (u8 *)(dmc_header) + header_len_bytes;
memcpy(dmc_info->payload, payload, payload_size);
diff --git a/drivers/gpu/drm/i915/display/intel_dp.c b/drivers/gpu/drm/i915/display/intel_dp.c
index a538db3870cf..b092649f01f5 100644
--- a/drivers/gpu/drm/i915/display/intel_dp.c
+++ b/drivers/gpu/drm/i915/display/intel_dp.c
@@ -3864,10 +3864,11 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp)
{
struct drm_i915_private *i915 = dp_to_i915(intel_dp);
bool link_ok = true;
+ int tries = 33;
drm_WARN_ON_ONCE(&i915->drm, intel_dp->active_mst_links < 0);
- for (;;) {
+ while (--tries) {
u8 esi[4] = {};
u8 ack[4] = {};
@@ -3900,6 +3901,11 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp)
drm_dp_mst_hpd_irq_send_new_request(&intel_dp->mst_mgr);
}
+ if (!tries) {
+ drm_dbg_kms(&i915->drm, "DPRX ESI not clearing, device may be stuck\n");
+ link_ok = false;
+ }
+
return link_ok;
}
diff --git a/drivers/gpu/drm/i915/display/intel_hdcp.c b/drivers/gpu/drm/i915/display/intel_hdcp.c
index cb7f86de967a..d31c1f1f0e50 100644
--- a/drivers/gpu/drm/i915/display/intel_hdcp.c
+++ b/drivers/gpu/drm/i915/display/intel_hdcp.c
@@ -182,11 +182,16 @@ int intel_hdcp_read_valid_bksv(struct intel_digital_port *dig_port,
/* Is HDCP1.4 capable on Platform and Sink */
bool intel_hdcp_capable(struct intel_connector *connector)
{
- struct intel_digital_port *dig_port = intel_attached_dig_port(connector);
+ struct intel_digital_port *dig_port;
const struct intel_hdcp_shim *shim = connector->hdcp.shim;
bool capable = false;
u8 bksv[5];
+ if (!intel_attached_encoder(connector))
+ return capable;
+
+ dig_port = intel_attached_dig_port(connector);
+
if (!shim)
return capable;
@@ -203,11 +208,16 @@ bool intel_hdcp_capable(struct intel_connector *connector)
/* Is HDCP2.2 capable on Platform and Sink */
bool intel_hdcp2_capable(struct intel_connector *connector)
{
- struct intel_digital_port *dig_port = intel_attached_dig_port(connector);
+ struct intel_digital_port *dig_port;
struct drm_i915_private *dev_priv = to_i915(connector->base.dev);
struct intel_hdcp *hdcp = &connector->hdcp;
bool capable = false;
+ if (!intel_attached_encoder(connector))
+ return capable;
+
+ dig_port = intel_attached_dig_port(connector);
+
/* I915 support for HDCP2.2 */
if (!hdcp->hdcp2_supported)
return false;
diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
index 58e64e21ae68..8dc53b4d1ebd 100644
--- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
+++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c
@@ -1752,6 +1752,7 @@ static int eb_reinit_userptr(struct i915_execbuffer *eb)
static noinline int eb_relocate_parse_slow(struct i915_execbuffer *eb)
{
+ const unsigned int count = eb->buffer_count;
bool have_copy = false;
struct eb_vma *ev;
int err = 0;
@@ -1854,7 +1855,6 @@ static noinline int eb_relocate_parse_slow(struct i915_execbuffer *eb)
out:
if (have_copy) {
- const unsigned int count = eb->buffer_count;
unsigned int i;
for (i = 0; i < count; i++) {
diff --git a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
index 321dbecba0f3..ca153f7f19dc 100644
--- a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
+++ b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c
@@ -3932,11 +3932,11 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
{
struct virtual_engine *ve;
unsigned int n;
- int err;
+ int err = -ENOMEM;
ve = kzalloc(struct_size(ve, siblings, count), GFP_KERNEL);
if (!ve)
- return ERR_PTR(-ENOMEM);
+ goto err;
ve->base.i915 = siblings[0]->i915;
ve->base.gt = siblings[0]->gt;
@@ -3968,10 +3968,8 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
intel_engine_init_execlists(&ve->base);
ve->base.sched_engine = i915_sched_engine_create(ENGINE_VIRTUAL);
- if (!ve->base.sched_engine) {
- err = -ENOMEM;
- goto err_put;
- }
+ if (!ve->base.sched_engine)
+ goto err_noput;
ve->base.sched_engine->private_data = &ve->base;
ve->base.cops = &virtual_context_ops;
@@ -3987,10 +3985,8 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
intel_context_init(&ve->context, &ve->base);
ve->base.breadcrumbs = intel_breadcrumbs_create(NULL);
- if (!ve->base.breadcrumbs) {
- err = -ENOMEM;
+ if (!ve->base.breadcrumbs)
goto err_put;
- }
for (n = 0; n < count; n++) {
struct intel_engine_cs *sibling = siblings[n];
@@ -4063,8 +4059,13 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count,
virtual_engine_initial_hint(ve);
return &ve->context;
+err_noput:
+ kfree(ve);
+ goto err;
+
err_put:
intel_context_put(&ve->context);
+err:
return ERR_PTR(err);
}
@@ -4153,17 +4154,6 @@ void intel_execlists_show_requests(struct intel_engine_cs *engine,
spin_unlock_irqrestore(&sched_engine->lock, flags);
}
-static unsigned long list_count(struct list_head *list)
-{
- struct list_head *pos;
- unsigned long count = 0;
-
- list_for_each(pos, list)
- count++;
-
- return count;
-}
-
void intel_execlists_dump_active_requests(struct intel_engine_cs *engine,
struct i915_request *hung_rq,
struct drm_printer *m)
@@ -4174,8 +4164,8 @@ void intel_execlists_dump_active_requests(struct intel_engine_cs *engine,
intel_engine_dump_active_requests(&engine->sched_engine->requests, hung_rq, m);
- drm_printf(m, "\tOn hold?: %lu\n",
- list_count(&engine->sched_engine->hold));
+ drm_printf(m, "\tOn hold?: %zu\n",
+ list_count_nodes(&engine->sched_engine->hold));
spin_unlock_irqrestore(&engine->sched_engine->lock, flags);
}
diff --git a/drivers/gpu/drm/i915/gt/intel_rps.c b/drivers/gpu/drm/i915/gt/intel_rps.c
index 6b86250c31ab..1e2f616dfc48 100644
--- a/drivers/gpu/drm/i915/gt/intel_rps.c
+++ b/drivers/gpu/drm/i915/gt/intel_rps.c
@@ -1874,7 +1874,8 @@ static void rps_work(struct work_struct *work)
out:
spin_lock_irq(gt->irq_lock);
- gen6_gt_pm_unmask_irq(gt, rps->pm_events);
+ if (intel_rps_is_active(rps))
+ gen6_gt_pm_unmask_irq(gt, rps->pm_events);
spin_unlock_irq(gt->irq_lock);
}
diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c
index 9e836fad4a65..2e918bd9a792 100644
--- a/drivers/gpu/drm/lima/lima_sched.c
+++ b/drivers/gpu/drm/lima/lima_sched.c
@@ -489,16 +489,21 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name)
{
unsigned int timeout = lima_sched_timeout_ms > 0 ?
lima_sched_timeout_ms : 500;
+ struct drm_sched_init_args args = {
+ .ops = &lima_sched_ops,
+ .credit_limit = 1,
+ .hang_limit = lima_job_hang_limit,
+ .timeout = msecs_to_jiffies(timeout),
+ .name = name,
+ .dev = pipe->ldev->dev,
+ };
pipe->fence_context = dma_fence_context_alloc(1);
spin_lock_init(&pipe->fence_lock);
INIT_WORK(&pipe->recover_work, lima_sched_recover_work);
- return drm_sched_init(&pipe->base, &lima_sched_ops, 1,
- lima_job_hang_limit,
- msecs_to_jiffies(timeout), NULL,
- NULL, name, pipe->ldev->dev);
+ return drm_sched_init(&pipe->base, &args);
}
void lima_sched_pipe_fini(struct lima_sched_pipe *pipe)
diff --git a/drivers/gpu/drm/mediatek/mtk_drm_crtc.c b/drivers/gpu/drm/mediatek/mtk_drm_crtc.c
index 59b310f7327c..8e72b3bb51eb 100644
--- a/drivers/gpu/drm/mediatek/mtk_drm_crtc.c
+++ b/drivers/gpu/drm/mediatek/mtk_drm_crtc.c
@@ -161,9 +161,8 @@ static void mtk_drm_crtc_destroy(struct drm_crtc *crtc)
mtk_mutex_put(mtk_crtc->mutex);
#if IS_REACHABLE(CONFIG_MTK_CMDQ)
- mtk_drm_cmdq_pkt_destroy(&mtk_crtc->cmdq_handle);
-
if (mtk_crtc->cmdq_client.chan) {
+ mtk_drm_cmdq_pkt_destroy(&mtk_crtc->cmdq_handle);
mbox_free_channel(mtk_crtc->cmdq_client.chan);
mtk_crtc->cmdq_client.chan = NULL;
}
diff --git a/drivers/gpu/drm/msm/adreno/adreno_gpu.c b/drivers/gpu/drm/msm/adreno/adreno_gpu.c
index 396f12b55047..8ff98efb0e8d 100644
--- a/drivers/gpu/drm/msm/adreno/adreno_gpu.c
+++ b/drivers/gpu/drm/msm/adreno/adreno_gpu.c
@@ -1052,6 +1052,8 @@ int adreno_gpu_init(struct drm_device *drm, struct platform_device *pdev,
adreno_gpu->revn = adreno_gpu->info->revn;
adreno_gpu->rev = *rev;
+ gpu->pdev = pdev;
+
if (adreno_read_speedbin(dev, &speedbin) || !speedbin)
speedbin = 0xffff;
adreno_gpu->speedbin = (uint16_t) (0xffff & speedbin);
diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys.h b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys.h
index f2af07d87f56..f1f022e9dcf7 100644
--- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys.h
+++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys.h
@@ -177,6 +177,7 @@ enum dpu_intr_idx {
* @hw_wb: Hardware interface to the wb registers
* @dpu_kms: Pointer to the dpu_kms top level
* @cached_mode: DRM mode cached at mode_set time, acted on in enable
+ * @vblank_ctl_lock: Vblank ctl mutex lock to protect vblank_refcount
* @enabled: Whether the encoder has enabled and running a mode
* @split_role: Role to play in a split-panel configuration
* @intf_mode: Interface mode
@@ -207,13 +208,14 @@ struct dpu_encoder_phys {
struct dpu_hw_wb *hw_wb;
struct dpu_kms *dpu_kms;
struct drm_display_mode cached_mode;
+ struct mutex vblank_ctl_lock;
enum dpu_enc_split_role split_role;
enum dpu_intf_mode intf_mode;
enum dpu_intf intf_idx;
enum dpu_wb wb_idx;
spinlock_t *enc_spinlock;
enum dpu_enc_enable_state enable_state;
- atomic_t vblank_refcount;
+ int vblank_refcount;
atomic_t vsync_cnt;
atomic_t underrun_cnt;
atomic_t pending_ctlstart_cnt;
diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_cmd.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_cmd.c
index 7581e418418f..b3cb507c052a 100644
--- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_cmd.c
+++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_cmd.c
@@ -246,7 +246,8 @@ static int dpu_encoder_phys_cmd_control_vblank_irq(
return -EINVAL;
}
- refcount = atomic_read(&phys_enc->vblank_refcount);
+ mutex_lock(&phys_enc->vblank_ctl_lock);
+ refcount = phys_enc->vblank_refcount;
/* Slave encoders don't report vblank */
if (!dpu_encoder_phys_cmd_is_master(phys_enc))
@@ -262,16 +263,24 @@ static int dpu_encoder_phys_cmd_control_vblank_irq(
phys_enc->hw_pp->idx - PINGPONG_0,
enable ? "true" : "false", refcount);
- if (enable && atomic_inc_return(&phys_enc->vblank_refcount) == 1)
- ret = dpu_core_irq_register_callback(phys_enc->dpu_kms,
- phys_enc->irq[INTR_IDX_RDPTR],
- dpu_encoder_phys_cmd_pp_rd_ptr_irq,
- phys_enc);
- else if (!enable && atomic_dec_return(&phys_enc->vblank_refcount) == 0)
- ret = dpu_core_irq_unregister_callback(phys_enc->dpu_kms,
- phys_enc->irq[INTR_IDX_RDPTR]);
+ if (enable) {
+ if (phys_enc->vblank_refcount == 0)
+ ret = dpu_core_irq_register_callback(phys_enc->dpu_kms,
+ phys_enc->irq[INTR_IDX_RDPTR],
+ dpu_encoder_phys_cmd_pp_rd_ptr_irq,
+ phys_enc);
+ if (!ret)
+ phys_enc->vblank_refcount++;
+ } else if (!enable) {
+ if (phys_enc->vblank_refcount == 1)
+ ret = dpu_core_irq_unregister_callback(phys_enc->dpu_kms,
+ phys_enc->irq[INTR_IDX_RDPTR]);
+ if (!ret)
+ phys_enc->vblank_refcount--;
+ }
end:
+ mutex_unlock(&phys_enc->vblank_ctl_lock);
if (ret) {
DRM_ERROR("vblank irq err id:%u pp:%d ret:%d, enable %s/%d\n",
DRMID(phys_enc->parent),
@@ -287,7 +296,7 @@ static void dpu_encoder_phys_cmd_irq_control(struct dpu_encoder_phys *phys_enc,
{
trace_dpu_enc_phys_cmd_irq_ctrl(DRMID(phys_enc->parent),
phys_enc->hw_pp->idx - PINGPONG_0,
- enable, atomic_read(&phys_enc->vblank_refcount));
+ enable, phys_enc->vblank_refcount);
if (enable) {
dpu_core_irq_register_callback(phys_enc->dpu_kms,
@@ -777,6 +786,9 @@ struct dpu_encoder_phys *dpu_encoder_phys_cmd_init(
phys_enc->hw_mdptop = p->dpu_kms->hw_mdp;
phys_enc->intf_idx = p->intf_idx;
+ mutex_init(&phys_enc->vblank_ctl_lock);
+ phys_enc->vblank_refcount = 0;
+
dpu_encoder_phys_cmd_init_ops(&phys_enc->ops);
phys_enc->parent = p->parent;
phys_enc->parent_ops = p->parent_ops;
@@ -789,7 +801,6 @@ struct dpu_encoder_phys *dpu_encoder_phys_cmd_init(
for (i = 0; i < ARRAY_SIZE(phys_enc->irq); i++)
phys_enc->irq[i] = -EINVAL;
- atomic_set(&phys_enc->vblank_refcount, 0);
atomic_set(&phys_enc->pending_kickoff_cnt, 0);
atomic_set(&phys_enc->pending_ctlstart_cnt, 0);
atomic_set(&cmd_enc->pending_vblank_cnt, 0);
diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_vid.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_vid.c
index 4b7952c4e639..3b45a50aa985 100644
--- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_vid.c
+++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_vid.c
@@ -375,7 +375,8 @@ static int dpu_encoder_phys_vid_control_vblank_irq(
int ret = 0;
int refcount;
- refcount = atomic_read(&phys_enc->vblank_refcount);
+ mutex_lock(&phys_enc->vblank_ctl_lock);
+ refcount = phys_enc->vblank_refcount;
/* Slave encoders don't report vblank */
if (!dpu_encoder_phys_vid_is_master(phys_enc))
@@ -388,18 +389,26 @@ static int dpu_encoder_phys_vid_control_vblank_irq(
}
DRM_DEBUG_VBL("id:%u enable=%d/%d\n", DRMID(phys_enc->parent), enable,
- atomic_read(&phys_enc->vblank_refcount));
+ refcount);
- if (enable && atomic_inc_return(&phys_enc->vblank_refcount) == 1)
- ret = dpu_core_irq_register_callback(phys_enc->dpu_kms,
- phys_enc->irq[INTR_IDX_VSYNC],
- dpu_encoder_phys_vid_vblank_irq,
- phys_enc);
- else if (!enable && atomic_dec_return(&phys_enc->vblank_refcount) == 0)
- ret = dpu_core_irq_unregister_callback(phys_enc->dpu_kms,
- phys_enc->irq[INTR_IDX_VSYNC]);
+ if (enable) {
+ if (phys_enc->vblank_refcount == 0)
+ ret = dpu_core_irq_register_callback(phys_enc->dpu_kms,
+ phys_enc->irq[INTR_IDX_VSYNC],
+ dpu_encoder_phys_vid_vblank_irq,
+ phys_enc);
+ if (!ret)
+ phys_enc->vblank_refcount++;
+ } else if (!enable) {
+ if (phys_enc->vblank_refcount == 1)
+ ret = dpu_core_irq_unregister_callback(phys_enc->dpu_kms,
+ phys_enc->irq[INTR_IDX_VSYNC]);
+ if (!ret)
+ phys_enc->vblank_refcount--;
+ }
end:
+ mutex_unlock(&phys_enc->vblank_ctl_lock);
if (ret) {
DRM_ERROR("failed: id:%u intf:%d ret:%d enable:%d refcnt:%d\n",
DRMID(phys_enc->parent),
@@ -607,7 +616,7 @@ static void dpu_encoder_phys_vid_irq_control(struct dpu_encoder_phys *phys_enc,
trace_dpu_enc_phys_vid_irq_ctrl(DRMID(phys_enc->parent),
phys_enc->hw_intf->idx - INTF_0,
enable,
- atomic_read(&phys_enc->vblank_refcount));
+ phys_enc->vblank_refcount);
if (enable) {
ret = dpu_encoder_phys_vid_control_vblank_irq(phys_enc, true);
@@ -703,6 +712,9 @@ struct dpu_encoder_phys *dpu_encoder_phys_vid_init(
DPU_DEBUG_VIDENC(phys_enc, "\n");
+ mutex_init(&phys_enc->vblank_ctl_lock);
+ phys_enc->vblank_refcount = 0;
+
dpu_encoder_phys_vid_init_ops(&phys_enc->ops);
phys_enc->parent = p->parent;
phys_enc->parent_ops = p->parent_ops;
@@ -713,7 +725,6 @@ struct dpu_encoder_phys *dpu_encoder_phys_vid_init(
for (i = 0; i < ARRAY_SIZE(phys_enc->irq); i++)
phys_enc->irq[i] = -EINVAL;
- atomic_set(&phys_enc->vblank_refcount, 0);
atomic_set(&phys_enc->pending_kickoff_cnt, 0);
init_waitqueue_head(&phys_enc->pending_kickoff_wq);
phys_enc->enable_state = DPU_ENC_DISABLED;
diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_wb.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_wb.c
index fd5f9d04f81e..d76c7a2e44bf 100644
--- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_wb.c
+++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_wb.c
@@ -212,14 +212,12 @@ static void dpu_encoder_phys_wb_setup_cdp(struct dpu_encoder_phys *phys_enc)
if (mode_3d && hw_pp && hw_pp->merge_3d)
intf_cfg.merge_3d = hw_pp->merge_3d->idx;
- if (phys_enc->hw_pp->merge_3d && phys_enc->hw_pp->merge_3d->ops.setup_3d_mode)
- phys_enc->hw_pp->merge_3d->ops.setup_3d_mode(phys_enc->hw_pp->merge_3d,
- mode_3d);
+ if (hw_pp && hw_pp->merge_3d && hw_pp->merge_3d->ops.setup_3d_mode)
+ hw_pp->merge_3d->ops.setup_3d_mode(hw_pp->merge_3d, mode_3d);
/* setup which pp blk will connect to this wb */
- if (hw_pp && phys_enc->hw_wb->ops.bind_pingpong_blk)
- phys_enc->hw_wb->ops.bind_pingpong_blk(phys_enc->hw_wb, true,
- phys_enc->hw_pp->idx);
+ if (hw_pp && hw_wb->ops.bind_pingpong_blk)
+ hw_wb->ops.bind_pingpong_blk(hw_wb, true, hw_pp->idx);
phys_enc->hw_ctl->ops.setup_intf_cfg(phys_enc->hw_ctl, &intf_cfg);
} else if (phys_enc->hw_ctl && phys_enc->hw_ctl->ops.setup_intf_cfg) {
@@ -735,7 +733,6 @@ struct dpu_encoder_phys *dpu_encoder_phys_wb_init(
phys_enc->irq[i] = -EINVAL;
atomic_set(&phys_enc->pending_kickoff_cnt, 0);
- atomic_set(&phys_enc->vblank_refcount, 0);
wb_enc->wb_done_timeout_cnt = 0;
init_waitqueue_head(&phys_enc->pending_kickoff_wq);
diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.c
index 75e1b89c9eac..346bb684a35d 100644
--- a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.c
+++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.c
@@ -156,8 +156,10 @@ static void dpu_core_irq_callback_handler(struct dpu_kms *dpu_kms, int irq_idx)
{
VERB("irq_idx=%d\n", irq_idx);
- if (!dpu_kms->hw_intr->irq_tbl[irq_idx].cb)
+ if (!dpu_kms->hw_intr->irq_tbl[irq_idx].cb) {
DRM_ERROR("no registered cb, idx:%d\n", irq_idx);
+ return;
+ }
atomic_inc(&dpu_kms->hw_intr->irq_tbl[irq_idx].count);
diff --git a/drivers/gpu/drm/msm/msm_gem_submit.c b/drivers/gpu/drm/msm/msm_gem_submit.c
index 6a02f790624a..df2f1348897c 100644
--- a/drivers/gpu/drm/msm/msm_gem_submit.c
+++ b/drivers/gpu/drm/msm/msm_gem_submit.c
@@ -856,8 +856,7 @@ int msm_ioctl_gem_submit(struct drm_device *dev, void *data,
goto out;
if (!submit->cmd[i].size ||
- ((submit->cmd[i].size + submit->cmd[i].offset) >
- msm_obj->base.size / 4)) {
+ (size_add(submit->cmd[i].size, submit->cmd[i].offset) > msm_obj->base.size / 4)) {
DRM_ERROR("invalid cmdstream size: %u\n", submit->cmd[i].size * 4);
ret = -EINVAL;
goto out;
diff --git a/drivers/gpu/drm/msm/msm_gpu.c b/drivers/gpu/drm/msm/msm_gpu.c
index 380249500325..55cbebe7a168 100644
--- a/drivers/gpu/drm/msm/msm_gpu.c
+++ b/drivers/gpu/drm/msm/msm_gpu.c
@@ -421,25 +421,23 @@ static void recover_worker(struct kthread_work *work)
msm_update_fence(ring->fctx, fence);
}
- if (msm_gpu_active(gpu)) {
- /* retire completed submits, plus the one that hung: */
- retire_submits(gpu);
+ gpu->funcs->recover(gpu);
- gpu->funcs->recover(gpu);
+ /* retire completed submits, plus the one that hung: */
+ retire_submits(gpu);
- /*
- * Replay all remaining submits starting with highest priority
- * ring
- */
- for (i = 0; i < gpu->nr_rings; i++) {
- struct msm_ringbuffer *ring = gpu->rb[i];
- unsigned long flags;
+ /*
+ * Replay all remaining submits starting with highest priority
+ * ring
+ */
+ for (i = 0; i < gpu->nr_rings; i++) {
+ struct msm_ringbuffer *ring = gpu->rb[i];
+ unsigned long flags;
- spin_lock_irqsave(&ring->submit_lock, flags);
- list_for_each_entry(submit, &ring->submits, node)
- gpu->funcs->submit(gpu, submit);
- spin_unlock_irqrestore(&ring->submit_lock, flags);
- }
+ spin_lock_irqsave(&ring->submit_lock, flags);
+ list_for_each_entry(submit, &ring->submits, node)
+ gpu->funcs->submit(gpu, submit);
+ spin_unlock_irqrestore(&ring->submit_lock, flags);
}
pm_runtime_put(&gpu->pdev->dev);
@@ -938,7 +936,6 @@ int msm_gpu_init(struct drm_device *drm, struct platform_device *pdev,
gpu->cx_collapse = devm_reset_control_get_optional_exclusive(&pdev->dev,
"cx_collapse");
- gpu->pdev = pdev;
platform_set_drvdata(pdev, &gpu->adreno_smmu);
msm_devfreq_init(gpu);
diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c
index 57a8e9564540..af9ba8bba445 100644
--- a/drivers/gpu/drm/msm/msm_ringbuffer.c
+++ b/drivers/gpu/drm/msm/msm_ringbuffer.c
@@ -56,8 +56,13 @@ static const struct drm_sched_backend_ops msm_sched_ops = {
struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id,
void *memptrs, uint64_t memptrs_iova)
{
+ struct drm_sched_init_args args = {
+ .ops = &msm_sched_ops,
+ .credit_limit = num_hw_submissions,
+ .timeout = MAX_SCHEDULE_TIMEOUT,
+ .dev = gpu->dev->dev,
+ };
struct msm_ringbuffer *ring;
- long sched_timeout;
char name[32];
int ret;
@@ -84,6 +89,7 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id,
}
msm_gem_object_set_name(ring->bo, "ring%d", id);
+ args.name = to_msm_bo(ring->bo)->name;
ring->end = ring->start + (MSM_GPU_RINGBUFFER_SZ >> 2);
ring->next = ring->start;
@@ -92,15 +98,10 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id,
ring->memptrs = memptrs;
ring->memptrs_iova = memptrs_iova;
- /* currently managing hangcheck ourselves: */
- sched_timeout = MAX_SCHEDULE_TIMEOUT;
-
- ret = drm_sched_init(&ring->sched, &msm_sched_ops,
- num_hw_submissions, 0, sched_timeout,
- NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev);
- if (ret) {
+ ret = drm_sched_init(&ring->sched, &args);
+ if (ret)
goto fail;
- }
+ ring->sched_initialized = true;
INIT_LIST_HEAD(&ring->submits);
spin_lock_init(&ring->submit_lock);
@@ -122,7 +123,8 @@ void msm_ringbuffer_destroy(struct msm_ringbuffer *ring)
if (IS_ERR_OR_NULL(ring))
return;
- drm_sched_fini(&ring->sched);
+ if (ring->sched_initialized)
+ drm_sched_fini(&ring->sched);
msm_fence_context_free(ring->fctx);
diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.h b/drivers/gpu/drm/msm/msm_ringbuffer.h
index 698b333abccd..eafaa7cef324 100644
--- a/drivers/gpu/drm/msm/msm_ringbuffer.h
+++ b/drivers/gpu/drm/msm/msm_ringbuffer.h
@@ -50,6 +50,7 @@ struct msm_ringbuffer {
* The job scheduler for this ring.
*/
struct drm_gpu_scheduler sched;
+ bool sched_initialized;
/*
* List of in-flight submits on this ring. Protected by submit_lock.
diff --git a/drivers/gpu/drm/nouveau/include/nvkm/core/client.h b/drivers/gpu/drm/nouveau/include/nvkm/core/client.h
index 2f86606e708c..e0d79bb7a1d8 100644
--- a/drivers/gpu/drm/nouveau/include/nvkm/core/client.h
+++ b/drivers/gpu/drm/nouveau/include/nvkm/core/client.h
@@ -12,6 +12,7 @@ struct nvkm_client {
struct nvkm_client_notify *notify[32];
struct rb_root objroot;
+ spinlock_t obj_lock;
void *data;
int (*ntfy)(const void *, u32, const void *, u32);
diff --git a/drivers/gpu/drm/nouveau/nouveau_chan.c b/drivers/gpu/drm/nouveau/nouveau_chan.c
index 48dea5d0c580..5b62959320b4 100644
--- a/drivers/gpu/drm/nouveau/nouveau_chan.c
+++ b/drivers/gpu/drm/nouveau/nouveau_chan.c
@@ -87,6 +87,14 @@ nouveau_channel_del(struct nouveau_channel **pchan)
if (chan) {
struct nouveau_cli *cli = (void *)chan->user.client;
+ /*
+ * Drop the kill-event subscription first. Its handler
+ * dereferences chan->fence, which the fence context teardown
+ * below frees, so leaving it armed across the teardown leaves
+ * a window for a use-after-free.
+ */
+ nvif_notify_dtor(&chan->kill);
+
if (chan->fence)
nouveau_fence(chan->drm)->context_del(chan);
@@ -96,7 +104,6 @@ nouveau_channel_del(struct nouveau_channel **pchan)
nvif_object_dtor(&chan->nvsw);
nvif_object_dtor(&chan->gart);
nvif_object_dtor(&chan->vram);
- nvif_notify_dtor(&chan->kill);
nvif_object_dtor(&chan->user);
nvif_object_dtor(&chan->push.ctxdma);
nouveau_vma_del(&chan->push.vma);
diff --git a/drivers/gpu/drm/nouveau/nouveau_gem.c b/drivers/gpu/drm/nouveau/nouveau_gem.c
index cde21cc0083b..4c9942f651b6 100644
--- a/drivers/gpu/drm/nouveau/nouveau_gem.c
+++ b/drivers/gpu/drm/nouveau/nouveau_gem.c
@@ -277,11 +277,20 @@ nouveau_gem_info(struct drm_file *file_priv, struct drm_gem_object *gem,
rep->domain = NOUVEAU_GEM_DOMAIN_VRAM;
rep->offset = nvbo->offset;
if (vmm->vmm.object.oclass >= NVIF_CLASS_VMM_NV50) {
+ int ret;
+
+ ret = ttm_bo_reserve(&nvbo->bo, false, false, NULL);
+ if (ret)
+ return ret;
+
vma = nouveau_vma_find(nvbo, vmm);
- if (!vma)
+ if (!vma) {
+ ttm_bo_unreserve(&nvbo->bo);
return -EINVAL;
+ }
rep->offset = vma->addr;
+ ttm_bo_unreserve(&nvbo->bo);
}
rep->size = nvbo->bo.base.size;
diff --git a/drivers/gpu/drm/nouveau/nvkm/core/client.c b/drivers/gpu/drm/nouveau/nvkm/core/client.c
index 0c8c55c73b12..db96699c5b2b 100644
--- a/drivers/gpu/drm/nouveau/nvkm/core/client.c
+++ b/drivers/gpu/drm/nouveau/nvkm/core/client.c
@@ -300,6 +300,7 @@ nvkm_client_new(const char *name, u64 device, const char *cfg,
client->device = device;
client->debug = nvkm_dbgopt(dbg, "CLIENT");
client->objroot = RB_ROOT;
+ spin_lock_init(&client->obj_lock);
client->ntfy = ntfy;
INIT_LIST_HEAD(&client->umem);
spin_lock_init(&client->lock);
diff --git a/drivers/gpu/drm/nouveau/nvkm/core/object.c b/drivers/gpu/drm/nouveau/nvkm/core/object.c
index 301a5e5b5f7f..4c038d5715b9 100644
--- a/drivers/gpu/drm/nouveau/nvkm/core/object.c
+++ b/drivers/gpu/drm/nouveau/nvkm/core/object.c
@@ -30,8 +30,10 @@ nvkm_object_search(struct nvkm_client *client, u64 handle,
const struct nvkm_object_func *func)
{
struct nvkm_object *object;
+ unsigned long flags;
if (handle) {
+ spin_lock_irqsave(&client->obj_lock, flags);
struct rb_node *node = client->objroot.rb_node;
while (node) {
object = rb_entry(node, typeof(*object), node);
@@ -40,9 +42,12 @@ nvkm_object_search(struct nvkm_client *client, u64 handle,
else
if (handle > object->object)
node = node->rb_right;
- else
+ else {
+ spin_unlock_irqrestore(&client->obj_lock, flags);
goto done;
+ }
}
+ spin_unlock_irqrestore(&client->obj_lock, flags);
return ERR_PTR(-ENOENT);
} else {
object = &client->object;
@@ -57,30 +62,39 @@ nvkm_object_search(struct nvkm_client *client, u64 handle,
void
nvkm_object_remove(struct nvkm_object *object)
{
+ unsigned long flags;
+
+ spin_lock_irqsave(&object->client->obj_lock, flags);
if (!RB_EMPTY_NODE(&object->node))
rb_erase(&object->node, &object->client->objroot);
+ spin_unlock_irqrestore(&object->client->obj_lock, flags);
}
bool
nvkm_object_insert(struct nvkm_object *object)
{
- struct rb_node **ptr = &object->client->objroot.rb_node;
+ struct rb_node **ptr;
struct rb_node *parent = NULL;
+ unsigned long flags;
+ spin_lock_irqsave(&object->client->obj_lock, flags);
+ ptr = &object->client->objroot.rb_node;
while (*ptr) {
struct nvkm_object *this = rb_entry(*ptr, typeof(*this), node);
parent = *ptr;
- if (object->object < this->object)
+ if (object->object < this->object) {
ptr = &parent->rb_left;
- else
- if (object->object > this->object)
+ } else if (object->object > this->object) {
ptr = &parent->rb_right;
- else
+ } else {
+ spin_unlock_irqrestore(&object->client->obj_lock, flags);
return false;
+ }
}
rb_link_node(&object->node, parent, ptr);
rb_insert_color(&object->node, &object->client->objroot);
+ spin_unlock_irqrestore(&object->client->obj_lock, flags);
return true;
}
diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c
index dbc597ab46fb..4b06edb6b05c 100644
--- a/drivers/gpu/drm/panfrost/panfrost_job.c
+++ b/drivers/gpu/drm/panfrost/panfrost_job.c
@@ -777,6 +777,12 @@ int panfrost_job_init(struct panfrost_device *pfdev)
{
struct panfrost_job_slot *js;
unsigned int nentries = 2;
+ struct drm_sched_init_args args = {
+ .ops = &panfrost_sched_ops,
+ .timeout = msecs_to_jiffies(JOB_TIMEOUT_MS),
+ .name = "pan_js",
+ .dev = pfdev->dev,
+ };
int ret, j;
/* All GPUs have two entries per queue, but without jobchain
@@ -811,15 +817,13 @@ int panfrost_job_init(struct panfrost_device *pfdev)
if (!pfdev->reset.wq)
return -ENOMEM;
+ args.credit_limit = nentries;
+ args.timeout_wq = pfdev->reset.wq;
+
for (j = 0; j < NUM_JOB_SLOTS; j++) {
js->queue[j].fence_context = dma_fence_context_alloc(1);
- ret = drm_sched_init(&js->queue[j].sched,
- &panfrost_sched_ops,
- nentries, 0,
- msecs_to_jiffies(JOB_TIMEOUT_MS),
- pfdev->reset.wq,
- NULL, "pan_js", pfdev->dev);
+ ret = drm_sched_init(&js->queue[j].sched, &args);
if (ret) {
dev_err(pfdev->dev, "Failed to create scheduler: %d.", ret);
goto err_sched;
diff --git a/drivers/gpu/drm/panfrost/panfrost_mmu.c b/drivers/gpu/drm/panfrost/panfrost_mmu.c
index e961fa27702c..5aa6d94e95c9 100644
--- a/drivers/gpu/drm/panfrost/panfrost_mmu.c
+++ b/drivers/gpu/drm/panfrost/panfrost_mmu.c
@@ -500,12 +500,19 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
mapping_set_unevictable(mapping);
for (i = page_offset; i < page_offset + NUM_FAULT_PAGES; i++) {
+ /* Can happen if the last fault only partially filled this
+ * section of the pages array before failing. In that case
+ * we skip already filled pages.
+ */
+ if (pages[i])
+ continue;
+
pages[i] = shmem_read_mapping_page(mapping, i);
if (IS_ERR(pages[i])) {
mutex_unlock(&bo->base.pages_lock);
ret = PTR_ERR(pages[i]);
pages[i] = NULL;
- goto err_pages;
+ goto err_bo;
}
}
@@ -515,7 +522,7 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
ret = sg_alloc_table_from_pages(sgt, pages + page_offset,
NUM_FAULT_PAGES, 0, SZ_2M, GFP_KERNEL);
if (ret)
- goto err_pages;
+ goto err_bo;
ret = dma_map_sgtable(pfdev->dev, sgt, DMA_BIDIRECTIONAL, 0);
if (ret)
@@ -535,8 +542,6 @@ static int panfrost_mmu_map_fault_addr(struct panfrost_device *pfdev, int as,
err_map:
sg_free_table(sgt);
-err_pages:
- drm_gem_shmem_put_pages(&bo->base);
err_bo:
panfrost_gem_mapping_put(bomapping);
return ret;
diff --git a/drivers/gpu/drm/radeon/radeon_bios.c b/drivers/gpu/drm/radeon/radeon_bios.c
index 63bdc9f6fc24..1fecb46d421f 100644
--- a/drivers/gpu/drm/radeon/radeon_bios.c
+++ b/drivers/gpu/drm/radeon/radeon_bios.c
@@ -49,6 +49,7 @@
static bool igp_read_bios_from_vram(struct radeon_device *rdev)
{
uint8_t __iomem *bios;
+ uint8_t val1, val2;
resource_size_t vram_base;
resource_size_t size = 256 * 1024; /* ??? */
@@ -63,7 +64,10 @@ static bool igp_read_bios_from_vram(struct radeon_device *rdev)
return false;
}
- if (size == 0 || bios[0] != 0x55 || bios[1] != 0xaa) {
+ val1 = readb(&bios[0]);
+ val2 = readb(&bios[1]);
+
+ if (size == 0 || val1 != 0x55 || val2 != 0xaa) {
iounmap(bios);
return false;
}
diff --git a/drivers/gpu/drm/radeon/radeon_fb.c b/drivers/gpu/drm/radeon/radeon_fb.c
index 6ccea51d4072..46800703cd20 100644
--- a/drivers/gpu/drm/radeon/radeon_fb.c
+++ b/drivers/gpu/drm/radeon/radeon_fb.c
@@ -25,6 +25,7 @@
*/
#include <linux/module.h>
+#include <linux/overflow.h>
#include <linux/pci.h>
#include <linux/pm_runtime.h>
#include <linux/slab.h>
@@ -91,6 +92,7 @@ int radeon_align_pitch(struct radeon_device *rdev, int width, int cpp, bool tile
int aligned = width;
int align_large = (ASIC_IS_AVIVO(rdev)) || tiled;
int pitch_mask = 0;
+ int pitch;
switch (cpp) {
case 1:
@@ -105,9 +107,12 @@ int radeon_align_pitch(struct radeon_device *rdev, int width, int cpp, bool tile
break;
}
- aligned += pitch_mask;
+ if (check_add_overflow(aligned, pitch_mask, &aligned))
+ return 0;
aligned &= ~pitch_mask;
- return aligned * cpp;
+ if (check_mul_overflow(aligned, cpp, &pitch))
+ return 0;
+ return pitch;
}
static void radeonfb_destroy_pinned_object(struct drm_gem_object *gobj)
diff --git a/drivers/gpu/drm/radeon/radeon_gem.c b/drivers/gpu/drm/radeon/radeon_gem.c
index 3388a3d21d2c..5e5d4de65bf7 100644
--- a/drivers/gpu/drm/radeon/radeon_gem.c
+++ b/drivers/gpu/drm/radeon/radeon_gem.c
@@ -831,8 +831,12 @@ int radeon_mode_dumb_create(struct drm_file *file_priv,
args->pitch = radeon_align_pitch(rdev, args->width,
DIV_ROUND_UP(args->bpp, 8), 0);
+ if (!args->pitch)
+ return -EINVAL;
args->size = (u64)args->pitch * args->height;
args->size = ALIGN(args->size, PAGE_SIZE);
+ if (!args->size)
+ return -EINVAL;
r = radeon_gem_object_create(rdev, args->size, 0,
RADEON_GEM_DOMAIN_VRAM, 0,
diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
index 1a17e9c1e833..608b0d681b21 100644
--- a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
+++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c
@@ -2409,6 +2409,10 @@ static int vop2_create_crtc(struct vop2 *vop2)
if (!vp->crtc.port)
continue;
+ if (!vp->primary_plane)
+ return dev_err_probe(drm->dev, -ENOENT,
+ "no primary plane for vp %d\n", i);
+
plane = &vp->primary_plane->base;
ret = drm_crtc_init_with_planes(drm, &vp->crtc, plane, NULL,
diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c
index a80001350411..566871f35cf1 100644
--- a/drivers/gpu/drm/scheduler/sched_main.c
+++ b/drivers/gpu/drm/scheduler/sched_main.c
@@ -994,33 +994,22 @@ static int drm_sched_main(void *param)
* drm_sched_init - Init a gpu scheduler instance
*
* @sched: scheduler instance
- * @ops: backend operations for this scheduler
- * @hw_submission: number of hw submissions that can be in flight
- * @hang_limit: number of times to allow a job to hang before dropping it
- * @timeout: timeout value in jiffies for the scheduler
- * @timeout_wq: workqueue to use for timeout work. If NULL, the system_wq is
- * used
- * @score: optional score atomic shared with other schedulers
- * @name: name used for debugging
- * @dev: target &struct device
+ * @args: scheduler initialization arguments
*
* Return 0 on success, otherwise error code.
*/
int drm_sched_init(struct drm_gpu_scheduler *sched,
- const struct drm_sched_backend_ops *ops,
- unsigned hw_submission, unsigned hang_limit,
- long timeout, struct workqueue_struct *timeout_wq,
- atomic_t *score, const char *name, struct device *dev)
+ const struct drm_sched_init_args *args)
{
int i, ret;
- sched->ops = ops;
- sched->hw_submission_limit = hw_submission;
- sched->name = name;
- sched->timeout = timeout;
- sched->timeout_wq = timeout_wq ? : system_wq;
- sched->hang_limit = hang_limit;
- sched->score = score ? score : &sched->_score;
- sched->dev = dev;
+ sched->ops = args->ops;
+ sched->hw_submission_limit = args->credit_limit;
+ sched->name = args->name;
+ sched->timeout = args->timeout;
+ sched->timeout_wq = args->timeout_wq ? : system_wq;
+ sched->hang_limit = args->hang_limit;
+ sched->score = args->score ? args->score : &sched->_score;
+ sched->dev = args->dev;
for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++)
drm_sched_rq_init(sched, &sched->sched_rq[i]);
@@ -1038,7 +1027,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched,
if (IS_ERR(sched->thread)) {
ret = PTR_ERR(sched->thread);
sched->thread = NULL;
- DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", name);
+ DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", sched->name);
return ret;
}
diff --git a/drivers/gpu/drm/tegra/gr2d.c b/drivers/gpu/drm/tegra/gr2d.c
index e3bb4c99ed39..9deed8345d76 100644
--- a/drivers/gpu/drm/tegra/gr2d.c
+++ b/drivers/gpu/drm/tegra/gr2d.c
@@ -282,16 +282,16 @@ static int gr2d_probe(struct platform_device *pdev)
if (err)
return err;
+ /* initialize address register map */
+ for (i = 0; i < ARRAY_SIZE(gr2d_addr_regs); i++)
+ set_bit(gr2d_addr_regs[i], gr2d->addr_regs);
+
err = host1x_client_register(&gr2d->client.base);
if (err < 0) {
dev_err(dev, "failed to register host1x client: %d\n", err);
return err;
}
- /* initialize address register map */
- for (i = 0; i < ARRAY_SIZE(gr2d_addr_regs); i++)
- set_bit(gr2d_addr_regs[i], gr2d->addr_regs);
-
return 0;
}
diff --git a/drivers/gpu/drm/tegra/gr3d.c b/drivers/gpu/drm/tegra/gr3d.c
index a1fd3113ea96..98dc2ed35915 100644
--- a/drivers/gpu/drm/tegra/gr3d.c
+++ b/drivers/gpu/drm/tegra/gr3d.c
@@ -536,6 +536,10 @@ static int gr3d_probe(struct platform_device *pdev)
if (err)
return err;
+ /* initialize address register map */
+ for (i = 0; i < ARRAY_SIZE(gr3d_addr_regs); i++)
+ set_bit(gr3d_addr_regs[i], gr3d->addr_regs);
+
err = host1x_client_register(&gr3d->client.base);
if (err < 0) {
dev_err(&pdev->dev, "failed to register host1x client: %d\n",
@@ -543,10 +547,6 @@ static int gr3d_probe(struct platform_device *pdev)
return err;
}
- /* initialize address register map */
- for (i = 0; i < ARRAY_SIZE(gr3d_addr_regs); i++)
- set_bit(gr3d_addr_regs[i], gr3d->addr_regs);
-
return 0;
}
diff --git a/drivers/gpu/drm/tiny/simpledrm.c b/drivers/gpu/drm/tiny/simpledrm.c
index 18489779fb8a..4b5ccbb13ecf 100644
--- a/drivers/gpu/drm/tiny/simpledrm.c
+++ b/drivers/gpu/drm/tiny/simpledrm.c
@@ -693,9 +693,15 @@ static struct simpledrm_device *simpledrm_device_create(struct drm_driver *drv,
return ERR_PTR(-ENODEV);
}
if (!stride) {
- stride = drm_format_info_min_pitch(format, 0, width);
- if (drm_WARN_ON(dev, !stride))
+ u64 pitch = drm_format_info_min_pitch(format, 0, width);
+
+ if (drm_WARN_ON(dev, !pitch)) {
+ return ERR_PTR(-EINVAL); /* driver bug */
+ } else if (pitch > INT_MAX) {
+ drm_warn(dev, "stride of %llu exceeds maximum\n", pitch);
return ERR_PTR(-EINVAL);
+ }
+ stride = pitch;
}
sdev->mode = simpledrm_mode(width, height);
diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c
index ab872bc81800..e31a3c8ff924 100644
--- a/drivers/gpu/drm/v3d/v3d_sched.c
+++ b/drivers/gpu/drm/v3d/v3d_sched.c
@@ -77,15 +77,11 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job)
struct v3d_bin_job *job = to_bin_job(sched_job);
struct v3d_dev *v3d = job->base.v3d;
struct drm_device *dev = &v3d->drm;
- struct dma_fence *fence;
+ struct dma_fence *fence = NULL;
unsigned long irqflags;
- if (unlikely(job->base.base.s_fence->finished.error)) {
- spin_lock_irqsave(&v3d->job_lock, irqflags);
- v3d->queue[V3D_BIN].active_job = NULL;
- spin_unlock_irqrestore(&v3d->job_lock, irqflags);
- return NULL;
- }
+ if (unlikely(job->base.base.s_fence->finished.error))
+ goto out_clean_job;
/* Lock required around bin_job update vs
* v3d_overflow_mem_work().
@@ -102,7 +98,7 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job)
fence = v3d_fence_create(v3d, V3D_BIN);
if (IS_ERR(fence))
- return NULL;
+ goto out_clean_job;
if (job->base.irq_fence)
dma_fence_put(job->base.irq_fence);
@@ -129,6 +125,12 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job)
V3D_CORE_WRITE(0, V3D_CLE_CT0QEA, job->end);
return fence;
+
+out_clean_job:
+ spin_lock_irqsave(&v3d->job_lock, irqflags);
+ v3d->queue[V3D_BIN].active_job = NULL;
+ spin_unlock_irqrestore(&v3d->job_lock, irqflags);
+ return fence;
}
static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job)
@@ -136,12 +138,10 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job)
struct v3d_render_job *job = to_render_job(sched_job);
struct v3d_dev *v3d = job->base.v3d;
struct drm_device *dev = &v3d->drm;
- struct dma_fence *fence;
+ struct dma_fence *fence = NULL;
- if (unlikely(job->base.base.s_fence->finished.error)) {
- v3d->queue[V3D_RENDER].active_job = NULL;
- return NULL;
- }
+ if (unlikely(job->base.base.s_fence->finished.error))
+ goto out_clean_job;
v3d->queue[V3D_RENDER].active_job = &job->base;
@@ -155,7 +155,7 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job)
fence = v3d_fence_create(v3d, V3D_RENDER);
if (IS_ERR(fence))
- return NULL;
+ goto out_clean_job;
if (job->base.irq_fence)
dma_fence_put(job->base.irq_fence);
@@ -175,6 +175,10 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job)
V3D_CORE_WRITE(0, V3D_CLE_CT1QEA, job->end);
return fence;
+
+out_clean_job:
+ v3d->queue[V3D_RENDER].active_job = NULL;
+ return fence;
}
static struct dma_fence *
@@ -183,18 +187,16 @@ v3d_tfu_job_run(struct drm_sched_job *sched_job)
struct v3d_tfu_job *job = to_tfu_job(sched_job);
struct v3d_dev *v3d = job->base.v3d;
struct drm_device *dev = &v3d->drm;
- struct dma_fence *fence;
+ struct dma_fence *fence = NULL;
- if (unlikely(job->base.base.s_fence->finished.error)) {
- v3d->queue[V3D_TFU].active_job = NULL;
- return NULL;
- }
+ if (unlikely(job->base.base.s_fence->finished.error))
+ goto out_clean_job;
v3d->queue[V3D_TFU].active_job = &job->base;
fence = v3d_fence_create(v3d, V3D_TFU);
if (IS_ERR(fence))
- return NULL;
+ goto out_clean_job;
if (job->base.irq_fence)
dma_fence_put(job->base.irq_fence);
@@ -218,6 +220,10 @@ v3d_tfu_job_run(struct drm_sched_job *sched_job)
V3D_WRITE(V3D_TFU_ICFG, job->args.icfg | V3D_TFU_ICFG_IOC);
return fence;
+
+out_clean_job:
+ v3d->queue[V3D_TFU].active_job = NULL;
+ return fence;
}
static struct dma_fence *
@@ -226,13 +232,11 @@ v3d_csd_job_run(struct drm_sched_job *sched_job)
struct v3d_csd_job *job = to_csd_job(sched_job);
struct v3d_dev *v3d = job->base.v3d;
struct drm_device *dev = &v3d->drm;
- struct dma_fence *fence;
+ struct dma_fence *fence = NULL;
int i;
- if (unlikely(job->base.base.s_fence->finished.error)) {
- v3d->queue[V3D_CSD].active_job = NULL;
- return NULL;
- }
+ if (unlikely(job->base.base.s_fence->finished.error))
+ goto out_clean_job;
/* The HW interprets a workgroup size of 0 as 65536; however, the
* user-space driver exposes a maximum of 65535. Therefore, a 0 in
@@ -250,7 +254,7 @@ v3d_csd_job_run(struct drm_sched_job *sched_job)
fence = v3d_fence_create(v3d, V3D_CSD);
if (IS_ERR(fence))
- return NULL;
+ goto out_clean_job;
if (job->base.irq_fence)
dma_fence_put(job->base.irq_fence);
@@ -266,6 +270,10 @@ v3d_csd_job_run(struct drm_sched_job *sched_job)
V3D_CORE_WRITE(0, V3D_CSD_QUEUED_CFG0, job->args.cfg[0]);
return fence;
+
+out_clean_job:
+ v3d->queue[V3D_CSD].active_job = NULL;
+ return fence;
}
static struct dma_fence *
@@ -426,46 +434,42 @@ v3d_sched_init(struct v3d_dev *v3d)
int hw_jobs_limit = 1;
int job_hang_limit = 0;
int hang_limit_ms = 500;
+ struct drm_sched_init_args args = {
+ .credit_limit = hw_jobs_limit,
+ .hang_limit = job_hang_limit,
+ .timeout = msecs_to_jiffies(hang_limit_ms),
+ .dev = v3d->drm.dev,
+ };
int ret;
- ret = drm_sched_init(&v3d->queue[V3D_BIN].sched,
- &v3d_bin_sched_ops,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_bin", v3d->drm.dev);
+ args.ops = &v3d_bin_sched_ops;
+ args.name = "v3d_bin";
+ ret = drm_sched_init(&v3d->queue[V3D_BIN].sched, &args);
if (ret)
return ret;
- ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched,
- &v3d_render_sched_ops,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_render", v3d->drm.dev);
+ args.ops = &v3d_render_sched_ops;
+ args.name = "v3d_render";
+ ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched, &args);
if (ret)
goto fail;
- ret = drm_sched_init(&v3d->queue[V3D_TFU].sched,
- &v3d_tfu_sched_ops,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_tfu", v3d->drm.dev);
+ args.ops = &v3d_tfu_sched_ops;
+ args.name = "v3d_tfu";
+ ret = drm_sched_init(&v3d->queue[V3D_TFU].sched, &args);
if (ret)
goto fail;
if (v3d_has_csd(v3d)) {
- ret = drm_sched_init(&v3d->queue[V3D_CSD].sched,
- &v3d_csd_sched_ops,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_csd", v3d->drm.dev);
+ args.ops = &v3d_csd_sched_ops;
+ args.name = "v3d_csd";
+ ret = drm_sched_init(&v3d->queue[V3D_CSD].sched, &args);
if (ret)
goto fail;
- ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched,
- &v3d_cache_clean_sched_ops,
- hw_jobs_limit, job_hang_limit,
- msecs_to_jiffies(hang_limit_ms), NULL,
- NULL, "v3d_cache_clean", v3d->drm.dev);
+ args.ops = &v3d_cache_clean_sched_ops;
+ args.name = "v3d_cache_clean";
+ ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched, &args);
if (ret)
goto fail;
}
diff --git a/drivers/gpu/drm/vc4/vc4_bo.c b/drivers/gpu/drm/vc4/vc4_bo.c
index 9028a56dd12b..df1b8bfb1c21 100644
--- a/drivers/gpu/drm/vc4/vc4_bo.c
+++ b/drivers/gpu/drm/vc4/vc4_bo.c
@@ -733,9 +733,13 @@ static int vc4_gem_object_mmap(struct drm_gem_object *obj, struct vm_area_struct
{
struct vc4_bo *bo = to_vc4_bo(obj);
- if (bo->validated_shader && (vma->vm_flags & VM_WRITE)) {
- DRM_DEBUG("mmaping of shader BOs for writing not allowed.\n");
- return -EINVAL;
+ if (bo->validated_shader) {
+ if (vma->vm_flags & VM_WRITE) {
+ DRM_DEBUG("mmaping of shader BOs for writing not allowed.\n");
+ return -EINVAL;
+ }
+
+ vma->vm_flags &= ~VM_MAYWRITE;
}
mutex_lock(&bo->madv_lock);
@@ -1050,10 +1054,11 @@ static void vc4_bo_cache_destroy(struct drm_device *dev, void *unused)
struct vc4_dev *vc4 = to_vc4_dev(dev);
int i;
- del_timer(&vc4->bo_cache.time_timer);
+ timer_shutdown_sync(&vc4->bo_cache.time_timer);
cancel_work_sync(&vc4->bo_cache.time_work);
vc4_bo_cache_purge(dev);
+ kfree(vc4->bo_cache.size_list);
for (i = 0; i < vc4->num_labels; i++) {
if (vc4->bo_labels[i].num_allocated) {
diff --git a/drivers/gpu/drm/vc4/vc4_drv.c b/drivers/gpu/drm/vc4/vc4_drv.c
index b6384a5dfdbc..ec5eca21df44 100644
--- a/drivers/gpu/drm/vc4/vc4_drv.c
+++ b/drivers/gpu/drm/vc4/vc4_drv.c
@@ -352,8 +352,11 @@ static int vc4_drm_bind(struct device *dev)
}
ret = drm_aperture_remove_framebuffers(driver);
- if (ret)
+ if (ret) {
+ if (firmware)
+ rpi_firmware_put(firmware);
return ret;
+ }
if (firmware) {
ret = rpi_firmware_property(firmware,
diff --git a/drivers/gpu/drm/vc4/vc4_hvs.c b/drivers/gpu/drm/vc4/vc4_hvs.c
index 88442cb65915..289485b25ffd 100644
--- a/drivers/gpu/drm/vc4/vc4_hvs.c
+++ b/drivers/gpu/drm/vc4/vc4_hvs.c
@@ -945,7 +945,7 @@ static int vc4_hvs_bind(struct device *dev, struct device *master, void *data)
static void vc4_hvs_unbind(struct device *dev, struct device *master,
void *data)
{
- struct drm_device *drm = dev_get_drvdata(master);
+ struct drm_device *drm = data;
struct vc4_dev *vc4 = to_vc4_dev(drm);
struct vc4_hvs *hvs = vc4->hvs;
struct drm_mm_node *node, *next;
diff --git a/drivers/gpu/drm/vc4/vc4_v3d.c b/drivers/gpu/drm/vc4/vc4_v3d.c
index a39e6f8eff0a..1dd41e2855d1 100644
--- a/drivers/gpu/drm/vc4/vc4_v3d.c
+++ b/drivers/gpu/drm/vc4/vc4_v3d.c
@@ -9,6 +9,7 @@
#include <linux/component.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
+#include <linux/timer.h>
#include "vc4_drv.h"
#include "vc4_regs.h"
@@ -510,11 +511,18 @@ static int vc4_v3d_bind(struct device *dev, struct device *master, void *data)
static void vc4_v3d_unbind(struct device *dev, struct device *master,
void *data)
{
- struct drm_device *drm = dev_get_drvdata(master);
+ struct drm_device *drm = data;
struct vc4_dev *vc4 = to_vc4_dev(drm);
+ /* A straggler vc4_reset() re-enables the interrupt. */
+ timer_shutdown_sync(&vc4->hangcheck.timer);
+ cancel_work_sync(&vc4->hangcheck.reset_work);
+
vc4_irq_uninstall(drm);
+ /* Flush rather than cancel, so queued completions release their jobs. */
+ flush_work(&vc4->job_done_work);
+
/* Disable the binner's overflow memory address, so the next
* driver probe (if any) doesn't try to reuse our old
* allocation.
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c b/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c
index ef2ee93f5e34..1be89105fc6e 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_execbuf.c
@@ -1282,9 +1282,13 @@ static int vmw_cmd_dx_bind_query(struct vmw_private *dev_priv,
SVGA3dCmdHeader *header)
{
VMW_DECLARE_CMD_VAR(*cmd, SVGA3dCmdDXBindQuery);
+ struct vmw_ctx_validation_info *ctx_node = VMW_GET_CTX_NODE(sw_context);
struct vmw_buffer_object *vmw_bo;
int ret;
+ if (!ctx_node)
+ return -EINVAL;
+
cmd = container_of(header, typeof(*cmd), header);
/*
@@ -1298,7 +1302,7 @@ static int vmw_cmd_dx_bind_query(struct vmw_private *dev_priv,
return ret;
sw_context->dx_query_mob = vmw_bo;
- sw_context->dx_query_ctx = sw_context->dx_ctx_node->ctx;
+ sw_context->dx_query_ctx = ctx_node->ctx;
return 0;
}
diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
index 5210b8084217..38abdaf663ff 100644
--- a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
+++ b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c
@@ -277,6 +277,7 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
unsigned long kmap_num;
SVGA3dCopyBox *box;
unsigned box_count;
+ u64 src_extent;
void *virtual;
bool dummy;
struct vmw_dma_cmd {
@@ -285,6 +286,8 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
} *cmd;
int i, ret;
+ BUILD_BUG_ON(sizeof(*srf->snooper.image) != 4);
+
cmd = container_of(header, struct vmw_dma_cmd, header);
/* No snooper installed */
@@ -322,8 +325,19 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
return;
}
+ if (box->w == 0 || box->h == 0)
+ return;
+
+ src_extent = (u64)(box->h - 1) * cmd->dma.guest.pitch +
+ (u64)box->w * 4;
+ if (src_extent > bo->base.size) {
+ DRM_ERROR("Cursor snoop source of %llu bytes exceeds the %zu byte buffer\n",
+ src_extent, bo->base.size);
+ return;
+ }
+
kmap_offset = cmd->dma.guest.ptr.offset >> PAGE_SHIFT;
- kmap_num = (64*64*4) >> PAGE_SHIFT;
+ kmap_num = PFN_UP(src_extent);
ret = ttm_bo_reserve(bo, true, false, NULL);
if (unlikely(ret != 0)) {
@@ -337,13 +351,13 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf,
virtual = ttm_kmap_obj_virtual(&map, &dummy);
- if (box->w == 64 && cmd->dma.guest.pitch == 64*4) {
+ if (box->w == 64 && box->h == 64 && cmd->dma.guest.pitch == 64*4) {
memcpy(srf->snooper.image, virtual, 64*64*4);
} else {
- /* Image is unsigned pointer. */
+ /* The destination stride is in pixels. */
for (i = 0; i < box->h; i++)
memcpy(srf->snooper.image + i * 64,
- virtual + i * cmd->dma.guest.pitch,
+ virtual + (size_t)i * cmd->dma.guest.pitch,
box->w * 4);
}
diff --git a/drivers/gpu/drm/xlnx/zynqmp_dpsub.c b/drivers/gpu/drm/xlnx/zynqmp_dpsub.c
index f34d4ec1d92f..463407d5f660 100644
--- a/drivers/gpu/drm/xlnx/zynqmp_dpsub.c
+++ b/drivers/gpu/drm/xlnx/zynqmp_dpsub.c
@@ -253,7 +253,7 @@ static int zynqmp_dpsub_remove(struct platform_device *pdev)
struct zynqmp_dpsub *dpsub = platform_get_drvdata(pdev);
struct drm_device *drm = &dpsub->drm;
- drm_dev_unregister(drm);
+ drm_dev_unplug(drm);
drm_atomic_helper_shutdown(drm);
drm_kms_helper_poll_fini(drm);
diff --git a/drivers/hid/hid-alps.c b/drivers/hid/hid-alps.c
index 72c21014de84..25da18017500 100644
--- a/drivers/hid/hid-alps.c
+++ b/drivers/hid/hid-alps.c
@@ -827,7 +827,20 @@ static int alps_probe(struct hid_device *hdev, const struct hid_device_id *id)
static void alps_remove(struct hid_device *hdev)
{
+ struct alps_dev *data = hid_get_drvdata(hdev);
+
+ /*
+ * input2 ("DualPoint Stick") is allocated separately and is not
+ * tracked in hdev->inputs, so the default remove path
+ * (hid_hw_stop -> hidinput_disconnect) does not unregister it.
+ *
+ * Stop the device first so that no URB callback can touch input2
+ * while it is being unregistered, then drop it explicitly.
+ */
hid_hw_stop(hdev);
+
+ if (data->input2)
+ input_unregister_device(data->input2);
}
static const struct hid_device_id alps_id[] = {
@@ -847,14 +860,12 @@ static struct hid_driver alps_driver = {
.name = "hid-alps",
.id_table = alps_id,
.probe = alps_probe,
- .remove = alps_remove,
.raw_event = alps_raw_event,
.input_mapping = alps_input_mapping,
.input_configured = alps_input_configured,
-#ifdef CONFIG_PM
- .resume = alps_post_resume,
- .reset_resume = alps_post_reset,
-#endif
+ .resume = pm_ptr(alps_post_resume),
+ .reset_resume = pm_ptr(alps_post_reset),
+ .remove = alps_remove,
};
module_hid_driver(alps_driver);
diff --git a/drivers/hid/hid-core.c b/drivers/hid/hid-core.c
index 61dde8e3fbb2..66deabcfb68b 100644
--- a/drivers/hid/hid-core.c
+++ b/drivers/hid/hid-core.c
@@ -640,7 +640,11 @@ static int hid_parser_main(struct hid_parser *parser, struct hid_item *item)
ret = hid_add_field(parser, HID_FEATURE_REPORT, data);
break;
default:
- hid_warn(parser->device, "unknown main item tag 0x%x\n", item->tag);
+ if (item->tag >= HID_MAIN_ITEM_TAG_RESERVED_MIN &&
+ item->tag <= HID_MAIN_ITEM_TAG_RESERVED_MAX)
+ hid_warn_ratelimited(parser->device, "reserved main item tag 0x%x\n", item->tag);
+ else
+ hid_warn_ratelimited(parser->device, "unknown main item tag 0x%x\n", item->tag);
ret = 0;
}
@@ -2048,9 +2052,9 @@ int hid_report_raw_event(struct hid_device *hid, enum hid_report_type type, u8 *
}
if (csize < rsize) {
- dbg_hid("report %d is too short, (%d < %d)\n", report->id,
- csize, rsize);
- memset(cdata + csize, 0, rsize - csize);
+ hid_warn_ratelimited(hid, "Event data for report %d was too short (%d vs %d)\n",
+ report->id, rsize, csize);
+ return -EINVAL;
}
if ((hid->claimed & HID_CLAIMED_HIDDEV) && hid->hiddev_report_event)
@@ -2085,14 +2089,36 @@ EXPORT_SYMBOL_GPL(hid_report_raw_event);
* @interrupt: distinguish between interrupt and control transfers
*
* This is data entry for lower layers.
+ * Legacy, please use hid_safe_input_report() instead.
*/
int hid_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data,
u32 size, int interrupt)
+{
+ return hid_safe_input_report(hid, type, data, size, size, interrupt);
+}
+EXPORT_SYMBOL_GPL(hid_input_report);
+
+/**
+ * hid_safe_input_report - report data from lower layer (usb, bt...)
+ *
+ * @hid: hid device
+ * @type: HID report type (HID_*_REPORT)
+ * @data: report contents
+ * @bufsize: allocated size of the data buffer
+ * @size: useful size of data parameter
+ * @interrupt: distinguish between interrupt and control transfers
+ *
+ * This is data entry for lower layers.
+ * Please use this function instead of the non safe version because we provide
+ * here the size of the buffer, allowing hid-core to make smarter decisions
+ * regarding the incoming buffer.
+ */
+int hid_safe_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data,
+ size_t bufsize, u32 size, int interrupt)
{
struct hid_report_enum *report_enum;
struct hid_driver *hdrv;
struct hid_report *report;
- size_t bufsize = size;
int ret = 0;
if (!hid)
@@ -2137,7 +2163,7 @@ int hid_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data
up(&hid->driver_input_lock);
return ret;
}
-EXPORT_SYMBOL_GPL(hid_input_report);
+EXPORT_SYMBOL_GPL(hid_safe_input_report);
bool hid_match_one_id(const struct hid_device *hdev,
const struct hid_device_id *id)
diff --git a/drivers/hid/hid-logitech-hidpp.c b/drivers/hid/hid-logitech-hidpp.c
index 37fe852dd2da..741be3c1fe5b 100644
--- a/drivers/hid/hid-logitech-hidpp.c
+++ b/drivers/hid/hid-logitech-hidpp.c
@@ -2634,18 +2634,19 @@ static int hidpp_ff_init(struct hidpp_device *hidpp,
* ownership to FF core
*/
data = kmemdup(data, sizeof(*data), GFP_KERNEL);
- if (!data)
- return -ENOMEM;
+ if (!data) {
+ error = -ENOMEM;
+ goto err_destroy_ff;
+ }
data->effect_ids = kcalloc(num_slots, sizeof(int), GFP_KERNEL);
if (!data->effect_ids) {
- kfree(data);
- return -ENOMEM;
+ error = -ENOMEM;
+ goto err_free_data;
}
data->wq = create_singlethread_workqueue("hidpp-ff-sendqueue");
if (!data->wq) {
- kfree(data->effect_ids);
- kfree(data);
- return -ENOMEM;
+ error = -ENOMEM;
+ goto err_free_effect_ids;
}
data->hidpp = hidpp;
@@ -2675,6 +2676,14 @@ static int hidpp_ff_init(struct hidpp_device *hidpp,
version);
return 0;
+
+err_free_effect_ids:
+ kfree(data->effect_ids);
+err_free_data:
+ kfree(data);
+err_destroy_ff:
+ input_ff_destroy(dev);
+ return error;
}
/* ************************************************************************** */
@@ -4278,10 +4287,12 @@ static int hidpp_probe(struct hid_device *hdev, const struct hid_device_id *id)
if (hidpp->quirks & HIDPP_QUIRK_CLASS_G920) {
ret = hidpp_ff_init(hidpp, &data);
- if (ret)
+ if (ret) {
hid_warn(hidpp->hid_dev,
"Unable to initialize force feedback support, errno %d\n",
ret);
+ ret = 0;
+ }
}
/*
diff --git a/drivers/hid/hid-mcp2221.c b/drivers/hid/hid-mcp2221.c
index 99a0f55a0277..2356f5d7b394 100644
--- a/drivers/hid/hid-mcp2221.c
+++ b/drivers/hid/hid-mcp2221.c
@@ -93,6 +93,7 @@ struct mcp2221 {
u8 *rxbuf;
u8 txbuf[64];
int rxbuf_idx;
+ int rxbuf_size;
int status;
u8 cur_i2c_clk_div;
struct gpio_chip *gc;
@@ -289,17 +290,19 @@ static int mcp_i2c_smbus_read(struct mcp2221 *mcp,
mcp->txbuf[3] = (u8)(msg->addr << 1);
total_len = msg->len;
mcp->rxbuf = msg->buf;
+ mcp->rxbuf_size = msg->len;
} else {
mcp->txbuf[1] = smbus_len;
mcp->txbuf[2] = 0;
mcp->txbuf[3] = (u8)(smbus_addr << 1);
total_len = smbus_len;
mcp->rxbuf = smbus_buf;
+ mcp->rxbuf_size = smbus_len;
}
ret = mcp_send_data_req_status(mcp, mcp->txbuf, 4);
if (ret)
- return ret;
+ goto out;
mcp->rxbuf_idx = 0;
@@ -321,7 +324,7 @@ static int mcp_i2c_smbus_read(struct mcp2221 *mcp,
} else {
usleep_range(980, 1000);
mcp_cancel_last_cmd(mcp);
- return ret;
+ goto out;
}
} else {
retries = 0;
@@ -331,6 +334,10 @@ static int mcp_i2c_smbus_read(struct mcp2221 *mcp,
usleep_range(980, 1000);
ret = mcp_chk_last_cmd_status_free_bus(mcp);
+out:
+ mcp->rxbuf = NULL;
+ mcp->rxbuf_size = 0;
+
return ret;
}
@@ -504,6 +511,7 @@ static int mcp_smbus_xfer(struct i2c_adapter *adapter, u16 addr,
mcp->rxbuf_idx = 0;
mcp->rxbuf = data->block;
+ mcp->rxbuf_size = sizeof(data->block);
mcp->txbuf[0] = MCP2221_I2C_GET_DATA;
ret = mcp_send_data_req_status(mcp, mcp->txbuf, 1);
if (ret)
@@ -527,6 +535,7 @@ static int mcp_smbus_xfer(struct i2c_adapter *adapter, u16 addr,
mcp->rxbuf_idx = 0;
mcp->rxbuf = data->block;
+ mcp->rxbuf_size = sizeof(data->block);
mcp->txbuf[0] = MCP2221_I2C_GET_DATA;
ret = mcp_send_data_req_status(mcp, mcp->txbuf, 1);
if (ret)
@@ -804,6 +813,10 @@ static int mcp2221_raw_event(struct hid_device *hdev,
mcp->status = -EINVAL;
break;
}
+ if (mcp->rxbuf_idx + data[3] > mcp->rxbuf_size) {
+ mcp->status = -EINVAL;
+ break;
+ }
buf = mcp->rxbuf;
memcpy(&buf[mcp->rxbuf_idx], &data[4], data[3]);
mcp->rxbuf_idx = mcp->rxbuf_idx + data[3];
diff --git a/drivers/hid/hid-nintendo.c b/drivers/hid/hid-nintendo.c
index 44edc50be6e5..6a57681fe123 100644
--- a/drivers/hid/hid-nintendo.c
+++ b/drivers/hid/hid-nintendo.c
@@ -301,6 +301,7 @@ enum joycon_ctlr_state {
JOYCON_CTLR_STATE_INIT,
JOYCON_CTLR_STATE_READ,
JOYCON_CTLR_STATE_REMOVED,
+ JOYCON_CTLR_STATE_SUSPENDED,
};
/* Controller type received as part of device info */
@@ -1718,10 +1719,6 @@ static int joycon_input_create(struct joycon_ctlr *ctlr)
ctlr->rumble_msecs = jiffies_to_msecs(jiffies);
#endif
- ret = input_register_device(ctlr->input);
- if (ret)
- return ret;
-
/* configure the imu input device */
ctlr->imu_input = devm_input_allocate_device(&hdev->dev);
if (!ctlr->imu_input)
@@ -1772,6 +1769,10 @@ static int joycon_input_create(struct joycon_ctlr *ctlr)
if (ret)
return ret;
+ ret = input_register_device(ctlr->input);
+ if (ret)
+ return ret;
+
return 0;
}
@@ -2341,14 +2342,46 @@ static void nintendo_hid_remove(struct hid_device *hdev)
static int nintendo_hid_resume(struct hid_device *hdev)
{
- int ret = joycon_init(hdev);
+ struct joycon_ctlr *ctlr = hid_get_drvdata(hdev);
+ int ret;
+
+ hid_dbg(hdev, "resume\n");
+ if (hdev->bus != BUS_USB) {
+ hid_dbg(hdev, "no-op resume for bt ctlr\n");
+ ctlr->ctlr_state = JOYCON_CTLR_STATE_READ;
+ return 0;
+ }
+ ret = joycon_init(hdev);
if (ret)
- hid_err(hdev, "Failed to restore controller after resume");
+ hid_err(hdev,
+ "Failed to restore controller after resume: %d\n",
+ ret);
+ else
+ ctlr->ctlr_state = JOYCON_CTLR_STATE_READ;
return ret;
}
+static int nintendo_hid_suspend(struct hid_device *hdev, pm_message_t message)
+{
+ struct joycon_ctlr *ctlr = hid_get_drvdata(hdev);
+
+ hid_dbg(hdev, "suspend: %d\n", message.event);
+ /*
+ * Avoid any blocking loops in suspend/resume transitions.
+ *
+ * joycon_enforce_subcmd_rate() can result in repeated retries if for
+ * whatever reason the controller stops providing input reports.
+ *
+ * This has been observed with bluetooth controllers which lose
+ * connectivity prior to suspend (but not long enough to result in
+ * complete disconnection).
+ */
+ ctlr->ctlr_state = JOYCON_CTLR_STATE_SUSPENDED;
+ return 0;
+}
+
#endif
static const struct hid_device_id nintendo_hid_devices[] = {
@@ -2375,6 +2408,7 @@ static struct hid_driver nintendo_hid_driver = {
#ifdef CONFIG_PM
.resume = nintendo_hid_resume,
+ .suspend = nintendo_hid_suspend,
#endif
};
module_hid_driver(nintendo_hid_driver);
diff --git a/drivers/hid/hid-sony.c b/drivers/hid/hid-sony.c
index f7f7252d839e..f1a861cb99ed 100644
--- a/drivers/hid/hid-sony.c
+++ b/drivers/hid/hid-sony.c
@@ -2564,13 +2564,10 @@ static int sony_check_add_dev_list(struct sony_sc *sc)
static void sony_remove_dev_list(struct sony_sc *sc)
{
- unsigned long flags;
+ guard(spinlock_irqsave)(&sony_dev_list_lock);
- if (sc->list_node.next) {
- spin_lock_irqsave(&sony_dev_list_lock, flags);
- list_del(&(sc->list_node));
- spin_unlock_irqrestore(&sony_dev_list_lock, flags);
- }
+ if (!list_empty(&sc->list_node))
+ list_del_init(&sc->list_node);
}
static int sony_get_bt_devaddr(struct sony_sc *sc)
@@ -2736,6 +2733,13 @@ static inline void sony_cancel_work_sync(struct sony_sc *sc)
}
}
+static void sony_cleanup(struct sony_sc *sc)
+{
+ sony_cancel_work_sync(sc);
+ sony_remove_dev_list(sc);
+ sony_release_device_id(sc);
+}
+
static int sony_input_configured(struct hid_device *hdev,
struct hid_input *hidinput)
{
@@ -2968,9 +2972,7 @@ static int sony_input_configured(struct hid_device *hdev,
device_remove_file(&sc->hdev->dev, &dev_attr_firmware_version);
if (sc->hw_version_created)
device_remove_file(&sc->hdev->dev, &dev_attr_hardware_version);
- sony_cancel_work_sync(sc);
- sony_remove_dev_list(sc);
- sony_release_device_id(sc);
+ sony_cleanup(sc);
return ret;
}
@@ -2996,6 +2998,8 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
}
spin_lock_init(&sc->lock);
+ INIT_LIST_HEAD(&sc->list_node);
+ sc->device_id = -1;
sc->quirks = quirks;
hid_set_drvdata(hdev, sc);
@@ -3024,6 +3028,7 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
ret = hid_hw_start(hdev, connect_mask);
if (ret) {
hid_err(hdev, "hw start failed\n");
+ sony_cleanup(sc);
return ret;
}
@@ -3075,7 +3080,7 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id)
err:
usb_free_urb(sc->ghl_urb);
-
+ sony_cleanup(sc);
hid_hw_stop(hdev);
return ret;
}
@@ -3085,7 +3090,9 @@ static void sony_remove(struct hid_device *hdev)
struct sony_sc *sc = hid_get_drvdata(hdev);
if (sc->quirks & (GHL_GUITAR_PS3WIIU | GHL_GUITAR_PS4)) {
- del_timer_sync(&sc->ghl_poke_timer);
+ /* poison, not kill: a pending timer must not re-submit during teardown */
+ usb_poison_urb(sc->ghl_urb);
+ timer_shutdown_sync(&sc->ghl_poke_timer);
usb_free_urb(sc->ghl_urb);
}
@@ -3100,12 +3107,7 @@ static void sony_remove(struct hid_device *hdev)
if (sc->hw_version_created)
device_remove_file(&sc->hdev->dev, &dev_attr_hardware_version);
- sony_cancel_work_sync(sc);
-
- sony_remove_dev_list(sc);
-
- sony_release_device_id(sc);
-
+ sony_cleanup(sc);
hid_hw_stop(hdev);
}
diff --git a/drivers/hid/hid-steam.c b/drivers/hid/hid-steam.c
index 8ee43cb225fc..a54bf625ae47 100644
--- a/drivers/hid/hid-steam.c
+++ b/drivers/hid/hid-steam.c
@@ -114,11 +114,13 @@ struct steam_device {
spinlock_t lock;
struct hid_device *hdev, *client_hdev;
struct mutex mutex;
- bool client_opened;
+ struct mutex registration_mutex;
+ unsigned long client_opened;
struct input_dev __rcu *input;
unsigned long quirks;
struct work_struct work_connect;
bool connected;
+ bool registered;
char serial_no[STEAM_SERIAL_LEN + 1];
struct power_supply_desc battery_desc;
struct power_supply __rcu *battery;
@@ -133,6 +135,13 @@ static int steam_recv_report(struct steam_device *steam,
u8 *buf;
int ret;
+ /*
+ * All reports start with a two byte header.
+ * We must read at least two bytes to get a sensible output.
+ */
+ if (size < 2)
+ return -EINVAL;
+
r = steam->hdev->report_enum[HID_FEATURE_REPORT].report_id_hash[0];
if (!r) {
hid_err(steam->hdev, "No HID_FEATURE_REPORT submitted - nothing to read\n");
@@ -155,10 +164,31 @@ static int steam_recv_report(struct steam_device *steam,
ret = hid_hw_raw_request(steam->hdev, 0x00,
buf, hid_report_len(r) + 1,
HID_FEATURE_REPORT, HID_REQ_GET_REPORT);
- if (ret > 0)
- memcpy(data, buf + 1, min(size, ret - 1));
+ if (ret > 0) {
+ /* Remove the report ID from the return buffer */
+ ret--;
+ size = min(size, ret);
+ memcpy(data, buf + 1, size);
+ }
kfree(buf);
- return ret;
+
+ if (ret < 0)
+ hid_err(steam->hdev, "%s: error %d\n", __func__, ret);
+ else
+ hid_dbg(steam->hdev, "Received report %*ph\n", size, data);
+ if (ret < 0)
+ return ret;
+
+ if (ret < 2) {
+ hid_err(steam->hdev, "%s: reply too short\n", __func__);
+ return -EPROTO;
+ }
+ if (ret < data[1] + 2) {
+ hid_err(steam->hdev, "%s: expected %u bytes, read %i\n",
+ __func__, data[1] + 2, ret);
+ return -EPROTO;
+ }
+ return size;
}
static int steam_send_report(struct steam_device *steam,
@@ -293,22 +323,30 @@ static void steam_set_lizard_mode(struct steam_device *steam, bool enable)
static int steam_input_open(struct input_dev *dev)
{
struct steam_device *steam = input_get_drvdata(dev);
+ unsigned long flags;
+ bool client_opened;
- mutex_lock(&steam->mutex);
- if (!steam->client_opened && lizard_mode)
+ spin_lock_irqsave(&steam->lock, flags);
+ client_opened = steam->client_opened;
+ spin_unlock_irqrestore(&steam->lock, flags);
+ guard(mutex)(&steam->mutex);
+ if (!client_opened && lizard_mode)
steam_set_lizard_mode(steam, false);
- mutex_unlock(&steam->mutex);
return 0;
}
static void steam_input_close(struct input_dev *dev)
{
struct steam_device *steam = input_get_drvdata(dev);
+ unsigned long flags;
+ bool client_opened;
- mutex_lock(&steam->mutex);
- if (!steam->client_opened && lizard_mode)
+ spin_lock_irqsave(&steam->lock, flags);
+ client_opened = steam->client_opened;
+ spin_unlock_irqrestore(&steam->lock, flags);
+ guard(mutex)(&steam->mutex);
+ if (!client_opened && lizard_mode)
steam_set_lizard_mode(steam, true);
- mutex_unlock(&steam->mutex);
}
static enum power_supply_property steam_battery_props[] = {
@@ -380,6 +418,7 @@ static int steam_battery_register(struct steam_device *steam)
&steam->battery_desc, &battery_cfg);
if (IS_ERR(battery)) {
ret = PTR_ERR(battery);
+ devm_kfree(&steam->hdev->dev, steam->battery_desc.name);
hid_err(steam->hdev,
"%s:power_supply_register failed with error %d\n",
__func__, ret);
@@ -504,69 +543,78 @@ static void steam_battery_unregister(struct steam_device *steam)
RCU_INIT_POINTER(steam->battery, NULL);
synchronize_rcu();
power_supply_unregister(battery);
+ devm_kfree(&steam->hdev->dev, steam->battery_desc.name);
}
static int steam_register(struct steam_device *steam)
{
int ret;
- bool client_opened;
+ mutex_lock(&steam->registration_mutex);
/*
* This function can be called several times in a row with the
* wireless adaptor, without steam_unregister() between them, because
* another client send a get_connection_status command, for example.
* The battery and serial number are set just once per device.
*/
- if (!steam->serial_no[0]) {
- /*
- * Unlikely, but getting the serial could fail, and it is not so
- * important, so make up a serial number and go on.
- */
- mutex_lock(&steam->mutex);
- if (steam_get_serial(steam) < 0)
- strscpy(steam->serial_no, "XXXXXXXXXX",
- sizeof(steam->serial_no));
- mutex_unlock(&steam->mutex);
-
- hid_info(steam->hdev, "Steam Controller '%s' connected",
- steam->serial_no);
-
- /* ignore battery errors, we can live without it */
- if (steam->quirks & STEAM_QUIRK_WIRELESS)
- steam_battery_register(steam);
-
- mutex_lock(&steam_devices_lock);
- if (list_empty(&steam->list))
- list_add(&steam->list, &steam_devices);
- mutex_unlock(&steam_devices_lock);
+ if (steam->registered) {
+ mutex_unlock(&steam->registration_mutex);
+ return 0;
}
mutex_lock(&steam->mutex);
- client_opened = steam->client_opened;
- if (!client_opened)
- steam_set_lizard_mode(steam, lizard_mode);
+ /*
+ * Unlikely, but getting the serial could fail, and it is not so
+ * important, so make up a serial number and go on.
+ */
+ if (steam_get_serial(steam) < 0 || !steam->serial_no[0])
+ strscpy(steam->serial_no, "XXXXXXXXXX",
+ sizeof(steam->serial_no));
+
+ hid_info(steam->hdev, "Steam Controller '%s' connected",
+ steam->serial_no);
+
+ steam_set_lizard_mode(steam, lizard_mode);
mutex_unlock(&steam->mutex);
- if (!client_opened)
- ret = steam_input_register(steam);
- else
- ret = 0;
+ /* ignore battery errors, we can live without it */
+ if (steam->quirks & STEAM_QUIRK_WIRELESS)
+ steam_battery_register(steam);
+ ret = steam_input_register(steam);
+ if (ret != 0)
+ goto steam_register_input_fail;
+
+ steam->registered = true;
+ mutex_unlock(&steam->registration_mutex);
+ mutex_lock(&steam_devices_lock);
+ if (list_empty(&steam->list))
+ list_add(&steam->list, &steam_devices);
+ mutex_unlock(&steam_devices_lock);
+ return 0;
+
+steam_register_input_fail:
+ steam_battery_unregister(steam);
+ mutex_unlock(&steam->registration_mutex);
return ret;
}
static void steam_unregister(struct steam_device *steam)
{
+ if (!steam->registered)
+ return;
+
+ hid_info(steam->hdev, "Steam Controller '%s' disconnected",
+ steam->serial_no);
+ mutex_lock(&steam->registration_mutex);
+ steam->registered = false;
steam_battery_unregister(steam);
steam_input_unregister(steam);
- if (steam->serial_no[0]) {
- hid_info(steam->hdev, "Steam Controller '%s' disconnected",
- steam->serial_no);
- mutex_lock(&steam_devices_lock);
- list_del_init(&steam->list);
- mutex_unlock(&steam_devices_lock);
- steam->serial_no[0] = 0;
- }
+ mutex_unlock(&steam->registration_mutex);
+ mutex_lock(&steam_devices_lock);
+ list_del_init(&steam->list);
+ mutex_unlock(&steam_devices_lock);
+ steam->serial_no[0] = 0;
}
static void steam_work_connect_cb(struct work_struct *work)
@@ -575,19 +623,20 @@ static void steam_work_connect_cb(struct work_struct *work)
work_connect);
unsigned long flags;
bool connected;
+ bool opened;
int ret;
spin_lock_irqsave(&steam->lock, flags);
+ opened = steam->client_opened;
connected = steam->connected;
spin_unlock_irqrestore(&steam->lock, flags);
- if (connected) {
+ if (connected && !opened) {
ret = steam_register(steam);
- if (ret) {
+ if (ret)
hid_err(steam->hdev,
"%s:steam_register failed with error %d\n",
__func__, ret);
- }
} else {
steam_unregister(steam);
}
@@ -632,12 +681,13 @@ static void steam_client_ll_stop(struct hid_device *hdev)
static int steam_client_ll_open(struct hid_device *hdev)
{
struct steam_device *steam = hdev->driver_data;
+ unsigned long flags;
- mutex_lock(&steam->mutex);
- steam->client_opened = true;
- mutex_unlock(&steam->mutex);
+ spin_lock_irqsave(&steam->lock, flags);
+ steam->client_opened++;
+ spin_unlock_irqrestore(&steam->lock, flags);
- steam_input_unregister(steam);
+ schedule_work(&steam->work_connect);
return 0;
}
@@ -645,22 +695,13 @@ static int steam_client_ll_open(struct hid_device *hdev)
static void steam_client_ll_close(struct hid_device *hdev)
{
struct steam_device *steam = hdev->driver_data;
-
unsigned long flags;
- bool connected;
spin_lock_irqsave(&steam->lock, flags);
- connected = steam->connected;
+ steam->client_opened--;
spin_unlock_irqrestore(&steam->lock, flags);
- mutex_lock(&steam->mutex);
- steam->client_opened = false;
- if (connected)
- steam_set_lizard_mode(steam, lizard_mode);
- mutex_unlock(&steam->mutex);
-
- if (connected)
- steam_input_register(steam);
+ schedule_work(&steam->work_connect);
}
static int steam_client_ll_raw_request(struct hid_device *hdev,
@@ -747,35 +788,25 @@ static int steam_probe(struct hid_device *hdev,
hid_set_drvdata(hdev, steam);
spin_lock_init(&steam->lock);
mutex_init(&steam->mutex);
+ mutex_init(&steam->registration_mutex);
steam->quirks = id->driver_data;
INIT_WORK(&steam->work_connect, steam_work_connect_cb);
INIT_LIST_HEAD(&steam->list);
- steam->client_hdev = steam_create_client_hid(hdev);
- if (IS_ERR(steam->client_hdev)) {
- ret = PTR_ERR(steam->client_hdev);
- goto client_hdev_fail;
- }
- steam->client_hdev->driver_data = steam;
-
/*
* With the real steam controller interface, do not connect hidraw.
* Instead, create the client_hid and connect that.
*/
ret = hid_hw_start(hdev, HID_CONNECT_DEFAULT & ~HID_CONNECT_HIDRAW);
if (ret)
- goto hid_hw_start_fail;
-
- ret = hid_add_device(steam->client_hdev);
- if (ret)
- goto client_hdev_add_fail;
+ goto err_cancel_work;
ret = hid_hw_open(hdev);
if (ret) {
hid_err(hdev,
"%s:hid_hw_open\n",
__func__);
- goto hid_hw_open_fail;
+ goto err_hw_stop;
}
if (steam->quirks & STEAM_QUIRK_WIRELESS) {
@@ -791,19 +822,33 @@ static int steam_probe(struct hid_device *hdev,
hid_err(hdev,
"%s:steam_register failed with error %d\n",
__func__, ret);
- goto input_register_fail;
+ goto err_hw_close;
}
}
+ steam->client_hdev = steam_create_client_hid(hdev);
+ if (IS_ERR(steam->client_hdev)) {
+ ret = PTR_ERR(steam->client_hdev);
+ goto err_stream_unregister;
+ }
+ steam->client_hdev->driver_data = steam;
+
+ ret = hid_add_device(steam->client_hdev);
+ if (ret)
+ goto err_destroy;
+
return 0;
-input_register_fail:
-hid_hw_open_fail:
-client_hdev_add_fail:
- hid_hw_stop(hdev);
-hid_hw_start_fail:
+err_destroy:
hid_destroy_device(steam->client_hdev);
-client_hdev_fail:
+err_stream_unregister:
+ if (steam->connected)
+ steam_unregister(steam);
+err_hw_close:
+ hid_hw_close(hdev);
+err_hw_stop:
+ hid_hw_stop(hdev);
+err_cancel_work:
cancel_work_sync(&steam->work_connect);
steam_alloc_fail:
hid_err(hdev, "%s: failed with error %d\n",
@@ -820,26 +865,24 @@ static void steam_remove(struct hid_device *hdev)
return;
}
+ hid_hw_close(hdev);
hid_destroy_device(steam->client_hdev);
- steam->client_opened = false;
+ steam->client_hdev = NULL;
+ steam->client_opened = 0;
cancel_work_sync(&steam->work_connect);
if (steam->quirks & STEAM_QUIRK_WIRELESS) {
hid_info(hdev, "Steam wireless receiver disconnected");
}
- hid_hw_close(hdev);
- hid_hw_stop(hdev);
steam_unregister(steam);
+ hid_hw_stop(hdev);
}
static void steam_do_connect_event(struct steam_device *steam, bool connected)
{
- unsigned long flags;
bool changed;
- spin_lock_irqsave(&steam->lock, flags);
changed = steam->connected != connected;
steam->connected = connected;
- spin_unlock_irqrestore(&steam->lock, flags);
if (changed && schedule_work(&steam->work_connect) == 0)
dbg_hid("%s: connected=%d event already queued\n",
@@ -1006,8 +1049,6 @@ static void steam_do_input_event(struct steam_device *steam,
static void steam_do_battery_event(struct steam_device *steam,
struct power_supply *battery, u8 *data)
{
- unsigned long flags;
-
s16 volts = steam_le16(data + 12);
u8 batt = data[14];
@@ -1015,10 +1056,8 @@ static void steam_do_battery_event(struct steam_device *steam,
rcu_read_lock();
battery = rcu_dereference(steam->battery);
if (likely(battery)) {
- spin_lock_irqsave(&steam->lock, flags);
steam->voltage = volts;
steam->battery_charge = batt;
- spin_unlock_irqrestore(&steam->lock, flags);
power_supply_changed(battery);
}
rcu_read_unlock();
@@ -1035,6 +1074,7 @@ static int steam_raw_event(struct hid_device *hdev,
if (!steam)
return 0;
+ guard(spinlock_irqsave)(&steam->lock);
if (steam->client_opened)
hid_input_report(steam->client_hdev, HID_FEATURE_REPORT,
data, size, 0);
@@ -1106,6 +1146,8 @@ static int steam_param_set_lizard_mode(const char *val,
{
struct steam_device *steam;
int ret;
+ bool client_opened;
+ unsigned long flags;
ret = param_set_bool(val, kp);
if (ret)
@@ -1113,10 +1155,13 @@ static int steam_param_set_lizard_mode(const char *val,
mutex_lock(&steam_devices_lock);
list_for_each_entry(steam, &steam_devices, list) {
- mutex_lock(&steam->mutex);
- if (!steam->client_opened)
+ spin_lock_irqsave(&steam->lock, flags);
+ client_opened = steam->client_opened;
+ spin_unlock_irqrestore(&steam->lock, flags);
+ if (!client_opened) {
+ guard(mutex)(&steam->mutex);
steam_set_lizard_mode(steam, lizard_mode);
- mutex_unlock(&steam->mutex);
+ }
}
mutex_unlock(&steam_devices_lock);
return 0;
diff --git a/drivers/hid/i2c-hid/i2c-hid-core.c b/drivers/hid/i2c-hid/i2c-hid-core.c
index 9559c96d44b4..d301dc5acd31 100644
--- a/drivers/hid/i2c-hid/i2c-hid-core.c
+++ b/drivers/hid/i2c-hid/i2c-hid-core.c
@@ -539,9 +539,10 @@ static void i2c_hid_get_input(struct i2c_hid *ihid)
if (ihid->hid->group != HID_GROUP_RMI)
pm_wakeup_event(&ihid->client->dev, 0);
- hid_input_report(ihid->hid, HID_INPUT_REPORT,
- ihid->inbuf + sizeof(__le16),
- ret_size - sizeof(__le16), 1);
+ hid_safe_input_report(ihid->hid, HID_INPUT_REPORT,
+ ihid->inbuf + sizeof(__le16),
+ ihid->bufsize - sizeof(__le16),
+ ret_size - sizeof(__le16), 1);
}
return;
diff --git a/drivers/hid/intel-ish-hid/ishtp/bus.c b/drivers/hid/intel-ish-hid/ishtp/bus.c
index 67e3215a332e..2bf5073ce128 100644
--- a/drivers/hid/intel-ish-hid/ishtp/bus.c
+++ b/drivers/hid/intel-ish-hid/ishtp/bus.c
@@ -730,6 +730,8 @@ void ishtp_bus_remove_all_clients(struct ishtp_device *ishtp_dev,
spin_lock_irqsave(&ishtp_dev->cl_list_lock, flags);
list_for_each_entry(cl, &ishtp_dev->cl_list, link) {
cl->state = ISHTP_CL_DISCONNECTED;
+ if (warm_reset && cl->device && cl->device->reference_count)
+ continue;
/*
* Wake any pending process. The waiter would check dev->state
diff --git a/drivers/hid/intel-ish-hid/ishtp/client.c b/drivers/hid/intel-ish-hid/ishtp/client.c
index df0a825694f5..d97c843cee0c 100644
--- a/drivers/hid/intel-ish-hid/ishtp/client.c
+++ b/drivers/hid/intel-ish-hid/ishtp/client.c
@@ -49,7 +49,9 @@ static void ishtp_read_list_flush(struct ishtp_cl *cl)
list_for_each_entry_safe(rb, next, &cl->dev->read_list.list, list)
if (rb->cl && ishtp_cl_cmp_id(cl, rb->cl)) {
list_del(&rb->list);
- ishtp_io_rb_free(rb);
+ spin_lock(&cl->free_list_spinlock);
+ list_add_tail(&rb->list, &cl->free_rb_list.list);
+ spin_unlock(&cl->free_list_spinlock);
}
spin_unlock_irqrestore(&cl->dev->read_list_spinlock, flags);
}
diff --git a/drivers/hid/uhid.c b/drivers/hid/uhid.c
index 59ac757c1d47..72baf8adf46c 100644
--- a/drivers/hid/uhid.c
+++ b/drivers/hid/uhid.c
@@ -597,8 +597,8 @@ static int uhid_dev_input(struct uhid_device *uhid, struct uhid_event *ev)
if (!READ_ONCE(uhid->running))
return -EINVAL;
- hid_input_report(uhid->hid, HID_INPUT_REPORT, ev->u.input.data,
- min_t(size_t, ev->u.input.size, UHID_DATA_MAX), 0);
+ hid_safe_input_report(uhid->hid, HID_INPUT_REPORT, ev->u.input.data, UHID_DATA_MAX,
+ min_t(size_t, ev->u.input.size, UHID_DATA_MAX), 0);
return 0;
}
@@ -608,8 +608,8 @@ static int uhid_dev_input2(struct uhid_device *uhid, struct uhid_event *ev)
if (!READ_ONCE(uhid->running))
return -EINVAL;
- hid_input_report(uhid->hid, HID_INPUT_REPORT, ev->u.input2.data,
- min_t(size_t, ev->u.input2.size, UHID_DATA_MAX), 0);
+ hid_safe_input_report(uhid->hid, HID_INPUT_REPORT, ev->u.input2.data, UHID_DATA_MAX,
+ min_t(size_t, ev->u.input2.size, UHID_DATA_MAX), 0);
return 0;
}
diff --git a/drivers/hid/usbhid/hid-core.c b/drivers/hid/usbhid/hid-core.c
index ccf0aad99a4b..3bfca6c3b528 100644
--- a/drivers/hid/usbhid/hid-core.c
+++ b/drivers/hid/usbhid/hid-core.c
@@ -281,9 +281,9 @@ static void hid_irq_in(struct urb *urb)
break;
usbhid_mark_busy(usbhid);
if (!test_bit(HID_RESUME_RUNNING, &usbhid->iofl)) {
- hid_input_report(urb->context, HID_INPUT_REPORT,
- urb->transfer_buffer,
- urb->actual_length, 1);
+ hid_safe_input_report(urb->context, HID_INPUT_REPORT,
+ urb->transfer_buffer, urb->transfer_buffer_length,
+ urb->actual_length, 1);
/*
* autosuspend refused while keys are pressed
* because most keyboards don't wake up when
@@ -480,9 +480,10 @@ static void hid_ctrl(struct urb *urb)
switch (status) {
case 0: /* success */
if (usbhid->ctrl[usbhid->ctrltail].dir == USB_DIR_IN)
- hid_input_report(urb->context,
+ hid_safe_input_report(urb->context,
usbhid->ctrl[usbhid->ctrltail].report->type,
- urb->transfer_buffer, urb->actual_length, 0);
+ urb->transfer_buffer, urb->transfer_buffer_length,
+ urb->actual_length, 0);
break;
case -ESHUTDOWN: /* unplug */
unplug = 1;
diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c
index 25e2958923af..e4b5ec18bf8a 100644
--- a/drivers/hv/channel_mgmt.c
+++ b/drivers/hv/channel_mgmt.c
@@ -743,13 +743,14 @@ static void init_vp_index(struct vmbus_channel *channel)
u32 i, ncpu = num_online_cpus();
cpumask_var_t available_mask;
struct cpumask *allocated_mask;
+ const struct cpumask *node_mask;
const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ);
u32 target_cpu;
int numa_node;
if (!perf_chn ||
!alloc_cpumask_var(&available_mask, GFP_KERNEL) ||
- cpumask_empty(hk_mask)) {
+ !cpumask_intersects(hk_mask, cpu_online_mask)) {
/*
* If the channel is not a performance critical
* channel, bind it to VMBUS_CONNECT_CPU.
@@ -771,14 +772,19 @@ static void init_vp_index(struct vmbus_channel *channel)
next_numa_node_id = 0;
continue;
}
- if (cpumask_empty(cpumask_of_node(numa_node)))
+ /*
+ * Try next NUMA node if current NUMA node has no CPU
+ * or doesn't contain any housekeeping CPU.
+ */
+ node_mask = cpumask_of_node(numa_node);
+ if (!cpumask_intersects(node_mask, hk_mask))
continue;
break;
}
allocated_mask = &hv_context.hv_numa_map[numa_node];
retry:
- cpumask_xor(available_mask, allocated_mask, cpumask_of_node(numa_node));
+ cpumask_xor(available_mask, allocated_mask, node_mask);
cpumask_and(available_mask, available_mask, hk_mask);
if (cpumask_empty(available_mask)) {
diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c
index 7bb94e0dd0f6..2eccdc9a6e78 100644
--- a/drivers/hv/connection.c
+++ b/drivers/hv/connection.c
@@ -322,7 +322,6 @@ int vmbus_connect(void)
cleanup:
pr_err("Unable to connect to host\n");
- vmbus_connection.conn_state = DISCONNECTED;
vmbus_disconnect();
kfree(msginfo);
diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c
index 8aa775c9382b..7d0d2c026127 100644
--- a/drivers/hv/vmbus_drv.c
+++ b/drivers/hv/vmbus_drv.c
@@ -2594,6 +2594,13 @@ static int __init hv_acpi_init(void)
return -ENODEV;
if (hv_root_partition)
+ /*
+ * A non-nested root partition does not need VMBus client
+ * functionality. However, the mshv_root module may have
+ * a dependency on the VMBus module as described in
+ * commit 840b740a35bf. Return success so the module
+ * loads even though no VMBus initialization is done.
+ */
return 0;
/*
@@ -2643,11 +2650,18 @@ static void __exit vmbus_exit(void)
{
int cpu;
+ if (hv_root_partition)
+ /*
+ * If a non-nested root partition loaded the VMBus module,
+ * hv_acpi_init() did not do any VMBus initialization.
+ * There's nothing to clean up, so just return.
+ */
+ return;
+
unregister_syscore_ops(&hv_synic_syscore_ops);
hv_remove_kexec_handler();
hv_remove_crash_handler();
- vmbus_connection.conn_state = DISCONNECTED;
hv_stimer_global_cleanup();
vmbus_disconnect();
if (vmbus_irq == -1) {
diff --git a/drivers/hwmon/drivetemp.c b/drivers/hwmon/drivetemp.c
index 5bac2b0fc7bb..b3de637a3a1f 100644
--- a/drivers/hwmon/drivetemp.c
+++ b/drivers/hwmon/drivetemp.c
@@ -165,6 +165,7 @@ static int drivetemp_scsi_command(struct drivetemp_data *st,
{
u8 scsi_cmd[MAX_COMMAND_SIZE];
int data_dir;
+ int err;
memset(scsi_cmd, 0, sizeof(scsi_cmd));
scsi_cmd[0] = ATA_16;
@@ -192,9 +193,12 @@ static int drivetemp_scsi_command(struct drivetemp_data *st,
scsi_cmd[12] = lba_high;
scsi_cmd[14] = ata_command;
- return scsi_execute_req(st->sdev, scsi_cmd, data_dir,
- st->smartdata, ATA_SECT_SIZE, NULL, HZ, 5,
- NULL);
+ err = scsi_execute_req(st->sdev, scsi_cmd, data_dir,
+ st->smartdata, ATA_SECT_SIZE, NULL, HZ, 5,
+ NULL);
+ if (err > 0)
+ err = -EIO;
+ return err;
}
static int drivetemp_ata_command(struct drivetemp_data *st, u8 feature,
diff --git a/drivers/hwmon/lm90.c b/drivers/hwmon/lm90.c
index 7ce75e64cc3c..7a8a47b9ba41 100644
--- a/drivers/hwmon/lm90.c
+++ b/drivers/hwmon/lm90.c
@@ -694,6 +694,7 @@ struct lm90_data {
struct mutex update_lock;
struct delayed_work alert_work;
struct work_struct report_work;
+ bool shutdown; /* true if shutting down */
bool valid; /* true if register values are valid */
bool alarms_valid; /* true if status register values are valid */
unsigned long last_updated; /* in jiffies */
@@ -1112,6 +1113,9 @@ static void lm90_report_alarms(struct work_struct *work)
static int lm90_update_alarms_locked(struct lm90_data *data, bool force)
{
+ if (data->shutdown)
+ return 0;
+
if (force || !data->alarms_valid ||
time_after(jiffies, data->alarms_updated + msecs_to_jiffies(data->update_interval))) {
struct i2c_client *client = data->client;
@@ -2557,15 +2561,23 @@ static void lm90_restore_conf(void *_data)
struct lm90_data *data = _data;
struct i2c_client *client = data->client;
- cancel_delayed_work_sync(&data->alert_work);
- cancel_work_sync(&data->report_work);
-
/* Restore initial configuration */
if (data->flags & LM90_HAVE_CONVRATE)
lm90_write_convrate(data, data->convrate_orig);
lm90_write_reg(client, LM90_REG_CONFIG1, data->config_orig);
}
+static void lm90_stop_work(void *_data)
+{
+ struct lm90_data *data = _data;
+
+ mutex_lock(&data->update_lock);
+ data->shutdown = true;
+ mutex_unlock(&data->update_lock);
+ cancel_delayed_work_sync(&data->alert_work);
+ cancel_work_sync(&data->report_work);
+}
+
static int lm90_init_client(struct i2c_client *client, struct lm90_data *data)
{
struct device_node *np = client->dev.of_node;
@@ -2914,6 +2926,10 @@ static int lm90_probe(struct i2c_client *client)
data->hwmon_dev = hwmon_dev;
+ err = devm_add_action_or_reset(&client->dev, lm90_stop_work, data);
+ if (err)
+ return err;
+
if (client->irq) {
dev_dbg(dev, "IRQ: %d\n", client->irq);
err = devm_request_threaded_irq(dev, client->irq,
@@ -2942,7 +2958,8 @@ static void lm90_alert(struct i2c_client *client, enum i2c_alert_protocol type,
*/
struct lm90_data *data = i2c_get_clientdata(client);
- if ((data->flags & LM90_HAVE_BROKEN_ALERT) &&
+ mutex_lock(&data->update_lock);
+ if (!data->shutdown && (data->flags & LM90_HAVE_BROKEN_ALERT) &&
(data->current_alarms & data->alert_alarms)) {
if (!(data->config & 0x80)) {
dev_dbg(&client->dev, "Disabling ALERT#\n");
@@ -2951,6 +2968,7 @@ static void lm90_alert(struct i2c_client *client, enum i2c_alert_protocol type,
schedule_delayed_work(&data->alert_work,
max_t(int, HZ, msecs_to_jiffies(data->update_interval)));
}
+ mutex_unlock(&data->update_lock);
} else {
dev_dbg(&client->dev, "Everything OK\n");
}
diff --git a/drivers/hwmon/pmbus/pmbus_core.c b/drivers/hwmon/pmbus/pmbus_core.c
index 11c95fc3305a..44e4d47372a6 100644
--- a/drivers/hwmon/pmbus/pmbus_core.c
+++ b/drivers/hwmon/pmbus/pmbus_core.c
@@ -19,7 +19,6 @@
#include <linux/pmbus.h>
#include <linux/regulator/driver.h>
#include <linux/regulator/machine.h>
-#include <linux/of.h>
#include <linux/thermal.h>
#include "pmbus.h"
@@ -2926,12 +2925,19 @@ static int pmbus_regulator_get_voltage(struct regulator_dev *rdev)
.class = PSC_VOLTAGE_OUT,
.convert = true,
};
+ int ret;
+ mutex_lock(&data->update_lock);
s.data = _pmbus_read_word_data(client, s.page, 0xff, PMBUS_READ_VOUT);
- if (s.data < 0)
- return s.data;
+ if (s.data < 0) {
+ ret = s.data;
+ goto unlock;
+ }
- return (int)pmbus_reg2data(data, &s) * 1000; /* unit is uV */
+ ret = (int)pmbus_reg2data(data, &s) * 1000; /* unit is uV */
+unlock:
+ mutex_unlock(&data->update_lock);
+ return ret;
}
static int pmbus_regulator_set_voltage(struct regulator_dev *rdev, int min_uv,
@@ -2948,16 +2954,22 @@ static int pmbus_regulator_set_voltage(struct regulator_dev *rdev, int min_uv,
};
int val = DIV_ROUND_CLOSEST(min_uv, 1000); /* convert to mV */
int low, high;
+ int ret;
*selector = 0;
+ mutex_lock(&data->update_lock);
low = pmbus_regulator_get_low_margin(client, s.page);
- if (low < 0)
- return low;
+ if (low < 0) {
+ ret = low;
+ goto unlock;
+ }
high = pmbus_regulator_get_high_margin(client, s.page);
- if (high < 0)
- return high;
+ if (high < 0) {
+ ret = high;
+ goto unlock;
+ }
/* Make sure we are within margins */
if (low > val)
@@ -2967,7 +2979,10 @@ static int pmbus_regulator_set_voltage(struct regulator_dev *rdev, int min_uv,
val = pmbus_data2reg(data, &s, val);
- return _pmbus_write_word_data(client, s.page, PMBUS_VOUT_COMMAND, (u16)val);
+ ret = _pmbus_write_word_data(client, s.page, PMBUS_VOUT_COMMAND, (u16)val);
+unlock:
+ mutex_unlock(&data->update_lock);
+ return ret;
}
static int pmbus_regulator_list_voltage(struct regulator_dev *rdev,
@@ -2975,7 +2990,9 @@ static int pmbus_regulator_list_voltage(struct regulator_dev *rdev,
{
struct device *dev = rdev_get_dev(rdev);
struct i2c_client *client = to_i2c_client(dev->parent);
+ struct pmbus_data *data = i2c_get_clientdata(client);
int val, low, high;
+ int ret;
if (selector >= rdev->desc->n_voltages ||
selector < rdev->desc->linear_min_sel)
@@ -2985,18 +3002,29 @@ static int pmbus_regulator_list_voltage(struct regulator_dev *rdev,
val = DIV_ROUND_CLOSEST(rdev->desc->min_uV +
(rdev->desc->uV_step * selector), 1000); /* convert to mV */
+ mutex_lock(&data->update_lock);
+
low = pmbus_regulator_get_low_margin(client, rdev_get_id(rdev));
- if (low < 0)
- return low;
+ if (low < 0) {
+ ret = low;
+ goto unlock;
+ }
high = pmbus_regulator_get_high_margin(client, rdev_get_id(rdev));
- if (high < 0)
- return high;
+ if (high < 0) {
+ ret = high;
+ goto unlock;
+ }
- if (val >= low && val <= high)
- return val * 1000; /* unit is uV */
+ if (val >= low && val <= high) {
+ ret = val * 1000; /* unit is uV */
+ goto unlock;
+ }
- return 0;
+ ret = 0;
+unlock:
+ mutex_unlock(&data->update_lock);
+ return ret;
}
const struct regulator_ops pmbus_regulator_ops = {
diff --git a/drivers/hwmon/pwm-fan.c b/drivers/hwmon/pwm-fan.c
index 83a347ca35da..a6449586d4d7 100644
--- a/drivers/hwmon/pwm-fan.c
+++ b/drivers/hwmon/pwm-fan.c
@@ -466,12 +466,18 @@ static void pwm_fan_cleanup(void *__ctx)
{
struct pwm_fan_ctx *ctx = __ctx;
- del_timer_sync(&ctx->rpm_timer);
/* Switch off everything */
ctx->enable_mode = pwm_disable_reg_disable;
pwm_fan_power_off(ctx);
}
+static void pwm_fan_timer_cleanup(void *__ctx)
+{
+ struct pwm_fan_ctx *ctx = __ctx;
+
+ timer_shutdown_sync(&ctx->rpm_timer);
+}
+
static int pwm_fan_probe(struct platform_device *pdev)
{
struct thermal_cooling_device *cdev;
@@ -598,6 +604,10 @@ static int pwm_fan_probe(struct platform_device *pdev)
}
if (ctx->tach_count > 0) {
+ ret = devm_add_action_or_reset(dev, pwm_fan_timer_cleanup, ctx);
+ if (ret)
+ return ret;
+
ctx->sample_start = ktime_get();
mod_timer(&ctx->rpm_timer, jiffies + HZ);
@@ -639,6 +649,7 @@ static void pwm_fan_shutdown(struct platform_device *pdev)
{
struct pwm_fan_ctx *ctx = platform_get_drvdata(pdev);
+ pwm_fan_timer_cleanup(ctx);
pwm_fan_cleanup(ctx);
}
diff --git a/drivers/hwmon/sht3x.c b/drivers/hwmon/sht3x.c
index 3f279aa1cee5..381e33c6b4a2 100644
--- a/drivers/hwmon/sht3x.c
+++ b/drivers/hwmon/sht3x.c
@@ -21,6 +21,7 @@
#include <linux/slab.h>
#include <linux/jiffies.h>
#include <linux/platform_data/sht3x.h>
+#include <asm/unaligned.h>
/* commands (high precision mode) */
static const unsigned char sht3x_cmd_measure_blocking_hpm[] = { 0x2c, 0x06 };
@@ -241,9 +242,9 @@ static struct sht3x_data *sht3x_update_client(struct device *dev)
if (ret)
goto out;
- val = be16_to_cpup((__be16 *)buf);
+ val = get_unaligned_be16(buf);
data->temperature = sht3x_extract_temperature(val);
- val = be16_to_cpup((__be16 *)(buf + 3));
+ val = get_unaligned_be16(buf + 3);
data->humidity = sht3x_extract_humidity(val);
data->last_update = jiffies;
}
@@ -302,7 +303,7 @@ static int limits_update(struct sht3x_data *data)
if (ret)
return ret;
- raw = be16_to_cpup((__be16 *)buffer);
+ raw = get_unaligned_be16(buffer);
temperature = sht3x_extract_temperature((raw & 0x01ff) << 7);
humidity = sht3x_extract_humidity(raw & 0xfe00);
data->temperature_limits[index] = temperature;
@@ -364,7 +365,7 @@ static size_t limit_store(struct device *dev,
raw = ((u32)(temperature + 45000) * 24543) >> (16 + 7);
raw |= ((humidity * 42950) >> 16) & 0xfe00;
- *((__be16 *)position) = cpu_to_be16(raw);
+ put_unaligned_be16(raw, position);
position += SHT3X_WORD_LEN;
*position = crc8(sht3x_crc8_table,
position - SHT3X_WORD_LEN,
diff --git a/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c b/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c
index fd81eca3ec18..a49b3c237c07 100644
--- a/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c
+++ b/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c
@@ -716,26 +716,19 @@ static DEVICE_ATTR_RW(cntr_rld_event);
static ssize_t cntr_val_show(struct device *dev,
struct device_attribute *attr, char *buf)
{
- int i, ret = 0;
u32 val;
struct etm_drvdata *drvdata = dev_get_drvdata(dev->parent);
struct etm_config *config = &drvdata->config;
if (!local_read(&drvdata->mode)) {
spin_lock(&drvdata->spinlock);
- for (i = 0; i < drvdata->nr_cntr; i++)
- ret += sprintf(buf, "counter %d: %x\n",
- i, config->cntr_val[i]);
+ val = config->cntr_val[config->cntr_idx];
spin_unlock(&drvdata->spinlock);
- return ret;
- }
-
- for (i = 0; i < drvdata->nr_cntr; i++) {
- val = etm_readl(drvdata, ETMCNTVRn(i));
- ret += sprintf(buf, "counter %d: %x\n", i, val);
+ } else {
+ val = etm_readl(drvdata, ETMCNTVRn(config->cntr_idx));
}
- return ret;
+ return sysfs_emit(buf, "%#x\n", val);
}
static ssize_t cntr_val_store(struct device *dev,
diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c
index 45f2f6db1588..0b77c180d855 100644
--- a/drivers/hwtracing/coresight/coresight-etm4x-core.c
+++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c
@@ -3,6 +3,7 @@
* Copyright (c) 2014, The Linux Foundation. All rights reserved.
*/
+#include <linux/acpi.h>
#include <linux/bitops.h>
#include <linux/kernel.h>
#include <linux/moduleparam.h>
@@ -51,10 +52,14 @@ MODULE_PARM_DESC(boot_enable, "Enable tracing on boot");
#define PARAM_PM_SAVE_NEVER 1 /* never save any state */
#define PARAM_PM_SAVE_SELF_HOSTED 2 /* save self-hosted state only */
+/*
+ * Save option for ETM4. ETE, sysreg ETM4s and ACPI boots ignore this option and
+ * will always save.
+ */
static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE;
module_param(pm_save_enable, int, 0444);
MODULE_PARM_DESC(pm_save_enable,
- "Save/restore state on power down: 1 = never, 2 = self-hosted");
+ "Save/restore state on power down: 1 = never, 2 = self-hosted. MMIO and DT only.");
static struct etmv4_drvdata *etmdrvdata[NR_CPUS];
static void etm4_set_default_config(struct etmv4_config *config);
@@ -664,8 +669,6 @@ static int etm4_parse_event_config(struct coresight_device *csdev,
struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
struct etmv4_config *config = &drvdata->config;
struct perf_event_attr *attr = &event->attr;
- unsigned long cfg_hash;
- int preset;
/* Clear configuration from previous run */
memset(config, 0, sizeof(struct etmv4_config));
@@ -735,20 +738,6 @@ static int etm4_parse_event_config(struct coresight_device *csdev,
/* bit[12], Return stack enable bit */
config->cfg |= TRCCONFIGR_RS;
- /*
- * Set any selected configuration and preset.
- *
- * This extracts the values of PMU_FORMAT_ATTR(configid) and PMU_FORMAT_ATTR(preset)
- * in the perf attributes defined in coresight-etm-perf.c.
- * configid uses bits 63:32 of attr->config2, preset uses bits 3:0 of attr->config.
- * A zero configid means no configuration active, preset = 0 means no preset selected.
- */
- if (attr->config2 & GENMASK_ULL(63, 32)) {
- cfg_hash = (u32)(attr->config2 >> 32);
- preset = attr->config & 0xF;
- ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset);
- }
-
/* branch broadcast - enable if selected and supported */
if (attr->config & BIT(ETM_OPT_BRANCH_BROADCAST)) {
if (!drvdata->trcbb) {
@@ -772,6 +761,9 @@ static int etm4_enable_perf(struct coresight_device *csdev,
{
int ret = 0;
struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent);
+ struct perf_event_attr *attr = &event->attr;
+ unsigned long cfg_hash = 0;
+ int preset;
if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id())) {
ret = -EINVAL;
@@ -782,8 +774,27 @@ static int etm4_enable_perf(struct coresight_device *csdev,
ret = etm4_parse_event_config(csdev, event);
if (ret)
goto out;
+
+ /*
+ * Set any selected configuration and preset.
+ *
+ * This extracts the values of PMU_FORMAT_ATTR(configid) and PMU_FORMAT_ATTR(preset)
+ * in the perf attributes defined in coresight-etm-perf.c.
+ * configid uses bits 63:32 of attr->config2, preset uses bits 3:0 of attr->config.
+ * A zero configid means no configuration active, preset = 0 means no preset selected.
+ */
+ if (attr->config2 & GENMASK_ULL(63, 32)) {
+ cfg_hash = (u32)(attr->config2 >> 32);
+ preset = attr->config & 0xF;
+ ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset);
+ if (ret)
+ goto out;
+ }
+
/* And enable it */
ret = etm4_enable_hw(drvdata);
+ if (ret && cfg_hash)
+ cscfg_csdev_disable_active_config(csdev);
out:
return ret;
@@ -1994,6 +2005,17 @@ static void etm4_pm_clear(void)
}
}
+static bool etm4x_always_pm_save(struct device *dev, struct csdev_access *csa)
+{
+ /*
+ * Only IO mem ETM devices will benefit from skipping PM save and only
+ * DT has the option to control it, not ACPI. Otherwise system register
+ * based ETMs and ETEs will always lose context on CPU power down, so
+ * always save.
+ */
+ return !csa->io_mem || is_acpi_device_node(dev_fwnode(dev));
+}
+
static int etm4_probe(struct device *dev, void __iomem *base, u32 etm_pid)
{
int ret;
@@ -2003,6 +2025,7 @@ static int etm4_probe(struct device *dev, void __iomem *base, u32 etm_pid)
struct etm4_init_arg init_arg = { 0 };
u8 major, minor;
char *type_name;
+ bool pm_save;
drvdata = devm_kzalloc(dev, sizeof(*drvdata), GFP_KERNEL);
if (!drvdata)
@@ -2010,17 +2033,6 @@ static int etm4_probe(struct device *dev, void __iomem *base, u32 etm_pid)
dev_set_drvdata(dev, drvdata);
- if (pm_save_enable == PARAM_PM_SAVE_FIRMWARE)
- pm_save_enable = coresight_loses_context_with_cpu(dev) ?
- PARAM_PM_SAVE_SELF_HOSTED : PARAM_PM_SAVE_NEVER;
-
- if (pm_save_enable != PARAM_PM_SAVE_NEVER) {
- drvdata->save_state = devm_kmalloc(dev,
- sizeof(struct etmv4_save_state), GFP_KERNEL);
- if (!drvdata->save_state)
- return -ENOMEM;
- }
-
drvdata->base = base;
spin_lock_init(&drvdata->spinlock);
@@ -2064,6 +2076,21 @@ static int etm4_probe(struct device *dev, void __iomem *base, u32 etm_pid)
etm4_init_trace_id(drvdata);
etm4_set_default(&drvdata->config);
+ if (etm4x_always_pm_save(dev, &desc.access))
+ pm_save = true;
+ else if (pm_save_enable == PARAM_PM_SAVE_FIRMWARE)
+ pm_save = coresight_loses_context_with_cpu(dev);
+ else
+ pm_save = pm_save_enable != PARAM_PM_SAVE_NEVER;
+
+ if (pm_save) {
+ drvdata->save_state = devm_kmalloc(dev,
+ sizeof(struct etmv4_save_state),
+ GFP_KERNEL);
+ if (!drvdata->save_state)
+ return -ENOMEM;
+ }
+
pdata = coresight_get_platform_data(dev);
if (IS_ERR(pdata))
return PTR_ERR(pdata);
diff --git a/drivers/hwtracing/coresight/coresight-platform.c b/drivers/hwtracing/coresight/coresight-platform.c
index 3f82ae07a18e..f19e6758ce59 100644
--- a/drivers/hwtracing/coresight/coresight-platform.c
+++ b/drivers/hwtracing/coresight/coresight-platform.c
@@ -520,19 +520,18 @@ static inline bool acpi_validate_dsd_graph(const union acpi_object *graph)
/* acpi_get_dsd_graph - Find the _DSD Graph property for the given device. */
static const union acpi_object *
-acpi_get_dsd_graph(struct acpi_device *adev)
+acpi_get_dsd_graph(struct acpi_device *adev, struct acpi_buffer *buf)
{
int i;
- struct acpi_buffer buf = { ACPI_ALLOCATE_BUFFER };
acpi_status status;
const union acpi_object *dsd;
status = acpi_evaluate_object_typed(adev->handle, "_DSD", NULL,
- &buf, ACPI_TYPE_PACKAGE);
+ buf, ACPI_TYPE_PACKAGE);
if (ACPI_FAILURE(status))
return NULL;
- dsd = buf.pointer;
+ dsd = buf->pointer;
/*
* _DSD property consists tuples { Prop_UUID, Package() }
@@ -583,12 +582,12 @@ acpi_validate_coresight_graph(const union acpi_object *cs_graph)
* returns NULL.
*/
static const union acpi_object *
-acpi_get_coresight_graph(struct acpi_device *adev)
+acpi_get_coresight_graph(struct acpi_device *adev, struct acpi_buffer *buf)
{
const union acpi_object *graph_list, *graph;
int i, nr_graphs;
- graph_list = acpi_get_dsd_graph(adev);
+ graph_list = acpi_get_dsd_graph(adev, buf);
if (!graph_list)
return graph_list;
@@ -688,18 +687,22 @@ static int acpi_coresight_parse_link(struct acpi_device *adev,
static int acpi_coresight_parse_graph(struct acpi_device *adev,
struct coresight_platform_data *pdata)
{
- int rc, i, nlinks;
+ int rc = 0;
+ int i, nlinks;
const union acpi_object *graph;
struct coresight_connection *conns, *ptr;
+ struct acpi_buffer buf = { ACPI_ALLOCATE_BUFFER, NULL };
pdata->nr_inport = pdata->nr_outport = 0;
- graph = acpi_get_coresight_graph(adev);
- if (!graph)
- return -ENOENT;
+ graph = acpi_get_coresight_graph(adev, &buf);
+ if (!graph) {
+ rc = -ENOENT;
+ goto free;
+ }
nlinks = graph->package.elements[2].integer.value;
if (!nlinks)
- return 0;
+ goto free;
/*
* To avoid scanning the table twice (once for finding the number of
@@ -708,16 +711,20 @@ static int acpi_coresight_parse_graph(struct acpi_device *adev,
* it to the pdata.
*/
conns = devm_kcalloc(&adev->dev, nlinks, sizeof(*conns), GFP_KERNEL);
- if (!conns)
- return -ENOMEM;
+ if (!conns) {
+ rc = -ENOMEM;
+ goto free;
+ }
ptr = conns;
for (i = 0; i < nlinks; i++) {
const union acpi_object *link = &graph->package.elements[3 + i];
int dir;
dir = acpi_coresight_parse_link(adev, link, ptr);
- if (dir < 0)
- return dir;
+ if (dir < 0) {
+ rc = dir;
+ goto free;
+ }
if (dir == ACPI_CORESIGHT_LINK_MASTER) {
if (ptr->outport >= pdata->nr_outport)
@@ -739,7 +746,7 @@ static int acpi_coresight_parse_graph(struct acpi_device *adev,
rc = coresight_alloc_conns(&adev->dev, pdata);
if (rc)
- return rc;
+ goto free;
/* Copy the connection information to the final location */
for (i = 0; conns + i < ptr; i++) {
@@ -751,7 +758,14 @@ static int acpi_coresight_parse_graph(struct acpi_device *adev,
}
devm_kfree(&adev->dev, conns);
- return 0;
+free:
+ /*
+ * When ACPI fails to alloc a buffer, it will free the buffer
+ * created via ACPI_ALLOCATE_BUFFER and set to NULL.
+ * ACPI_FREE can handle NULL pointers, so free it directly.
+ */
+ ACPI_FREE(buf.pointer);
+ return rc;
}
/*
diff --git a/drivers/hwtracing/coresight/coresight-syscfg.c b/drivers/hwtracing/coresight/coresight-syscfg.c
index 30a561d87481..55f36ed1af13 100644
--- a/drivers/hwtracing/coresight/coresight-syscfg.c
+++ b/drivers/hwtracing/coresight/coresight-syscfg.c
@@ -395,6 +395,8 @@ static void cscfg_remove_owned_csdev_configs(struct coresight_device *csdev, voi
if (list_empty(&csdev->config_csdev_list))
return;
+ guard(spinlock_irqsave)(&csdev->cscfg_csdev_lock);
+
list_for_each_entry_safe(config_csdev, tmp, &csdev->config_csdev_list, node) {
if (config_csdev->config_desc->load_owner == load_owner)
list_del(&config_csdev->node);
diff --git a/drivers/i2c/busses/i2c-at91-master.c b/drivers/i2c/busses/i2c-at91-master.c
index 28ed1bde3d68..593881902a0d 100644
--- a/drivers/i2c/busses/i2c-at91-master.c
+++ b/drivers/i2c/busses/i2c-at91-master.c
@@ -919,8 +919,14 @@ int at91_twi_probe_master(struct platform_device *pdev,
at91_calc_twi_clock(dev);
rc = at91_init_twi_recovery_info(pdev, dev);
- if (rc == -EPROBE_DEFER)
+ if (rc == -EPROBE_DEFER) {
+ /*
+ * The DMA channels were already set up above, release them
+ * before deferring the probe so they are not leaked.
+ */
+ at91_twi_dma_release(dev);
return rc;
+ }
dev->adapter.algo = &at91_twi_algorithm;
dev->adapter.quirks = &at91_twi_quirks;
diff --git a/drivers/i2c/busses/i2c-mxs.c b/drivers/i2c/busses/i2c-mxs.c
index e0f3b3545cfe..d792bcc11a89 100644
--- a/drivers/i2c/busses/i2c-mxs.c
+++ b/drivers/i2c/busses/i2c-mxs.c
@@ -832,7 +832,7 @@ static int mxs_i2c_probe(struct platform_device *pdev)
}
/* Setup the DMA */
- i2c->dmach = dma_request_chan(dev, "rx-tx");
+ i2c->dmach = devm_dma_request_chan(dev, "rx-tx");
if (IS_ERR(i2c->dmach)) {
return dev_err_probe(dev, PTR_ERR(i2c->dmach),
"Failed to request dma\n");
@@ -870,9 +870,6 @@ static int mxs_i2c_remove(struct platform_device *pdev)
i2c_del_adapter(&i2c->adapter);
- if (i2c->dmach)
- dma_release_channel(i2c->dmach);
-
writel(MXS_I2C_CTRL0_SFTRST, i2c->regs + MXS_I2C_CTRL0_SET);
return 0;
diff --git a/drivers/i2c/busses/i2c-qcom-cci.c b/drivers/i2c/busses/i2c-qcom-cci.c
index 4989d0e705d7..f64622e725d5 100644
--- a/drivers/i2c/busses/i2c-qcom-cci.c
+++ b/drivers/i2c/busses/i2c-qcom-cci.c
@@ -168,12 +168,12 @@ static irqreturn_t cci_isr(int irq, void *dev)
}
if (unlikely(val & CCI_IRQ_STATUS_0_I2C_M0_Q0Q1_HALT_ACK)) {
- reset = CCI_RESET_CMD_M0_MASK;
+ reset |= CCI_RESET_CMD_M0_MASK;
ret = IRQ_HANDLED;
}
if (unlikely(val & CCI_IRQ_STATUS_0_I2C_M1_Q0Q1_HALT_ACK)) {
- reset = CCI_RESET_CMD_M1_MASK;
+ reset |= CCI_RESET_CMD_M1_MASK;
ret = IRQ_HANDLED;
}
diff --git a/drivers/i2c/busses/i2c-wmt.c b/drivers/i2c/busses/i2c-wmt.c
index 7d4bc8736079..a1716dddfca8 100644
--- a/drivers/i2c/busses/i2c-wmt.c
+++ b/drivers/i2c/busses/i2c-wmt.c
@@ -391,7 +391,7 @@ static int wmt_i2c_probe(struct platform_device *pdev)
return -EINVAL;
}
- i2c_dev->clk = of_clk_get(np, 0);
+ i2c_dev->clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(i2c_dev->clk)) {
dev_err(&pdev->dev, "unable to request clock\n");
return PTR_ERR(i2c_dev->clk);
diff --git a/drivers/i2c/i2c-core-base.c b/drivers/i2c/i2c-core-base.c
index e8fb84c7cfd8..cfb1454c48b9 100644
--- a/drivers/i2c/i2c-core-base.c
+++ b/drivers/i2c/i2c-core-base.c
@@ -2432,6 +2432,20 @@ int i2c_get_device_id(const struct i2c_client *client,
}
EXPORT_SYMBOL_GPL(i2c_get_device_id);
+/**
+ * i2c_client_get_device_id - get the driver match table entry of a device
+ * @client: the device to query. The device must be bound to a driver
+ *
+ * Returns a pointer to the matching entry if found, NULL otherwise.
+ */
+const struct i2c_device_id *i2c_client_get_device_id(const struct i2c_client *client)
+{
+ const struct i2c_driver *drv = to_i2c_driver(client->dev.driver);
+
+ return i2c_match_id(drv->id_table, client);
+}
+EXPORT_SYMBOL_GPL(i2c_client_get_device_id);
+
/* ----------------------------------------------------
* the i2c address scanning function
* Will not work for 10-bit addresses!
diff --git a/drivers/i3c/master.c b/drivers/i3c/master.c
index 059ddd50e859..62d0bbcf560f 100644
--- a/drivers/i3c/master.c
+++ b/drivers/i3c/master.c
@@ -667,6 +667,11 @@ static struct attribute *i3c_masterdev_attrs[] = {
};
ATTRIBUTE_GROUPS(i3c_masterdev);
+static void i3c_master_free_i3c_dev(struct i3c_dev_desc *dev)
+{
+ kfree(dev);
+}
+
static void i3c_masterdev_release(struct device *dev)
{
struct i3c_master_controller *master = dev_to_i3cmaster(dev);
@@ -679,6 +684,8 @@ static void i3c_masterdev_release(struct device *dev)
i3c_bus_cleanup(bus);
of_node_put(dev->of_node);
+
+ i3c_master_free_i3c_dev(master->this);
}
static const struct device_type i3c_masterdev_type = {
@@ -866,11 +873,6 @@ static void i3c_device_release(struct device *dev)
kfree(i3cdev);
}
-static void i3c_master_free_i3c_dev(struct i3c_dev_desc *dev)
-{
- kfree(dev);
-}
-
static struct i3c_dev_desc *
i3c_master_alloc_i3c_dev(struct i3c_master_controller *master,
const struct i3c_device_info *info)
@@ -1613,12 +1615,15 @@ static int i3c_master_early_i3c_dev_add(struct i3c_master_controller *master,
static void
i3c_master_register_new_i3c_devs(struct i3c_master_controller *master)
{
+ struct i3c_device *i3cdev, *tmp;
struct i3c_dev_desc *desc;
+ LIST_HEAD(i3c_unreg_devs);
int ret;
if (!master->init_done)
return;
+ i3c_bus_maintenance_lock(&master->bus);
i3c_bus_for_each_i3cdev(&master->bus, desc) {
if (desc->dev || !desc->info.dyn_addr || desc == master->this)
continue;
@@ -1639,15 +1644,30 @@ i3c_master_register_new_i3c_devs(struct i3c_master_controller *master)
if (desc->boardinfo)
desc->dev->dev.of_node = desc->boardinfo->of_node;
- ret = device_register(&desc->dev->dev);
- if (ret) {
- dev_err(&master->dev,
- "Failed to add I3C device (err = %d)\n", ret);
- desc->dev->desc = NULL;
- put_device(&desc->dev->dev);
- desc->dev = NULL;
- }
+ list_add_tail(&desc->dev->node, &i3c_unreg_devs);
}
+
+ i3c_bus_maintenance_unlock(&master->bus);
+
+ list_for_each_entry_safe(i3cdev, tmp, &i3c_unreg_devs, node) {
+ ret = device_register(&i3cdev->dev);
+ if (ret)
+ dev_err(&master->dev, "Failed to add I3C device (err = %d)\n", ret);
+ else
+ list_del_init(&i3cdev->node);
+ }
+
+ i3c_bus_maintenance_lock(&master->bus);
+
+ list_for_each_entry_safe(i3cdev, tmp, &i3c_unreg_devs, node) {
+ list_del(&i3cdev->node);
+ desc = i3cdev->desc;
+ i3cdev->desc = NULL;
+ put_device(&i3cdev->dev);
+ desc->dev = NULL;
+ }
+
+ i3c_bus_maintenance_unlock(&master->bus);
}
/**
@@ -1676,9 +1696,7 @@ int i3c_master_do_daa(struct i3c_master_controller *master)
if (ret)
return ret;
- i3c_bus_normaluse_lock(&master->bus);
i3c_master_register_new_i3c_devs(master);
- i3c_bus_normaluse_unlock(&master->bus);
return 0;
}
@@ -1738,6 +1756,8 @@ int i3c_master_set_info(struct i3c_master_controller *master,
return 0;
err_free_dev:
+ master->bus.cur_master = NULL;
+ master->this = NULL;
i3c_master_free_i3c_dev(i3cdev);
return ret;
@@ -1758,7 +1778,8 @@ static void i3c_master_detach_free_devs(struct i3c_master_controller *master)
i3cdev->boardinfo->init_dyn_addr,
I3C_ADDR_SLOT_FREE);
- i3c_master_free_i3c_dev(i3cdev);
+ if (i3cdev != master->this)
+ i3c_master_free_i3c_dev(i3cdev);
}
list_for_each_entry_safe(i2cdev, i2ctmp, &master->bus.devs.i2c,
@@ -1978,7 +1999,8 @@ i3c_master_search_i3c_dev_duplicate(struct i3c_dev_desc *refdev)
struct i3c_dev_desc *i3cdev;
i3c_bus_for_each_i3cdev(&master->bus, i3cdev) {
- if (i3cdev != refdev && i3cdev->info.pid == refdev->info.pid)
+ if (i3cdev != refdev && i3cdev->info.pid == refdev->info.pid &&
+ i3cdev != master->this)
return i3cdev;
}
@@ -2846,9 +2868,7 @@ int i3c_master_register(struct i3c_master_controller *master,
* register I3C devices discovered during the initial DAA.
*/
master->init_done = true;
- i3c_bus_normaluse_lock(&master->bus);
i3c_master_register_new_i3c_devs(master);
- i3c_bus_normaluse_unlock(&master->bus);
return 0;
diff --git a/drivers/i3c/master/mipi-i3c-hci/core.c b/drivers/i3c/master/mipi-i3c-hci/core.c
index 3c5ce4a64f80..b9a6073499ee 100644
--- a/drivers/i3c/master/mipi-i3c-hci/core.c
+++ b/drivers/i3c/master/mipi-i3c-hci/core.c
@@ -751,6 +751,8 @@ static int i3c_hci_probe(struct platform_device *pdev)
if (IS_ERR(hci->base_regs))
return PTR_ERR(hci->base_regs);
+ mutex_init(&hci->control_mutex);
+
platform_set_drvdata(pdev, hci);
/* temporary for dev_printk's, to be replaced in i3c_master_register */
hci->master.dev.init_name = dev_name(&pdev->dev);
diff --git a/drivers/i3c/master/mipi-i3c-hci/dma.c b/drivers/i3c/master/mipi-i3c-hci/dma.c
index 2cf4974fffa7..d1d92be40325 100644
--- a/drivers/i3c/master/mipi-i3c-hci/dma.c
+++ b/drivers/i3c/master/mipi-i3c-hci/dma.c
@@ -448,16 +448,25 @@ static bool hci_dma_dequeue_xfer(struct i3c_hci *hci,
struct hci_rh_data *rh = &rings->headers[xfer_list[0].ring_number];
unsigned int i;
bool did_unqueue = false;
-
- /* stop the ring */
- rh_reg_write(RING_CONTROL, RING_CTRL_ABORT);
- if (wait_for_completion_timeout(&rh->op_done, HZ) == 0) {
- /*
- * We're deep in it if ever this condition is ever met.
- * Hardware might still be writing to memory, etc.
- */
- dev_crit(&hci->master.dev, "unable to abort the ring\n");
- WARN_ON(1);
+ u32 ring_status;
+
+ guard(mutex)(&hci->control_mutex);
+
+ ring_status = rh_reg_read(RING_STATUS);
+ if (ring_status & RING_STATUS_RUNNING) {
+ /* stop the ring */
+ reinit_completion(&rh->op_done);
+ rh_reg_write(RING_CONTROL, rh_reg_read(RING_CONTROL) | RING_CTRL_ABORT);
+ wait_for_completion_timeout(&rh->op_done, HZ);
+ ring_status = rh_reg_read(RING_STATUS);
+ if (ring_status & RING_STATUS_RUNNING) {
+ /*
+ * We're deep in it if ever this condition is ever met.
+ * Hardware might still be writing to memory, etc.
+ */
+ dev_crit(&hci->master.dev, "unable to abort the ring\n");
+ WARN_ON(1);
+ }
}
for (i = 0; i < n; i++) {
diff --git a/drivers/i3c/master/mipi-i3c-hci/hci.h b/drivers/i3c/master/mipi-i3c-hci/hci.h
index f109923f6c3f..39490bc9f84f 100644
--- a/drivers/i3c/master/mipi-i3c-hci/hci.h
+++ b/drivers/i3c/master/mipi-i3c-hci/hci.h
@@ -43,6 +43,7 @@ struct i3c_hci {
const struct hci_io_ops *io;
void *io_data;
const struct hci_cmd_ops *cmd;
+ struct mutex control_mutex;
atomic_t next_cmd_tid;
u32 caps;
unsigned int quirks;
diff --git a/drivers/iio/accel/kxcjk-1013.c b/drivers/iio/accel/kxcjk-1013.c
index adc66b3615c0..6f594c42b337 100644
--- a/drivers/iio/accel/kxcjk-1013.c
+++ b/drivers/iio/accel/kxcjk-1013.c
@@ -1051,7 +1051,7 @@ static int kxcjk1013_write_event_config(struct iio_dev *indio_dev,
struct kxcjk1013_data *data = iio_priv(indio_dev);
int ret;
- if (state && data->ev_enable_state)
+ if (state == data->ev_enable_state)
return 0;
mutex_lock(&data->mutex);
diff --git a/drivers/iio/accel/sca3000.c b/drivers/iio/accel/sca3000.c
index 2b87f7f5508b..4b0497f63f83 100644
--- a/drivers/iio/accel/sca3000.c
+++ b/drivers/iio/accel/sca3000.c
@@ -652,7 +652,7 @@ static int sca3000_write_raw_samp_freq(struct sca3000_state *st, int val)
if (val == base_freq / 2)
ctrlval |= SCA3000_REG_OUT_CTRL_BUF_DIV_2;
- if (val == base_freq / 4)
+ else if (val == base_freq / 4)
ctrlval |= SCA3000_REG_OUT_CTRL_BUF_DIV_4;
else if (val != base_freq)
return -EINVAL;
diff --git a/drivers/iio/accel/st_accel_core.c b/drivers/iio/accel/st_accel_core.c
index c4143c567874..a5c0ecbae271 100644
--- a/drivers/iio/accel/st_accel_core.c
+++ b/drivers/iio/accel/st_accel_core.c
@@ -1269,6 +1269,7 @@ static int apply_acpi_orientation(struct iio_dev *indio_dev)
union acpi_object *ont;
union acpi_object *elements;
acpi_status status;
+ struct device *parent = indio_dev->dev.parent;
int ret = -EINVAL;
unsigned int val;
int i, j;
@@ -1287,7 +1288,7 @@ static int apply_acpi_orientation(struct iio_dev *indio_dev)
};
- adev = ACPI_COMPANION(indio_dev->dev.parent);
+ adev = ACPI_COMPANION(parent);
if (!adev)
return -ENXIO;
@@ -1296,8 +1297,7 @@ static int apply_acpi_orientation(struct iio_dev *indio_dev)
if (status == AE_NOT_FOUND) {
return -ENXIO;
} else if (ACPI_FAILURE(status)) {
- dev_warn(&indio_dev->dev, "failed to execute _ONT: %d\n",
- status);
+ dev_warn(parent, "failed to execute _ONT: %d\n", status);
return status;
}
@@ -1373,12 +1373,12 @@ static int apply_acpi_orientation(struct iio_dev *indio_dev)
}
ret = 0;
- dev_info(&indio_dev->dev, "computed mount matrix from ACPI\n");
+ dev_info(parent, "computed mount matrix from ACPI\n");
out:
kfree(buffer.pointer);
if (ret)
- dev_dbg(&indio_dev->dev,
+ dev_dbg(parent,
"failed to apply ACPI orientation data: %d\n", ret);
return ret;
diff --git a/drivers/iio/adc/ad_sigma_delta.c b/drivers/iio/adc/ad_sigma_delta.c
index 914274ed899e..45e0f638c0ed 100644
--- a/drivers/iio/adc/ad_sigma_delta.c
+++ b/drivers/iio/adc/ad_sigma_delta.c
@@ -343,9 +343,8 @@ EXPORT_SYMBOL_NS_GPL(ad_sigma_delta_single_conversion, IIO_AD_SIGMA_DELTA);
static int ad_sd_buffer_postenable(struct iio_dev *indio_dev)
{
struct ad_sigma_delta *sigma_delta = iio_device_get_drvdata(indio_dev);
- unsigned int i, slot, samples_buf_size;
+ unsigned int i, slot;
unsigned int channel;
- uint8_t *samples_buf;
int ret;
if (sigma_delta->num_slots == 1) {
@@ -378,15 +377,6 @@ static int ad_sd_buffer_postenable(struct iio_dev *indio_dev)
return ret;
}
- samples_buf_size = ALIGN(slot * indio_dev->channels[0].scan_type.storagebits / 8, 8);
- samples_buf_size += sizeof(int64_t);
- samples_buf = devm_krealloc(&sigma_delta->spi->dev, sigma_delta->samples_buf,
- samples_buf_size, GFP_KERNEL);
- if (!samples_buf)
- return -ENOMEM;
-
- sigma_delta->samples_buf = samples_buf;
-
spi_bus_lock(sigma_delta->spi->master);
sigma_delta->bus_locked = true;
sigma_delta->keep_cs_asserted = true;
@@ -620,6 +610,7 @@ static int devm_ad_sd_probe_trigger(struct device *dev, struct iio_dev *indio_de
int devm_ad_sd_setup_buffer_and_trigger(struct device *dev, struct iio_dev *indio_dev)
{
struct ad_sigma_delta *sigma_delta = iio_device_get_drvdata(indio_dev);
+ unsigned int samples_buf_size;
int ret;
sigma_delta->slots = devm_kcalloc(dev, sigma_delta->num_slots,
@@ -627,6 +618,19 @@ int devm_ad_sd_setup_buffer_and_trigger(struct device *dev, struct iio_dev *indi
if (!sigma_delta->slots)
return -ENOMEM;
+ /*
+ * Worst-case size: all sequencer slots can be active, capped
+ * at num_slots by ad_sd_validate_scan_mask().
+ */
+ samples_buf_size =
+ ALIGN(sigma_delta->num_slots *
+ BITS_TO_BYTES(indio_dev->channels[0].scan_type.storagebits),
+ sizeof(s64));
+ samples_buf_size += sizeof(s64);
+ sigma_delta->samples_buf = devm_kzalloc(dev, samples_buf_size, GFP_KERNEL);
+ if (!sigma_delta->samples_buf)
+ return -ENOMEM;
+
ret = devm_iio_triggered_buffer_setup(dev, indio_dev,
&iio_pollfunc_store_time,
&ad_sd_trigger_handler,
diff --git a/drivers/iio/adc/aspeed_adc.c b/drivers/iio/adc/aspeed_adc.c
index 998e8bcc06e1..4cb2210cb38d 100644
--- a/drivers/iio/adc/aspeed_adc.c
+++ b/drivers/iio/adc/aspeed_adc.c
@@ -549,12 +549,13 @@ static int aspeed_adc_probe(struct platform_device *pdev)
return PTR_ERR(data->clk_scaler);
data->rst = devm_reset_control_get_shared(&pdev->dev, NULL);
- if (IS_ERR(data->rst)) {
- dev_err(&pdev->dev,
- "invalid or missing reset controller device tree entry");
- return PTR_ERR(data->rst);
- }
- reset_control_deassert(data->rst);
+ if (IS_ERR(data->rst))
+ return dev_err_probe(&pdev->dev, PTR_ERR(data->rst),
+ "invalid or missing reset controller device tree entry");
+
+ ret = reset_control_deassert(data->rst);
+ if (ret)
+ return ret;
ret = devm_add_action_or_reset(data->dev, aspeed_adc_reset_assert,
data->rst);
diff --git a/drivers/iio/adc/axp288_adc.c b/drivers/iio/adc/axp288_adc.c
index 580361bd9849..3ccdfe9dea3f 100644
--- a/drivers/iio/adc/axp288_adc.c
+++ b/drivers/iio/adc/axp288_adc.c
@@ -204,6 +204,14 @@ static const struct dmi_system_id axp288_adc_ts_bias_override[] = {
},
.driver_data = (void *)(uintptr_t)AXP288_ADC_TS_BIAS_80UA,
},
+ {
+ /* Haier HV103H */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Haier"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "HV103H"),
+ },
+ .driver_data = (void *)(uintptr_t)AXP288_ADC_TS_BIAS_80UA,
+ },
{}
};
diff --git a/drivers/iio/adc/max1363.c b/drivers/iio/adc/max1363.c
index 875f5ff20208..bbb2336006a9 100644
--- a/drivers/iio/adc/max1363.c
+++ b/drivers/iio/adc/max1363.c
@@ -408,6 +408,14 @@ static int max1363_read_single_chan(struct iio_dev *indio_dev,
}
data = rxbuf[0];
}
+
+ /*
+ * Differential channels are bipolar and the device returns the sample
+ * in two's complement, so sign-extend it from the resolution bit.
+ */
+ if (chan->differential)
+ data = sign_extend32(data, st->chip_info->bits - 1);
+
*val = data;
error_ret:
diff --git a/drivers/iio/adc/stm32-adc.c b/drivers/iio/adc/stm32-adc.c
index 6fede34091cc..6a666d14a609 100644
--- a/drivers/iio/adc/stm32-adc.c
+++ b/drivers/iio/adc/stm32-adc.c
@@ -1372,11 +1372,16 @@ static int stm32_adc_read_raw(struct iio_dev *indio_dev,
ret = stm32_adc_single_conv(indio_dev, chan, val);
else
ret = -EINVAL;
+ iio_device_release_direct_mode(indio_dev);
+ if (ret < 0)
+ return ret;
- if (mask == IIO_CHAN_INFO_PROCESSED)
+ if (mask == IIO_CHAN_INFO_PROCESSED) {
+ if (*val == 0)
+ return -EINVAL;
*val = STM32_ADC_VREFINT_VOLTAGE * adc->vrefint.vrefint_cal / *val;
+ }
- iio_device_release_direct_mode(indio_dev);
return ret;
case IIO_CHAN_INFO_SCALE:
diff --git a/drivers/iio/adc/sun4i-gpadc-iio.c b/drivers/iio/adc/sun4i-gpadc-iio.c
index a6ade70dedf8..43239b43cfa3 100644
--- a/drivers/iio/adc/sun4i-gpadc-iio.c
+++ b/drivers/iio/adc/sun4i-gpadc-iio.c
@@ -645,10 +645,10 @@ static int sun4i_gpadc_probe(struct platform_device *pdev)
* thermal because no thermal DT node is found.
*/
if (IS_ERR(info->tzd) && PTR_ERR(info->tzd) != -ENODEV) {
- dev_err(&pdev->dev,
- "could not register thermal sensor: %ld\n",
- PTR_ERR(info->tzd));
- return PTR_ERR(info->tzd);
+ ret = dev_err_probe(&pdev->dev,
+ PTR_ERR(info->tzd),
+ "could not register thermal sensor\n");
+ goto err_map;
}
}
@@ -664,7 +664,6 @@ static int sun4i_gpadc_probe(struct platform_device *pdev)
if (!info->no_irq && IS_ENABLED(CONFIG_THERMAL_OF))
iio_map_array_unregister(indio_dev);
- pm_runtime_put(&pdev->dev);
pm_runtime_disable(&pdev->dev);
return ret;
@@ -675,7 +674,6 @@ static int sun4i_gpadc_remove(struct platform_device *pdev)
struct iio_dev *indio_dev = platform_get_drvdata(pdev);
struct sun4i_gpadc_iio *info = iio_priv(indio_dev);
- pm_runtime_put(&pdev->dev);
pm_runtime_disable(&pdev->dev);
if (!IS_ENABLED(CONFIG_THERMAL_OF))
diff --git a/drivers/iio/adc/xilinx-xadc-core.c b/drivers/iio/adc/xilinx-xadc-core.c
index 683a91f81aa5..58f4d6d0e5fc 100644
--- a/drivers/iio/adc/xilinx-xadc-core.c
+++ b/drivers/iio/adc/xilinx-xadc-core.c
@@ -1406,13 +1406,14 @@ static int xadc_probe(struct platform_device *pdev)
}
if (irq > 0) {
- ret = devm_request_irq(dev, irq, xadc->ops->interrupt_handler,
- 0, dev_name(dev), indio_dev);
+ /* Clear out any work queued by the interrupt */
+ ret = devm_add_action_or_reset(dev, xadc_cancel_delayed_work,
+ &xadc->zynq_unmask_work);
if (ret)
return ret;
- ret = devm_add_action_or_reset(dev, xadc_cancel_delayed_work,
- &xadc->zynq_unmask_work);
+ ret = devm_request_irq(dev, irq, xadc->ops->interrupt_handler,
+ 0, dev_name(dev), indio_dev);
if (ret)
return ret;
}
diff --git a/drivers/iio/cdc/ad7150.c b/drivers/iio/cdc/ad7150.c
index ebe112b4618b..023b4fd09b25 100644
--- a/drivers/iio/cdc/ad7150.c
+++ b/drivers/iio/cdc/ad7150.c
@@ -653,11 +653,13 @@ static const struct i2c_device_id ad7150_id[] = {
MODULE_DEVICE_TABLE(i2c, ad7150_id);
static const struct of_device_id ad7150_of_match[] = {
- { "adi,ad7150" },
- { "adi,ad7151" },
- { "adi,ad7156" },
+ { .compatible = "adi,ad7150" },
+ { .compatible = "adi,ad7151" },
+ { .compatible = "adi,ad7156" },
{}
};
+MODULE_DEVICE_TABLE(of, ad7150_of_match);
+
static struct i2c_driver ad7150_driver = {
.driver = {
.name = "ad7150",
diff --git a/drivers/iio/common/st_sensors/st_sensors_core.c b/drivers/iio/common/st_sensors/st_sensors_core.c
index dcc032f68740..2a645c359fb5 100644
--- a/drivers/iio/common/st_sensors/st_sensors_core.c
+++ b/drivers/iio/common/st_sensors/st_sensors_core.c
@@ -154,7 +154,7 @@ static int st_sensors_set_fullscale(struct iio_dev *indio_dev, unsigned int fs)
return err;
st_accel_set_fullscale_error:
- dev_err(&indio_dev->dev, "failed to set new fullscale.\n");
+ dev_err(indio_dev->dev.parent, "failed to set new fullscale.\n");
return err;
}
@@ -219,60 +219,35 @@ int st_sensors_set_axis_enable(struct iio_dev *indio_dev, u8 axis_enable)
}
EXPORT_SYMBOL_NS(st_sensors_set_axis_enable, IIO_ST_SENSORS);
-static void st_reg_disable(void *reg)
-{
- regulator_disable(reg);
-}
int st_sensors_power_enable(struct iio_dev *indio_dev)
{
- struct st_sensor_data *pdata = iio_priv(indio_dev);
+ static const char * const regulator_names[] = { "vdd", "vddio" };
struct device *parent = indio_dev->dev.parent;
int err;
/* Regulators not mandatory, but if requested we should enable them. */
- pdata->vdd = devm_regulator_get(parent, "vdd");
- if (IS_ERR(pdata->vdd))
- return dev_err_probe(&indio_dev->dev, PTR_ERR(pdata->vdd),
- "unable to get Vdd supply\n");
-
- err = regulator_enable(pdata->vdd);
- if (err != 0) {
- dev_warn(&indio_dev->dev,
- "Failed to enable specified Vdd supply\n");
- return err;
- }
-
- err = devm_add_action_or_reset(parent, st_reg_disable, pdata->vdd);
+ err = devm_regulator_bulk_get_enable(parent,
+ ARRAY_SIZE(regulator_names),
+ regulator_names);
if (err)
- return err;
-
- pdata->vdd_io = devm_regulator_get(parent, "vddio");
- if (IS_ERR(pdata->vdd_io))
- return dev_err_probe(&indio_dev->dev, PTR_ERR(pdata->vdd_io),
- "unable to get Vdd_IO supply\n");
-
- err = regulator_enable(pdata->vdd_io);
- if (err != 0) {
- dev_warn(&indio_dev->dev,
- "Failed to enable specified Vdd_IO supply\n");
- return err;
- }
+ return dev_err_probe(parent, err, "unable to enable supplies\n");
- return devm_add_action_or_reset(parent, st_reg_disable, pdata->vdd_io);
+ return 0;
}
EXPORT_SYMBOL_NS(st_sensors_power_enable, IIO_ST_SENSORS);
static int st_sensors_set_drdy_int_pin(struct iio_dev *indio_dev,
struct st_sensors_platform_data *pdata)
{
+ struct device *parent = indio_dev->dev.parent;
struct st_sensor_data *sdata = iio_priv(indio_dev);
/* Sensor does not support interrupts */
if (!sdata->sensor_settings->drdy_irq.int1.addr &&
!sdata->sensor_settings->drdy_irq.int2.addr) {
if (pdata->drdy_int_pin)
- dev_info(&indio_dev->dev,
+ dev_info(parent,
"DRDY on pin INT%d specified, but sensor does not support interrupts\n",
pdata->drdy_int_pin);
return 0;
@@ -281,29 +256,27 @@ static int st_sensors_set_drdy_int_pin(struct iio_dev *indio_dev,
switch (pdata->drdy_int_pin) {
case 1:
if (!sdata->sensor_settings->drdy_irq.int1.mask) {
- dev_err(&indio_dev->dev,
- "DRDY on INT1 not available.\n");
+ dev_err(parent, "DRDY on INT1 not available.\n");
return -EINVAL;
}
sdata->drdy_int_pin = 1;
break;
case 2:
if (!sdata->sensor_settings->drdy_irq.int2.mask) {
- dev_err(&indio_dev->dev,
- "DRDY on INT2 not available.\n");
+ dev_err(parent, "DRDY on INT2 not available.\n");
return -EINVAL;
}
sdata->drdy_int_pin = 2;
break;
default:
- dev_err(&indio_dev->dev, "DRDY on pdata not valid.\n");
+ dev_err(parent, "DRDY on pdata not valid.\n");
return -EINVAL;
}
if (pdata->open_drain) {
if (!sdata->sensor_settings->drdy_irq.int1.addr_od &&
!sdata->sensor_settings->drdy_irq.int2.addr_od)
- dev_err(&indio_dev->dev,
+ dev_err(parent,
"open drain requested but unsupported.\n");
else
sdata->int_pin_open_drain = true;
@@ -361,6 +334,7 @@ EXPORT_SYMBOL_NS(st_sensors_dev_name_probe, IIO_ST_SENSORS);
int st_sensors_init_sensor(struct iio_dev *indio_dev,
struct st_sensors_platform_data *pdata)
{
+ struct device *parent = indio_dev->dev.parent;
struct st_sensor_data *sdata = iio_priv(indio_dev);
struct st_sensors_platform_data *of_pdata;
int err = 0;
@@ -368,7 +342,7 @@ int st_sensors_init_sensor(struct iio_dev *indio_dev,
mutex_init(&sdata->odr_lock);
/* If OF/DT pdata exists, it will take precedence of anything else */
- of_pdata = st_sensors_dev_probe(indio_dev->dev.parent, pdata);
+ of_pdata = st_sensors_dev_probe(parent, pdata);
if (IS_ERR(of_pdata))
return PTR_ERR(of_pdata);
if (of_pdata)
@@ -395,7 +369,7 @@ int st_sensors_init_sensor(struct iio_dev *indio_dev,
if (err < 0)
return err;
} else
- dev_info(&indio_dev->dev, "Full-scale not possible\n");
+ dev_info(parent, "Full-scale not possible\n");
err = st_sensors_set_odr(indio_dev, sdata->odr);
if (err < 0)
@@ -430,7 +404,7 @@ int st_sensors_init_sensor(struct iio_dev *indio_dev,
mask = sdata->sensor_settings->drdy_irq.int2.mask_od;
}
- dev_info(&indio_dev->dev,
+ dev_info(parent,
"set interrupt line to open drain mode on pin %d\n",
sdata->drdy_int_pin);
err = st_sensors_write_data_with_mask(indio_dev, addr,
@@ -630,22 +604,20 @@ EXPORT_SYMBOL_NS(st_sensors_get_settings_index, IIO_ST_SENSORS);
int st_sensors_verify_id(struct iio_dev *indio_dev)
{
struct st_sensor_data *sdata = iio_priv(indio_dev);
+ struct device *parent = indio_dev->dev.parent;
int wai, err;
if (sdata->sensor_settings->wai_addr) {
err = regmap_read(sdata->regmap,
sdata->sensor_settings->wai_addr, &wai);
if (err < 0) {
- dev_err(&indio_dev->dev,
- "failed to read Who-Am-I register.\n");
- return err;
+ return dev_err_probe(parent, err,
+ "failed to read Who-Am-I register.\n");
}
if (sdata->sensor_settings->wai != wai) {
- dev_err(&indio_dev->dev,
- "%s: WhoAmI mismatch (0x%x).\n",
- indio_dev->name, wai);
- return -EINVAL;
+ dev_warn(parent, "%s: WhoAmI mismatch (0x%x).\n",
+ indio_dev->name, wai);
}
}
diff --git a/drivers/iio/common/st_sensors/st_sensors_trigger.c b/drivers/iio/common/st_sensors/st_sensors_trigger.c
index 899b640c0a70..6672946702ed 100644
--- a/drivers/iio/common/st_sensors/st_sensors_trigger.c
+++ b/drivers/iio/common/st_sensors/st_sensors_trigger.c
@@ -127,7 +127,7 @@ int st_sensors_allocate_trigger(struct iio_dev *indio_dev,
sdata->trig = devm_iio_trigger_alloc(parent, "%s-trigger",
indio_dev->name);
if (sdata->trig == NULL) {
- dev_err(&indio_dev->dev, "failed to allocate iio trigger.\n");
+ dev_err(parent, "failed to allocate iio trigger.\n");
return -ENOMEM;
}
@@ -143,7 +143,7 @@ int st_sensors_allocate_trigger(struct iio_dev *indio_dev,
case IRQF_TRIGGER_FALLING:
case IRQF_TRIGGER_LOW:
if (!sdata->sensor_settings->drdy_irq.addr_ihl) {
- dev_err(&indio_dev->dev,
+ dev_err(parent,
"falling/low specified for IRQ but hardware supports only rising/high: will request rising/high\n");
if (irq_trig == IRQF_TRIGGER_FALLING)
irq_trig = IRQF_TRIGGER_RISING;
@@ -156,21 +156,19 @@ int st_sensors_allocate_trigger(struct iio_dev *indio_dev,
sdata->sensor_settings->drdy_irq.mask_ihl, 1);
if (err < 0)
return err;
- dev_info(&indio_dev->dev,
+ dev_info(parent,
"interrupts on the falling edge or active low level\n");
}
break;
case IRQF_TRIGGER_RISING:
- dev_info(&indio_dev->dev,
- "interrupts on the rising edge\n");
+ dev_info(parent, "interrupts on the rising edge\n");
break;
case IRQF_TRIGGER_HIGH:
- dev_info(&indio_dev->dev,
- "interrupts active high level\n");
+ dev_info(parent, "interrupts active high level\n");
break;
default:
/* This is the most preferred mode, if possible */
- dev_err(&indio_dev->dev,
+ dev_err(parent,
"unsupported IRQ trigger specified (%lx), enforce rising edge\n", irq_trig);
irq_trig = IRQF_TRIGGER_RISING;
}
@@ -179,7 +177,7 @@ int st_sensors_allocate_trigger(struct iio_dev *indio_dev,
if (irq_trig == IRQF_TRIGGER_FALLING ||
irq_trig == IRQF_TRIGGER_RISING) {
if (!sdata->sensor_settings->drdy_irq.stat_drdy.addr) {
- dev_err(&indio_dev->dev,
+ dev_err(parent,
"edge IRQ not supported w/o stat register.\n");
return -EOPNOTSUPP;
}
@@ -214,13 +212,13 @@ int st_sensors_allocate_trigger(struct iio_dev *indio_dev,
sdata->trig->name,
sdata->trig);
if (err) {
- dev_err(&indio_dev->dev, "failed to request trigger IRQ.\n");
+ dev_err(parent, "failed to request trigger IRQ.\n");
return err;
}
err = devm_iio_trigger_register(parent, sdata->trig);
if (err < 0) {
- dev_err(&indio_dev->dev, "failed to register iio trigger.\n");
+ dev_err(parent, "failed to register iio trigger.\n");
return err;
}
indio_dev->trig = iio_trigger_get(sdata->trig);
diff --git a/drivers/iio/frequency/admv1013.c b/drivers/iio/frequency/admv1013.c
index d15b85377159..9c147b9e8982 100644
--- a/drivers/iio/frequency/admv1013.c
+++ b/drivers/iio/frequency/admv1013.c
@@ -197,7 +197,7 @@ static int admv1013_read_raw(struct iio_dev *indio_dev,
switch (info) {
case IIO_CHAN_INFO_CALIBBIAS:
- switch (chan->channel) {
+ switch (chan->channel2) {
case IIO_MOD_I:
addr = ADMV1013_REG_OFFSET_ADJUST_I;
break;
diff --git a/drivers/iio/gyro/adis16136.c b/drivers/iio/gyro/adis16136.c
index 71295709f2b9..eb7ab53824ea 100644
--- a/drivers/iio/gyro/adis16136.c
+++ b/drivers/iio/gyro/adis16136.c
@@ -145,7 +145,7 @@ static int adis16136_debugfs_init(struct iio_dev *indio_dev)
struct adis16136 *adis16136 = iio_priv(indio_dev);
struct dentry *d = iio_get_debugfs_dentry(indio_dev);
- debugfs_create_file_unsafe("serial_number", 0400,
+ debugfs_create_file("serial_number", 0400,
d, adis16136, &adis16136_serial_fops);
debugfs_create_file_unsafe("product_id", 0400,
d, adis16136, &adis16136_product_id_fops);
diff --git a/drivers/iio/imu/adis16400.c b/drivers/iio/imu/adis16400.c
index 17bb0c40a149..f31d568efbc8 100644
--- a/drivers/iio/imu/adis16400.c
+++ b/drivers/iio/imu/adis16400.c
@@ -279,7 +279,7 @@ static int adis16400_debugfs_init(struct iio_dev *indio_dev)
struct dentry *d = iio_get_debugfs_dentry(indio_dev);
if (st->variant->flags & ADIS16400_HAS_SERIAL_NUMBER)
- debugfs_create_file_unsafe("serial_number", 0400,
+ debugfs_create_file("serial_number", 0400,
d, st, &adis16400_serial_number_fops);
if (st->variant->flags & ADIS16400_HAS_PROD_ID)
debugfs_create_file_unsafe("product_id", 0400,
diff --git a/drivers/iio/imu/adis16480.c b/drivers/iio/imu/adis16480.c
index fe520194a837..ef78f7074576 100644
--- a/drivers/iio/imu/adis16480.c
+++ b/drivers/iio/imu/adis16480.c
@@ -302,9 +302,9 @@ static int adis16480_debugfs_init(struct iio_dev *indio_dev)
struct adis16480 *adis16480 = iio_priv(indio_dev);
struct dentry *d = iio_get_debugfs_dentry(indio_dev);
- debugfs_create_file_unsafe("firmware_revision", 0400,
+ debugfs_create_file("firmware_revision", 0400,
d, adis16480, &adis16480_firmware_revision_fops);
- debugfs_create_file_unsafe("firmware_date", 0400,
+ debugfs_create_file("firmware_date", 0400,
d, adis16480, &adis16480_firmware_date_fops);
debugfs_create_file_unsafe("serial_number", 0400,
d, adis16480, &adis16480_serial_number_fops);
diff --git a/drivers/iio/imu/st_lsm6dsx/st_lsm6dsx_core.c b/drivers/iio/imu/st_lsm6dsx/st_lsm6dsx_core.c
index d692b0f1f762..3b3a04e5d949 100644
--- a/drivers/iio/imu/st_lsm6dsx/st_lsm6dsx_core.c
+++ b/drivers/iio/imu/st_lsm6dsx/st_lsm6dsx_core.c
@@ -34,6 +34,13 @@
* - Gyroscope supported full-scale [dps]: +-125/+-245/+-500/+-1000/+-2000
* - FIFO size: 3KB
*
+ * - LSM6DSV/LSM6DSV16X:
+ * - Accelerometer/Gyroscope supported ODR [Hz]: 7.5, 15, 30, 60, 120, 240,
+ * 480, 960
+ * - Accelerometer supported full-scale [g]: +-2/+-4/+-8/+-16
+ * - Gyroscope supported full-scale [dps]: +-125/+-250/+-500/+-1000/+-2000
+ * - FIFO size: 3KB
+ *
* - LSM9DS1/LSM6DS0:
* - Accelerometer supported ODR [Hz]: 10, 50, 119, 238, 476, 952
* - Accelerometer supported full-scale [g]: +-2/+-4/+-8/+-16
@@ -76,6 +83,13 @@ static const struct iio_chan_spec st_lsm6dsx_acc_channels[] = {
IIO_CHAN_SOFT_TIMESTAMP(3),
};
+static const struct iio_chan_spec st_lsm6ds0_acc_channels[] = {
+ ST_LSM6DSX_CHANNEL(IIO_ACCEL, 0x28, IIO_MOD_X, 0),
+ ST_LSM6DSX_CHANNEL(IIO_ACCEL, 0x2a, IIO_MOD_Y, 1),
+ ST_LSM6DSX_CHANNEL(IIO_ACCEL, 0x2c, IIO_MOD_Z, 2),
+ IIO_CHAN_SOFT_TIMESTAMP(3),
+};
+
static const struct iio_chan_spec st_lsm6dsx_gyro_channels[] = {
ST_LSM6DSX_CHANNEL(IIO_ANGL_VEL, 0x22, IIO_MOD_X, 0),
ST_LSM6DSX_CHANNEL(IIO_ANGL_VEL, 0x24, IIO_MOD_Y, 1),
@@ -117,8 +131,8 @@ static const struct st_lsm6dsx_settings st_lsm6dsx_sensor_settings[] = {
},
.channels = {
[ST_LSM6DSX_ID_ACC] = {
- .chan = st_lsm6dsx_acc_channels,
- .len = ARRAY_SIZE(st_lsm6dsx_acc_channels),
+ .chan = st_lsm6ds0_acc_channels,
+ .len = ARRAY_SIZE(st_lsm6ds0_acc_channels),
},
[ST_LSM6DSX_ID_GYRO] = {
.chan = st_lsm6ds0_gyro_channels,
diff --git a/drivers/iio/imu/st_lsm9ds0/st_lsm9ds0_core.c b/drivers/iio/imu/st_lsm9ds0/st_lsm9ds0_core.c
index ae7bc815382f..e887b45cdbcd 100644
--- a/drivers/iio/imu/st_lsm9ds0/st_lsm9ds0_core.c
+++ b/drivers/iio/imu/st_lsm9ds0/st_lsm9ds0_core.c
@@ -18,58 +18,6 @@
#include "st_lsm9ds0.h"
-static int st_lsm9ds0_power_enable(struct device *dev, struct st_lsm9ds0 *lsm9ds0)
-{
- int ret;
-
- /* Regulators not mandatory, but if requested we should enable them. */
- lsm9ds0->vdd = devm_regulator_get(dev, "vdd");
- if (IS_ERR(lsm9ds0->vdd))
- return dev_err_probe(dev, PTR_ERR(lsm9ds0->vdd),
- "unable to get Vdd supply\n");
-
- ret = regulator_enable(lsm9ds0->vdd);
- if (ret) {
- dev_warn(dev, "Failed to enable specified Vdd supply\n");
- return ret;
- }
-
- lsm9ds0->vdd_io = devm_regulator_get(dev, "vddio");
- if (IS_ERR(lsm9ds0->vdd_io)) {
- regulator_disable(lsm9ds0->vdd);
- return dev_err_probe(dev, PTR_ERR(lsm9ds0->vdd_io),
- "unable to get Vdd_IO supply\n");
- }
- ret = regulator_enable(lsm9ds0->vdd_io);
- if (ret) {
- dev_warn(dev, "Failed to enable specified Vdd_IO supply\n");
- regulator_disable(lsm9ds0->vdd);
- return ret;
- }
-
- return 0;
-}
-
-static void st_lsm9ds0_power_disable(void *data)
-{
- struct st_lsm9ds0 *lsm9ds0 = data;
-
- regulator_disable(lsm9ds0->vdd_io);
- regulator_disable(lsm9ds0->vdd);
-}
-
-static int devm_st_lsm9ds0_power_enable(struct st_lsm9ds0 *lsm9ds0)
-{
- struct device *dev = lsm9ds0->dev;
- int ret;
-
- ret = st_lsm9ds0_power_enable(dev, lsm9ds0);
- if (ret)
- return ret;
-
- return devm_add_action_or_reset(dev, st_lsm9ds0_power_disable, lsm9ds0);
-}
-
static int st_lsm9ds0_probe_accel(struct st_lsm9ds0 *lsm9ds0, struct regmap *regmap)
{
const struct st_sensor_settings *settings;
@@ -92,8 +40,6 @@ static int st_lsm9ds0_probe_accel(struct st_lsm9ds0 *lsm9ds0, struct regmap *reg
data->sensor_settings = (struct st_sensor_settings *)settings;
data->irq = lsm9ds0->irq;
data->regmap = regmap;
- data->vdd = lsm9ds0->vdd;
- data->vdd_io = lsm9ds0->vdd_io;
return st_accel_common_probe(lsm9ds0->accel);
}
@@ -120,19 +66,22 @@ static int st_lsm9ds0_probe_magn(struct st_lsm9ds0 *lsm9ds0, struct regmap *regm
data->sensor_settings = (struct st_sensor_settings *)settings;
data->irq = lsm9ds0->irq;
data->regmap = regmap;
- data->vdd = lsm9ds0->vdd;
- data->vdd_io = lsm9ds0->vdd_io;
return st_magn_common_probe(lsm9ds0->magn);
}
int st_lsm9ds0_probe(struct st_lsm9ds0 *lsm9ds0, struct regmap *regmap)
{
+ struct device *dev = lsm9ds0->dev;
+ static const char * const regulator_names[] = { "vdd", "vddio" };
int ret;
- ret = devm_st_lsm9ds0_power_enable(lsm9ds0);
+ /* Regulators not mandatory, but if requested we should enable them. */
+ ret = devm_regulator_bulk_get_enable(dev, ARRAY_SIZE(regulator_names),
+ regulator_names);
if (ret)
- return ret;
+ return dev_err_probe(dev, ret,
+ "unable to enable Vdd supply\n");
/* Setup accelerometer device */
ret = st_lsm9ds0_probe_accel(lsm9ds0, regmap);
diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c
index d0bcb9c57f96..37ea4a2d85f8 100644
--- a/drivers/iio/industrialio-buffer.c
+++ b/drivers/iio/industrialio-buffer.c
@@ -1291,6 +1291,8 @@ EXPORT_SYMBOL_GPL(iio_update_buffers);
void iio_disable_all_buffers(struct iio_dev *indio_dev)
{
+ guard(mutex)(&indio_dev->mlock);
+
iio_disable_buffers(indio_dev);
iio_buffer_deactivate_all(indio_dev);
}
@@ -1905,7 +1907,8 @@ int iio_push_to_buffers_with_ts_unaligned(struct iio_dev *indio_dev,
bb = devm_krealloc(&indio_dev->dev,
iio_dev_opaque->bounce_buffer,
- indio_dev->scan_bytes, GFP_KERNEL);
+ indio_dev->scan_bytes,
+ GFP_KERNEL | __GFP_ZERO);
if (!bb)
return -ENOMEM;
iio_dev_opaque->bounce_buffer = bb;
diff --git a/drivers/iio/industrialio-trigger.c b/drivers/iio/industrialio-trigger.c
index 6885a186fe27..97c89e4b04b6 100644
--- a/drivers/iio/industrialio-trigger.c
+++ b/drivers/iio/industrialio-trigger.c
@@ -506,6 +506,8 @@ static void iio_trig_release(struct device *device)
struct iio_trigger *trig = to_iio_trigger(device);
int i;
+ cancel_work_sync(&trig->reenable_work);
+
if (trig->subirq_base) {
for (i = 0; i < CONFIG_IIO_CONSUMERS_PER_TRIGGER; i++) {
irq_modify_status(trig->subirq_base + i,
diff --git a/drivers/iio/light/gp2ap020a00f.c b/drivers/iio/light/gp2ap020a00f.c
index 826439299e8b..b931b08fc978 100644
--- a/drivers/iio/light/gp2ap020a00f.c
+++ b/drivers/iio/light/gp2ap020a00f.c
@@ -1565,6 +1565,7 @@ static int gp2ap020a00f_probe(struct i2c_client *client,
iio_trigger_unregister(data->trig);
error_free_irq:
free_irq(client->irq, indio_dev);
+ irq_work_sync(&data->work);
error_uninit_buffer:
iio_triggered_buffer_cleanup(indio_dev);
error_regulator_disable:
@@ -1587,6 +1588,7 @@ static void gp2ap020a00f_remove(struct i2c_client *client)
iio_device_unregister(indio_dev);
iio_trigger_unregister(data->trig);
free_irq(client->irq, indio_dev);
+ irq_work_sync(&data->work);
iio_triggered_buffer_cleanup(indio_dev);
regulator_disable(data->vled_reg);
}
diff --git a/drivers/iio/light/ltrf216a.c b/drivers/iio/light/ltrf216a.c
index 4b8ef36b6912..85d8f5329bce 100644
--- a/drivers/iio/light/ltrf216a.c
+++ b/drivers/iio/light/ltrf216a.c
@@ -234,22 +234,20 @@ static int ltrf216a_read_data(struct ltrf216a_data *data, u8 addr)
static int ltrf216a_get_lux(struct ltrf216a_data *data)
{
int ret, greendata;
- u64 lux, div;
+ u64 lux;
ret = ltrf216a_set_power_state(data, true);
if (ret)
return ret;
greendata = ltrf216a_read_data(data, LTRF216A_ALS_DATA_0);
+ ltrf216a_set_power_state(data, false);
if (greendata < 0)
return greendata;
- ltrf216a_set_power_state(data, false);
-
- lux = greendata * 45 * LTRF216A_WIN_FAC * 100;
- div = data->als_gain_fac * data->int_time_fac * 100;
+ lux = greendata * 45 * LTRF216A_WIN_FAC;
- return div_u64(lux, div);
+ return lux;
}
static int ltrf216a_read_raw(struct iio_dev *indio_dev,
@@ -279,7 +277,8 @@ static int ltrf216a_read_raw(struct iio_dev *indio_dev,
if (ret < 0)
return ret;
*val = ret;
- return IIO_VAL_INT;
+ *val2 = data->als_gain_fac * data->int_time_fac;
+ return IIO_VAL_FRACTIONAL;
case IIO_CHAN_INFO_INT_TIME:
mutex_lock(&data->lock);
ret = ltrf216a_get_int_time(data, val, val2);
diff --git a/drivers/iio/pressure/bmp280-core.c b/drivers/iio/pressure/bmp280-core.c
index d8321da5c6a7..cf81c529a276 100644
--- a/drivers/iio/pressure/bmp280-core.c
+++ b/drivers/iio/pressure/bmp280-core.c
@@ -725,7 +725,7 @@ static int bmp280_write_sampling_frequency(struct bmp280_data *data,
int val, int val2)
{
const int (*avail)[2] = data->chip_info->sampling_freq_avail;
- const int n = data->chip_info->num_sampling_freq_avail;
+ const int n = data->chip_info->num_sampling_freq_avail / 2;
int ret, prev;
int i;
diff --git a/drivers/iio/pressure/dps310.c b/drivers/iio/pressure/dps310.c
index 519fcd425b6a..0057e14a45dd 100644
--- a/drivers/iio/pressure/dps310.c
+++ b/drivers/iio/pressure/dps310.c
@@ -843,7 +843,7 @@ static int dps310_probe(struct i2c_client *client,
data->client = client;
mutex_init(&data->lock);
- iio->name = id->name;
+ iio->name = DPS310_DEV_NAME;
iio->channels = dps310_channels;
iio->num_channels = ARRAY_SIZE(dps310_channels);
iio->info = &dps310_info;
diff --git a/drivers/iio/proximity/isl29501.c b/drivers/iio/proximity/isl29501.c
index 3ccc95cf645c..7fa6bdcc6cb4 100644
--- a/drivers/iio/proximity/isl29501.c
+++ b/drivers/iio/proximity/isl29501.c
@@ -227,7 +227,7 @@ static int isl29501_register_read(struct isl29501_private *isl29501,
return ret;
}
-static u32 isl29501_register_write(struct isl29501_private *isl29501,
+static int isl29501_register_write(struct isl29501_private *isl29501,
enum isl29501_register_name name,
u32 value)
{
diff --git a/drivers/iio/proximity/pulsedlight-lidar-lite-v2.c b/drivers/iio/proximity/pulsedlight-lidar-lite-v2.c
index 791a33d5286c..b5015448ce02 100644
--- a/drivers/iio/proximity/pulsedlight-lidar-lite-v2.c
+++ b/drivers/iio/proximity/pulsedlight-lidar-lite-v2.c
@@ -299,12 +299,14 @@ static int lidar_probe(struct i2c_client *client,
ret = pm_runtime_set_active(&client->dev);
if (ret)
- goto error_unreg_buffer;
+ goto error_unreg_dev;
pm_runtime_enable(&client->dev);
pm_runtime_idle(&client->dev);
return 0;
+error_unreg_dev:
+ iio_device_unregister(indio_dev);
error_unreg_buffer:
iio_triggered_buffer_cleanup(indio_dev);
diff --git a/drivers/iio/proximity/sx9324.c b/drivers/iio/proximity/sx9324.c
index 977cf17cec52..d9c540f2b862 100644
--- a/drivers/iio/proximity/sx9324.c
+++ b/drivers/iio/proximity/sx9324.c
@@ -223,7 +223,7 @@ static const struct iio_chan_spec_ext_info sx9324_channel_ext_info[] = {
.scan_index = idx, \
.scan_type = { \
.sign = 's', \
- .realbits = 12, \
+ .realbits = 16, \
.storagebits = 16, \
.endianness = IIO_BE, \
}, \
diff --git a/drivers/iio/proximity/vcnl3020.c b/drivers/iio/proximity/vcnl3020.c
index cbc8400c773c..ace22182e6a4 100644
--- a/drivers/iio/proximity/vcnl3020.c
+++ b/drivers/iio/proximity/vcnl3020.c
@@ -584,16 +584,26 @@ static irqreturn_t vcnl3020_handle_irq_thread(int irq, void *p)
return IRQ_HANDLED;
}
- if (!(isr & VCNL_ICR_THRES_EN))
+ if (!(isr & (VCNL_INT_TH_HI | VCNL_INT_TH_LOW)))
return IRQ_NONE;
- iio_push_event(indio_dev,
- IIO_UNMOD_EVENT_CODE(IIO_PROXIMITY, 1,
- IIO_EV_TYPE_THRESH,
- IIO_EV_DIR_RISING),
- iio_get_time_ns(indio_dev));
+ if (isr & VCNL_INT_TH_HI) {
+ iio_push_event(indio_dev,
+ IIO_UNMOD_EVENT_CODE(IIO_PROXIMITY, 0,
+ IIO_EV_TYPE_THRESH,
+ IIO_EV_DIR_RISING),
+ iio_get_time_ns(indio_dev));
+ }
+
+ if (isr & VCNL_INT_TH_LOW) {
+ iio_push_event(indio_dev,
+ IIO_UNMOD_EVENT_CODE(IIO_PROXIMITY, 0,
+ IIO_EV_TYPE_THRESH,
+ IIO_EV_DIR_FALLING),
+ iio_get_time_ns(indio_dev));
+ }
- rc = regmap_write(data->regmap, VCNL_ISR, isr & VCNL_ICR_THRES_EN);
+ rc = regmap_write(data->regmap, VCNL_ISR, isr & (VCNL_INT_TH_HI | VCNL_INT_TH_LOW));
if (rc)
dev_err(data->dev, "Error (%d) writing in reg (0x%x)\n",
rc, VCNL_ISR);
diff --git a/drivers/infiniband/core/counters.c b/drivers/infiniband/core/counters.c
index 8cbca897fc53..78a76b4fa6f6 100644
--- a/drivers/infiniband/core/counters.c
+++ b/drivers/infiniband/core/counters.c
@@ -190,12 +190,19 @@ static struct rdma_counter *alloc_and_bind(struct ib_device *dev, u32 port,
ret = __rdma_counter_bind_qp(counter, qp);
if (ret)
- goto err_mode;
+ goto err_bind;
rdma_restrack_parent_name(&counter->res, &qp->res);
rdma_restrack_add(&counter->res);
return counter;
+err_bind:
+ mutex_lock(&port_counter->lock);
+ port_counter->num_counters--;
+ if (!port_counter->num_counters &&
+ port_counter->mode.mode == RDMA_COUNTER_MODE_MANUAL)
+ __counter_set_mode(port_counter, RDMA_COUNTER_MODE_NONE, 0);
+ mutex_unlock(&port_counter->lock);
err_mode:
rdma_free_hw_stats_struct(counter->stats);
err_stats:
diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
index 63fae7263c89..9db0d824d462 100644
--- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c
+++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c
@@ -1659,6 +1659,8 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq,
srq->qplib_srq.threshold = srq_init_attr->attr.srq_limit;
srq->srq_limit = srq_init_attr->attr.srq_limit;
srq->qplib_srq.eventq_hw_ring_id = rdev->nq[0].ring_id;
+ srq->qplib_srq.sg_info.pgsize = PAGE_SIZE;
+ srq->qplib_srq.sg_info.pgshft = PAGE_SHIFT;
nq = &rdev->nq[0];
if (udata) {
diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c
index 9c24789bbb54..9ed43d23c183 100644
--- a/drivers/infiniband/hw/cxgb4/mem.c
+++ b/drivers/infiniband/hw/cxgb4/mem.c
@@ -199,7 +199,8 @@ static int _c4iw_write_mem_dma(struct c4iw_rdev *rdev, u32 addr, u32 len,
daddr = dma_map_single(&rdev->lldi.pdev->dev, data, len, DMA_TO_DEVICE);
if (dma_mapping_error(&rdev->lldi.pdev->dev, daddr))
- return -1;
+ return _c4iw_write_mem_inline(rdev, addr, len, data, skb,
+ wr_waitp);
save = daddr;
while (remain > inline_threshold) {
@@ -235,30 +236,12 @@ static int write_adapter_mem(struct c4iw_rdev *rdev, u32 addr, u32 len,
void *data, struct sk_buff *skb,
struct c4iw_wr_wait *wr_waitp)
{
- int ret;
-
- if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl) {
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
- wr_waitp);
- goto out;
- }
-
- if (len <= inline_threshold) {
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
+ if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl ||
+ len <= inline_threshold)
+ return _c4iw_write_mem_inline(rdev, addr, len, data, skb,
wr_waitp);
- goto out;
- }
-
- ret = _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp);
- if (ret) {
- pr_warn_ratelimited("%s: dma map failure (non fatal)\n",
- pci_name(rdev->lldi.pdev));
- ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb,
- wr_waitp);
- }
-out:
- return ret;
+ return _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp);
}
/*
@@ -280,12 +263,16 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
bool stag_idx_allocated = false;
static atomic_t key;
- if (c4iw_fatal_error(rdev))
+ if (c4iw_fatal_error(rdev)) {
+ kfree_skb(skb);
return -EIO;
+ }
tpt = kmalloc(sizeof(*tpt), GFP_KERNEL);
- if (!tpt)
+ if (!tpt) {
+ kfree_skb(skb);
return -ENOMEM;
+ }
stag_state = stag_state > 0;
stag_idx = (*stag) >> 8;
@@ -297,6 +284,7 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry,
rdev->stats.stag.fail++;
mutex_unlock(&rdev->stats.lock);
kfree(tpt);
+ kfree_skb(skb);
return -ENOMEM;
}
mutex_lock(&rdev->stats.lock);
@@ -471,8 +459,10 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc)
FW_RI_STAG_NSMR, mhp->attr.perms,
mhp->attr.mw_bind_enable, 0, 0, ~0ULL, 0, 0, 0,
NULL, mhp->wr_waitp);
- if (ret)
- goto err_free_skb;
+ if (ret) {
+ kfree_skb(mhp->dereg_skb);
+ goto err_free_wr_wait;
+ }
ret = finish_mem_reg(mhp, stag);
if (ret)
@@ -481,8 +471,6 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc)
err_dereg_mem:
dereg_mem(&rhp->rdev, mhp->attr.stag, mhp->attr.pbl_size,
mhp->attr.pbl_addr, mhp->dereg_skb, mhp->wr_waitp);
-err_free_skb:
- kfree_skb(mhp->dereg_skb);
err_free_wr_wait:
c4iw_put_wr_wait(mhp->wr_waitp);
err_free_mhp:
diff --git a/drivers/infiniband/hw/erdma/erdma_cm.c b/drivers/infiniband/hw/erdma/erdma_cm.c
index 1156ae62c468..dfa4664bcadb 100644
--- a/drivers/infiniband/hw/erdma/erdma_cm.c
+++ b/drivers/infiniband/hw/erdma/erdma_cm.c
@@ -1014,10 +1014,9 @@ int erdma_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
if (laddr->sa_family != AF_INET || raddr->sa_family != AF_INET)
return -EAFNOSUPPORT;
- qp = find_qp_by_qpn(dev, params->qpn);
+ qp = erdma_qp_get_by_qpn(dev, params->qpn);
if (!qp)
return -ENOENT;
- erdma_qp_get(qp);
ret = sock_create(AF_INET, SOCK_STREAM, IPPROTO_TCP, &s);
if (ret < 0)
@@ -1146,10 +1145,9 @@ int erdma_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
return -ECONNRESET;
}
- qp = find_qp_by_qpn(dev, params->qpn);
+ qp = erdma_qp_get_by_qpn(dev, params->qpn);
if (!qp)
return -ENOENT;
- erdma_qp_get(qp);
down_write(&qp->state_lock);
if (qp->attrs.state > ERDMA_QP_STATE_RTR) {
diff --git a/drivers/infiniband/hw/erdma/erdma_eq.c b/drivers/infiniband/hw/erdma/erdma_eq.c
index da146516db3e..9abcd2effeda 100644
--- a/drivers/infiniband/hw/erdma/erdma_eq.c
+++ b/drivers/infiniband/hw/erdma/erdma_eq.c
@@ -65,7 +65,7 @@ void erdma_aeq_event_handler(struct erdma_dev *dev)
erdma_cq_put(cq);
} else {
qpn = le32_to_cpu(aeqe->event_data0);
- qp = find_qp_by_qpn(dev, qpn);
+ qp = erdma_qp_get_by_qpn(dev, qpn);
if (!qp)
continue;
@@ -75,6 +75,7 @@ void erdma_aeq_event_handler(struct erdma_dev *dev)
if (qp->ibqp.event_handler)
qp->ibqp.event_handler(&event,
qp->ibqp.qp_context);
+ erdma_qp_put(qp);
}
}
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c
index 0894517d33be..2ddc56335b9a 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.c
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.c
@@ -971,8 +971,15 @@ int erdma_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
FIELD_PREP(ERDMA_CMD_MR_KEY_MASK, ibmr->lkey & 0xFF);
ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
+ /*
+ * A timeout disables the command queue, so retry cannot succeed. Treat
+ * terminal command failures as diagnostic; propagating them can make
+ * forced uverbs cleanup discard the last software resource pointers.
+ */
if (ret)
- return ret;
+ ibdev_warn_ratelimited(&dev->ibdev,
+ "failed to deregister MR 0x%x: %d\n",
+ ibmr->lkey, ret);
erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_STAG_IDX], ibmr->lkey >> 8);
@@ -998,7 +1005,9 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata)
err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
if (err)
- return err;
+ ibdev_warn_ratelimited(&dev->ibdev,
+ "failed to destroy CQ %u: %d\n",
+ cq->cqn, err);
xa_lock_irqsave(&dev->cq_xa, flags);
__xa_erase(&dev->cq_xa, cq->cqn);
@@ -1026,6 +1035,7 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
struct erdma_ucontext *ctx = rdma_udata_to_drv_context(
udata, struct erdma_ucontext, ibucontext);
struct erdma_qp_attrs qp_attrs;
+ unsigned long flags;
int err;
struct erdma_cmdq_destroy_qp_req req;
@@ -1040,7 +1050,13 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL);
if (err)
- return err;
+ ibdev_warn_ratelimited(&dev->ibdev,
+ "failed to destroy QP %u: %d\n",
+ QP_ID(qp), err);
+
+ xa_lock_irqsave(&dev->qp_xa, flags);
+ __xa_erase(&dev->qp_xa, QP_ID(qp));
+ xa_unlock_irqrestore(&dev->qp_xa, flags);
erdma_qp_put(qp);
wait_for_completion(&qp->safe_free);
@@ -1064,7 +1080,6 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata)
if (qp->cep)
erdma_cep_put(qp->cep);
- xa_erase(&dev->qp_xa, QP_ID(qp));
return 0;
}
diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.h b/drivers/infiniband/hw/erdma/erdma_verbs.h
index 03ce6e171be8..67becf90d53f 100644
--- a/drivers/infiniband/hw/erdma/erdma_verbs.h
+++ b/drivers/infiniband/hw/erdma/erdma_verbs.h
@@ -250,6 +250,21 @@ static inline struct erdma_qp *find_qp_by_qpn(struct erdma_dev *dev, int id)
return (struct erdma_qp *)xa_load(&dev->qp_xa, id);
}
+static inline struct erdma_qp *erdma_qp_get_by_qpn(struct erdma_dev *dev,
+ int id)
+{
+ struct erdma_qp *qp;
+ unsigned long flags;
+
+ xa_lock_irqsave(&dev->qp_xa, flags);
+ qp = xa_load(&dev->qp_xa, id);
+ if (qp && !kref_get_unless_zero(&qp->ref))
+ qp = NULL;
+ xa_unlock_irqrestore(&dev->qp_xa, flags);
+
+ return qp;
+}
+
static inline struct erdma_cq *erdma_cq_get_by_cqn(struct erdma_dev *dev,
int id)
{
diff --git a/drivers/infiniband/hw/hns/hns_roce_cq.c b/drivers/infiniband/hw/hns/hns_roce_cq.c
index 5e0d78f4e545..c72bbf21f1ae 100644
--- a/drivers/infiniband/hw/hns/hns_roce_cq.c
+++ b/drivers/infiniband/hw/hns/hns_roce_cq.c
@@ -119,9 +119,9 @@ static int hns_roce_create_cqc(struct hns_roce_dev *hr_dev,
ret = hns_roce_create_hw_ctx(hr_dev, mailbox, HNS_ROCE_CMD_CREATE_CQC,
hr_cq->cqn);
if (ret)
- ibdev_err(ibdev,
- "failed to send create cmd for CQ(0x%lx), ret = %d.\n",
- hr_cq->cqn, ret);
+ ibdev_err_ratelimited(ibdev,
+ "failed to send create cmd for CQ(0x%lx), ret = %d.\n",
+ hr_cq->cqn, ret);
hns_roce_free_cmd_mailbox(hr_dev, mailbox);
diff --git a/drivers/infiniband/hw/hns/hns_roce_device.h b/drivers/infiniband/hw/hns/hns_roce_device.h
index 060797001353..fec4084e2dff 100644
--- a/drivers/infiniband/hw/hns/hns_roce_device.h
+++ b/drivers/infiniband/hw/hns/hns_roce_device.h
@@ -718,6 +718,7 @@ struct hns_roce_eq {
int shift;
int event_type;
int sub_type;
+ struct work_struct work;
};
struct hns_roce_eq_table {
diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
index 1c180a6b1c07..538b3261614a 100644
--- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
+++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c
@@ -36,6 +36,7 @@
#include <linux/iopoll.h>
#include <linux/kernel.h>
#include <linux/types.h>
+#include <linux/workqueue.h>
#include <net/addrconf.h>
#include <rdma/ib_addr.h>
#include <rdma/ib_cache.h>
@@ -3013,14 +3014,22 @@ static int hns_roce_v2_init(struct hns_roce_dev *hr_dev)
{
int ret;
+ if (hr_dev->pci_dev->revision == PCI_REVISION_ID_HIP08) {
+ ret = free_mr_init(hr_dev);
+ if (ret) {
+ dev_err(hr_dev->dev, "failed to init free mr!\n");
+ return ret;
+ }
+ }
+
/* The hns ROCEE requires the extdb info to be cleared before using */
ret = hns_roce_clear_extdb_list_info(hr_dev);
if (ret)
- return ret;
+ goto err_clear_extdb_failed;
ret = get_hem_table(hr_dev);
if (ret)
- return ret;
+ goto err_clear_extdb_failed;
if (hr_dev->is_vf)
return 0;
@@ -3035,6 +3044,9 @@ static int hns_roce_v2_init(struct hns_roce_dev *hr_dev)
err_llm_init_failed:
put_hem_table(hr_dev);
+err_clear_extdb_failed:
+ if (hr_dev->pci_dev->revision == PCI_REVISION_ID_HIP08)
+ free_mr_exit(hr_dev);
return ret;
}
@@ -5841,9 +5853,9 @@ static int hns_roce_v2_modify_srq(struct ib_srq *ibsrq,
HNS_ROCE_CMD_MODIFY_SRQC, srq->srqn);
hns_roce_free_cmd_mailbox(hr_dev, mailbox);
if (ret) {
- ibdev_err(&hr_dev->ib_dev,
- "failed to handle cmd of modifying SRQ, ret = %d.\n",
- ret);
+ ibdev_err_ratelimited(&hr_dev->ib_dev,
+ "failed to handle cmd of modifying SRQ, ret = %d.\n",
+ ret);
return ret;
}
}
@@ -5867,9 +5879,9 @@ static int hns_roce_v2_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr)
ret = hns_roce_cmd_mbox(hr_dev, 0, mailbox->dma,
HNS_ROCE_CMD_QUERY_SRQC, srq->srqn);
if (ret) {
- ibdev_err(&hr_dev->ib_dev,
- "failed to process cmd of querying SRQ, ret = %d.\n",
- ret);
+ ibdev_err_ratelimited(&hr_dev->ib_dev,
+ "failed to process cmd of querying SRQ, ret = %d.\n",
+ ret);
goto out;
}
@@ -5970,9 +5982,9 @@ static int hns_roce_v2_query_mpt(struct hns_roce_dev *hr_dev, u32 key,
ret = hns_roce_cmd_mbox(hr_dev, 0, mailbox->dma, HNS_ROCE_CMD_QUERY_MPT,
key_to_hw_index(key));
if (ret) {
- ibdev_err(&hr_dev->ib_dev,
- "failed to process cmd when querying MPT, ret = %d.\n",
- ret);
+ ibdev_err_ratelimited(&hr_dev->ib_dev,
+ "failed to process cmd when querying MPT, ret = %d.\n",
+ ret);
goto err_mailbox;
}
@@ -6206,32 +6218,11 @@ static struct hns_roce_ceqe *next_ceqe_sw_v2(struct hns_roce_eq *eq)
!!(eq->cons_index & eq->entries)) ? ceqe : NULL;
}
-static irqreturn_t hns_roce_v2_ceq_int(struct hns_roce_dev *hr_dev,
- struct hns_roce_eq *eq)
+static irqreturn_t hns_roce_v2_ceq_int(struct hns_roce_eq *eq)
{
- struct hns_roce_ceqe *ceqe = next_ceqe_sw_v2(eq);
- irqreturn_t ceqe_found = IRQ_NONE;
- u32 cqn;
-
- while (ceqe) {
- /* Make sure we read CEQ entry after we have checked the
- * ownership bit
- */
- dma_rmb();
-
- cqn = hr_reg_read(ceqe, CEQE_CQN);
-
- hns_roce_cq_completion(hr_dev, cqn);
-
- ++eq->cons_index;
- ceqe_found = IRQ_HANDLED;
-
- ceqe = next_ceqe_sw_v2(eq);
- }
-
- update_eq_db(eq);
+ queue_work(system_highpri_wq, &eq->work);
- return IRQ_RETVAL(ceqe_found);
+ return IRQ_HANDLED;
}
static irqreturn_t hns_roce_v2_msix_interrupt_eq(int irq, void *eq_ptr)
@@ -6242,7 +6233,7 @@ static irqreturn_t hns_roce_v2_msix_interrupt_eq(int irq, void *eq_ptr)
if (eq->type_flag == HNS_ROCE_CEQ)
/* Completion event interrupt */
- int_work = hns_roce_v2_ceq_int(hr_dev, eq);
+ int_work = hns_roce_v2_ceq_int(eq);
else
/* Asynchronous event interrupt */
int_work = hns_roce_v2_aeq_int(hr_dev, eq);
@@ -6617,6 +6608,33 @@ static int hns_roce_v2_create_eq(struct hns_roce_dev *hr_dev,
return ret;
}
+static void hns_roce_ceq_work(struct work_struct *work)
+{
+ struct hns_roce_eq *eq = from_work(eq, work, work);
+ struct hns_roce_ceqe *ceqe = next_ceqe_sw_v2(eq);
+ struct hns_roce_dev *hr_dev = eq->hr_dev;
+ int ceqe_num = 0;
+ u32 cqn;
+
+ while (ceqe && ceqe_num < hr_dev->caps.ceqe_depth) {
+ /* Make sure we read CEQ entry after we have checked the
+ * ownership bit
+ */
+ dma_rmb();
+
+ cqn = hr_reg_read(ceqe, CEQE_CQN);
+
+ hns_roce_cq_completion(hr_dev, cqn);
+
+ ++eq->cons_index;
+ ++ceqe_num;
+
+ ceqe = next_ceqe_sw_v2(eq);
+ }
+
+ update_eq_db(eq);
+}
+
static int __hns_roce_request_irq(struct hns_roce_dev *hr_dev, int irq_num,
int comp_num, int aeq_num, int other_num)
{
@@ -6647,21 +6665,24 @@ static int __hns_roce_request_irq(struct hns_roce_dev *hr_dev, int irq_num,
"hns-ceq-%d", j - other_num - aeq_num);
for (j = 0; j < irq_num; j++) {
- if (j < other_num)
+ if (j < other_num) {
ret = request_irq(hr_dev->irq[j],
hns_roce_v2_msix_interrupt_abn,
0, hr_dev->irq_names[j], hr_dev);
-
- else if (j < (other_num + comp_num))
+ } else if (j < (other_num + comp_num)) {
+ INIT_WORK(&eq_table->eq[j - other_num].work,
+ hns_roce_ceq_work);
ret = request_irq(eq_table->eq[j - other_num].irq,
hns_roce_v2_msix_interrupt_eq,
0, hr_dev->irq_names[j + aeq_num],
&eq_table->eq[j - other_num]);
- else
+ } else {
ret = request_irq(eq_table->eq[j - other_num].irq,
hns_roce_v2_msix_interrupt_eq,
0, hr_dev->irq_names[j - comp_num],
&eq_table->eq[j - other_num]);
+ }
+
if (ret) {
dev_err(hr_dev->dev, "request irq error!\n");
goto err_request_failed;
@@ -6671,12 +6692,16 @@ static int __hns_roce_request_irq(struct hns_roce_dev *hr_dev, int irq_num,
return 0;
err_request_failed:
- for (j -= 1; j >= 0; j--)
- if (j < other_num)
+ for (j -= 1; j >= 0; j--) {
+ if (j < other_num) {
free_irq(hr_dev->irq[j], hr_dev);
- else
- free_irq(eq_table->eq[j - other_num].irq,
- &eq_table->eq[j - other_num]);
+ continue;
+ }
+ free_irq(eq_table->eq[j - other_num].irq,
+ &eq_table->eq[j - other_num]);
+ if (j < other_num + comp_num)
+ cancel_work_sync(&eq_table->eq[j - other_num].work);
+ }
err_kzalloc_failed:
for (i -= 1; i >= 0; i--)
@@ -6697,8 +6722,11 @@ static void __hns_roce_free_irq(struct hns_roce_dev *hr_dev)
for (i = 0; i < hr_dev->caps.num_other_vectors; i++)
free_irq(hr_dev->irq[i], hr_dev);
- for (i = 0; i < eq_num; i++)
+ for (i = 0; i < eq_num; i++) {
free_irq(hr_dev->eq_table.eq[i].irq, &hr_dev->eq_table.eq[i]);
+ if (i < hr_dev->caps.num_comp_vectors)
+ cancel_work_sync(&hr_dev->eq_table.eq[i].work);
+ }
for (i = 0; i < irq_num; i++)
kfree(hr_dev->irq_names[i]);
@@ -6939,21 +6967,11 @@ static int __hns_roce_hw_v2_init_instance(struct hnae3_handle *handle)
goto error_failed_roce_init;
}
- if (hr_dev->pci_dev->revision == PCI_REVISION_ID_HIP08) {
- ret = free_mr_init(hr_dev);
- if (ret) {
- dev_err(hr_dev->dev, "failed to init free mr!\n");
- goto error_failed_free_mr_init;
- }
- }
handle->priv = hr_dev;
return 0;
-error_failed_free_mr_init:
- hns_roce_exit(hr_dev);
-
error_failed_roce_init:
kfree(hr_dev->priv);
diff --git a/drivers/infiniband/hw/hns/hns_roce_main.c b/drivers/infiniband/hw/hns/hns_roce_main.c
index 5702176a733a..e6eca4105c1a 100644
--- a/drivers/infiniband/hw/hns/hns_roce_main.c
+++ b/drivers/infiniband/hw/hns/hns_roce_main.c
@@ -835,6 +835,9 @@ static int hns_roce_setup_hca(struct hns_roce_dev *hr_dev)
spin_lock_init(&hr_dev->sm_lock);
+ INIT_LIST_HEAD(&hr_dev->qp_list);
+ spin_lock_init(&hr_dev->qp_list_lock);
+
if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_CQ_RECORD_DB ||
hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_QP_RECORD_DB) {
INIT_LIST_HEAD(&hr_dev->pgdir_list);
@@ -979,8 +982,6 @@ int hns_roce_init(struct hns_roce_dev *hr_dev)
}
}
- INIT_LIST_HEAD(&hr_dev->qp_list);
- spin_lock_init(&hr_dev->qp_list_lock);
INIT_LIST_HEAD(&hr_dev->dip_list);
spin_lock_init(&hr_dev->dip_list_lock);
diff --git a/drivers/infiniband/hw/hns/hns_roce_mr.c b/drivers/infiniband/hw/hns/hns_roce_mr.c
index 408ef2a96149..105ebe4c8b9b 100644
--- a/drivers/infiniband/hw/hns/hns_roce_mr.c
+++ b/drivers/infiniband/hw/hns/hns_roce_mr.c
@@ -163,7 +163,7 @@ static int hns_roce_mr_enable(struct hns_roce_dev *hr_dev,
ret = hns_roce_create_hw_ctx(hr_dev, mailbox, HNS_ROCE_CMD_CREATE_MPT,
mtpt_idx & (hr_dev->caps.num_mtpts - 1));
if (ret) {
- dev_err(dev, "failed to create mpt, ret = %d.\n", ret);
+ dev_err_ratelimited(dev, "failed to create mpt, ret = %d.\n", ret);
goto err_page;
}
@@ -291,7 +291,7 @@ struct ib_mr *hns_roce_rereg_user_mr(struct ib_mr *ibmr, int flags, u64 start,
ret = hns_roce_destroy_hw_ctx(hr_dev, HNS_ROCE_CMD_DESTROY_MPT,
mtpt_idx);
if (ret)
- ibdev_warn(ib_dev, "failed to destroy MPT, ret = %d.\n", ret);
+ ibdev_warn_ratelimited(ib_dev, "failed to destroy MPT, ret = %d.\n", ret);
mr->enabled = 0;
mr->iova = virt_addr;
@@ -322,7 +322,7 @@ struct ib_mr *hns_roce_rereg_user_mr(struct ib_mr *ibmr, int flags, u64 start,
ret = hns_roce_create_hw_ctx(hr_dev, mailbox, HNS_ROCE_CMD_CREATE_MPT,
mtpt_idx);
if (ret) {
- ibdev_err(ib_dev, "failed to create MPT, ret = %d.\n", ret);
+ ibdev_err_ratelimited(ib_dev, "failed to create MPT, ret = %d.\n", ret);
goto free_cmd_mbox;
}
diff --git a/drivers/infiniband/hw/hns/hns_roce_srq.c b/drivers/infiniband/hw/hns/hns_roce_srq.c
index 80fcb1b0e8fd..c699bf48adb4 100644
--- a/drivers/infiniband/hw/hns/hns_roce_srq.c
+++ b/drivers/infiniband/hw/hns/hns_roce_srq.c
@@ -103,7 +103,7 @@ static int hns_roce_create_srqc(struct hns_roce_dev *hr_dev,
ret = hns_roce_create_hw_ctx(hr_dev, mailbox, HNS_ROCE_CMD_CREATE_SRQ,
srq->srqn);
if (ret)
- ibdev_err(ibdev, "failed to config SRQC, ret = %d.\n", ret);
+ ibdev_err_ratelimited(ibdev, "failed to config SRQC, ret = %d.\n", ret);
err_mbox:
hns_roce_free_cmd_mailbox(hr_dev, mailbox);
diff --git a/drivers/infiniband/hw/irdma/hw.c b/drivers/infiniband/hw/irdma/hw.c
index 918a2d783141..6f71edf4e5e1 100644
--- a/drivers/infiniband/hw/irdma/hw.c
+++ b/drivers/infiniband/hw/irdma/hw.c
@@ -994,9 +994,9 @@ static int irdma_create_cqp(struct irdma_pci_f *rf)
INIT_LIST_HEAD(&cqp->cqp_avail_reqs);
INIT_LIST_HEAD(&cqp->cqp_pending_reqs);
- /* init the waitqueue of the cqp_requests and add them to the list */
+ /* init the completion of the cqp_requests and add them to the list */
for (i = 0; i < sqsize; i++) {
- init_waitqueue_head(&cqp->cqp_requests[i].waitq);
+ init_completion(&cqp->cqp_requests[i].comp);
list_add_tail(&cqp->cqp_requests[i].list, &cqp->cqp_avail_reqs);
}
init_waitqueue_head(&cqp->remove_wq);
@@ -2086,8 +2086,7 @@ void irdma_cqp_ce_handler(struct irdma_pci_f *rf, struct irdma_sc_cq *cq)
cqp_request->compl_info.error = info.error;
if (cqp_request->waiting) {
- WRITE_ONCE(cqp_request->request_done, true);
- wake_up(&cqp_request->waitq);
+ complete_all(&cqp_request->comp);
irdma_put_cqp_request(&rf->cqp, cqp_request);
} else {
if (cqp_request->callback_fcn)
diff --git a/drivers/infiniband/hw/irdma/main.h b/drivers/infiniband/hw/irdma/main.h
index e38172af6939..7f8428d00a9f 100644
--- a/drivers/infiniband/hw/irdma/main.h
+++ b/drivers/infiniband/hw/irdma/main.h
@@ -153,13 +153,12 @@ struct irdma_cqp_compl_info {
struct irdma_cqp_request {
struct cqp_cmds_info info;
- wait_queue_head_t waitq;
+ struct completion comp;
struct list_head list;
refcount_t refcnt;
void (*callback_fcn)(struct irdma_cqp_request *cqp_request);
void *param;
struct irdma_cqp_compl_info compl_info;
- bool request_done; /* READ/WRITE_ONCE macros operate on it */
bool waiting:1;
bool dynamic:1;
};
diff --git a/drivers/infiniband/hw/irdma/utils.c b/drivers/infiniband/hw/irdma/utils.c
index b66910b2942c..9187614e7442 100644
--- a/drivers/infiniband/hw/irdma/utils.c
+++ b/drivers/infiniband/hw/irdma/utils.c
@@ -445,7 +445,7 @@ struct irdma_cqp_request *irdma_alloc_and_get_cqp_request(struct irdma_cqp *cqp,
if (cqp_request) {
cqp_request->dynamic = true;
if (wait)
- init_waitqueue_head(&cqp_request->waitq);
+ init_completion(&cqp_request->comp);
}
}
if (!cqp_request) {
@@ -482,7 +482,7 @@ void irdma_free_cqp_request(struct irdma_cqp *cqp,
if (cqp_request->dynamic) {
kfree(cqp_request);
} else {
- WRITE_ONCE(cqp_request->request_done, false);
+ reinit_completion(&cqp_request->comp);
cqp_request->callback_fcn = NULL;
cqp_request->waiting = false;
@@ -516,8 +516,7 @@ irdma_free_pending_cqp_request(struct irdma_cqp *cqp,
{
if (cqp_request->waiting) {
cqp_request->compl_info.error = true;
- WRITE_ONCE(cqp_request->request_done, true);
- wake_up(&cqp_request->waitq);
+ complete_all(&cqp_request->comp);
}
wait_event_timeout(cqp->remove_wq,
refcount_read(&cqp_request->refcnt) == 1, 1000);
@@ -571,9 +570,8 @@ static int irdma_wait_event(struct irdma_pci_f *rf,
cqp_timeout.compl_cqp_cmds = atomic64_read(&rf->sc_dev.cqp->completed_ops);
do {
irdma_cqp_ce_handler(rf, &rf->ccq.sc_cq);
- if (wait_event_timeout(cqp_request->waitq,
- READ_ONCE(cqp_request->request_done),
- msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS)))
+ if (wait_for_completion_timeout(&cqp_request->comp,
+ msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS)))
break;
irdma_check_cqp_progress(&cqp_timeout, &rf->sc_dev);
diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c
index a8856a8613f4..246dfae3ce0e 100644
--- a/drivers/infiniband/hw/irdma/verbs.c
+++ b/drivers/infiniband/hw/irdma/verbs.c
@@ -599,18 +599,17 @@ static int irdma_setup_umode_qp(struct ib_udata *udata,
iwqp->ctx_info.qp_compl_ctx = req.user_compl_ctx;
iwqp->user_mode = 1;
- if (req.user_wqe_bufs) {
- info->qp_uk_init_info.legacy_mode = ucontext->legacy_mode;
- spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags);
- iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs,
- &ucontext->qp_reg_mem_list);
- spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
+ info->qp_uk_init_info.legacy_mode = ucontext->legacy_mode;
- if (!iwqp->iwpbl) {
- ret = -ENODATA;
- ibdev_dbg(&iwdev->ibdev, "VERBS: no pbl info\n");
- return ret;
- }
+ spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags);
+ iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs,
+ &ucontext->qp_reg_mem_list);
+ spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags);
+
+ if (!iwqp->iwpbl) {
+ ret = -ENODATA;
+ ibdev_dbg(&iwdev->ibdev, "VERBS: no pbl info\n");
+ return ret;
}
if (!ucontext->use_raw_attrs) {
diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c
index 886e9560e913..b5c2e0e1d82b 100644
--- a/drivers/infiniband/hw/mlx5/cq.c
+++ b/drivers/infiniband/hw/mlx5/cq.c
@@ -1009,15 +1009,18 @@ int mlx5_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr,
if (cq->create_flags & IB_UVERBS_CQ_FLAGS_IGNORE_OVERRUN)
MLX5_SET(cqc, cqc, oi, 1);
+ if (udata) {
+ cq->mcq.comp = mlx5_add_cq_to_tasklet;
+ cq->mcq.tasklet_ctx.comp = mlx5_ib_cq_comp;
+ } else {
+ cq->mcq.comp = mlx5_ib_cq_comp;
+ }
+
err = mlx5_core_create_cq(dev->mdev, &cq->mcq, cqb, inlen, out, sizeof(out));
if (err)
goto err_cqb;
mlx5_ib_dbg(dev, "cqn 0x%x\n", cq->mcq.cqn);
- if (udata)
- cq->mcq.tasklet_ctx.comp = mlx5_ib_cq_comp;
- else
- cq->mcq.comp = mlx5_ib_cq_comp;
cq->mcq.event = mlx5_ib_cq_event;
INIT_LIST_HEAD(&cq->wc_list);
diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c
index 9da42edecb05..72bba5f05d24 100644
--- a/drivers/infiniband/hw/mlx5/main.c
+++ b/drivers/infiniband/hw/mlx5/main.c
@@ -1743,9 +1743,13 @@ static int mlx5_ib_alloc_transport_domain(struct mlx5_ib_dev *dev, u32 *tdn,
if ((MLX5_CAP_GEN(dev->mdev, port_type) != MLX5_CAP_PORT_TYPE_ETH) ||
(!MLX5_CAP_GEN(dev->mdev, disable_local_lb_uc) &&
!MLX5_CAP_GEN(dev->mdev, disable_local_lb_mc)))
- return err;
+ return 0;
+
+ err = mlx5_ib_enable_lb(dev, true, false);
+ if (err)
+ mlx5_cmd_dealloc_transport_domain(dev->mdev, *tdn, uid);
- return mlx5_ib_enable_lb(dev, true, false);
+ return err;
}
static void mlx5_ib_dealloc_transport_domain(struct mlx5_ib_dev *dev, u32 tdn,
@@ -2802,7 +2806,6 @@ static int mlx5_ib_dev_res_init(struct mlx5_ib_dev *dev)
struct ib_srq_init_attr attr;
struct ib_device *ibdev;
struct ib_cq_init_attr cq_attr = {.cqe = 1};
- int port;
int ret = 0;
ibdev = &dev->ib_dev;
@@ -2851,10 +2854,6 @@ static int mlx5_ib_dev_res_init(struct mlx5_ib_dev *dev)
goto error6;
}
- for (port = 0; port < ARRAY_SIZE(devr->ports); ++port)
- INIT_WORK(&devr->ports[port].pkey_change_work,
- pkey_change_handler);
-
return 0;
error6:
@@ -2873,16 +2872,6 @@ static int mlx5_ib_dev_res_init(struct mlx5_ib_dev *dev)
static void mlx5_ib_dev_res_cleanup(struct mlx5_ib_dev *dev)
{
struct mlx5_ib_resources *devr = &dev->devr;
- int port;
-
- /*
- * Make sure no change P_Key work items are still executing.
- *
- * At this stage, the mlx5_ib_event should be unregistered
- * and it ensures that no new works are added.
- */
- for (port = 0; port < ARRAY_SIZE(devr->ports); ++port)
- cancel_work_sync(&devr->ports[port].pkey_change_work);
ib_destroy_srq(devr->s1);
ib_destroy_srq(devr->s0);
@@ -3650,6 +3639,8 @@ static int mlx5_ib_stage_init_init(struct mlx5_ib_dev *dev)
dev->port[i].roce.last_port_state = IB_PORT_DOWN;
}
+ mutex_init(&dev->lb.mutex);
+
err = mlx5_ib_init_multiport_master(dev);
if (err)
return err;
@@ -3856,11 +3847,6 @@ static int mlx5_ib_stage_caps_init(struct mlx5_ib_dev *dev)
if (err)
return err;
- if ((MLX5_CAP_GEN(dev->mdev, port_type) == MLX5_CAP_PORT_TYPE_ETH) &&
- (MLX5_CAP_GEN(dev->mdev, disable_local_lb_uc) ||
- MLX5_CAP_GEN(dev->mdev, disable_local_lb_mc)))
- mutex_init(&dev->lb.mutex);
-
if (MLX5_CAP_GEN_64(dev->mdev, general_obj_types) &
MLX5_GENERAL_OBJ_TYPES_CAP_VIRTIO_NET_Q) {
err = mlx5_ib_init_var_table(dev);
@@ -4089,6 +4075,13 @@ static void mlx5_ib_stage_delay_drop_cleanup(struct mlx5_ib_dev *dev)
static int mlx5_ib_stage_dev_notifier_init(struct mlx5_ib_dev *dev)
{
+ struct mlx5_ib_resources *devr = &dev->devr;
+ int port;
+
+ for (port = 0; port < ARRAY_SIZE(devr->ports); ++port)
+ INIT_WORK(&devr->ports[port].pkey_change_work,
+ pkey_change_handler);
+
dev->mdev_events.notifier_call = mlx5_ib_event;
mlx5_notifier_register(dev->mdev, &dev->mdev_events);
return 0;
@@ -4096,7 +4089,13 @@ static int mlx5_ib_stage_dev_notifier_init(struct mlx5_ib_dev *dev)
static void mlx5_ib_stage_dev_notifier_cleanup(struct mlx5_ib_dev *dev)
{
+ struct mlx5_ib_resources *devr = &dev->devr;
+ int port;
+
mlx5_notifier_unregister(dev->mdev, &dev->mdev_events);
+
+ for (port = 0; port < ARRAY_SIZE(devr->ports); ++port)
+ cancel_work_sync(&devr->ports[port].pkey_change_work);
}
void __mlx5_ib_remove(struct mlx5_ib_dev *dev,
@@ -4170,9 +4169,6 @@ static const struct mlx5_ib_profile pf_profile = {
STAGE_CREATE(MLX5_IB_STAGE_DEVICE_RESOURCES,
mlx5_ib_dev_res_init,
mlx5_ib_dev_res_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_DEVICE_NOTIFIER,
- mlx5_ib_stage_dev_notifier_init,
- mlx5_ib_stage_dev_notifier_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_ODP,
mlx5_ib_odp_init_one,
mlx5_ib_odp_cleanup_one),
@@ -4197,6 +4193,9 @@ static const struct mlx5_ib_profile pf_profile = {
STAGE_CREATE(MLX5_IB_STAGE_IB_REG,
mlx5_ib_stage_ib_reg_init,
mlx5_ib_stage_ib_reg_cleanup),
+ STAGE_CREATE(MLX5_IB_STAGE_DEVICE_NOTIFIER,
+ mlx5_ib_stage_dev_notifier_init,
+ mlx5_ib_stage_dev_notifier_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_POST_IB_REG_UMR,
mlx5_ib_stage_post_ib_reg_umr_init,
NULL),
@@ -4233,9 +4232,6 @@ const struct mlx5_ib_profile raw_eth_profile = {
STAGE_CREATE(MLX5_IB_STAGE_DEVICE_RESOURCES,
mlx5_ib_dev_res_init,
mlx5_ib_dev_res_cleanup),
- STAGE_CREATE(MLX5_IB_STAGE_DEVICE_NOTIFIER,
- mlx5_ib_stage_dev_notifier_init,
- mlx5_ib_stage_dev_notifier_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_COUNTERS,
mlx5_ib_counters_init,
mlx5_ib_counters_cleanup),
@@ -4257,6 +4253,9 @@ const struct mlx5_ib_profile raw_eth_profile = {
STAGE_CREATE(MLX5_IB_STAGE_IB_REG,
mlx5_ib_stage_ib_reg_init,
mlx5_ib_stage_ib_reg_cleanup),
+ STAGE_CREATE(MLX5_IB_STAGE_DEVICE_NOTIFIER,
+ mlx5_ib_stage_dev_notifier_init,
+ mlx5_ib_stage_dev_notifier_cleanup),
STAGE_CREATE(MLX5_IB_STAGE_POST_IB_REG_UMR,
mlx5_ib_stage_post_ib_reg_umr_init,
NULL),
diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h
index 0ef347e91ffe..9b6fa6b1c1ca 100644
--- a/drivers/infiniband/hw/mlx5/mlx5_ib.h
+++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h
@@ -637,6 +637,13 @@ enum mlx5_mkey_type {
MLX5_MKEY_INDIRECT_DEVX,
};
+struct mlx5r_cache_rb_key {
+ u8 ats:1;
+ unsigned int access_mode;
+ unsigned int access_flags;
+ unsigned int ndescs;
+};
+
struct mlx5_ib_mkey {
u32 key;
enum mlx5_mkey_type type;
@@ -757,10 +764,9 @@ struct mlx5_cache_ent {
unsigned long reserved;
char name[4];
- u32 order;
- u32 access_mode;
- u32 page;
- unsigned int ndescs;
+
+ struct rb_node node;
+ struct mlx5r_cache_rb_key rb_key;
u8 disabled:1;
u8 fill_to_high_water:1;
@@ -791,8 +797,9 @@ struct mlx5r_async_create_mkey {
struct mlx5_mkey_cache {
struct workqueue_struct *wq;
- struct mlx5_cache_ent ent[MAX_MKEY_CACHE_ENTRIES];
- struct dentry *root;
+ struct rb_root rb_root;
+ struct mutex rb_lock;
+ struct dentry *fs_root;
unsigned long last_add;
};
@@ -927,7 +934,6 @@ enum mlx5_ib_stages {
MLX5_IB_STAGE_QP,
MLX5_IB_STAGE_SRQ,
MLX5_IB_STAGE_DEVICE_RESOURCES,
- MLX5_IB_STAGE_DEVICE_NOTIFIER,
MLX5_IB_STAGE_ODP,
MLX5_IB_STAGE_COUNTERS,
MLX5_IB_STAGE_CONG_DEBUGFS,
@@ -936,6 +942,7 @@ enum mlx5_ib_stages {
MLX5_IB_STAGE_PRE_IB_REG_UMR,
MLX5_IB_STAGE_WHITELIST_UID,
MLX5_IB_STAGE_IB_REG,
+ MLX5_IB_STAGE_DEVICE_NOTIFIER,
MLX5_IB_STAGE_POST_IB_REG_UMR,
MLX5_IB_STAGE_DELAY_DROP,
MLX5_IB_STAGE_RESTRACK,
@@ -1337,10 +1344,13 @@ void mlx5_ib_copy_pas(u64 *old, u64 *new, int step, int num);
int mlx5_ib_get_cqe_size(struct ib_cq *ibcq);
int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev);
int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev);
+struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev,
+ struct mlx5r_cache_rb_key rb_key,
+ bool persistent_entry);
struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
- struct mlx5_cache_ent *ent,
- int access_flags);
+ int access_flags, int access_mode,
+ int ndescs);
int mlx5_ib_check_mr_status(struct ib_mr *ibmr, u32 check_mask,
struct ib_mr_status *mr_status);
@@ -1364,7 +1374,7 @@ int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq);
void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev);
int __init mlx5_ib_odp_init(void);
void mlx5_ib_odp_cleanup(void);
-void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent);
+int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev);
void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries,
struct mlx5_ib_mr *mr, int flags);
@@ -1383,7 +1393,10 @@ static inline int mlx5r_odp_create_eq(struct mlx5_ib_dev *dev,
static inline void mlx5_ib_odp_cleanup_one(struct mlx5_ib_dev *ibdev) {}
static inline int mlx5_ib_odp_init(void) { return 0; }
static inline void mlx5_ib_odp_cleanup(void) {}
-static inline void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent) {}
+static inline int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev)
+{
+ return 0;
+}
static inline void mlx5_odp_populate_xlt(void *xlt, size_t idx, size_t nentries,
struct mlx5_ib_mr *mr, int flags) {}
diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c
index 2cb4310cc3d6..dacda8d63c5a 100644
--- a/drivers/infiniband/hw/mlx5/mr.c
+++ b/drivers/infiniband/hw/mlx5/mr.c
@@ -292,12 +292,15 @@ static void set_cache_mkc(struct mlx5_cache_ent *ent, void *mkc)
set_mkc_access_pd_addr_fields(mkc, 0, 0, ent->dev->umrc.pd);
MLX5_SET(mkc, mkc, free, 1);
MLX5_SET(mkc, mkc, umr_en, 1);
- MLX5_SET(mkc, mkc, access_mode_1_0, ent->access_mode & 0x3);
- MLX5_SET(mkc, mkc, access_mode_4_2, (ent->access_mode >> 2) & 0x7);
+ MLX5_SET(mkc, mkc, access_mode_1_0, ent->rb_key.access_mode & 0x3);
+ MLX5_SET(mkc, mkc, access_mode_4_2,
+ (ent->rb_key.access_mode >> 2) & 0x7);
+ MLX5_SET(mkc, mkc, ma_translation_mode, !!ent->rb_key.ats);
MLX5_SET(mkc, mkc, translations_octword_size,
- get_mkc_octo_size(ent->access_mode, ent->ndescs));
- MLX5_SET(mkc, mkc, log_page_size, ent->page);
+ get_mkc_octo_size(ent->rb_key.access_mode,
+ ent->rb_key.ndescs));
+ MLX5_SET(mkc, mkc, log_page_size, PAGE_SHIFT);
}
/* Asynchronously schedule new MRs to be populated in the cache. */
@@ -515,18 +518,22 @@ static const struct file_operations limit_fops = {
static bool someone_adding(struct mlx5_mkey_cache *cache)
{
- unsigned int i;
-
- for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) {
- struct mlx5_cache_ent *ent = &cache->ent[i];
- bool ret;
+ struct mlx5_cache_ent *ent;
+ struct rb_node *node;
+ bool ret;
+ mutex_lock(&cache->rb_lock);
+ for (node = rb_first(&cache->rb_root); node; node = rb_next(node)) {
+ ent = rb_entry(node, struct mlx5_cache_ent, node);
xa_lock_irq(&ent->mkeys);
ret = ent->stored < ent->limit;
xa_unlock_irq(&ent->mkeys);
- if (ret)
+ if (ret) {
+ mutex_unlock(&cache->rb_lock);
return true;
+ }
}
+ mutex_unlock(&cache->rb_lock);
return false;
}
@@ -590,8 +597,8 @@ static void __cache_work_func(struct mlx5_cache_ent *ent)
if (err != -EAGAIN) {
mlx5_ib_warn(
dev,
- "command failed order %d, err %d\n",
- ent->order, err);
+ "add keys command failed, err %d\n",
+ err);
queue_delayed_work(cache->wq, &ent->dwork,
msecs_to_jiffies(1000));
}
@@ -637,16 +644,101 @@ static void delayed_cache_work_func(struct work_struct *work)
__cache_work_func(ent);
}
-struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
- struct mlx5_cache_ent *ent,
- int access_flags)
+static int cache_ent_key_cmp(struct mlx5r_cache_rb_key key1,
+ struct mlx5r_cache_rb_key key2)
+{
+ int res;
+
+ res = key1.ats - key2.ats;
+ if (res)
+ return res;
+
+ res = key1.access_mode - key2.access_mode;
+ if (res)
+ return res;
+
+ res = key1.access_flags - key2.access_flags;
+ if (res)
+ return res;
+
+ /*
+ * keep ndescs the last in the compare table since the find function
+ * searches for an exact match on all properties and only closest
+ * match in size.
+ */
+ return key1.ndescs - key2.ndescs;
+}
+
+static int mlx5_cache_ent_insert(struct mlx5_mkey_cache *cache,
+ struct mlx5_cache_ent *ent)
+{
+ struct rb_node **new = &cache->rb_root.rb_node, *parent = NULL;
+ struct mlx5_cache_ent *cur;
+ int cmp;
+
+ mutex_lock(&cache->rb_lock);
+ /* Figure out where to put new node */
+ while (*new) {
+ cur = rb_entry(*new, struct mlx5_cache_ent, node);
+ parent = *new;
+ cmp = cache_ent_key_cmp(cur->rb_key, ent->rb_key);
+ if (cmp > 0)
+ new = &((*new)->rb_left);
+ if (cmp < 0)
+ new = &((*new)->rb_right);
+ if (cmp == 0) {
+ mutex_unlock(&cache->rb_lock);
+ return -EEXIST;
+ }
+ }
+
+ /* Add new node and rebalance tree. */
+ rb_link_node(&ent->node, parent, new);
+ rb_insert_color(&ent->node, &cache->rb_root);
+
+ mutex_unlock(&cache->rb_lock);
+ return 0;
+}
+
+static struct mlx5_cache_ent *
+mkey_cache_ent_from_rb_key(struct mlx5_ib_dev *dev,
+ struct mlx5r_cache_rb_key rb_key)
+{
+ struct rb_node *node = dev->cache.rb_root.rb_node;
+ struct mlx5_cache_ent *cur, *smallest = NULL;
+ int cmp;
+
+ /*
+ * Find the smallest ent with order >= requested_order.
+ */
+ while (node) {
+ cur = rb_entry(node, struct mlx5_cache_ent, node);
+ cmp = cache_ent_key_cmp(cur->rb_key, rb_key);
+ if (cmp > 0) {
+ smallest = cur;
+ node = node->rb_left;
+ }
+ if (cmp < 0)
+ node = node->rb_right;
+ if (cmp == 0)
+ return cur;
+ }
+
+ return (smallest &&
+ smallest->rb_key.access_mode == rb_key.access_mode &&
+ smallest->rb_key.access_flags == rb_key.access_flags &&
+ smallest->rb_key.ats == rb_key.ats) ?
+ smallest :
+ NULL;
+}
+
+static struct mlx5_ib_mr *_mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
+ struct mlx5_cache_ent *ent,
+ int access_flags)
{
struct mlx5_ib_mr *mr;
int err;
- if (!mlx5r_umr_can_reconfig(dev, 0, access_flags))
- return ERR_PTR(-EOPNOTSUPP);
-
mr = kzalloc(sizeof(*mr), GFP_KERNEL);
if (!mr)
return ERR_PTR(-ENOMEM);
@@ -677,10 +769,48 @@ struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
return mr;
}
-static void clean_keys(struct mlx5_ib_dev *dev, int c)
+static int get_unchangeable_access_flags(struct mlx5_ib_dev *dev,
+ int access_flags)
+{
+ int ret = 0;
+
+ if ((access_flags & IB_ACCESS_REMOTE_ATOMIC) &&
+ MLX5_CAP_GEN(dev->mdev, atomic) &&
+ MLX5_CAP_GEN(dev->mdev, umr_modify_atomic_disabled))
+ ret |= IB_ACCESS_REMOTE_ATOMIC;
+
+ if ((access_flags & IB_ACCESS_RELAXED_ORDERING) &&
+ MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write) &&
+ !MLX5_CAP_GEN(dev->mdev, relaxed_ordering_write_umr))
+ ret |= IB_ACCESS_RELAXED_ORDERING;
+
+ if ((access_flags & IB_ACCESS_RELAXED_ORDERING) &&
+ MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read) &&
+ !MLX5_CAP_GEN(dev->mdev, relaxed_ordering_read_umr))
+ ret |= IB_ACCESS_RELAXED_ORDERING;
+
+ return ret;
+}
+
+struct mlx5_ib_mr *mlx5_mr_cache_alloc(struct mlx5_ib_dev *dev,
+ int access_flags, int access_mode,
+ int ndescs)
+{
+ struct mlx5r_cache_rb_key rb_key = {
+ .ndescs = ndescs,
+ .access_mode = access_mode,
+ .access_flags = get_unchangeable_access_flags(dev, access_flags)
+ };
+ struct mlx5_cache_ent *ent = mkey_cache_ent_from_rb_key(dev, rb_key);
+
+ if (!ent)
+ return ERR_PTR(-EOPNOTSUPP);
+
+ return _mlx5_mr_cache_alloc(dev, ent, access_flags);
+}
+
+static void clean_keys(struct mlx5_ib_dev *dev, struct mlx5_cache_ent *ent)
{
- struct mlx5_mkey_cache *cache = &dev->cache;
- struct mlx5_cache_ent *ent = &cache->ent[c];
u32 mkey;
cancel_delayed_work(&ent->dwork);
@@ -699,31 +829,36 @@ static void mlx5_mkey_cache_debugfs_cleanup(struct mlx5_ib_dev *dev)
if (!mlx5_debugfs_root || dev->is_rep)
return;
- debugfs_remove_recursive(dev->cache.root);
- dev->cache.root = NULL;
+ debugfs_remove_recursive(dev->cache.fs_root);
+ dev->cache.fs_root = NULL;
+}
+
+static void mlx5_mkey_cache_debugfs_add_ent(struct mlx5_ib_dev *dev,
+ struct mlx5_cache_ent *ent)
+{
+ int order = order_base_2(ent->rb_key.ndescs);
+ struct dentry *dir;
+
+ if (ent->rb_key.access_mode == MLX5_MKC_ACCESS_MODE_KSM)
+ order = MLX5_IMR_KSM_CACHE_ENTRY + 2;
+
+ sprintf(ent->name, "%d", order);
+ dir = debugfs_create_dir(ent->name, dev->cache.fs_root);
+ debugfs_create_file("size", 0600, dir, ent, &size_fops);
+ debugfs_create_file("limit", 0600, dir, ent, &limit_fops);
+ debugfs_create_ulong("cur", 0400, dir, &ent->stored);
+ debugfs_create_u32("miss", 0600, dir, &ent->miss);
}
static void mlx5_mkey_cache_debugfs_init(struct mlx5_ib_dev *dev)
{
+ struct dentry *dbg_root = mlx5_debugfs_get_dev_root(dev->mdev);
struct mlx5_mkey_cache *cache = &dev->cache;
- struct mlx5_cache_ent *ent;
- struct dentry *dir;
- int i;
if (!mlx5_debugfs_root || dev->is_rep)
return;
- cache->root = debugfs_create_dir("mr_cache", mlx5_debugfs_get_dev_root(dev->mdev));
-
- for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) {
- ent = &cache->ent[i];
- sprintf(ent->name, "%d", ent->order);
- dir = debugfs_create_dir(ent->name, cache->root);
- debugfs_create_file("size", 0600, dir, ent, &size_fops);
- debugfs_create_file("limit", 0600, dir, ent, &limit_fops);
- debugfs_create_ulong("cur", 0400, dir, &ent->stored);
- debugfs_create_u32("miss", 0600, dir, &ent->miss);
- }
+ cache->fs_root = debugfs_create_dir("mr_cache", dbg_root);
}
static void delay_time_func(struct timer_list *t)
@@ -733,13 +868,64 @@ static void delay_time_func(struct timer_list *t)
WRITE_ONCE(dev->fill_delay, 0);
}
+struct mlx5_cache_ent *mlx5r_cache_create_ent(struct mlx5_ib_dev *dev,
+ struct mlx5r_cache_rb_key rb_key,
+ bool persistent_entry)
+{
+ struct mlx5_cache_ent *ent;
+ int order;
+ int ret;
+
+ ent = kzalloc(sizeof(*ent), GFP_KERNEL);
+ if (!ent)
+ return ERR_PTR(-ENOMEM);
+
+ xa_init_flags(&ent->mkeys, XA_FLAGS_LOCK_IRQ);
+ ent->rb_key = rb_key;
+ ent->dev = dev;
+
+ INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func);
+
+ ret = mlx5_cache_ent_insert(&dev->cache, ent);
+ if (ret) {
+ kfree(ent);
+ return ERR_PTR(ret);
+ }
+
+ if (persistent_entry) {
+ if (rb_key.access_mode == MLX5_MKC_ACCESS_MODE_KSM)
+ order = MLX5_IMR_KSM_CACHE_ENTRY;
+ else
+ order = order_base_2(rb_key.ndescs) - 2;
+
+ if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) &&
+ !dev->is_rep && mlx5_core_is_pf(dev->mdev) &&
+ mlx5r_umr_can_load_pas(dev, 0))
+ ent->limit = dev->mdev->profile.mr_cache[order].limit;
+ else
+ ent->limit = 0;
+
+ mlx5_mkey_cache_debugfs_add_ent(dev, ent);
+ }
+
+ return ent;
+}
+
int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev)
{
struct mlx5_mkey_cache *cache = &dev->cache;
+ struct rb_root *root = &dev->cache.rb_root;
+ struct mlx5r_cache_rb_key rb_key = {
+ .access_mode = MLX5_MKC_ACCESS_MODE_MTT,
+ };
struct mlx5_cache_ent *ent;
+ struct rb_node *node;
+ int ret;
int i;
mutex_init(&dev->slow_path_mutex);
+ mutex_init(&dev->cache.rb_lock);
+ dev->cache.rb_root = RB_ROOT;
cache->wq = alloc_ordered_workqueue("mkey_cache", WQ_MEM_RECLAIM);
if (!cache->wq) {
mlx5_ib_warn(dev, "failed to create work queue\n");
@@ -748,52 +934,46 @@ int mlx5_mkey_cache_init(struct mlx5_ib_dev *dev)
mlx5_cmd_init_async_ctx(dev->mdev, &dev->async_ctx);
timer_setup(&dev->delay_timer, delay_time_func, 0);
- for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) {
- ent = &cache->ent[i];
- xa_init_flags(&ent->mkeys, XA_FLAGS_LOCK_IRQ);
- ent->order = i + 2;
- ent->dev = dev;
- ent->limit = 0;
-
- INIT_DELAYED_WORK(&ent->dwork, delayed_cache_work_func);
-
- if (i > MKEY_CACHE_LAST_STD_ENTRY) {
- mlx5_odp_init_mkey_cache_entry(ent);
- continue;
+ mlx5_mkey_cache_debugfs_init(dev);
+ for (i = 0; i <= mkey_cache_max_order(dev); i++) {
+ rb_key.ndescs = 1 << (i + 2);
+ ent = mlx5r_cache_create_ent(dev, rb_key, true);
+ if (IS_ERR(ent)) {
+ ret = PTR_ERR(ent);
+ goto err;
}
+ }
- if (ent->order > mkey_cache_max_order(dev))
- continue;
+ ret = mlx5_odp_init_mkey_cache(dev);
+ if (ret)
+ goto err;
- ent->page = PAGE_SHIFT;
- ent->ndescs = 1 << ent->order;
- ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT;
- if ((dev->mdev->profile.mask & MLX5_PROF_MASK_MR_CACHE) &&
- !dev->is_rep && mlx5_core_is_pf(dev->mdev) &&
- mlx5r_umr_can_load_pas(dev, 0))
- ent->limit = dev->mdev->profile.mr_cache[i].limit;
- else
- ent->limit = 0;
+ for (node = rb_first(root); node; node = rb_next(node)) {
+ ent = rb_entry(node, struct mlx5_cache_ent, node);
xa_lock_irq(&ent->mkeys);
queue_adjust_cache_locked(ent);
xa_unlock_irq(&ent->mkeys);
}
- mlx5_mkey_cache_debugfs_init(dev);
-
return 0;
+
+err:
+ mlx5_ib_warn(dev, "failed to create mkey cache entry\n");
+ return ret;
}
int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev)
{
- unsigned int i;
+ struct rb_root *root = &dev->cache.rb_root;
+ struct mlx5_cache_ent *ent;
+ struct rb_node *node;
if (!dev->cache.wq)
return 0;
- for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++) {
- struct mlx5_cache_ent *ent = &dev->cache.ent[i];
-
+ mutex_lock(&dev->cache.rb_lock);
+ for (node = rb_first(root); node; node = rb_next(node)) {
+ ent = rb_entry(node, struct mlx5_cache_ent, node);
xa_lock_irq(&ent->mkeys);
ent->disabled = true;
xa_unlock_irq(&ent->mkeys);
@@ -803,8 +983,15 @@ int mlx5_mkey_cache_cleanup(struct mlx5_ib_dev *dev)
mlx5_mkey_cache_debugfs_cleanup(dev);
mlx5_cmd_cleanup_async_ctx(&dev->async_ctx);
- for (i = 0; i < MAX_MKEY_CACHE_ENTRIES; i++)
- clean_keys(dev, i);
+ node = rb_first(root);
+ while (node) {
+ ent = rb_entry(node, struct mlx5_cache_ent, node);
+ node = rb_next(node);
+ clean_keys(dev, ent);
+ rb_erase(&ent->node, root);
+ kfree(ent);
+ }
+ mutex_unlock(&dev->cache.rb_lock);
destroy_workqueue(dev->cache.wq);
del_timer_sync(&dev->delay_timer);
@@ -873,23 +1060,10 @@ static int get_octo_len(u64 addr, u64 len, int page_shift)
static int mkey_cache_max_order(struct mlx5_ib_dev *dev)
{
if (MLX5_CAP_GEN(dev->mdev, umr_extended_translation_offset))
- return MKEY_CACHE_LAST_STD_ENTRY + 2;
+ return MKEY_CACHE_LAST_STD_ENTRY;
return MLX5_MAX_UMR_SHIFT;
}
-static struct mlx5_cache_ent *mkey_cache_ent_from_order(struct mlx5_ib_dev *dev,
- unsigned int order)
-{
- struct mlx5_mkey_cache *cache = &dev->cache;
-
- if (order < cache->ent[0].order)
- return &cache->ent[0];
- order = order - cache->ent[0].order;
- if (order > MKEY_CACHE_LAST_STD_ENTRY)
- return NULL;
- return &cache->ent[order];
-}
-
static void set_mr_fields(struct mlx5_ib_dev *dev, struct mlx5_ib_mr *mr,
u64 length, int access_flags, u64 iova)
{
@@ -916,6 +1090,9 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd,
struct ib_umem *umem, u64 iova,
int access_flags)
{
+ struct mlx5r_cache_rb_key rb_key = {
+ .access_mode = MLX5_MKC_ACCESS_MODE_MTT,
+ };
struct mlx5_ib_dev *dev = to_mdev(pd->device);
struct mlx5_cache_ent *ent;
struct mlx5_ib_mr *mr;
@@ -928,8 +1105,11 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd,
0, iova);
if (WARN_ON(!page_size))
return ERR_PTR(-EINVAL);
- ent = mkey_cache_ent_from_order(
- dev, order_base_2(ib_umem_num_dma_blocks(umem, page_size)));
+
+ rb_key.ndescs = ib_umem_num_dma_blocks(umem, page_size);
+ rb_key.ats = mlx5_umem_needs_ats(dev, umem, access_flags);
+ rb_key.access_flags = get_unchangeable_access_flags(dev, access_flags);
+ ent = mkey_cache_ent_from_rb_key(dev, rb_key);
/*
* Matches access in alloc_cache_mr(). If the MR can't come from the
* cache then synchronously create an uncached one.
@@ -943,7 +1123,7 @@ static struct mlx5_ib_mr *alloc_cacheable_mr(struct ib_pd *pd,
return mr;
}
- mr = mlx5_mr_cache_alloc(dev, ent, access_flags);
+ mr = _mlx5_mr_cache_alloc(dev, ent, access_flags);
if (IS_ERR(mr))
return mr;
@@ -1373,7 +1553,7 @@ static bool can_use_umr_rereg_pas(struct mlx5_ib_mr *mr,
mlx5_umem_find_best_pgsz(new_umem, mkc, log_page_size, 0, iova);
if (WARN_ON(!*page_size))
return false;
- return (1ULL << mr->mmkey.cache_ent->order) >=
+ return (mr->mmkey.cache_ent->rb_key.ndescs) >=
ib_umem_num_dma_blocks(new_umem, *page_size);
}
@@ -1571,7 +1751,7 @@ mlx5_alloc_priv_descs(struct ib_device *device,
static void
mlx5_free_priv_descs(struct mlx5_ib_mr *mr)
{
- if (!mr->umem && mr->descs) {
+ if (!mr->umem && mr->ibmr.type != IB_MR_TYPE_DM && mr->descs) {
struct ib_device *device = mr->ibmr.device;
int size = mr->max_descs * mr->desc_size;
struct mlx5_ib_dev *dev = to_mdev(device);
@@ -1587,6 +1767,8 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
{
struct mlx5_ib_mr *mr = to_mmr(ibmr);
struct mlx5_ib_dev *dev = to_mdev(ibmr->device);
+ bool is_odp = is_odp_mr(mr);
+ bool odp_locked = is_odp;
int rc;
/*
@@ -1628,25 +1810,42 @@ int mlx5_ib_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata)
}
/* Stop DMA */
+ if (odp_locked)
+ mutex_lock(&to_ib_umem_odp(mr->umem)->umem_mutex);
+
if (mr->mmkey.cache_ent) {
xa_lock_irq(&mr->mmkey.cache_ent->mkeys);
mr->mmkey.cache_ent->in_use--;
xa_unlock_irq(&mr->mmkey.cache_ent->mkeys);
- if (mlx5r_umr_revoke_mr(mr) ||
- push_mkey(mr->mmkey.cache_ent, false,
- xa_mk_value(mr->mmkey.key)))
+ if (mlx5r_umr_revoke_mr(mr)) {
mr->mmkey.cache_ent = NULL;
+ } else {
+ if (odp_locked) {
+ to_ib_umem_odp(mr->umem)->private = NULL;
+ mutex_unlock(&to_ib_umem_odp(mr->umem)->umem_mutex);
+ odp_locked = false;
+ }
+ if (push_mkey(mr->mmkey.cache_ent, false,
+ xa_mk_value(mr->mmkey.key)))
+ mr->mmkey.cache_ent = NULL;
+ }
}
if (!mr->mmkey.cache_ent) {
rc = destroy_mkey(to_mdev(mr->ibmr.device), mr);
- if (rc)
+ if (rc) {
+ if (odp_locked)
+ mutex_unlock(&to_ib_umem_odp(mr->umem)->umem_mutex);
return rc;
+ }
}
- if (mr->umem) {
- bool is_odp = is_odp_mr(mr);
+ if (odp_locked) {
+ to_ib_umem_odp(mr->umem)->private = NULL;
+ mutex_unlock(&to_ib_umem_odp(mr->umem)->umem_mutex);
+ }
+ if (mr->umem) {
if (!is_odp)
atomic_sub(ib_umem_num_pages(mr->umem),
&dev->mdev->priv.reg_pages);
diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c
index 89682fe0747f..d2ef2ff88ccd 100644
--- a/drivers/infiniband/hw/mlx5/odp.c
+++ b/drivers/infiniband/hw/mlx5/odp.c
@@ -213,10 +213,25 @@ static void destroy_unused_implicit_child_mr(struct mlx5_ib_mr *mr)
unsigned long idx = ib_umem_start(odp) >> MLX5_IMR_MTT_SHIFT;
struct mlx5_ib_mr *imr = mr->parent;
+ /*
+ * If userspace is racing freeing the parent implicit ODP MR then we can
+ * loose the race with parent destruction. In this case
+ * mlx5_ib_free_odp_mr() will free everything in the implicit_children
+ * xarray so NOP is fine. This child MR cannot be destroyed here because
+ * we are under its umem_mutex.
+ */
if (!refcount_inc_not_zero(&imr->mmkey.usecount))
return;
- xa_erase(&imr->implicit_children, idx);
+ xa_lock(&imr->implicit_children);
+ if (__xa_cmpxchg(&imr->implicit_children, idx, mr, NULL, GFP_KERNEL) !=
+ mr) {
+ xa_unlock(&imr->implicit_children);
+ mlx5r_deref_odp_mkey(&imr->mmkey);
+ return;
+ }
+
+ xa_unlock(&imr->implicit_children);
/* Freeing a MR is a sleeping operation, so bounce to a work queue */
INIT_WORK(&mr->odp_destroy.work, free_implicit_child_mr_work);
@@ -251,6 +266,8 @@ static bool mlx5_ib_invalidate_range(struct mmu_interval_notifier *mni,
if (!umem_odp->npages)
goto out;
mr = umem_odp->private;
+ if (!mr)
+ goto out;
start = max_t(u64, ib_umem_start(umem_odp), range->start);
end = min_t(u64, ib_umem_end(umem_odp), range->end);
@@ -420,8 +437,9 @@ static struct mlx5_ib_mr *implicit_get_child_mr(struct mlx5_ib_mr *imr,
if (IS_ERR(odp))
return ERR_CAST(odp);
- mr = mlx5_mr_cache_alloc(dev, &dev->cache.ent[MLX5_IMR_MTT_CACHE_ENTRY],
- imr->access_flags);
+ mr = mlx5_mr_cache_alloc(dev, imr->access_flags,
+ MLX5_MKC_ACCESS_MODE_MTT,
+ MLX5_IMR_MTT_ENTRIES);
if (IS_ERR(mr)) {
ib_umem_odp_release(odp);
return mr;
@@ -495,9 +513,8 @@ struct mlx5_ib_mr *mlx5_ib_alloc_implicit_mr(struct mlx5_ib_pd *pd,
if (IS_ERR(umem_odp))
return ERR_CAST(umem_odp);
- imr = mlx5_mr_cache_alloc(dev,
- &dev->cache.ent[MLX5_IMR_KSM_CACHE_ENTRY],
- access_flags);
+ imr = mlx5_mr_cache_alloc(dev, access_flags, MLX5_MKC_ACCESS_MODE_KSM,
+ mlx5_imr_ksm_entries);
if (IS_ERR(imr)) {
ib_umem_odp_release(umem_odp);
return imr;
@@ -1593,26 +1610,22 @@ mlx5_ib_odp_destroy_eq(struct mlx5_ib_dev *dev, struct mlx5_ib_pf_eq *eq)
return err;
}
-void mlx5_odp_init_mkey_cache_entry(struct mlx5_cache_ent *ent)
+int mlx5_odp_init_mkey_cache(struct mlx5_ib_dev *dev)
{
- if (!(ent->dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT))
- return;
+ struct mlx5r_cache_rb_key rb_key = {
+ .access_mode = MLX5_MKC_ACCESS_MODE_KSM,
+ .ndescs = mlx5_imr_ksm_entries,
+ };
+ struct mlx5_cache_ent *ent;
- switch (ent->order - 2) {
- case MLX5_IMR_MTT_CACHE_ENTRY:
- ent->page = PAGE_SHIFT;
- ent->ndescs = MLX5_IMR_MTT_ENTRIES;
- ent->access_mode = MLX5_MKC_ACCESS_MODE_MTT;
- ent->limit = 0;
- break;
+ if (!(dev->odp_caps.general_caps & IB_ODP_SUPPORT_IMPLICIT))
+ return 0;
- case MLX5_IMR_KSM_CACHE_ENTRY:
- ent->page = MLX5_KSM_PAGE_SHIFT;
- ent->ndescs = mlx5_imr_ksm_entries;
- ent->access_mode = MLX5_MKC_ACCESS_MODE_KSM;
- ent->limit = 0;
- break;
- }
+ ent = mlx5r_cache_create_ent(dev, rb_key, true);
+ if (IS_ERR(ent))
+ return PTR_ERR(ent);
+
+ return 0;
}
static const struct ib_device_ops mlx5_ib_dev_odp_ops = {
diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c
index 219e89242d88..2e7029181382 100644
--- a/drivers/infiniband/hw/mlx5/umr.c
+++ b/drivers/infiniband/hw/mlx5/umr.c
@@ -200,30 +200,6 @@ void mlx5r_umr_resource_cleanup(struct mlx5_ib_dev *dev)
ib_dealloc_pd(dev->umrc.pd);
}
-static int mlx5r_umr_recover(struct mlx5_ib_dev *dev)
-{
- struct umr_common *umrc = &dev->umrc;
- struct ib_qp_attr attr;
- int err;
-
- attr.qp_state = IB_QPS_RESET;
- err = ib_modify_qp(umrc->qp, &attr, IB_QP_STATE);
- if (err) {
- mlx5_ib_dbg(dev, "Couldn't modify UMR QP\n");
- goto err;
- }
-
- err = mlx5r_umr_qp_rst2rts(dev, umrc->qp);
- if (err)
- goto err;
-
- umrc->state = MLX5_UMR_STATE_ACTIVE;
- return 0;
-
-err:
- umrc->state = MLX5_UMR_STATE_ERR;
- return err;
-}
static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
struct mlx5r_umr_wqe *wqe, bool with_data)
@@ -271,6 +247,61 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe,
return err;
}
+static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey,
+ struct mlx5r_umr_context *umr_context,
+ struct mlx5r_umr_wqe *wqe, bool with_data)
+{
+ struct umr_common *umrc = &dev->umrc;
+ struct ib_qp_attr attr;
+ int err;
+
+ mutex_lock(&umrc->lock);
+ /* Preventing any further WRs to be sent now */
+ if (umrc->state != MLX5_UMR_STATE_RECOVER) {
+ mlx5_ib_warn(dev, "UMR recovery encountered an unexpected state=%d\n",
+ umrc->state);
+ umrc->state = MLX5_UMR_STATE_RECOVER;
+ }
+ mutex_unlock(&umrc->lock);
+
+ /* Sending a final/barrier WR (the failed one) and wait for its completion.
+ * This will ensure that all the previous WRs got a completion before
+ * we set the QP state to RESET.
+ */
+ err = mlx5r_umr_post_send(umrc->qp, mkey, &umr_context->cqe, wqe,
+ with_data);
+ if (err) {
+ mlx5_ib_warn(dev, "UMR recovery post send failed, err %d\n", err);
+ goto err;
+ }
+
+ /* Since the QP is in an error state, it will only receive
+ * IB_WC_WR_FLUSH_ERR. However, as it serves only as a barrier
+ * we don't care about its status.
+ */
+ wait_for_completion(&umr_context->done);
+
+ attr.qp_state = IB_QPS_RESET;
+ err = ib_modify_qp(umrc->qp, &attr, IB_QP_STATE);
+ if (err) {
+ mlx5_ib_warn(dev, "Couldn't modify UMR QP to RESET, err=%d\n", err);
+ goto err;
+ }
+
+ err = mlx5r_umr_qp_rst2rts(dev, umrc->qp);
+ if (err) {
+ mlx5_ib_warn(dev, "Couldn't modify UMR QP to RTS, err=%d\n", err);
+ goto err;
+ }
+
+ umrc->state = MLX5_UMR_STATE_ACTIVE;
+ return 0;
+
+err:
+ umrc->state = MLX5_UMR_STATE_ERR;
+ return err;
+}
+
static void mlx5r_umr_done(struct ib_cq *cq, struct ib_wc *wc)
{
struct mlx5_ib_umr_context *context =
@@ -333,11 +364,9 @@ static int mlx5r_umr_post_send_wait(struct mlx5_ib_dev *dev, u32 mkey,
WARN_ON_ONCE(1);
mlx5_ib_warn(dev,
- "reg umr failed (%u). Trying to recover and resubmit the flushed WQEs\n",
- umr_context.status);
- mutex_lock(&umrc->lock);
- err = mlx5r_umr_recover(dev);
- mutex_unlock(&umrc->lock);
+ "reg umr failed (%u). Trying to recover and resubmit the flushed WQEs, mkey = %u\n",
+ umr_context.status, mkey);
+ err = mlx5r_umr_recover(dev, mkey, &umr_context, wqe, with_data);
if (err)
mlx5_ib_warn(dev, "couldn't recover UMR, err %d\n",
err);
diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c
index d61f8de7f21c..ddc47691791e 100644
--- a/drivers/infiniband/sw/rdmavt/vt.c
+++ b/drivers/infiniband/sw/rdmavt/vt.c
@@ -54,8 +54,10 @@ struct rvt_dev_info *rvt_alloc_device(size_t size, int nports)
return rdi;
rdi->ports = kcalloc(nports, sizeof(*rdi->ports), GFP_KERNEL);
- if (!rdi->ports)
+ if (!rdi->ports) {
ib_dealloc_device(&rdi->ibdev);
+ return NULL;
+ }
return rdi;
}
diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c
index ab42bfea92a6..a468e9bdc736 100644
--- a/drivers/infiniband/sw/rxe/rxe_qp.c
+++ b/drivers/infiniband/sw/rxe/rxe_qp.c
@@ -124,6 +124,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp)
}
kfree(qp->resp.resources);
qp->resp.resources = NULL;
+ qp->resp.res = NULL;
}
}
@@ -583,10 +584,18 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask,
int max_dest_rd_atomic = attr->max_dest_rd_atomic ?
roundup_pow_of_two(attr->max_dest_rd_atomic) : 0;
+ /*
+ * Not gated by IB_QP_STATE, so the responder task is live.
+ * Quiesce resp.task like rxe_qp_reset() before swapping the
+ * rd_atomic array, so rxe_responder() cannot race the free/
+ * realloc.
+ */
+ rxe_disable_task(&qp->resp.task);
free_rd_atomic_resources(qp);
qp->attr.max_dest_rd_atomic = max_dest_rd_atomic;
err = alloc_rd_atomic_resources(qp, max_dest_rd_atomic);
+ rxe_enable_task(&qp->resp.task);
if (err)
return err;
}
diff --git a/drivers/infiniband/sw/rxe/rxe_srq.c b/drivers/infiniband/sw/rxe/rxe_srq.c
index 038a9cd55413..82e872fd8f45 100644
--- a/drivers/infiniband/sw/rxe/rxe_srq.c
+++ b/drivers/infiniband/sw/rxe/rxe_srq.c
@@ -161,18 +161,13 @@ int rxe_srq_from_attr(struct rxe_dev *rxe, struct rxe_srq *srq,
&srq->rq.producer_lock,
&srq->rq.consumer_lock);
if (err)
- goto err2;
+ return err;
}
if (mask & IB_SRQ_LIMIT)
srq->limit = attr->srq_limit;
return 0;
-
-err2:
- rxe_queue_cleanup(q);
- srq->rq.queue = NULL;
- return err;
}
void rxe_srq_cleanup(struct rxe_pool_elem *elem)
diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c
index 19932fc8856d..2ffac1c94df3 100644
--- a/drivers/infiniband/sw/siw/siw_cm.c
+++ b/drivers/infiniband/sw/siw/siw_cm.c
@@ -93,6 +93,7 @@ static void siw_socket_disassoc(struct socket *s)
cep = sk_to_cep(sk);
if (cep) {
siw_sk_restore_upcalls(sk, cep);
+ cep->sock = NULL;
siw_cep_put(cep);
} else {
pr_warn("siw: cannot restore sk callbacks: no ep\n");
@@ -370,6 +371,16 @@ static void siw_free_cm_id(struct siw_cep *cep)
cep->cm_id = NULL;
}
+static void siw_destroy_cep_sock(struct siw_cep *cep)
+{
+ struct socket *s = cep->sock;
+
+ if (s) {
+ siw_socket_disassoc(s);
+ sock_release(s);
+ }
+}
+
/*
* siw_qp_cm_drop()
*
@@ -427,14 +438,7 @@ void siw_qp_cm_drop(struct siw_qp *qp, int schedule)
}
cep->state = SIW_EPSTATE_CLOSED;
- if (cep->sock) {
- siw_socket_disassoc(cep->sock);
- /*
- * Immediately close socket
- */
- sock_release(cep->sock);
- cep->sock = NULL;
- }
+ siw_destroy_cep_sock(cep);
if (cep->qp) {
cep->qp = NULL;
siw_qp_put(qp);
@@ -1003,7 +1007,6 @@ static void siw_accept_newconn(struct siw_cep *cep)
if (new_s) {
siw_socket_disassoc(new_s);
sock_release(new_s);
- new_cep->sock = NULL;
}
siw_dbg_cep(cep, "error %d\n", rv);
}
@@ -1155,6 +1158,8 @@ static void siw_cm_work_handler(struct work_struct *w)
WARN(1, "Undefined CM work type: %d\n", work->type);
}
if (release_cep) {
+ struct socket *s = cep->sock;
+
siw_dbg_cep(cep,
"release: timer=%s, QP[%u]\n",
cep->mpa_timer ? "y" : "n",
@@ -1180,10 +1185,9 @@ static void siw_cm_work_handler(struct work_struct *w)
cep->qp = NULL;
siw_qp_put(qp);
}
- if (cep->sock) {
- siw_socket_disassoc(cep->sock);
- sock_release(cep->sock);
- cep->sock = NULL;
+ if (s) {
+ siw_socket_disassoc(s);
+ sock_release(s);
}
if (cep->cm_id) {
siw_free_cm_id(cep);
@@ -1511,7 +1515,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params)
if (cep) {
siw_socket_disassoc(s);
sock_release(s);
- cep->sock = NULL;
cep->qp = NULL;
@@ -1698,9 +1701,7 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
error_unlock:
up_write(&qp->state_lock);
error:
- siw_socket_disassoc(cep->sock);
- sock_release(cep->sock);
- cep->sock = NULL;
+ siw_destroy_cep_sock(cep);
cep->state = SIW_EPSTATE_CLOSED;
@@ -1745,9 +1746,7 @@ int siw_reject(struct iw_cm_id *id, const void *pdata, u8 pd_len)
cep->mpa.hdr.params.bits |= MPA_RR_FLAG_REJECT; /* reject */
siw_send_mpareqrep(cep, pdata, pd_len);
}
- siw_socket_disassoc(cep->sock);
- sock_release(cep->sock);
- cep->sock = NULL;
+ siw_destroy_cep_sock(cep);
cep->state = SIW_EPSTATE_CLOSED;
@@ -1879,7 +1878,6 @@ int siw_create_listen(struct iw_cm_id *id, int backlog)
siw_cep_set_inuse(cep);
siw_free_cm_id(cep);
- cep->sock = NULL;
siw_socket_disassoc(s);
cep->state = SIW_EPSTATE_CLOSED;
@@ -1900,6 +1898,7 @@ static void siw_drop_listeners(struct iw_cm_id *id)
*/
list_for_each_safe(p, tmp, (struct list_head *)id->provider_data) {
struct siw_cep *cep = list_entry(p, struct siw_cep, listenq);
+ struct socket *s = cep->sock;
list_del(p);
@@ -1908,10 +1907,9 @@ static void siw_drop_listeners(struct iw_cm_id *id)
siw_cep_set_inuse(cep);
siw_free_cm_id(cep);
- if (cep->sock) {
- siw_socket_disassoc(cep->sock);
- sock_release(cep->sock);
- cep->sock = NULL;
+ if (s) {
+ siw_socket_disassoc(s);
+ sock_release(s);
}
cep->state = SIW_EPSTATE_CLOSED;
siw_cep_set_free_and_put(cep);
diff --git a/drivers/infiniband/sw/siw/siw_qp_tx.c b/drivers/infiniband/sw/siw/siw_qp_tx.c
index b98a4100aeeb..1f06a7cb7650 100644
--- a/drivers/infiniband/sw/siw/siw_qp_tx.c
+++ b/drivers/infiniband/sw/siw/siw_qp_tx.c
@@ -338,6 +338,8 @@ static int siw_tcp_sendpages(struct socket *s, struct page **page, int offset,
msg.msg_flags &= ~MSG_MORE;
tcp_rate_check_app_limited(sk);
+ if (!sendpage_ok(page[i]))
+ msg.msg_flags &= ~MSG_SPLICE_PAGES;
bvec_set_page(&bvec, page[i], bytes, offset);
iov_iter_bvec(&msg.msg_iter, ITER_SOURCE, &bvec, 1, bytes);
diff --git a/drivers/input/keyboard/atkbd.c b/drivers/input/keyboard/atkbd.c
index 9333da3cabaf..2e0c69de35e9 100644
--- a/drivers/input/keyboard/atkbd.c
+++ b/drivers/input/keyboard/atkbd.c
@@ -1954,6 +1954,14 @@ static const struct dmi_system_id atkbd_dmi_quirk_table[] __initconst = {
},
.callback = atkbd_deactivate_fixup,
},
+ {
+ /* Lenovo IdeaPad Slim 3 15IWC11 (83RR) */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "83RR"),
+ },
+ .callback = atkbd_deactivate_fixup,
+ },
{
/* Xiaomi Book Pro 14 (TM2424) */
.matches = {
diff --git a/drivers/input/keyboard/lkkbd.c b/drivers/input/keyboard/lkkbd.c
index 047b654b3752..7b57577a1715 100644
--- a/drivers/input/keyboard/lkkbd.c
+++ b/drivers/input/keyboard/lkkbd.c
@@ -671,6 +671,7 @@ static int lkkbd_connect(struct serio *serio, struct serio_driver *drv)
return 0;
fail3: serio_close(serio);
+ cancel_work_sync(&lk->tq);
fail2: serio_set_drvdata(serio, NULL);
fail1: input_free_device(input_dev);
kfree(lk);
@@ -687,6 +688,7 @@ static void lkkbd_disconnect(struct serio *serio)
input_get_device(lk->dev);
input_unregister_device(lk->dev);
serio_close(serio);
+ cancel_work_sync(&lk->tq);
serio_set_drvdata(serio, NULL);
input_put_device(lk->dev);
kfree(lk);
diff --git a/drivers/input/mouse/alps.c b/drivers/input/mouse/alps.c
index dd08ce97e7c9..b23f8cf72fe9 100644
--- a/drivers/input/mouse/alps.c
+++ b/drivers/input/mouse/alps.c
@@ -2971,6 +2971,8 @@ static void alps_disconnect(struct psmouse *psmouse)
psmouse_reset(psmouse);
del_timer_sync(&priv->timer);
+ psmouse_set_state(psmouse, PSMOUSE_IGNORE);
+ cancel_delayed_work_sync(&priv->dev3_register_work);
if (priv->dev2)
input_unregister_device(priv->dev2);
if (!IS_ERR_OR_NULL(priv->dev3))
diff --git a/drivers/input/mouse/synaptics.c b/drivers/input/mouse/synaptics.c
index 8839c36c7342..d4a229481b1b 100644
--- a/drivers/input/mouse/synaptics.c
+++ b/drivers/input/mouse/synaptics.c
@@ -190,6 +190,7 @@ static const char * const smbus_pnp_ids[] = {
"LEN2054", /* E480 */
"LEN2055", /* E580 */
"LEN2058", /* E490 */
+ "LEN205b", /* T490 */
"LEN2068", /* T14 Gen 1 */
"SYN1221", /* TUXEDO InfinityBook Pro 14 v5 */
"SYN3003", /* HP EliteBook 850 G1 */
@@ -1810,8 +1811,10 @@ static int synaptics_setup_intertouch(struct psmouse *psmouse,
return -ENXIO;
}
- /* Disable intertouch on known-broken board revisions */
- if (info->board_id == 2722) {
+ /* Disable intertouch on known-broken T440p board revisions */
+ if (info->board_id == 2722 &&
+ psmouse_matches_pnp_id(psmouse,
+ (const char * const []){"LEN0036", NULL})) {
psmouse_info(psmouse,
"Disabling intertouch for board id %u\n",
info->board_id);
diff --git a/drivers/input/serio/i8042-acpipnpio.h b/drivers/input/serio/i8042-acpipnpio.h
index 37f7fcd8d88c..d44d17d3190a 100644
--- a/drivers/input/serio/i8042-acpipnpio.h
+++ b/drivers/input/serio/i8042-acpipnpio.h
@@ -670,6 +670,15 @@ static const struct dmi_system_id i8042_dmi_quirk_table[] __initconst = {
},
.driver_data = (void *)(SERIO_QUIRK_NOAUX)
},
+ {
+ /* Fujitsu LIFEBOOK U7410 */
+ /* Mux mode also disturbs the EC keyboard, causing spurious ACKs */
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "FUJITSU CLIENT COMPUTING LIMITED"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "LIFEBOOK U7410"),
+ },
+ .driver_data = (void *)(SERIO_QUIRK_NOMUX)
+ },
{
/* Gigabyte M912 */
.matches = {
diff --git a/drivers/input/touchscreen/s6sy761.c b/drivers/input/touchscreen/s6sy761.c
index 1a7d00289b4c..dbf9af973b1e 100644
--- a/drivers/input/touchscreen/s6sy761.c
+++ b/drivers/input/touchscreen/s6sy761.c
@@ -508,10 +508,22 @@ static int __maybe_unused s6sy761_suspend(struct device *dev)
static int __maybe_unused s6sy761_resume(struct device *dev)
{
struct s6sy761_data *sdata = dev_get_drvdata(dev);
+ int err;
+
+ err = s6sy761_power_on(sdata);
+ if (err)
+ return err;
enable_irq(sdata->client->irq);
- return s6sy761_power_on(sdata);
+ guard(mutex)(&sdata->input->mutex);
+
+ if (!input_device_enabled(sdata->input))
+ return 0;
+
+ err = i2c_smbus_write_byte(sdata->client, S6SY761_SENSE_ON);
+
+ return err;
}
static const struct dev_pm_ops s6sy761_pm_ops = {
diff --git a/drivers/interconnect/core.c b/drivers/interconnect/core.c
index 09143aae680e..180015512aa4 100644
--- a/drivers/interconnect/core.c
+++ b/drivers/interconnect/core.c
@@ -29,6 +29,7 @@ static LIST_HEAD(icc_providers);
static int providers_count;
static bool synced_state;
static DEFINE_MUTEX(icc_lock);
+static DEFINE_MUTEX(icc_bw_lock);
static struct dentry *icc_debugfs_dir;
static void icc_summary_show_one(struct seq_file *s, struct icc_node *n)
@@ -161,6 +162,8 @@ static struct icc_path *path_init(struct device *dev, struct icc_node *dst,
path->num_nodes = num_nodes;
+ mutex_lock(&icc_bw_lock);
+
for (i = num_nodes - 1; i >= 0; i--) {
node->provider->users++;
hlist_add_head(&path->reqs[i].req_node, &node->req_list);
@@ -171,6 +174,8 @@ static struct icc_path *path_init(struct device *dev, struct icc_node *dst,
node = node->reverse;
}
+ mutex_unlock(&icc_bw_lock);
+
return path;
}
@@ -635,7 +640,7 @@ int icc_set_bw(struct icc_path *path, u32 avg_bw, u32 peak_bw)
if (WARN_ON(IS_ERR(path) || !path->num_nodes))
return -EINVAL;
- mutex_lock(&icc_lock);
+ mutex_lock(&icc_bw_lock);
old_avg = path->reqs[0].avg_bw;
old_peak = path->reqs[0].peak_bw;
@@ -667,7 +672,7 @@ int icc_set_bw(struct icc_path *path, u32 avg_bw, u32 peak_bw)
apply_constraints(path);
}
- mutex_unlock(&icc_lock);
+ mutex_unlock(&icc_bw_lock);
trace_icc_set_bw_end(path, ret);
@@ -778,12 +783,16 @@ void icc_put(struct icc_path *path)
pr_err("%s: error (%d)\n", __func__, ret);
mutex_lock(&icc_lock);
+ mutex_lock(&icc_bw_lock);
+
for (i = 0; i < path->num_nodes; i++) {
node = path->reqs[i].node;
hlist_del(&path->reqs[i].req_node);
if (!WARN_ON(!node->provider->users))
node->provider->users--;
}
+
+ mutex_unlock(&icc_bw_lock);
mutex_unlock(&icc_lock);
kfree_const(path->name);
@@ -971,6 +980,7 @@ void icc_node_add(struct icc_node *node, struct icc_provider *provider)
return;
mutex_lock(&icc_lock);
+ mutex_lock(&icc_bw_lock);
node->provider = provider;
list_add_tail(&node->node_list, &provider->nodes);
@@ -996,6 +1006,7 @@ void icc_node_add(struct icc_node *node, struct icc_provider *provider)
node->avg_bw = 0;
node->peak_bw = 0;
+ mutex_unlock(&icc_bw_lock);
mutex_unlock(&icc_lock);
}
EXPORT_SYMBOL_GPL(icc_node_add);
@@ -1135,6 +1146,7 @@ void icc_sync_state(struct device *dev)
return;
mutex_lock(&icc_lock);
+ mutex_lock(&icc_bw_lock);
synced_state = true;
list_for_each_entry(p, &icc_providers, provider_list) {
dev_dbg(p->dev, "interconnect provider is in synced state\n");
diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c
index d4b6a32f258e..dc47692db143 100644
--- a/drivers/iommu/amd/iommu.c
+++ b/drivers/iommu/amd/iommu.c
@@ -217,10 +217,11 @@ static struct iommu_dev_data *search_dev_data(struct amd_iommu *iommu, u16 devid
return NULL;
}
-static int clone_alias(struct pci_dev *pdev, u16 alias, void *data)
+static int clone_alias(struct pci_dev *pdev_origin, u16 alias, void *data)
{
struct amd_iommu *iommu;
struct dev_table_entry *dev_table;
+ struct pci_dev *pdev = data;
u16 devid = pci_dev_id(pdev);
if (devid == alias)
@@ -252,9 +253,9 @@ static void clone_aliases(struct amd_iommu *iommu, struct device *dev)
* part of the PCI DMA aliases if it's bus differs
* from the original device.
*/
- clone_alias(pdev, iommu->pci_seg->alias_table[pci_dev_id(pdev)], NULL);
+ clone_alias(pdev, iommu->pci_seg->alias_table[pci_dev_id(pdev)], pdev);
- pci_for_each_dma_alias(pdev, clone_alias, NULL);
+ pci_for_each_dma_alias(pdev, clone_alias, pdev);
}
static void setup_aliases(struct amd_iommu *iommu, struct device *dev)
@@ -918,7 +919,12 @@ static int wait_on_sem(struct amd_iommu *iommu, u64 data)
{
int i = 0;
- while (*iommu->cmd_sem != data && i < LOOP_TIMEOUT) {
+ /*
+ * cmd_sem holds a monotonically non-decreasing completion sequence
+ * number.
+ */
+ while ((__s64)(READ_ONCE(*iommu->cmd_sem) - data) < 0 &&
+ i < LOOP_TIMEOUT) {
udelay(1);
i += 1;
}
@@ -1178,24 +1184,33 @@ static int iommu_completion_wait(struct amd_iommu *iommu)
int ret;
u64 data;
- if (!iommu->need_sync)
- return 0;
-
raw_spin_lock_irqsave(&iommu->lock, flags);
+ if (!iommu->need_sync) {
+ /*
+ * No command has been queued since the last completion-wait.
+ * A concurrent CPU may have already queued that CWAIT and
+ * cleared need_sync; need_sync == false only means a covering
+ * CWAIT is queued, not that all prior commands have completed.
+ * Wait for the last allocated sequence number so that any
+ * command queued before this call (possibly on another CPU)
+ * is guaranteed to have completed before returning.
+ */
+ data = iommu->cmd_sem_val;
+ raw_spin_unlock_irqrestore(&iommu->lock, flags);
+ return wait_on_sem(iommu, data);
+ }
+
data = ++iommu->cmd_sem_val;
build_completion_wait(&cmd, iommu, data);
ret = __iommu_queue_command_sync(iommu, &cmd, false);
- if (ret)
- goto out_unlock;
-
- ret = wait_on_sem(iommu, data);
-
-out_unlock:
raw_spin_unlock_irqrestore(&iommu->lock, flags);
- return ret;
+ if (ret)
+ return ret;
+
+ return wait_on_sem(iommu, data);
}
static int iommu_flush_dte(struct amd_iommu *iommu, u16 devid)
diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
index 33f3c5e6b332..79d907c57b6b 100644
--- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
+++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c
@@ -2968,6 +2968,13 @@ static int arm_smmu_cmdq_init(struct arm_smmu_device *smmu)
return 0;
}
+static void arm_smmu_free_iopf_action(void *data)
+{
+ struct iopf_queue *queue = data;
+
+ iopf_queue_free(queue);
+}
+
static int arm_smmu_init_queues(struct arm_smmu_device *smmu)
{
int ret;
@@ -2995,6 +3002,11 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu)
smmu->evtq.iopf = iopf_queue_alloc(dev_name(smmu->dev));
if (!smmu->evtq.iopf)
return -ENOMEM;
+ ret = devm_add_action_or_reset(smmu->dev,
+ arm_smmu_free_iopf_action,
+ smmu->evtq.iopf);
+ if (ret)
+ return ret;
}
/* priq */
@@ -3317,6 +3329,13 @@ static int arm_smmu_device_disable(struct arm_smmu_device *smmu)
return ret;
}
+static void arm_smmu_disable_action(void *data)
+{
+ struct arm_smmu_device *smmu = data;
+
+ arm_smmu_device_disable(smmu);
+}
+
static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass)
{
int ret;
@@ -3623,6 +3642,7 @@ static int arm_smmu_device_hw_probe(struct arm_smmu_device *smmu)
/* SID/SSID sizes */
smmu->ssid_bits = FIELD_GET(IDR1_SSIDSIZE, reg);
smmu->sid_bits = FIELD_GET(IDR1_SIDSIZE, reg);
+ smmu->iommu.max_pasids = 1UL << smmu->ssid_bits;
/*
* If the SMMU supports fewer bits than would fill a single L2 stream
@@ -3897,7 +3917,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
/* Initialise in-memory data structures */
ret = arm_smmu_init_structures(smmu);
if (ret)
- goto err_free_iopf;
+ return ret;
/* Record our private device structure */
platform_set_drvdata(pdev, smmu);
@@ -3907,30 +3927,30 @@ static int arm_smmu_device_probe(struct platform_device *pdev)
/* Reset the device */
ret = arm_smmu_device_reset(smmu, bypass);
+ if (ret) {
+ arm_smmu_device_disable(smmu);
+ return ret;
+ }
+
+ /* Register last so it unwinds first, while the CMDQ is still up. */
+ ret = devm_add_action_or_reset(smmu->dev, arm_smmu_disable_action, smmu);
if (ret)
- goto err_disable;
+ return ret;
/* And we're up. Go go go! */
ret = iommu_device_sysfs_add(&smmu->iommu, dev, NULL,
"smmu3.%pa", &ioaddr);
if (ret)
- goto err_disable;
+ return ret;
ret = iommu_device_register(&smmu->iommu, &arm_smmu_ops, dev);
if (ret) {
dev_err(dev, "Failed to register iommu\n");
- goto err_free_sysfs;
+ iommu_device_sysfs_remove(&smmu->iommu);
+ return ret;
}
return 0;
-
-err_free_sysfs:
- iommu_device_sysfs_remove(&smmu->iommu);
-err_disable:
- arm_smmu_device_disable(smmu);
-err_free_iopf:
- iopf_queue_free(smmu->evtq.iopf);
- return ret;
}
static int arm_smmu_device_remove(struct platform_device *pdev)
@@ -3939,8 +3959,6 @@ static int arm_smmu_device_remove(struct platform_device *pdev)
iommu_device_unregister(&smmu->iommu);
iommu_device_sysfs_remove(&smmu->iommu);
- arm_smmu_device_disable(smmu);
- iopf_queue_free(smmu->evtq.iopf);
return 0;
}
diff --git a/drivers/iommu/intel/dmar.c b/drivers/iommu/intel/dmar.c
index 47ac10fb76db..1d330852fb0b 100644
--- a/drivers/iommu/intel/dmar.c
+++ b/drivers/iommu/intel/dmar.c
@@ -912,6 +912,18 @@ dmar_validate_one_drhd(struct acpi_dmar_header *entry, void *arg)
return 0;
}
+static bool dmar_required(void)
+{
+ /* tboot supersedes any user/platform opt */
+ if (!intel_iommu_tboot_noforce && tboot_enabled())
+ return true;
+
+ if (!no_iommu && (!dmar_disabled || dmar_platform_optin()))
+ return true;
+
+ return false;
+}
+
void __init detect_intel_iommu(void)
{
int ret;
@@ -925,8 +937,7 @@ void __init detect_intel_iommu(void)
if (!ret)
ret = dmar_walk_dmar_table((struct acpi_table_dmar *)dmar_tbl,
&validate_drhd_cb);
- if (!ret && !no_iommu && !iommu_detected &&
- (!dmar_disabled || dmar_platform_optin())) {
+ if (!ret && !iommu_detected && dmar_required()) {
iommu_detected = 1;
/* Make sure ACS will be enabled */
pci_request_acs();
@@ -1106,6 +1117,13 @@ static int alloc_iommu(struct dmar_drhd_unit *drhd)
raw_spin_lock_init(&iommu->register_lock);
+ /*
+ * A value of N in PSS field of eCap register indicates hardware
+ * supports PASID field of N+1 bits.
+ */
+ if (pasid_supported(iommu))
+ iommu->iommu.max_pasids = 2UL << ecap_pss(iommu->ecap);
+
/*
* This is only for hotplug; at boot time intel_iommu_enabled won't
* be set yet. When intel_iommu_init() runs, it registers the units
diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c
index 9ef81870b20c..48b9d5e4a3cd 100644
--- a/drivers/iommu/intel/iommu.c
+++ b/drivers/iommu/intel/iommu.c
@@ -138,7 +138,7 @@ static int rwbf_quirk;
* (used when kernel is launched w/ TXT)
*/
static int force_on = 0;
-static int intel_iommu_tboot_noforce;
+int intel_iommu_tboot_noforce;
static int no_platform_optin;
#define ROOT_ENTRY_NR (VTD_PAGE_SIZE/sizeof(struct root_entry))
@@ -169,7 +169,26 @@ static phys_addr_t root_entry_uctp(struct root_entry *re)
static inline void context_set_present(struct context_entry *context)
{
- context->lo |= 1;
+ u64 val;
+
+ dma_wmb();
+ val = READ_ONCE(context->lo) | 1;
+ WRITE_ONCE(context->lo, val);
+}
+
+/*
+ * Clear the Present (P) bit (bit 0) of a context table entry. This initiates
+ * the transition of the entry's ownership from hardware to software. The
+ * caller is responsible for fulfilling the invalidation handshake recommended
+ * by the VT-d spec, Section 6.5.3.3 (Guidance to Software for Invalidations).
+ */
+static inline void context_clear_present(struct context_entry *context)
+{
+ u64 val;
+
+ val = READ_ONCE(context->lo) & GENMASK_ULL(63, 1);
+ WRITE_ONCE(context->lo, val);
+ dma_wmb();
}
static inline void context_set_fault_enable(struct context_entry *context)
@@ -2355,7 +2374,7 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8
did_old = context_domain_id(context);
}
- context_clear_entry(context);
+ context_clear_present(context);
__iommu_flush_cache(iommu, context, sizeof(*context));
spin_unlock(&iommu->lock);
iommu->flush.flush_context(iommu,
@@ -2374,6 +2393,9 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8
DMA_TLB_DSI_FLUSH);
__iommu_flush_dev_iotlb(info, 0, MAX_AGAW_PFN_WIDTH);
+
+ context_clear_entry(context);
+ __iommu_flush_cache(iommu, context, sizeof(*context));
}
static int domain_setup_first_level(struct intel_iommu *iommu,
diff --git a/drivers/iommu/intel/iommu.h b/drivers/iommu/intel/iommu.h
index c99cb715bd9a..080cbaafec43 100644
--- a/drivers/iommu/intel/iommu.h
+++ b/drivers/iommu/intel/iommu.h
@@ -480,8 +480,6 @@ enum {
#define VTD_FLAG_IRQ_REMAP_PRE_ENABLED (1 << 1)
#define VTD_FLAG_SVM_CAPABLE (1 << 2)
-extern int intel_iommu_sm;
-
#define sm_supported(iommu) (intel_iommu_sm && ecap_smts((iommu)->ecap))
#define pasid_supported(iommu) (sm_supported(iommu) && \
ecap_pasid((iommu)->ecap))
@@ -799,10 +797,12 @@ struct context_entry *iommu_context_addr(struct intel_iommu *iommu, u8 bus,
extern const struct iommu_ops intel_iommu_ops;
#ifdef CONFIG_INTEL_IOMMU
+extern int intel_iommu_sm;
extern int iommu_calculate_agaw(struct intel_iommu *iommu);
extern int iommu_calculate_max_sagaw(struct intel_iommu *iommu);
extern int dmar_disabled;
extern int intel_iommu_enabled;
+extern int intel_iommu_tboot_noforce;
#else
static inline int iommu_calculate_agaw(struct intel_iommu *iommu)
{
@@ -814,6 +814,8 @@ static inline int iommu_calculate_max_sagaw(struct intel_iommu *iommu)
}
#define dmar_disabled (1)
#define intel_iommu_enabled (0)
+#define intel_iommu_sm (0)
+#define intel_iommu_tboot_noforce (0)
#endif
static inline const char *decode_prq_descriptor(char *str, size_t size,
diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c
index 178ca6fadb73..e0647142c6af 100644
--- a/drivers/iommu/intel/pasid.c
+++ b/drivers/iommu/intel/pasid.c
@@ -262,7 +262,7 @@ intel_pasid_clear_entry(struct device *dev, u32 pasid, bool fault_ignore)
if (WARN_ON(!pe))
return;
- if (fault_ignore && pasid_pte_is_present(pe))
+ if (fault_ignore)
pasid_clear_entry_with_fpd(pe);
else
pasid_clear_entry(pe);
@@ -358,9 +358,23 @@ static inline void pasid_set_wpe(struct pasid_entry *pe)
*/
static inline void pasid_set_present(struct pasid_entry *pe)
{
+ dma_wmb();
pasid_set_bits(&pe->val[0], 1 << 0, 1);
}
+/*
+ * Clear the Present (P) bit (bit 0) of a scalable-mode PASID table entry.
+ * This initiates the transition of the entry's ownership from hardware
+ * to software. The caller is responsible for fulfilling the invalidation
+ * handshake recommended by the VT-d spec, Section 6.5.3.3 (Guidance to
+ * Software for Invalidations).
+ */
+static inline void pasid_clear_present(struct pasid_entry *pe)
+{
+ pasid_set_bits(&pe->val[0], 1 << 0, 0);
+ dma_wmb();
+}
+
/*
* Setup Page Walk Snoop bit (Bit 87) of a scalable mode PASID
* entry.
@@ -470,7 +484,7 @@ void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
did = pasid_get_domain_id(pte);
pgtt = pasid_pte_get_pgtt(pte);
- intel_pasid_clear_entry(dev, pasid, fault_ignore);
+ pasid_clear_present(pte);
spin_unlock(&iommu->lock);
if (!ecap_coherent(iommu->ecap))
@@ -486,6 +500,10 @@ void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev,
/* Device IOTLB doesn't need to be flushed in caching mode. */
if (!cap_caching_mode(iommu->cap))
devtlb_invalidation_with_pasid(iommu, dev, pasid);
+
+ intel_pasid_clear_entry(dev, pasid, fault_ignore);
+ if (!ecap_coherent(iommu->ecap))
+ clflush_cache_range(pte, sizeof(*pte));
}
/*
diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c
index 256ac3617d70..3d86c9d2c3d1 100644
--- a/drivers/iommu/msm_iommu.c
+++ b/drivers/iommu/msm_iommu.c
@@ -783,19 +783,24 @@ static int msm_iommu_probe(struct platform_device *pdev)
"msm-smmu.%pa", &ioaddr);
if (ret) {
pr_err("Could not add msm-smmu at %pa to sysfs\n", &ioaddr);
- goto fail;
+ goto err_remove_list;
}
ret = iommu_device_register(&iommu->iommu, &msm_iommu_ops, &pdev->dev);
if (ret) {
pr_err("Could not register msm-smmu at %pa\n", &ioaddr);
- goto fail;
+ goto err_remove_sysfs;
}
pr_info("device mapped at %p, irq %d with %d ctx banks\n",
iommu->base, iommu->irq, iommu->ncb);
return ret;
+
+err_remove_sysfs:
+ iommu_device_sysfs_remove(&iommu->iommu);
+err_remove_list:
+ list_del(&iommu->dev_node);
fail:
clk_unprepare(iommu->clk);
clk_unprepare(iommu->pclk);
diff --git a/drivers/leds/leds-an30259a.c b/drivers/leds/leds-an30259a.c
index e072ee5409f7..374011e3121e 100644
--- a/drivers/leds/leds-an30259a.c
+++ b/drivers/leds/leds-an30259a.c
@@ -296,7 +296,10 @@ static int an30259a_probe(struct i2c_client *client)
if (err < 0)
return err;
- mutex_init(&chip->mutex);
+ err = devm_mutex_init(&client->dev, &chip->mutex);
+ if (err)
+ return err;
+
chip->client = client;
i2c_set_clientdata(client, chip);
@@ -330,17 +333,9 @@ static int an30259a_probe(struct i2c_client *client)
return 0;
exit:
- mutex_destroy(&chip->mutex);
return err;
}
-static void an30259a_remove(struct i2c_client *client)
-{
- struct an30259a *chip = i2c_get_clientdata(client);
-
- mutex_destroy(&chip->mutex);
-}
-
static const struct of_device_id an30259a_match_table[] = {
{ .compatible = "panasonic,an30259a", },
{ /* sentinel */ },
@@ -360,7 +355,6 @@ static struct i2c_driver an30259a_driver = {
.of_match_table = of_match_ptr(an30259a_match_table),
},
.probe_new = an30259a_probe,
- .remove = an30259a_remove,
.id_table = an30259a_id,
};
diff --git a/drivers/mailbox/arm_mhu.c b/drivers/mailbox/arm_mhu.c
index 22243cabe056..537f7bfb7b06 100644
--- a/drivers/mailbox/arm_mhu.c
+++ b/drivers/mailbox/arm_mhu.c
@@ -12,6 +12,7 @@
#include <linux/io.h>
#include <linux/mailbox_controller.h>
#include <linux/module.h>
+#include <linux/of.h>
#define INTR_STAT_OFS 0x0
#define INTR_SET_OFS 0x8
diff --git a/drivers/mailbox/arm_mhu_db.c b/drivers/mailbox/arm_mhu_db.c
index aa0a4d83880f..27a510d46908 100644
--- a/drivers/mailbox/arm_mhu_db.c
+++ b/drivers/mailbox/arm_mhu_db.c
@@ -15,7 +15,6 @@
#include <linux/mailbox_controller.h>
#include <linux/module.h>
#include <linux/of.h>
-#include <linux/of_device.h>
#define INTR_STAT_OFS 0x0
#define INTR_SET_OFS 0x8
diff --git a/drivers/mailbox/hi3660-mailbox.c b/drivers/mailbox/hi3660-mailbox.c
index ab24e731a782..17c29e960fbf 100644
--- a/drivers/mailbox/hi3660-mailbox.c
+++ b/drivers/mailbox/hi3660-mailbox.c
@@ -11,6 +11,7 @@
#include <linux/iopoll.h>
#include <linux/mailbox_controller.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/platform_device.h>
#include <linux/slab.h>
diff --git a/drivers/mailbox/hi6220-mailbox.c b/drivers/mailbox/hi6220-mailbox.c
index fca61f5312d9..cf92523fd30b 100644
--- a/drivers/mailbox/hi6220-mailbox.c
+++ b/drivers/mailbox/hi6220-mailbox.c
@@ -15,6 +15,7 @@
#include <linux/kfifo.h>
#include <linux/mailbox_controller.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/platform_device.h>
#include <linux/slab.h>
diff --git a/drivers/mailbox/imx-mailbox.c b/drivers/mailbox/imx-mailbox.c
index 3292fb2dc3bf..5eef59f00abd 100644
--- a/drivers/mailbox/imx-mailbox.c
+++ b/drivers/mailbox/imx-mailbox.c
@@ -14,7 +14,8 @@
#include <linux/kernel.h>
#include <linux/mailbox_controller.h>
#include <linux/module.h>
-#include <linux/of_device.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
#include <linux/suspend.h>
#include <linux/slab.h>
diff --git a/drivers/mailbox/mailbox-mpfs.c b/drivers/mailbox/mailbox-mpfs.c
index 08aa840cccac..5fde9b9e6fea 100644
--- a/drivers/mailbox/mailbox-mpfs.c
+++ b/drivers/mailbox/mailbox-mpfs.c
@@ -14,6 +14,7 @@
#include <linux/module.h>
#include <linux/kernel.h>
#include <linux/interrupt.h>
+#include <linux/mod_devicetable.h>
#include <linux/platform_device.h>
#include <linux/mailbox_controller.h>
#include <soc/microchip/mpfs.h>
diff --git a/drivers/mailbox/mailbox.c b/drivers/mailbox/mailbox.c
index a00a02b6709c..0af89103b034 100644
--- a/drivers/mailbox/mailbox.c
+++ b/drivers/mailbox/mailbox.c
@@ -17,6 +17,7 @@
#include <linux/bitops.h>
#include <linux/mailbox_client.h>
#include <linux/mailbox_controller.h>
+#include <linux/of.h>
#include "mailbox.h"
diff --git a/drivers/mailbox/mtk-adsp-mailbox.c b/drivers/mailbox/mtk-adsp-mailbox.c
index 182045dd90df..8bcecddee0eb 100644
--- a/drivers/mailbox/mtk-adsp-mailbox.c
+++ b/drivers/mailbox/mtk-adsp-mailbox.c
@@ -10,7 +10,8 @@
#include <linux/kernel.h>
#include <linux/mailbox_controller.h>
#include <linux/module.h>
-#include <linux/of_device.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
#include <linux/slab.h>
struct mtk_adsp_mbox_priv {
diff --git a/drivers/mailbox/mtk-cmdq-mailbox.c b/drivers/mailbox/mtk-cmdq-mailbox.c
index 3d369c23970c..1eaab345f4a6 100644
--- a/drivers/mailbox/mtk-cmdq-mailbox.c
+++ b/drivers/mailbox/mtk-cmdq-mailbox.c
@@ -15,7 +15,7 @@
#include <linux/platform_device.h>
#include <linux/mailbox_controller.h>
#include <linux/mailbox/mtk-cmdq-mailbox.h>
-#include <linux/of_device.h>
+#include <linux/of.h>
#define CMDQ_OP_CODE_MASK (0xff << CMDQ_OP_CODE_SHIFT)
#define CMDQ_NUM_CMD(t) (t->cmd_buf_size / CMDQ_INST_SIZE)
diff --git a/drivers/mailbox/omap-mailbox.c b/drivers/mailbox/omap-mailbox.c
index 098c82d87137..2f8334c1ef5f 100644
--- a/drivers/mailbox/omap-mailbox.c
+++ b/drivers/mailbox/omap-mailbox.c
@@ -16,7 +16,7 @@
#include <linux/kfifo.h>
#include <linux/err.h>
#include <linux/module.h>
-#include <linux/of_device.h>
+#include <linux/of.h>
#include <linux/platform_device.h>
#include <linux/pm_runtime.h>
#include <linux/omap-mailbox.h>
diff --git a/drivers/mailbox/platform_mhu.c b/drivers/mailbox/platform_mhu.c
index a5922ac0b0bf..54161303a36b 100644
--- a/drivers/mailbox/platform_mhu.c
+++ b/drivers/mailbox/platform_mhu.c
@@ -15,6 +15,7 @@
#include <linux/slab.h>
#include <linux/err.h>
#include <linux/io.h>
+#include <linux/mod_devicetable.h>
#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/mailbox_controller.h>
diff --git a/drivers/mailbox/rockchip-mailbox.c b/drivers/mailbox/rockchip-mailbox.c
index e0e9ba8f8999..99f049f76f0c 100644
--- a/drivers/mailbox/rockchip-mailbox.c
+++ b/drivers/mailbox/rockchip-mailbox.c
@@ -8,8 +8,8 @@
#include <linux/io.h>
#include <linux/kernel.h>
#include <linux/mailbox_controller.h>
+#include <linux/of.h>
#include <linux/module.h>
-#include <linux/of_device.h>
#include <linux/platform_device.h>
#define MAILBOX_A2B_INTEN 0x00
diff --git a/drivers/mailbox/sprd-mailbox.c b/drivers/mailbox/sprd-mailbox.c
index 2a43fb375bcb..109ff60c4ac7 100644
--- a/drivers/mailbox/sprd-mailbox.c
+++ b/drivers/mailbox/sprd-mailbox.c
@@ -11,7 +11,7 @@
#include <linux/io.h>
#include <linux/mailbox_controller.h>
#include <linux/module.h>
-#include <linux/of_device.h>
+#include <linux/of.h>
#include <linux/platform_device.h>
#include <linux/clk.h>
diff --git a/drivers/mailbox/stm32-ipcc.c b/drivers/mailbox/stm32-ipcc.c
index 15d538fe2113..4ad3653f3866 100644
--- a/drivers/mailbox/stm32-ipcc.c
+++ b/drivers/mailbox/stm32-ipcc.c
@@ -11,6 +11,7 @@
#include <linux/io.h>
#include <linux/mailbox_controller.h>
#include <linux/module.h>
+#include <linux/of.h>
#include <linux/platform_device.h>
#include <linux/pm_wakeirq.h>
diff --git a/drivers/mailbox/tegra-hsp.c b/drivers/mailbox/tegra-hsp.c
index e12cd1a62155..96d034992aaf 100644
--- a/drivers/mailbox/tegra-hsp.c
+++ b/drivers/mailbox/tegra-hsp.c
@@ -8,7 +8,6 @@
#include <linux/io.h>
#include <linux/mailbox_controller.h>
#include <linux/of.h>
-#include <linux/of_device.h>
#include <linux/platform_device.h>
#include <linux/pm.h>
#include <linux/slab.h>
diff --git a/drivers/mailbox/zynqmp-ipi-mailbox.c b/drivers/mailbox/zynqmp-ipi-mailbox.c
index 6a48b832950d..c89989c23411 100644
--- a/drivers/mailbox/zynqmp-ipi-mailbox.c
+++ b/drivers/mailbox/zynqmp-ipi-mailbox.c
@@ -16,8 +16,6 @@
#include <linux/module.h>
#include <linux/of.h>
#include <linux/of_address.h>
-#include <linux/of_device.h>
-#include <linux/of_irq.h>
#include <linux/platform_device.h>
/* IPI agent ID any */
@@ -74,6 +72,10 @@ struct zynqmp_ipi_mchan {
unsigned int chan_type;
};
+struct zynqmp_ipi_mbox;
+
+typedef int (*setup_ipi_fn)(struct zynqmp_ipi_mbox *ipi_mbox, struct device_node *node);
+
/**
* struct zynqmp_ipi_mbox - Description of a ZynqMP IPI mailbox
* platform data.
@@ -83,7 +85,7 @@ struct zynqmp_ipi_mchan {
* @remote_id: remote IPI agent ID
* @mbox: mailbox Controller
* @mchans: array for channels, tx channel and rx channel.
- * @irq: IPI agent interrupt ID
+ * @setup_ipi_fn: Function Pointer to set up IPI Channels
*/
struct zynqmp_ipi_mbox {
struct zynqmp_ipi_pdata *pdata;
@@ -91,6 +93,7 @@ struct zynqmp_ipi_mbox {
u32 remote_id;
struct mbox_controller mbox;
struct zynqmp_ipi_mchan mchans[2];
+ setup_ipi_fn setup_ipi_fn;
};
/**
@@ -418,12 +421,6 @@ static struct mbox_chan *zynqmp_ipi_of_xlate(struct mbox_controller *mbox,
return chan;
}
-static const struct of_device_id zynqmp_ipi_of_match[] = {
- { .compatible = "xlnx,zynqmp-ipi-mailbox" },
- {},
-};
-MODULE_DEVICE_TABLE(of, zynqmp_ipi_of_match);
-
/**
* zynqmp_ipi_mbox_get_buf_res - Get buffer resource from the IPI dev node
*
@@ -473,12 +470,9 @@ static void zynqmp_ipi_mbox_dev_release(struct device *dev)
static int zynqmp_ipi_mbox_probe(struct zynqmp_ipi_mbox *ipi_mbox,
struct device_node *node)
{
- struct zynqmp_ipi_mchan *mchan;
struct mbox_chan *chans;
struct mbox_controller *mbox;
- struct resource res;
struct device *dev, *mdev;
- const char *name;
int ret;
dev = ipi_mbox->pdata->dev;
@@ -498,6 +492,73 @@ static int zynqmp_ipi_mbox_probe(struct zynqmp_ipi_mbox *ipi_mbox,
}
mdev = &ipi_mbox->dev;
+ /* Get the IPI remote agent ID */
+ ret = of_property_read_u32(node, "xlnx,ipi-id", &ipi_mbox->remote_id);
+ if (ret < 0) {
+ dev_err(dev, "No IPI remote ID is specified.\n");
+ return ret;
+ }
+
+ ret = ipi_mbox->setup_ipi_fn(ipi_mbox, node);
+ if (ret) {
+ dev_err(dev, "Failed to set up IPI Buffers.\n");
+ return ret;
+ }
+
+ mbox = &ipi_mbox->mbox;
+ mbox->dev = mdev;
+ mbox->ops = &zynqmp_ipi_chan_ops;
+ mbox->num_chans = 2;
+ mbox->txdone_irq = false;
+ mbox->txdone_poll = true;
+ mbox->txpoll_period = 5;
+ mbox->of_xlate = zynqmp_ipi_of_xlate;
+ chans = devm_kzalloc(mdev, 2 * sizeof(*chans), GFP_KERNEL);
+ if (!chans)
+ return -ENOMEM;
+ mbox->chans = chans;
+ chans[IPI_MB_CHNL_TX].con_priv = &ipi_mbox->mchans[IPI_MB_CHNL_TX];
+ chans[IPI_MB_CHNL_RX].con_priv = &ipi_mbox->mchans[IPI_MB_CHNL_RX];
+ ipi_mbox->mchans[IPI_MB_CHNL_TX].chan_type = IPI_MB_CHNL_TX;
+ ipi_mbox->mchans[IPI_MB_CHNL_RX].chan_type = IPI_MB_CHNL_RX;
+ ret = devm_mbox_controller_register(mdev, mbox);
+ if (ret)
+ dev_err(mdev,
+ "Failed to register mbox_controller(%d)\n", ret);
+ else
+ dev_info(mdev,
+ "Registered ZynqMP IPI mbox with TX/RX channels.\n");
+ return ret;
+}
+
+/**
+ * zynqmp_ipi_setup - set up IPI Buffers for classic flow
+ *
+ * @ipi_mbox: pointer to IPI mailbox private data structure
+ * @node: IPI mailbox device node
+ *
+ * This will be used to set up IPI Buffers for ZynqMP SOC if user
+ * wishes to use classic driver usage model on new SOC's with only
+ * buffered IPIs.
+ *
+ * Note that bufferless IPIs and mixed usage of buffered and bufferless
+ * IPIs are not supported with this flow.
+ *
+ * This will be invoked with compatible string "xlnx,zynqmp-ipi-mailbox".
+ *
+ * Return: 0 for success, negative value for failure
+ */
+static int zynqmp_ipi_setup(struct zynqmp_ipi_mbox *ipi_mbox,
+ struct device_node *node)
+{
+ struct zynqmp_ipi_mchan *mchan;
+ struct device *mdev;
+ struct resource res;
+ const char *name;
+ int ret;
+
+ mdev = &ipi_mbox->dev;
+
mchan = &ipi_mbox->mchans[IPI_MB_CHNL_TX];
name = "local_request_region";
ret = zynqmp_ipi_mbox_get_buf_res(node, name, &res);
@@ -572,37 +633,7 @@ static int zynqmp_ipi_mbox_probe(struct zynqmp_ipi_mbox *ipi_mbox,
if (!mchan->rx_buf)
return -ENOMEM;
- /* Get the IPI remote agent ID */
- ret = of_property_read_u32(node, "xlnx,ipi-id", &ipi_mbox->remote_id);
- if (ret < 0) {
- dev_err(dev, "No IPI remote ID is specified.\n");
- return ret;
- }
-
- mbox = &ipi_mbox->mbox;
- mbox->dev = mdev;
- mbox->ops = &zynqmp_ipi_chan_ops;
- mbox->num_chans = 2;
- mbox->txdone_irq = false;
- mbox->txdone_poll = true;
- mbox->txpoll_period = 5;
- mbox->of_xlate = zynqmp_ipi_of_xlate;
- chans = devm_kzalloc(mdev, 2 * sizeof(*chans), GFP_KERNEL);
- if (!chans)
- return -ENOMEM;
- mbox->chans = chans;
- chans[IPI_MB_CHNL_TX].con_priv = &ipi_mbox->mchans[IPI_MB_CHNL_TX];
- chans[IPI_MB_CHNL_RX].con_priv = &ipi_mbox->mchans[IPI_MB_CHNL_RX];
- ipi_mbox->mchans[IPI_MB_CHNL_TX].chan_type = IPI_MB_CHNL_TX;
- ipi_mbox->mchans[IPI_MB_CHNL_RX].chan_type = IPI_MB_CHNL_RX;
- ret = devm_mbox_controller_register(mdev, mbox);
- if (ret)
- dev_err(mdev,
- "Failed to register mbox_controller(%d)\n", ret);
- else
- dev_info(mdev,
- "Registered ZynqMP IPI mbox with TX/RX channels.\n");
- return ret;
+ return 0;
}
/**
@@ -615,7 +646,7 @@ static void zynqmp_ipi_free_mboxes(struct zynqmp_ipi_pdata *pdata)
struct zynqmp_ipi_mbox *ipi_mbox;
int i;
- i = pdata->num_mboxes;
+ i = pdata->num_mboxes - 1;
for (; i >= 0; i--) {
ipi_mbox = &pdata->ipi_mboxes[i];
if (device_is_registered(&ipi_mbox->dev))
@@ -630,6 +661,7 @@ static int zynqmp_ipi_probe(struct platform_device *pdev)
struct zynqmp_ipi_pdata *pdata;
struct zynqmp_ipi_mbox *mbox;
int num_mboxes, ret = -EINVAL;
+ setup_ipi_fn ipi_fn;
num_mboxes = of_get_available_child_count(np);
if (num_mboxes == 0) {
@@ -650,9 +682,18 @@ static int zynqmp_ipi_probe(struct platform_device *pdev)
return ret;
}
+ ipi_fn = (setup_ipi_fn)device_get_match_data(&pdev->dev);
+ if (!ipi_fn) {
+ dev_err(dev,
+ "Mbox Compatible String is missing IPI Setup fn.\n");
+ return -ENODEV;
+ }
+
pdata->num_mboxes = num_mboxes;
mbox = pdata->ipi_mboxes;
+ mbox->setup_ipi_fn = ipi_fn;
+
for_each_available_child_of_node(np, nc) {
mbox->pdata = pdata;
ret = zynqmp_ipi_mbox_probe(mbox, nc);
@@ -697,6 +738,14 @@ static int zynqmp_ipi_remove(struct platform_device *pdev)
return 0;
}
+static const struct of_device_id zynqmp_ipi_of_match[] = {
+ { .compatible = "xlnx,zynqmp-ipi-mailbox",
+ .data = &zynqmp_ipi_setup,
+ },
+ {},
+};
+MODULE_DEVICE_TABLE(of, zynqmp_ipi_of_match);
+
static struct platform_driver zynqmp_ipi_driver = {
.probe = zynqmp_ipi_probe,
.remove = zynqmp_ipi_remove,
diff --git a/drivers/md/dm-bio-prison-v1.c b/drivers/md/dm-bio-prison-v1.c
index 138067abe14b..b9bb3ce2ee84 100644
--- a/drivers/md/dm-bio-prison-v1.c
+++ b/drivers/md/dm-bio-prison-v1.c
@@ -77,9 +77,9 @@ static void __setup_new_cell(struct dm_cell_key *key,
struct bio *holder,
struct dm_bio_prison_cell *cell)
{
- memcpy(&cell->key, key, sizeof(cell->key));
- cell->holder = holder;
- bio_list_init(&cell->bios);
+ memcpy(&cell->key, key, sizeof(cell->key));
+ cell->holder = holder;
+ bio_list_init(&cell->bios);
}
static int cmp_keys(struct dm_cell_key *lhs,
diff --git a/drivers/md/dm-bufio.c b/drivers/md/dm-bufio.c
index f1d34e427ef1..bbcc33828c63 100644
--- a/drivers/md/dm-bufio.c
+++ b/drivers/md/dm-bufio.c
@@ -258,9 +258,11 @@ static void buffer_record_stack(struct dm_buffer *b)
}
#endif
-/*----------------------------------------------------------------
+/*
+ *----------------------------------------------------------------
* A red/black tree acts as an index for all the buffers.
- *--------------------------------------------------------------*/
+ *----------------------------------------------------------------
+ */
static struct dm_buffer *__find(struct dm_bufio_client *c, sector_t block)
{
struct rb_node *n = c->buffer_tree.rb_node;
@@ -561,7 +563,8 @@ static void __relink_lru(struct dm_buffer *b, int dirty)
b->last_accessed = jiffies;
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------------------
* Submit I/O on the buffer.
*
* Bio interface is faster but it has some problems:
@@ -577,7 +580,8 @@ static void __relink_lru(struct dm_buffer *b, int dirty)
* rejects the bio because it is too large, use dm-io layer to do the I/O.
* The dm-io layer splits the I/O into multiple requests, avoiding the above
* shortcomings.
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------------------
+ */
/*
* dm-io completion routine. It just calls b->bio.bi_end_io, pretending
@@ -720,9 +724,11 @@ static void submit_io(struct dm_buffer *b, enum req_op op,
use_dmio(b, op, sector, n_sectors, offset);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Writing dirty buffers
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
/*
* The endio routine for write.
@@ -1003,9 +1009,11 @@ static void __check_watermark(struct dm_bufio_client *c,
__write_dirty_buffers_async(c, 1, write_list);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Getting a buffer
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static struct dm_buffer *__bufio_new(struct dm_bufio_client *c, sector_t block,
enum new_flag nf, int *need_submit,
@@ -2080,9 +2088,11 @@ static void work_fn(struct work_struct *w)
DM_BUFIO_WORK_TIMER_SECS * HZ);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Module setup
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
/*
* This is called only once for the whole dm_bufio module.
diff --git a/drivers/md/dm-cache-metadata.c b/drivers/md/dm-cache-metadata.c
index 4ffe3f0bf1cd..351dff329f3c 100644
--- a/drivers/md/dm-cache-metadata.c
+++ b/drivers/md/dm-cache-metadata.c
@@ -162,10 +162,11 @@ struct dm_cache_metadata {
struct dm_bitset_cursor dirty_cursor;
};
-/*-------------------------------------------------------------------
+/*
+ *-----------------------------------------------------------------
* superblock validator
- *-----------------------------------------------------------------*/
-
+ *-----------------------------------------------------------------
+ */
#define SUPERBLOCK_CSUM_XOR 9031977
static void sb_prepare_for_write(struct dm_block_validator *v,
diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c
index f0eb64f04340..d51417d284d5 100644
--- a/drivers/md/dm-cache-target.c
+++ b/drivers/md/dm-cache-target.c
@@ -180,14 +180,14 @@ static void continue_after_commit(struct batcher *b, struct continuation *k)
*/
static void issue_after_commit(struct batcher *b, struct bio *bio)
{
- bool commit_scheduled;
+ bool commit_scheduled;
- spin_lock_irq(&b->lock);
- commit_scheduled = b->commit_scheduled;
- bio_list_add(&b->bios, bio);
- spin_unlock_irq(&b->lock);
+ spin_lock_irq(&b->lock);
+ commit_scheduled = b->commit_scheduled;
+ bio_list_add(&b->bios, bio);
+ spin_unlock_irq(&b->lock);
- if (commit_scheduled)
+ if (commit_scheduled)
async_commit(b);
}
@@ -338,8 +338,6 @@ struct cache {
struct list_head invalidation_requests;
sector_t migration_threshold;
- wait_queue_head_t migration_wait;
- atomic_t nr_allocated_migrations;
/*
* The number of in flight migrations that are performing
@@ -395,7 +393,11 @@ struct cache {
bool loaded_mappings:1;
bool loaded_discards:1;
- struct rw_semaphore background_work_lock;
+ /* background work management */
+ bool background_work_allowed;
+ unsigned background_work_nr;
+ spinlock_t background_work_lock;
+ wait_queue_head_t background_work_wait;
struct batcher committer;
struct work_struct commit_ws;
@@ -486,19 +488,13 @@ static struct dm_cache_migration *alloc_migration(struct cache *cache)
memset(mg, 0, sizeof(*mg));
mg->cache = cache;
- atomic_inc(&cache->nr_allocated_migrations);
return mg;
}
static void free_migration(struct dm_cache_migration *mg)
{
- struct cache *cache = mg->cache;
-
- if (atomic_dec_and_test(&cache->nr_allocated_migrations))
- wake_up(&cache->migration_wait);
-
- mempool_free(mg, &cache->migration_pool);
+ mempool_free(mg, &mg->cache->migration_pool);
}
/*----------------------------------------------------------------*/
@@ -530,9 +526,11 @@ static unsigned int lock_level(struct bio *bio)
READ_WRITE_LOCK_LEVEL;
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Per bio data
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static struct per_bio_data *get_per_bio_data(struct bio *bio)
{
@@ -714,9 +712,11 @@ static bool is_discarded_oblock(struct cache *cache, dm_oblock_t b)
return r;
}
-/*----------------------------------------------------------------
+/*
+ * -------------------------------------------------------------
* Remapping
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void remap_to_origin(struct cache *cache, struct bio *bio)
{
bio_set_dev(bio, cache->origin_dev->bdev);
@@ -843,9 +843,11 @@ static void remap_to_origin_and_cache(struct cache *cache, struct bio *bio,
remap_to_cache(cache, bio, cblock);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Failure modes
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static enum cache_metadata_mode get_cache_mode(struct cache *cache)
{
return cache->features.mode;
@@ -982,13 +984,14 @@ static void update_stats(struct cache_stats *stats, enum policy_operation op)
}
}
-/*----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------------
* Migration processing
*
* Migration covers moving data from the origin device to the cache, or
* vice versa.
- *--------------------------------------------------------------*/
-
+ *---------------------------------------------------------------------
+ */
static void inc_io_migrations(struct cache *cache)
{
atomic_inc(&cache->nr_io_migrations);
@@ -1022,34 +1025,39 @@ static void calc_discard_block_range(struct cache *cache, struct bio *bio,
static void prevent_background_work(struct cache *cache)
{
- lockdep_off();
- down_write(&cache->background_work_lock);
- lockdep_on();
+ spin_lock_irq(&cache->background_work_lock);
+ cache->background_work_allowed = false;
+ wait_event_lock_irq(cache->background_work_wait,
+ cache->background_work_nr == 0,
+ cache->background_work_lock);
+ spin_unlock_irq(&cache->background_work_lock);
}
static void allow_background_work(struct cache *cache)
{
- lockdep_off();
- up_write(&cache->background_work_lock);
- lockdep_on();
+ spin_lock_irq(&cache->background_work_lock);
+ cache->background_work_allowed = true;
+ spin_unlock_irq(&cache->background_work_lock);
}
static bool background_work_begin(struct cache *cache)
{
bool r;
- lockdep_off();
- r = down_read_trylock(&cache->background_work_lock);
- lockdep_on();
-
+ spin_lock_irq(&cache->background_work_lock);
+ r = cache->background_work_allowed;
+ if (r)
+ cache->background_work_nr++;
+ spin_unlock_irq(&cache->background_work_lock);
return r;
}
static void background_work_end(struct cache *cache)
{
- lockdep_off();
- up_read(&cache->background_work_lock);
- lockdep_on();
+ spin_lock_irq(&cache->background_work_lock);
+ if (--cache->background_work_nr == 0)
+ wake_up(&cache->background_work_wait);
+ spin_unlock_irq(&cache->background_work_lock);
}
/*----------------------------------------------------------------*/
@@ -1443,15 +1451,20 @@ static int mg_start(struct cache *cache, struct policy_work *op, struct bio *bio
return mg_lock_writes(mg);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* invalidation processing
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void invalidate_complete(struct dm_cache_migration *mg, bool success)
{
struct bio_list bios;
struct cache *cache = mg->cache;
+ if (success)
+ atomic_inc(&cache->stats.demotion);
+
bio_list_init(&bios);
if (mg->cell) {
if (dm_cell_unlock_v2(cache->prison, mg->cell, &bios))
@@ -1602,9 +1615,11 @@ static int invalidate_start(struct cache *cache, dm_cblock_t cblock,
return invalidate_lock(mg);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* bio processing
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
enum busy {
IDLE,
@@ -1721,7 +1736,6 @@ static int map_bio(struct cache *cache, struct bio *bio, dm_oblock_t block,
if (passthrough_mode(cache)) {
if (bio_data_dir(bio) == WRITE) {
bio_drop_shared_lock(cache, bio);
- atomic_inc(&cache->stats.demotion);
invalidate_start(cache, cblock, block, bio);
return DM_MAPIO_SUBMITTED;
} else
@@ -1813,9 +1827,11 @@ static bool process_discard_bio(struct cache *cache, struct bio *bio)
{
dm_dblock_t b, e;
- // FIXME: do we need to lock the region? Or can we just assume the
- // user wont be so foolish as to issue discard concurrently with
- // other IO?
+ /*
+ * FIXME: do we need to lock the region? Or can we just assume the
+ * user wont be so foolish as to issue discard concurrently with
+ * other IO?
+ */
calc_discard_block_range(cache, bio, &b, &e);
while (b != e) {
set_discard(cache, b);
@@ -1862,10 +1878,11 @@ static void process_deferred_bios(struct work_struct *ws)
schedule_commit(&cache->committer);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Main worker loop
- *--------------------------------------------------------------*/
-
+ *--------------------------------------------------------------
+ */
static void requeue_deferred_bios(struct cache *cache)
{
struct bio *bio;
@@ -1924,9 +1941,11 @@ static void check_migrations(struct work_struct *ws)
}
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Target methods
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
/*
* This function gets called on the error paths of the constructor, so we
@@ -2499,9 +2518,7 @@ static int cache_create(struct cache_args *ca, struct cache **result)
spin_lock_init(&cache->lock);
bio_list_init(&cache->deferred_bios);
- atomic_set(&cache->nr_allocated_migrations, 0);
atomic_set(&cache->nr_io_migrations, 0);
- init_waitqueue_head(&cache->migration_wait);
r = -ENOMEM;
atomic_set(&cache->nr_dirty, 0);
@@ -2583,8 +2600,10 @@ static int cache_create(struct cache_args *ca, struct cache **result)
issue_op, cache, cache->wq);
dm_iot_init(&cache->tracker);
- init_rwsem(&cache->background_work_lock);
- prevent_background_work(cache);
+ init_waitqueue_head(&cache->background_work_wait);
+ spin_lock_init(&cache->background_work_lock);
+ cache->background_work_allowed = false;
+ cache->background_work_nr = 0;
*result = cache;
return 0;
diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
index 449e012ade53..ade39bbe4911 100644
--- a/drivers/md/dm-crypt.c
+++ b/drivers/md/dm-crypt.c
@@ -368,9 +368,10 @@ static int crypt_iv_benbi_ctr(struct crypt_config *cc, struct dm_target *ti,
bs = crypto_skcipher_blocksize(any_tfm(cc));
log = ilog2(bs);
- /* we need to calculate how far we must shift the sector count
- * to get the cipher block count, we use this shift in _gen */
-
+ /*
+ * We need to calculate how far we must shift the sector count
+ * to get the cipher block count, we use this shift in _gen.
+ */
if (1 << log != bs) {
ti->error = "cypher blocksize is not a power of 2";
return -EINVAL;
@@ -1757,7 +1758,6 @@ static void crypt_dec_pending(struct dm_crypt_io *io)
{
struct crypt_config *cc = io->cc;
struct bio *base_bio = io->base_bio;
- blk_status_t error = io->error;
if (!atomic_dec_and_test(&io->io_pending))
return;
@@ -1779,7 +1779,7 @@ static void crypt_dec_pending(struct dm_crypt_io *io)
else
kfree(io->integrity_metadata);
- base_bio->bi_status = error;
+ base_bio->bi_status = io->error;
bio_endio(base_bio);
}
@@ -2914,8 +2914,13 @@ static int crypt_ctr_cipher_new(struct dm_target *ti, char *cipher_in, char *key
}
}
- if (*ivmode && !strcmp(*ivmode, "lmk"))
+ if (*ivmode && !strcmp(*ivmode, "lmk")) {
+ if (crypt_integrity_aead(cc)) {
+ ti->error = "AEAD transforms not supported for LMK";
+ return -EINVAL;
+ }
cc->tfms_count = 64;
+ }
if (*ivmode && !strcmp(*ivmode, "essiv")) {
if (!*ivopts) {
diff --git a/drivers/md/dm-ebs-target.c b/drivers/md/dm-ebs-target.c
index 8bf44aa2f283..2c6cccca5f88 100644
--- a/drivers/md/dm-ebs-target.c
+++ b/drivers/md/dm-ebs-target.c
@@ -241,7 +241,7 @@ static void __ebs_process_bios(struct work_struct *ws)
* <offset>: offset in 512 bytes sectors into <dev_path>
* <ebs>: emulated block size in units of 512 bytes exposed to the upper layer
* [<ubs>]: underlying block size in units of 512 bytes imposed on the lower layer;
- * optional, if not supplied, retrieve logical block size from underlying device
+ * optional, if not supplied, retrieve logical block size from underlying device
*/
static int ebs_ctr(struct dm_target *ti, unsigned int argc, char **argv)
{
@@ -264,8 +264,7 @@ static int ebs_ctr(struct dm_target *ti, unsigned int argc, char **argv)
r = -EINVAL;
if (sscanf(argv[1], "%llu%c", &tmp, &dummy) != 1 ||
- tmp != (sector_t)tmp ||
- (sector_t)tmp >= ti->len) {
+ tmp != (sector_t)tmp) {
ti->error = "Invalid device offset sector";
goto bad;
}
diff --git a/drivers/md/dm-era-target.c b/drivers/md/dm-era-target.c
index a1ccee10bb24..f3b229e55020 100644
--- a/drivers/md/dm-era-target.c
+++ b/drivers/md/dm-era-target.c
@@ -22,9 +22,11 @@
#define INVALID_WRITESET_ROOT SUPERBLOCK_LOCATION
#define MIN_BLOCK_SIZE 8
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Writeset
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
struct writeset_metadata {
uint32_t nr_bits;
dm_block_t root;
@@ -149,9 +151,11 @@ static int writeset_test_and_set(struct dm_disk_bitset *info,
return 1;
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* On disk metadata layout
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
#define SPACE_MAP_ROOT_SIZE 128
#define UUID_LEN 16
@@ -187,9 +191,11 @@ struct superblock_disk {
__le64 metadata_snap;
} __packed;
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Superblock validation
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void sb_prepare_for_write(struct dm_block_validator *v,
struct dm_block *b,
size_t sb_block_size)
@@ -254,9 +260,11 @@ static struct dm_block_validator sb_validator = {
.check = sb_check
};
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Low level metadata handling
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
#define DM_ERA_METADATA_BLOCK_SIZE 4096
#define ERA_MAX_CONCURRENT_LOCKS 5
@@ -661,13 +669,15 @@ static void swap_writeset(struct era_metadata *md, struct writeset *new_writeset
synchronize_rcu();
}
-/*----------------------------------------------------------------
+/*
+ *------------------------------------------------------------------------
* Writesets get 'digested' into the main era array.
*
* We're using a coroutine here so the worker thread can do the digestion,
* thus avoiding synchronisation of the metadata. Digesting a whole
* writeset in one go would cause too much latency.
- *--------------------------------------------------------------*/
+ *------------------------------------------------------------------------
+ */
struct digest {
uint32_t era;
unsigned int nr_bits, current_bit;
@@ -787,10 +797,12 @@ static int metadata_digest_start(struct era_metadata *md, struct digest *d)
return 0;
}
-/*----------------------------------------------------------------
- * High level metadata interface. Target methods should use these, and not
- * the lower level ones.
- *--------------------------------------------------------------*/
+/*
+ *-----------------------------------------------------------------
+ * High level metadata interface. Target methods should use these,
+ * and not the lower level ones.
+ *-----------------------------------------------------------------
+ */
static struct era_metadata *metadata_open(struct block_device *bdev,
sector_t block_size,
bool may_format)
@@ -1199,9 +1211,11 @@ struct rpc {
struct completion complete;
};
-/*----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Remapping.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static bool block_size_is_power_of_two(struct era *era)
{
return era->sectors_per_block_shift >= 0;
@@ -1225,9 +1239,11 @@ static void remap_to_origin(struct era *era, struct bio *bio)
bio->bi_iter.bi_sector = dm_target_offset(era->ti, bio->bi_iter.bi_sector);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Worker thread
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void wake_worker(struct era *era)
{
if (!atomic_read(&era->suspended))
@@ -1416,9 +1432,11 @@ static void stop_worker(struct era *era)
drain_workqueue(era->wq);
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Target methods
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void era_destroy(struct era *era)
{
if (era->md)
diff --git a/drivers/md/dm-exception-store.h b/drivers/md/dm-exception-store.h
index 862df68a7db0..07fedb55676c 100644
--- a/drivers/md/dm-exception-store.h
+++ b/drivers/md/dm-exception-store.h
@@ -43,36 +43,36 @@ struct dm_exception_store_type {
const char *name;
struct module *module;
- int (*ctr) (struct dm_exception_store *store, char *options);
+ int (*ctr)(struct dm_exception_store *store, char *options);
/*
* Destroys this object when you've finished with it.
*/
- void (*dtr) (struct dm_exception_store *store);
+ void (*dtr)(struct dm_exception_store *store);
/*
* The target shouldn't read the COW device until this is
* called. As exceptions are read from the COW, they are
* reported back via the callback.
*/
- int (*read_metadata) (struct dm_exception_store *store,
- int (*callback)(void *callback_context,
- chunk_t old, chunk_t new),
- void *callback_context);
+ int (*read_metadata)(struct dm_exception_store *store,
+ int (*callback)(void *callback_context,
+ chunk_t old, chunk_t new),
+ void *callback_context);
/*
* Find somewhere to store the next exception.
*/
- int (*prepare_exception) (struct dm_exception_store *store,
- struct dm_exception *e);
+ int (*prepare_exception)(struct dm_exception_store *store,
+ struct dm_exception *e);
/*
* Update the metadata with this exception.
*/
- void (*commit_exception) (struct dm_exception_store *store,
- struct dm_exception *e, int valid,
- void (*callback) (void *, int success),
- void *callback_context);
+ void (*commit_exception)(struct dm_exception_store *store,
+ struct dm_exception *e, int valid,
+ void (*callback)(void *, int success),
+ void *callback_context);
/*
* Returns 0 if the exception store is empty.
@@ -82,30 +82,30 @@ struct dm_exception_store_type {
* still-to-be-merged chunk and returns the number of
* consecutive previous ones.
*/
- int (*prepare_merge) (struct dm_exception_store *store,
- chunk_t *last_old_chunk, chunk_t *last_new_chunk);
+ int (*prepare_merge)(struct dm_exception_store *store,
+ chunk_t *last_old_chunk, chunk_t *last_new_chunk);
/*
* Clear the last n exceptions.
* nr_merged must be <= the value returned by prepare_merge.
*/
- int (*commit_merge) (struct dm_exception_store *store, int nr_merged);
+ int (*commit_merge)(struct dm_exception_store *store, int nr_merged);
/*
* The snapshot is invalid, note this in the metadata.
*/
- void (*drop_snapshot) (struct dm_exception_store *store);
+ void (*drop_snapshot)(struct dm_exception_store *store);
- unsigned int (*status) (struct dm_exception_store *store,
- status_type_t status, char *result,
- unsigned int maxlen);
+ unsigned int (*status)(struct dm_exception_store *store,
+ status_type_t status, char *result,
+ unsigned int maxlen);
/*
* Return how full the snapshot is.
*/
- void (*usage) (struct dm_exception_store *store,
- sector_t *total_sectors, sector_t *sectors_allocated,
- sector_t *metadata_sectors);
+ void (*usage)(struct dm_exception_store *store,
+ sector_t *total_sectors, sector_t *sectors_allocated,
+ sector_t *metadata_sectors);
/* For internal device-mapper use only. */
struct list_head list;
diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c
index f552c402b3c1..8a7422a54a89 100644
--- a/drivers/md/dm-integrity.c
+++ b/drivers/md/dm-integrity.c
@@ -1830,7 +1830,7 @@ static void integrity_metadata(struct work_struct *w)
if (unlikely(dio->op == REQ_OP_DISCARD)) {
unsigned int bi_size = dio->bio_details.bi_iter.bi_size;
- unsigned int max_size = likely(checksums != checksums_onstack) ? PAGE_SIZE : HASH_MAX_DIGESTSIZE;
+ unsigned int max_size = likely(checksums != checksums_onstack) ? PAGE_SIZE : sizeof(checksums_onstack);
unsigned int max_blocks = max_size / ic->tag_size;
memset(checksums, DISCARD_FILLER, max_size);
@@ -3503,6 +3503,9 @@ static void dm_integrity_io_hints(struct dm_target *ti, struct queue_limits *lim
blk_limits_io_min(limits, ic->sectors_per_block << SECTOR_SHIFT);
limits->dma_alignment = limits->logical_block_size - 1;
}
+
+ if (ic->internal_hash && (ic->mode == 'D' || ic->mode == 'B'))
+ ti->needs_stable_writes = true;
}
static void calculate_journal_section_size(struct dm_integrity_c *ic)
diff --git a/drivers/md/dm-io.c b/drivers/md/dm-io.c
index 87adb94dfcb2..2735a6ba5198 100644
--- a/drivers/md/dm-io.c
+++ b/drivers/md/dm-io.c
@@ -80,13 +80,15 @@ void dm_io_client_destroy(struct dm_io_client *client)
}
EXPORT_SYMBOL(dm_io_client_destroy);
-/*-----------------------------------------------------------------
+/*
+ *-------------------------------------------------------------------
* We need to keep track of which region a bio is doing io for.
* To avoid a memory allocation to store just 5 or 6 bits, we
* ensure the 'struct io' pointer is aligned so enough low bits are
* always zero and then combine it with the region number directly in
* bi_private.
- *---------------------------------------------------------------*/
+ *-------------------------------------------------------------------
+ */
static void store_io_and_region_in_bio(struct bio *bio, struct io *io,
unsigned int region)
{
@@ -107,10 +109,12 @@ static void retrieve_io_and_region_from_bio(struct bio *bio, struct io **io,
*region = val & (DM_IO_MAX_REGIONS - 1);
}
-/*-----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* We need an io object to keep track of the number of bios that
* have been dispatched for a particular io.
- *---------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void complete_io(struct io *io)
{
unsigned long error_bits = io->error_bits;
@@ -154,10 +158,12 @@ static void endio(struct bio *bio)
dec_count(io, region, error);
}
-/*-----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* These little objects provide an abstraction for getting a new
* destination page for io.
- *---------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
struct dpages {
void (*get_page)(struct dpages *dp,
struct page **p, unsigned long *len, unsigned int *offset);
@@ -291,9 +297,11 @@ static void km_dp_init(struct dpages *dp, void *data)
dp->context_ptr = data;
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* IO routines that accept a list of pages.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static void do_region(const blk_opf_t opf, unsigned int region,
struct dm_io_region *where, struct dpages *dp,
struct io *io, unsigned short ioprio)
diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c
index 93dc77c2f39f..2153f93a26d0 100644
--- a/drivers/md/dm-ioctl.c
+++ b/drivers/md/dm-ioctl.c
@@ -34,10 +34,12 @@ struct dm_file {
volatile unsigned int global_event_nr;
};
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* The ioctl interface needs to be able to look up devices by
* name or uuid.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
struct hash_cell {
struct rb_node name_node;
struct rb_node uuid_node;
@@ -51,10 +53,10 @@ struct hash_cell {
};
struct vers_iter {
- size_t param_size;
- struct dm_target_versions *vers, *old_vers;
- char *end;
- uint32_t flags;
+ size_t param_size;
+ struct dm_target_versions *vers, *old_vers;
+ char *end;
+ uint32_t flags;
};
@@ -78,9 +80,11 @@ static void dm_hash_exit(void)
dm_hash_remove_all(false, false, false);
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Code for looking up a device by name
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static struct hash_cell *__get_name_cell(const char *str)
{
struct rb_node *n = name_rb_tree.rb_node;
@@ -203,9 +207,11 @@ static struct hash_cell *__get_dev_cell(uint64_t dev)
return hc;
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Inserting, removing and renaming a device.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static struct hash_cell *alloc_cell(const char *name, const char *uuid,
struct mapped_device *md)
{
@@ -507,9 +513,11 @@ void dm_deferred_remove(void)
dm_hash_remove_all(true, false, true);
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Implementation of the ioctl commands
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
/*
* All the ioctl commands get dispatched to functions with this
* prototype.
@@ -545,10 +553,16 @@ static void *get_result_buffer(struct dm_ioctl *param, size_t param_size,
{
param->data_start = align_ptr(param + 1) - (void *) param;
- if (param->data_start < param_size)
+ if (param->data_start < param_size) {
*len = param_size - param->data_start;
- else
+ } else {
+ /*
+ * The buffer can be as small as offsetof(*param, data), which
+ * is less than sizeof(*param), so don't run off the end of it.
+ */
*len = 0;
+ param->data_start = param_size;
+ }
return ((void *) param) + param->data_start;
}
@@ -660,36 +674,34 @@ static int list_devices(struct file *filp, struct dm_ioctl *param, size_t param_
static void list_version_get_needed(struct target_type *tt, void *needed_param)
{
- size_t *needed = needed_param;
+ size_t *needed = needed_param;
- *needed += sizeof(struct dm_target_versions);
- *needed += strlen(tt->name) + 1;
- *needed += ALIGN_MASK;
+ *needed += sizeof(struct dm_target_versions);
+ *needed += strlen(tt->name) + 1;
+ *needed += ALIGN_MASK;
}
static void list_version_get_info(struct target_type *tt, void *param)
{
- struct vers_iter *info = param;
+ struct vers_iter *info = param;
- /* Check space - it might have changed since the first iteration */
- if ((char *)info->vers + sizeof(tt->version) + strlen(tt->name) + 1 >
- info->end) {
+ /* Check space - it might have changed since the first iteration */
+ if ((char *)info->vers + sizeof(struct dm_target_versions) + strlen(tt->name) + 1 > info->end) {
+ info->flags = DM_BUFFER_FULL_FLAG;
+ return;
+ }
- info->flags = DM_BUFFER_FULL_FLAG;
- return;
- }
+ if (info->old_vers)
+ info->old_vers->next = (uint32_t) ((void *)info->vers - (void *)info->old_vers);
- if (info->old_vers)
- info->old_vers->next = (uint32_t) ((void *)info->vers -
- (void *)info->old_vers);
- info->vers->version[0] = tt->version[0];
- info->vers->version[1] = tt->version[1];
- info->vers->version[2] = tt->version[2];
- info->vers->next = 0;
- strcpy(info->vers->name, tt->name);
+ info->vers->version[0] = tt->version[0];
+ info->vers->version[1] = tt->version[1];
+ info->vers->version[2] = tt->version[2];
+ info->vers->next = 0;
+ strcpy(info->vers->name, tt->name);
- info->old_vers = info->vers;
- info->vers = align_ptr(((void *) ++info->vers) + strlen(tt->name) + 1);
+ info->old_vers = info->vers;
+ info->vers = align_ptr((void *)(info->vers + 1) + strlen(tt->name) + 1);
}
static int __list_versions(struct dm_ioctl *param, size_t param_size, const char *name)
@@ -1160,6 +1172,7 @@ static int do_resume(struct dm_ioctl *param)
/* Do we need to load a new map ? */
if (new_map) {
sector_t old_size, new_size;
+ fmode_t new_map_mode;
/* Suspend if it isn't already suspended */
if (param->flags & DM_SKIP_LOCKFS_FLAG)
@@ -1187,6 +1200,7 @@ static int do_resume(struct dm_ioctl *param)
}
}
+ new_map_mode = dm_table_get_mode(new_map);
old_size = dm_get_size(md);
old_map = dm_swap_table(md, new_map);
if (IS_ERR(old_map)) {
@@ -1199,7 +1213,7 @@ static int do_resume(struct dm_ioctl *param)
if (old_size && new_size && old_size != new_size)
need_resize_uevent = true;
- if (dm_table_get_mode(new_map) & FMODE_WRITE)
+ if (new_map_mode & FMODE_WRITE)
set_disk_ro(dm_disk(md), 0);
else
set_disk_ro(dm_disk(md), 1);
@@ -1804,10 +1818,11 @@ static int target_message(struct file *filp, struct dm_ioctl *param, size_t para
#define IOCTL_FLAGS_NO_PARAMS 1
#define IOCTL_FLAGS_ISSUE_GLOBAL_EVENT 2
-/*-----------------------------------------------------------------
- * Implementation of open/close/ioctl on the special char
- * device.
- *---------------------------------------------------------------*/
+/*
+ *---------------------------------------------------------------
+ * Implementation of open/close/ioctl on the special char device.
+ *---------------------------------------------------------------
+ */
static ioctl_fn lookup_ioctl(unsigned int cmd, int *ioctl_flags)
{
static const struct {
@@ -2135,9 +2150,9 @@ static const struct file_operations _ctl_fops = {
static struct miscdevice _dm_misc = {
.minor = MAPPER_CTRL_MINOR,
- .name = DM_NAME,
+ .name = DM_NAME,
.nodename = DM_DIR "/" DM_CONTROL_NODE,
- .fops = &_ctl_fops
+ .fops = &_ctl_fops
};
MODULE_ALIAS_MISCDEV(MAPPER_CTRL_MINOR);
diff --git a/drivers/md/dm-kcopyd.c b/drivers/md/dm-kcopyd.c
index 5a4237a679a6..a2219bb89a48 100644
--- a/drivers/md/dm-kcopyd.c
+++ b/drivers/md/dm-kcopyd.c
@@ -50,10 +50,12 @@ static unsigned int dm_get_kcopyd_subjob_size(void)
return sub_job_size_kb << 1;
}
-/*-----------------------------------------------------------------
+/*
+ *----------------------------------------------------------------
* Each kcopyd client has its own little pool of preallocated
* pages for kcopyd io.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
struct dm_kcopyd_client {
struct page_list *pages;
unsigned int nr_reserved_pages;
@@ -334,11 +336,13 @@ static void client_free_pages(struct dm_kcopyd_client *kc)
kc->nr_free_pages = kc->nr_reserved_pages = 0;
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* kcopyd_jobs need to be allocated by the *clients* of kcopyd,
* for this reason we use a mempool to prevent the client from
* ever having to do io (which could cause a deadlock).
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
struct kcopyd_job {
struct dm_kcopyd_client *kc;
struct list_head list;
@@ -604,7 +608,7 @@ static int run_pages_job(struct kcopyd_job *job)
* of successful jobs.
*/
static int process_jobs(struct list_head *jobs, struct dm_kcopyd_client *kc,
- int (*fn) (struct kcopyd_job *))
+ int (*fn)(struct kcopyd_job *))
{
struct kcopyd_job *job;
int r, count = 0;
@@ -902,9 +906,11 @@ int kcopyd_cancel(struct kcopyd_job *job, int block)
}
#endif /* 0 */
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Client setup
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
struct dm_kcopyd_client *dm_kcopyd_client_create(struct dm_kcopyd_throttle *throttle)
{
int r;
diff --git a/drivers/md/dm-log.c b/drivers/md/dm-log.c
index 6d44e91a526f..77da6cd56e0e 100644
--- a/drivers/md/dm-log.c
+++ b/drivers/md/dm-log.c
@@ -182,10 +182,12 @@ void dm_dirty_log_destroy(struct dm_dirty_log *log)
}
EXPORT_SYMBOL(dm_dirty_log_destroy);
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Persistent and core logs share a lot of their implementation.
* FIXME: need a reload method to be called from a resume
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
/*
* Magic for persistent mirrors: "MiRr"
*/
@@ -274,9 +276,11 @@ static inline void log_clear_bit(struct log_c *l,
l->touched_dirtied = 1;
}
-/*----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Header IO
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void header_to_disk(struct log_header_core *core, struct log_header_disk *disk)
{
disk->magic = cpu_to_le32(core->magic);
@@ -352,11 +356,13 @@ static int _check_region_size(struct dm_target *ti, uint32_t region_size)
return 1;
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* core log constructor/destructor
*
* argv contains region_size followed optionally by [no]sync
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
#define BYTE_SHIFT 3
static int create_log_context(struct dm_dirty_log *log, struct dm_target *ti,
unsigned int argc, char **argv,
@@ -537,11 +543,13 @@ static void core_dtr(struct dm_dirty_log *log)
destroy_log_context(lc);
}
-/*----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------------
* disk log constructor/destructor
*
* argv contains log_device region_size followed optionally by [no]sync
- *--------------------------------------------------------------*/
+ *---------------------------------------------------------------------
+ */
static int disk_ctr(struct dm_dirty_log *log, struct dm_target *ti,
unsigned int argc, char **argv)
{
diff --git a/drivers/md/dm-mpath.c b/drivers/md/dm-mpath.c
index 8b97695fac49..dc8cbc354831 100644
--- a/drivers/md/dm-mpath.c
+++ b/drivers/md/dm-mpath.c
@@ -117,10 +117,11 @@ static void activate_path_work(struct work_struct *work);
static void process_queued_bios(struct work_struct *work);
static void queue_if_no_path_timeout_work(struct timer_list *t);
-/*-----------------------------------------------
+/*
+ *-----------------------------------------------
* Multipath state flags.
- *-----------------------------------------------*/
-
+ *-----------------------------------------------
+ */
#define MPATHF_QUEUE_IO 0 /* Must we queue all I/O? */
#define MPATHF_QUEUE_IF_NO_PATH 1 /* Queue I/O if last path fails? */
#define MPATHF_SAVED_QUEUE_IF_NO_PATH 2 /* Saved state during suspension */
@@ -144,10 +145,11 @@ static bool mpath_double_check_test_bit(int MPATHF_bit, struct multipath *m)
return r;
}
-/*-----------------------------------------------
+/*
+ *-----------------------------------------------
* Allocation routines
- *-----------------------------------------------*/
-
+ *-----------------------------------------------
+ */
static struct pgpath *alloc_pgpath(void)
{
struct pgpath *pgpath = kzalloc(sizeof(*pgpath), GFP_KERNEL);
@@ -303,10 +305,11 @@ static void multipath_init_per_bio_data(struct bio *bio, struct dm_mpath_io **mp
dm_bio_record(bio_details, bio);
}
-/*-----------------------------------------------
+/*
+ *-----------------------------------------------
* Path selection
- *-----------------------------------------------*/
-
+ *-----------------------------------------------
+ */
static int __pg_init_all_paths(struct multipath *m)
{
struct pgpath *pgpath;
@@ -827,7 +830,8 @@ static void trigger_event(struct work_struct *work)
dm_table_event(m->ti->table);
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Constructor/argument parsing:
* <#multipath feature args> [<arg>]*
* <#hw_handler args> [hw_handler [<arg>]*]
@@ -836,7 +840,8 @@ static void trigger_event(struct work_struct *work)
* [<selector> <#selector args> [<arg>]*
* <#paths> <#per-path selector args>
* [<path> [<arg>]* ]+ ]+
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static int parse_path_selector(struct dm_arg_set *as, struct priority_group *pg,
struct dm_target *ti)
{
@@ -2171,9 +2176,11 @@ static int multipath_busy(struct dm_target *ti)
return busy;
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Module setup
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static struct target_type multipath_target = {
.name = "multipath",
.version = {1, 14, 0},
diff --git a/drivers/md/dm-path-selector.h b/drivers/md/dm-path-selector.h
index 0f2b37af8766..8e166de9c594 100644
--- a/drivers/md/dm-path-selector.h
+++ b/drivers/md/dm-path-selector.h
@@ -52,44 +52,43 @@ struct path_selector_type {
/*
* Constructs a path selector object, takes custom arguments
*/
- int (*create) (struct path_selector *ps, unsigned int argc, char **argv);
- void (*destroy) (struct path_selector *ps);
+ int (*create)(struct path_selector *ps, unsigned int argc, char **argv);
+ void (*destroy)(struct path_selector *ps);
/*
* Add an opaque path object, along with some selector specific
* path args (eg, path priority).
*/
- int (*add_path) (struct path_selector *ps, struct dm_path *path,
- int argc, char **argv, char **error);
+ int (*add_path)(struct path_selector *ps, struct dm_path *path,
+ int argc, char **argv, char **error);
/*
* Chooses a path for this io, if no paths are available then
* NULL will be returned.
*/
- struct dm_path *(*select_path) (struct path_selector *ps,
- size_t nr_bytes);
+ struct dm_path *(*select_path)(struct path_selector *ps, size_t nr_bytes);
/*
* Notify the selector that a path has failed.
*/
- void (*fail_path) (struct path_selector *ps, struct dm_path *p);
+ void (*fail_path)(struct path_selector *ps, struct dm_path *p);
/*
* Ask selector to reinstate a path.
*/
- int (*reinstate_path) (struct path_selector *ps, struct dm_path *p);
+ int (*reinstate_path)(struct path_selector *ps, struct dm_path *p);
/*
* Table content based on parameters added in ps_add_path_fn
* or path selector status
*/
- int (*status) (struct path_selector *ps, struct dm_path *path,
- status_type_t type, char *result, unsigned int maxlen);
+ int (*status)(struct path_selector *ps, struct dm_path *path,
+ status_type_t type, char *result, unsigned int maxlen);
- int (*start_io) (struct path_selector *ps, struct dm_path *path,
- size_t nr_bytes);
- int (*end_io) (struct path_selector *ps, struct dm_path *path,
- size_t nr_bytes, u64 start_time);
+ int (*start_io)(struct path_selector *ps, struct dm_path *path,
+ size_t nr_bytes);
+ int (*end_io)(struct path_selector *ps, struct dm_path *path,
+ size_t nr_bytes, u64 start_time);
};
/* Register a path selector */
diff --git a/drivers/md/dm-ps-queue-length.c b/drivers/md/dm-ps-queue-length.c
index 8e298570c8d2..fbe175780e1b 100644
--- a/drivers/md/dm-ps-queue-length.c
+++ b/drivers/md/dm-ps-queue-length.c
@@ -122,8 +122,8 @@ static int ql_add_path(struct path_selector *ps, struct dm_path *path,
/*
* Arguments: [<repeat_count>]
- * <repeat_count>: The number of I/Os before switching path.
- * If not given, default (QL_MIN_IO) is used.
+ * <repeat_count>: The number of I/Os before switching path.
+ * If not given, default (QL_MIN_IO) is used.
*/
if (argc > 1) {
*error = "queue-length ps: incorrect number of arguments";
diff --git a/drivers/md/dm-ps-round-robin.c b/drivers/md/dm-ps-round-robin.c
index 22c68ca81a24..733737477f32 100644
--- a/drivers/md/dm-ps-round-robin.c
+++ b/drivers/md/dm-ps-round-robin.c
@@ -20,9 +20,11 @@
#define RR_MIN_IO 1
#define RR_VERSION "1.2.0"
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Path-handling code, paths are held in lists
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
struct path_info {
struct list_head list;
struct dm_path *path;
@@ -39,10 +41,11 @@ static void free_paths(struct list_head *paths)
}
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Round-robin selector
- *---------------------------------------------------------------*/
-
+ *---------------------------------------------------------------
+ */
struct selector {
struct list_head valid_paths;
struct list_head invalid_paths;
diff --git a/drivers/md/dm-ps-service-time.c b/drivers/md/dm-ps-service-time.c
index d1e77eefaf2b..bc8c3042b4a8 100644
--- a/drivers/md/dm-ps-service-time.c
+++ b/drivers/md/dm-ps-service-time.c
@@ -120,11 +120,11 @@ static int st_add_path(struct path_selector *ps, struct dm_path *path,
/*
* Arguments: [<repeat_count> [<relative_throughput>]]
- * <repeat_count>: The number of I/Os before switching path.
- * If not given, default (ST_MIN_IO) is used.
- * <relative_throughput>: The relative throughput value of
+ * <repeat_count>: The number of I/Os before switching path.
+ * If not given, default (ST_MIN_IO) is used.
+ * <relative_throughput>: The relative throughput value of
* the path among all paths in the path-group.
- * The valid range: 0-<ST_MAX_RELATIVE_THROUGHPUT>
+ * The valid range: 0-<ST_MAX_RELATIVE_THROUGHPUT>
* If not given, minimum value '1' is used.
* If '0' is given, the path isn't selected while
* other paths having a positive value are available.
diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
index 3d9be184d460..ae3a6713da51 100644
--- a/drivers/md/dm-raid.c
+++ b/drivers/md/dm-raid.c
@@ -3185,12 +3185,12 @@ static int raid_ctr(struct dm_target *ti, unsigned int argc, char **argv)
/* Out-of-place space has to be available to allow for a reshape unless raid1! */
if (reshape_sectors || rs_is_raid1(rs)) {
/*
- * We can only prepare for a reshape here, because the
- * raid set needs to run to provide the repective reshape
- * check functions via its MD personality instance.
- *
- * So do the reshape check after md_run() succeeded.
- */
+ * We can only prepare for a reshape here, because the
+ * raid set needs to run to provide the repective reshape
+ * check functions via its MD personality instance.
+ *
+ * So do the reshape check after md_run() succeeded.
+ */
r = rs_prepare_reshape(rs);
if (r)
goto bad;
diff --git a/drivers/md/dm-raid1.c b/drivers/md/dm-raid1.c
index b9c0510f3fe3..aff6db13f592 100644
--- a/drivers/md/dm-raid1.c
+++ b/drivers/md/dm-raid1.c
@@ -32,9 +32,11 @@
static DECLARE_WAIT_QUEUE_HEAD(_kmirrord_recovery_stopped);
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Mirror set structures.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
enum dm_raid1_error {
DM_RAID1_WRITE_ERROR,
DM_RAID1_FLUSH_ERROR,
@@ -284,13 +286,15 @@ static int mirror_flush(struct dm_target *ti)
return 0;
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Recovery.
*
* When a mirror is first activated we may find that some regions
* are in the no-sync state. We have to recover these by
* recopying from the default mirror to all the others.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static void recovery_complete(int read_err, unsigned long write_err,
void *context)
{
@@ -407,9 +411,11 @@ static void do_recovery(struct mirror_set *ms)
}
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Reads
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static struct mirror *choose_mirror(struct mirror_set *ms, sector_t sector)
{
struct mirror *m = get_default_mirror(ms);
@@ -497,9 +503,11 @@ static void hold_bio(struct mirror_set *ms, struct bio *bio)
spin_unlock_irq(&ms->lock);
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Reads
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static void read_callback(unsigned long error, void *context)
{
struct bio *bio = context;
@@ -577,18 +585,18 @@ static void do_reads(struct mirror_set *ms, struct bio_list *reads)
}
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------------
* Writes.
*
* We do different things with the write io depending on the
* state of the region that it's in:
*
- * SYNC: increment pending, use kcopyd to write to *all* mirrors
+ * SYNC: increment pending, use kcopyd to write to *all* mirrors
* RECOVERING: delay the io until recovery completes
* NOSYNC: increment pending, just write to the default mirror
- *---------------------------------------------------------------*/
-
-
+ *---------------------------------------------------------------------
+ */
static void write_callback(unsigned long error, void *context)
{
unsigned int i;
@@ -840,9 +848,11 @@ static void trigger_event(struct work_struct *work)
dm_table_event(ms->ti->table);
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* kmirrord
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static void do_mirror(struct work_struct *work)
{
struct mirror_set *ms = container_of(work, struct mirror_set,
@@ -866,9 +876,11 @@ static void do_mirror(struct work_struct *work)
do_failures(ms, &failures);
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Target functions
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static struct mirror_set *alloc_context(unsigned int nr_mirrors,
uint32_t region_size,
struct dm_target *ti,
diff --git a/drivers/md/dm-region-hash.c b/drivers/md/dm-region-hash.c
index adbdb4b67137..3a67743015cf 100644
--- a/drivers/md/dm-region-hash.c
+++ b/drivers/md/dm-region-hash.c
@@ -18,7 +18,8 @@
#define DM_MSG_PREFIX "region hash"
-/*-----------------------------------------------------------------
+/*
+ *------------------------------------------------------------------
* Region hash
*
* The mirror splits itself up into discrete regions. Each
@@ -53,7 +54,8 @@
* lists in the region_hash, with the 'state', 'list' and
* 'delayed_bios' fields of the regions. This is used from irq
* context, so all other uses will have to suspend local irqs.
- *---------------------------------------------------------------*/
+ *------------------------------------------------------------------
+ */
struct dm_region_hash {
uint32_t region_size;
unsigned int region_shift;
diff --git a/drivers/md/dm-snap-persistent.c b/drivers/md/dm-snap-persistent.c
index 04bbe1506b9e..1b61a7a54c11 100644
--- a/drivers/md/dm-snap-persistent.c
+++ b/drivers/md/dm-snap-persistent.c
@@ -21,10 +21,12 @@
#define DM_PREFETCH_CHUNKS 12
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Persistent snapshots, by persistent we mean that the snapshot
* will survive a reboot.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
/*
* We need to store a record of which parts of the origin have
@@ -693,7 +695,7 @@ static int persistent_prepare_exception(struct dm_exception_store *store,
static void persistent_commit_exception(struct dm_exception_store *store,
struct dm_exception *e, int valid,
- void (*callback) (void *, int success),
+ void (*callback)(void *, int success),
void *callback_context)
{
unsigned int i;
diff --git a/drivers/md/dm-snap-transient.c b/drivers/md/dm-snap-transient.c
index 11de107f5f46..bbe2fa60ac00 100644
--- a/drivers/md/dm-snap-transient.c
+++ b/drivers/md/dm-snap-transient.c
@@ -16,9 +16,11 @@
#define DM_MSG_PREFIX "transient snapshot"
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Implementation of the store for non-persistent snapshots.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
struct transient_c {
sector_t next_free;
};
@@ -53,7 +55,7 @@ static int transient_prepare_exception(struct dm_exception_store *store,
static void transient_commit_exception(struct dm_exception_store *store,
struct dm_exception *e, int valid,
- void (*callback) (void *, int success),
+ void (*callback)(void *, int success),
void *callback_context)
{
/* Just succeed */
diff --git a/drivers/md/dm-snap.c b/drivers/md/dm-snap.c
index 2a4f914f53e6..bf73e17cd307 100644
--- a/drivers/md/dm-snap.c
+++ b/drivers/md/dm-snap.c
@@ -122,11 +122,11 @@ struct dm_snapshot {
* The merge operation failed if this flag is set.
* Failure modes are handled as follows:
* - I/O error reading the header
- * => don't load the target; abort.
+ * => don't load the target; abort.
* - Header does not have "valid" flag set
- * => use the origin; forget about the snapshot.
+ * => use the origin; forget about the snapshot.
* - I/O error when reading exceptions
- * => don't load the target; abort.
+ * => don't load the target; abort.
* (We can't use the intermediate origin state.)
* - I/O error while merging
* => stop merging; set merge_failed; process I/O normally.
@@ -1306,6 +1306,20 @@ static int snapshot_ctr(struct dm_target *ti, unsigned int argc, char **argv)
goto bad_store;
}
+ /*
+ * snapshot-merge requires the optional prepare_merge/commit_merge
+ * callbacks; the transient exception store does not implement them.
+ * The handover path checks this in __validate_exception_handover(),
+ * but a directly constructed snapshot-merge never goes through it.
+ */
+ if (dm_target_is_snapshot_merge(ti) &&
+ (!s->store->type->prepare_merge ||
+ !s->store->type->commit_merge)) {
+ ti->error = "Snapshot exception store does not support snapshot-merge.";
+ r = -EINVAL;
+ goto bad_hash_tables;
+ }
+
argv += args_used;
argc -= args_used;
@@ -2419,10 +2433,11 @@ static void snapshot_io_hints(struct dm_target *ti, struct queue_limits *limits)
}
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Origin methods
- *---------------------------------------------------------------*/
-
+ *---------------------------------------------------------------
+ */
/*
* If no exceptions need creating, DM_MAPIO_REMAPPED is returned and any
* supplied bio was ignored. The caller may submit it immediately.
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index 8e0b9783f3da..214a7af330f8 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -1941,7 +1941,7 @@ static int device_requires_stable_pages(struct dm_target *ti,
struct dm_dev *dev, sector_t start,
sector_t len, void *data)
{
- return bdev_stable_writes(dev->bdev);
+ return ti->needs_stable_writes || bdev_stable_writes(dev->bdev);
}
int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q,
diff --git a/drivers/md/dm-thin-metadata.c b/drivers/md/dm-thin-metadata.c
index c143daeee5b3..52b0b263e363 100644
--- a/drivers/md/dm-thin-metadata.c
+++ b/drivers/md/dm-thin-metadata.c
@@ -14,7 +14,8 @@
#include <linux/device-mapper.h>
#include <linux/workqueue.h>
-/*--------------------------------------------------------------------------
+/*
+ *--------------------------------------------------------------------------
* As far as the metadata goes, there is:
*
* - A superblock in block zero, taking up fewer than 512 bytes for
@@ -70,7 +71,8 @@
*
* All metadata io is in THIN_METADATA_BLOCK_SIZE sized/aligned chunks
* from the block manager.
- *--------------------------------------------------------------------------*/
+ *--------------------------------------------------------------------------
+ */
#define DM_MSG_PREFIX "thin metadata"
@@ -240,10 +242,11 @@ struct dm_thin_device {
uint32_t snapshotted_time;
};
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* superblock validator
- *--------------------------------------------------------------*/
-
+ *--------------------------------------------------------------
+ */
#define SUPERBLOCK_CSUM_XOR 160774
static void sb_prepare_for_write(struct dm_block_validator *v,
@@ -297,10 +300,11 @@ static struct dm_block_validator sb_validator = {
.check = sb_check
};
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Methods for the btree value types
- *--------------------------------------------------------------*/
-
+ *--------------------------------------------------------------
+ */
static uint64_t pack_block_time(dm_block_t b, uint32_t t)
{
return (b << 24) | t;
diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c
index b78cadaa17dd..677a299dce1d 100644
--- a/drivers/md/dm-thin.c
+++ b/drivers/md/dm-thin.c
@@ -2804,9 +2804,11 @@ static void requeue_bios(struct pool *pool)
rcu_read_unlock();
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Binding of control targets to a pool object
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static bool is_factor(sector_t block_size, uint32_t n)
{
return !sector_div(block_size, n);
@@ -2870,9 +2872,11 @@ static void unbind_control_target(struct pool *pool, struct dm_target *ti)
pool->ti = NULL;
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Pool creation
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
/* Initialize pool features. */
static void pool_features_init(struct pool_features *pf)
{
@@ -3096,9 +3100,11 @@ static struct pool *__pool_find(struct mapped_device *pool_md,
return pool;
}
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Pool target methods
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void pool_dtr(struct dm_target *ti)
{
struct pool_c *pt = ti->private;
@@ -4102,9 +4108,11 @@ static struct target_type pool_target = {
.io_hints = pool_io_hints,
};
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Thin target methods
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void thin_get(struct thin_c *tc)
{
refcount_inc(&tc->refcount);
diff --git a/drivers/md/dm-uevent.c b/drivers/md/dm-uevent.c
index a02b3f6ea47a..491738263292 100644
--- a/drivers/md/dm-uevent.c
+++ b/drivers/md/dm-uevent.c
@@ -3,7 +3,7 @@
* Device Mapper Uevent Support (dm-uevent)
*
* Copyright IBM Corporation, 2007
- * Author: Mike Anderson <andmike@linux.vnet.ibm.com>
+ * Author: Mike Anderson <andmike@linux.vnet.ibm.com>
*/
#include <linux/list.h>
#include <linux/slab.h>
diff --git a/drivers/md/dm-verity-fec.c b/drivers/md/dm-verity-fec.c
index 39bc0c686519..b5b70cb522ac 100644
--- a/drivers/md/dm-verity-fec.c
+++ b/drivers/md/dm-verity-fec.c
@@ -272,7 +272,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io,
(unsigned long long)block, PTR_ERR(bbuf));
/* assume the block is corrupted */
- if (neras && *neras <= v->fec->roots)
+ if (neras && *neras < v->fec->roots)
fio->erasures[(*neras)++] = i;
continue;
@@ -290,7 +290,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io,
* skip if we have already found the theoretical
* maximum number (i.e. fec->roots) of erasures
*/
- if (neras && *neras <= v->fec->roots &&
+ if (neras && *neras < v->fec->roots &&
fec_is_erasure(v, io, want_digest, bbuf))
fio->erasures[(*neras)++] = i;
}
diff --git a/drivers/md/dm.c b/drivers/md/dm.c
index 92165417fe33..53acb28f8633 100644
--- a/drivers/md/dm.c
+++ b/drivers/md/dm.c
@@ -1880,9 +1880,11 @@ static int dm_poll_bio(struct bio *bio, struct io_comp_batch *iob,
return 1;
}
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* An IDR is used to keep track of allocated minor numbers.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
static void free_minor(int minor)
{
spin_lock(&_minor_lock);
@@ -2997,9 +2999,11 @@ void dm_internal_resume_fast(struct mapped_device *md)
}
EXPORT_SYMBOL_GPL(dm_internal_resume_fast);
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Event notification.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
int dm_kobject_uevent(struct mapped_device *md, enum kobject_action action,
unsigned int cookie, bool need_resize_uevent)
{
diff --git a/drivers/md/dm.h b/drivers/md/dm.h
index a7917df09caf..22eaed188907 100644
--- a/drivers/md/dm.h
+++ b/drivers/md/dm.h
@@ -48,9 +48,11 @@ struct dm_md_mempools;
struct dm_target_io;
struct dm_io;
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Internal table functions.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
void dm_table_event_callback(struct dm_table *t,
void (*fn)(void *), void *context);
struct dm_target *dm_table_find_target(struct dm_table *t, sector_t sector);
@@ -119,9 +121,11 @@ static inline int dm_zone_map_bio(struct dm_target_io *tio)
}
#endif
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* A registry of target types.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
int dm_target_init(void);
void dm_target_exit(void);
struct target_type *dm_get_target_type(const char *name);
diff --git a/drivers/md/md-cluster.c b/drivers/md/md-cluster.c
index 7484bb83171a..c3394112e91c 100644
--- a/drivers/md/md-cluster.c
+++ b/drivers/md/md-cluster.c
@@ -526,11 +526,20 @@ static void process_add_new_disk(struct mddev *mddev, struct cluster_msg *cmsg)
static void process_metadata_update(struct mddev *mddev, struct cluster_msg *msg)
{
int got_lock = 0;
+ struct md_thread *thread;
struct md_cluster_info *cinfo = mddev->cluster_info;
mddev->good_device_nr = le32_to_cpu(msg->raid_slot);
dlm_lock_sync(cinfo->no_new_dev_lockres, DLM_LOCK_CR);
- wait_event(mddev->thread->wqueue,
+
+ thread = mddev->thread;
+ if (!thread) {
+ pr_warn("md-cluster: Received metadata update but MD thread is not ready\n");
+ dlm_unlock_sync(cinfo->no_new_dev_lockres);
+ return;
+ }
+
+ wait_event(thread->wqueue,
(got_lock = mddev_trylock(mddev)) ||
test_bit(MD_CLUSTER_HOLDING_MUTEX_FOR_RECVD, &cinfo->state));
md_reload_sb(mddev, mddev->good_device_nr);
diff --git a/drivers/md/md.c b/drivers/md/md.c
index ce5063114449..e268eb0be6da 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -213,7 +213,8 @@ void mddev_create_serial_pool(struct mddev *mddev, struct md_rdev *rdev,
{
int ret = 0;
- if (rdev && !rdev_need_serial(rdev) &&
+ if (!mddev->serialize_policy &&
+ rdev && !rdev_need_serial(rdev) &&
!test_bit(CollisionCheck, &rdev->flags))
return;
diff --git a/drivers/md/persistent-data/dm-array.c b/drivers/md/persistent-data/dm-array.c
index a1522d6805d4..9f85dbcffe3a 100644
--- a/drivers/md/persistent-data/dm-array.c
+++ b/drivers/md/persistent-data/dm-array.c
@@ -37,6 +37,14 @@ struct array_block {
*/
#define CSUM_XOR 595846735
+/*
+ * Each array block can hold this many values.
+ */
+static uint32_t calc_max_entries(size_t value_size, size_t size_of_block)
+{
+ return (size_of_block - sizeof(struct array_block)) / value_size;
+}
+
static void array_block_prepare_for_write(struct dm_block_validator *v,
struct dm_block *b,
size_t size_of_block)
@@ -54,6 +62,7 @@ static int array_block_check(struct dm_block_validator *v,
size_t size_of_block)
{
struct array_block *bh_le = dm_block_data(b);
+ uint32_t nr_entries, max_entries, value_size;
__le32 csum_disk;
if (dm_block_location(b) != le64_to_cpu(bh_le->blocknr)) {
@@ -73,6 +82,26 @@ static int array_block_check(struct dm_block_validator *v,
return -EILSEQ;
}
+ nr_entries = le32_to_cpu(bh_le->nr_entries);
+ max_entries = le32_to_cpu(bh_le->max_entries);
+ value_size = le32_to_cpu(bh_le->value_size);
+
+ if (!value_size) {
+ DMERR_LIMIT("%s failed: value_size is zero", __func__);
+ return -EILSEQ;
+ }
+
+ if (max_entries != calc_max_entries(value_size, size_of_block)) {
+ DMERR_LIMIT("%s failed: max_entries %u invalid for value_size %u",
+ __func__, max_entries, value_size);
+ return -EILSEQ;
+ }
+
+ if (nr_entries > max_entries) {
+ DMERR_LIMIT("%s failed: too many entries", __func__);
+ return -EILSEQ;
+ }
+
return 0;
}
@@ -137,14 +166,6 @@ static void dec_ablock_entries(struct dm_array_info *info, struct array_block *a
on_entries(info, ab, vt->dec);
}
-/*
- * Each array block can hold this many values.
- */
-static uint32_t calc_max_entries(size_t value_size, size_t size_of_block)
-{
- return (size_of_block - sizeof(struct array_block)) / value_size;
-}
-
/*
* Allocate a new array block. The caller will need to unlock block.
*/
diff --git a/drivers/md/persistent-data/dm-block-manager.c b/drivers/md/persistent-data/dm-block-manager.c
index 6a1fb5ea990f..f41ed928f912 100644
--- a/drivers/md/persistent-data/dm-block-manager.c
+++ b/drivers/md/persistent-data/dm-block-manager.c
@@ -370,9 +370,11 @@ static void dm_block_manager_write_callback(struct dm_buffer *buf)
}
}
-/*----------------------------------------------------------------
+/*
+ * -------------------------------------------------------------
* Public interface
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
struct dm_block_manager {
struct dm_bufio_client *bufio;
bool read_only:1;
diff --git a/drivers/md/persistent-data/dm-btree.c b/drivers/md/persistent-data/dm-btree.c
index 50df2693d615..3393e12878a3 100644
--- a/drivers/md/persistent-data/dm-btree.c
+++ b/drivers/md/persistent-data/dm-btree.c
@@ -13,9 +13,11 @@
#define DM_MSG_PREFIX "btree"
-/*----------------------------------------------------------------
+/*
+ *--------------------------------------------------------------
* Array manipulation
- *--------------------------------------------------------------*/
+ *--------------------------------------------------------------
+ */
static void memcpy_disk(void *dest, const void *src, size_t len)
__dm_written_to_disk(src)
{
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index f6196a3c8945..8c496f72d4ff 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -1331,7 +1331,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio,
submit_bio_noacct(read_bio);
}
-static void raid1_write_request(struct mddev *mddev, struct bio *bio,
+static bool raid1_write_request(struct mddev *mddev, struct bio *bio,
int max_write_sectors)
{
struct r1conf *conf = mddev->private;
@@ -1351,7 +1351,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
DEFINE_WAIT(w);
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
- return;
+ return false;
}
for (;;) {
prepare_to_wait(&conf->wait_barrier,
@@ -1373,7 +1373,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
if (!wait_barrier(conf, bio->bi_iter.bi_sector,
bio->bi_opf & REQ_NOWAIT)) {
bio_wouldblock_error(bio);
- return;
+ return false;
}
r1_bio = alloc_r1bio(mddev, bio);
@@ -1476,7 +1476,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
- return;
+ return false;
}
raid1_log(mddev, "wait rdev %d blocked", blocked_rdev->raid_disk);
md_wait_for_blocked_rdev(blocked_rdev, mddev);
@@ -1580,6 +1580,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio,
/* In case raid1d snuck in to freeze_array */
wake_up(&conf->wait_barrier);
+ return true;
}
static bool raid1_make_request(struct mddev *mddev, struct bio *bio)
@@ -1605,7 +1606,8 @@ static bool raid1_make_request(struct mddev *mddev, struct bio *bio)
else {
if (!md_write_start(mddev,bio))
return false;
- raid1_write_request(mddev, bio, sectors);
+ if (!raid1_write_request(mddev, bio, sectors))
+ md_write_end(mddev);
}
return true;
}
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 508b461eabe8..e5d6ad9f9ab0 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1181,6 +1181,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
int slot = r10_bio->read_slot;
struct md_rdev *err_rdev = NULL;
gfp_t gfp = GFP_NOIO;
+ int error;
if (slot >= 0 && r10_bio->devs[slot].rdev) {
/*
@@ -1233,6 +1234,10 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
if (max_sectors < bio_sectors(bio)) {
struct bio *split = bio_split(bio, max_sectors,
gfp, &conf->bio_split);
+ if (IS_ERR(split)) {
+ error = PTR_ERR(split);
+ goto err_handle;
+ }
bio_chain(split, bio);
allow_barrier(conf);
submit_bio_noacct(bio);
@@ -1265,6 +1270,11 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio,
r10_bio->sector);
submit_bio_noacct(read_bio);
return;
+err_handle:
+ atomic_dec(&rdev->nr_pending);
+ bio->bi_status = errno_to_blk_status(error);
+ set_bit(R10BIO_Uptodate, &r10_bio->state);
+ raid_end_bio_io(r10_bio);
}
static void raid10_write_one_disk(struct mddev *mddev, struct r10bio *r10_bio,
@@ -1405,13 +1415,14 @@ static void wait_blocked_dev(struct mddev *mddev, struct r10bio *r10_bio)
}
}
-static void raid10_write_request(struct mddev *mddev, struct bio *bio,
+static bool raid10_write_request(struct mddev *mddev, struct bio *bio,
struct r10bio *r10_bio)
{
struct r10conf *conf = mddev->private;
- int i;
+ int i, k;
sector_t sectors;
int max_sectors;
+ int error;
if ((mddev_is_clustered(mddev) &&
md_cluster_ops->area_resyncing(mddev, WRITE,
@@ -1421,7 +1432,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
/* Bail out if REQ_NOWAIT is set for the bio */
if (bio->bi_opf & REQ_NOWAIT) {
bio_wouldblock_error(bio);
- return;
+ return false;
}
for (;;) {
prepare_to_wait(&conf->wait_barrier,
@@ -1437,7 +1448,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
sectors = r10_bio->sectors;
if (!regular_request_wait(mddev, conf, bio, sectors)) {
free_r10bio(r10_bio);
- return;
+ return false;
}
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) &&
@@ -1454,7 +1465,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
if (bio->bi_opf & REQ_NOWAIT) {
allow_barrier(conf);
bio_wouldblock_error(bio);
- return;
+ return false;
}
raid10_log(conf->mddev, "wait reshape metadata");
wait_event(mddev->sb_wait,
@@ -1547,6 +1558,10 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
if (r10_bio->sectors < bio_sectors(bio)) {
struct bio *split = bio_split(bio, r10_bio->sectors,
GFP_NOIO, &conf->bio_split);
+ if (IS_ERR(split)) {
+ error = PTR_ERR(split);
+ goto err_handle;
+ }
bio_chain(split, bio);
allow_barrier(conf);
submit_bio_noacct(bio);
@@ -1567,10 +1582,33 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
raid10_write_one_disk(mddev, r10_bio, bio, true, i);
}
one_write_done(r10_bio);
+ return true;
+
+err_handle:
+ for (k = 0; k < i; k++) {
+ int d = r10_bio->devs[k].devnum;
+ struct md_rdev *rdev = conf->mirrors[d].rdev;
+ struct md_rdev *rrdev = conf->mirrors[d].replacement;
+
+ if (r10_bio->devs[k].bio) {
+ rdev_dec_pending(rdev, mddev);
+ r10_bio->devs[k].bio = NULL;
+ }
+ if (r10_bio->devs[k].repl_bio) {
+ rdev_dec_pending(rrdev, mddev);
+ r10_bio->devs[k].repl_bio = NULL;
+ }
+ }
+
+ bio->bi_status = errno_to_blk_status(error);
+ set_bit(R10BIO_Uptodate, &r10_bio->state);
+ raid_end_bio_io(r10_bio);
+ return false;
}
-static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
+static bool __make_request(struct mddev *mddev, struct bio *bio, int sectors)
{
+ bool ret;
struct r10conf *conf = mddev->private;
struct r10bio *r10_bio;
@@ -1587,10 +1625,13 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) *
conf->geo.raid_disks);
+ ret = true;
if (bio_data_dir(bio) == READ)
raid10_read_request(mddev, bio, r10_bio);
else
- raid10_write_request(mddev, bio, r10_bio);
+ ret = raid10_write_request(mddev, bio, r10_bio);
+
+ return ret;
}
static void raid_end_discard_bio(struct r10bio *r10bio)
@@ -1676,11 +1717,11 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery))
return -EAGAIN;
- if (WARN_ON_ONCE(bio->bi_opf & REQ_NOWAIT)) {
+ if (!wait_barrier(conf, bio->bi_opf & REQ_NOWAIT)) {
bio_wouldblock_error(bio);
+ md_write_end(mddev);
return 0;
}
- wait_barrier(conf, false);
/*
* Check reshape again to avoid reshape happens after checking
@@ -1718,6 +1759,11 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
if (remainder) {
split_size = stripe_size - remainder;
split = bio_split(bio, split_size, GFP_NOIO, &conf->bio_split);
+ if (IS_ERR(split)) {
+ bio->bi_status = errno_to_blk_status(PTR_ERR(split));
+ bio_endio(bio);
+ return 0;
+ }
bio_chain(split, bio);
allow_barrier(conf);
/* Resend the fist split part */
@@ -1728,6 +1774,11 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
if (remainder) {
split_size = bio_sectors(bio) - remainder;
split = bio_split(bio, split_size, GFP_NOIO, &conf->bio_split);
+ if (IS_ERR(split)) {
+ bio->bi_status = errno_to_blk_status(PTR_ERR(split));
+ bio_endio(bio);
+ return 0;
+ }
bio_chain(split, bio);
allow_barrier(conf);
/* Resend the second split part */
@@ -1934,7 +1985,8 @@ static bool raid10_make_request(struct mddev *mddev, struct bio *bio)
sectors = chunk_sects -
(bio->bi_iter.bi_sector &
(chunk_sects - 1));
- __make_request(mddev, bio, sectors);
+ if (!__make_request(mddev, bio, sectors))
+ md_write_end(mddev);
/* In case raid10d snuck in to freeze_array */
wake_up_barrier(conf);
diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
index dd0864df1d8c..092cf1d0008e 100644
--- a/drivers/md/raid5.c
+++ b/drivers/md/raid5.c
@@ -4958,25 +4958,27 @@ static void break_stripe_batch_list(struct stripe_head *head_sh,
sh->check_state = head_sh->check_state;
sh->reconstruct_state = head_sh->reconstruct_state;
spin_lock_irq(&sh->stripe_lock);
- sh->batch_head = NULL;
- spin_unlock_irq(&sh->stripe_lock);
for (i = 0; i < sh->disks; i++) {
if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags))
do_wakeup = 1;
- sh->dev[i].flags = head_sh->dev[i].flags &
+ sh->dev[i].flags = READ_ONCE(head_sh->dev[i].flags) &
(~((1 << R5_WriteError) | (1 << R5_Overlap)));
}
+ sh->batch_head = NULL;
+ spin_unlock_irq(&sh->stripe_lock);
+
if (handle_flags == 0 ||
sh->state & handle_flags)
set_bit(STRIPE_HANDLE, &sh->state);
raid5_release_stripe(sh);
}
spin_lock_irq(&head_sh->stripe_lock);
- head_sh->batch_head = NULL;
- spin_unlock_irq(&head_sh->stripe_lock);
for (i = 0; i < head_sh->disks; i++)
if (test_and_clear_bit(R5_Overlap, &head_sh->dev[i].flags))
do_wakeup = 1;
+ head_sh->batch_head = NULL;
+ spin_unlock_irq(&head_sh->stripe_lock);
+
if (head_sh->state & handle_flags)
set_bit(STRIPE_HANDLE, &head_sh->state);
diff --git a/drivers/media/common/videobuf2/videobuf2-dma-sg.c b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
index 7442fa736724..7ef8d224d8ef 100644
--- a/drivers/media/common/videobuf2/videobuf2-dma-sg.c
+++ b/drivers/media/common/videobuf2/videobuf2-dma-sg.c
@@ -343,6 +343,7 @@ static int vb2_dma_sg_mmap(void *buf_priv, struct vm_area_struct *vma)
return err;
}
+ vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP;
/*
* Use common vm_area operations to track buffer refcount.
*/
diff --git a/drivers/media/pci/ivtv/ivtvfb.c b/drivers/media/pci/ivtv/ivtvfb.c
index a642becdc0d7..19f5abd06907 100644
--- a/drivers/media/pci/ivtv/ivtvfb.c
+++ b/drivers/media/pci/ivtv/ivtvfb.c
@@ -378,8 +378,8 @@ static ssize_t ivtvfb_write(struct fb_info *info, const char __user *buf,
unsigned long dma_size;
u16 lead = 0, tail = 0;
- if (info->state != FBINFO_STATE_RUNNING)
- return -EPERM;
+ if (!info->screen_base)
+ return -ENODEV;
total_size = info->screen_size;
diff --git a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-core.c b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-core.c
index 97edcd9d1c81..9a11064ecfa9 100644
--- a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-core.c
+++ b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-core.c
@@ -241,6 +241,11 @@ static int mdp_probe(struct platform_device *pdev)
goto err_destroy_clock_wq;
}
mdp->scp = platform_get_drvdata(mm_pdev);
+ if (!mdp->scp) {
+ ret = -EPROBE_DEFER;
+ goto err_destroy_clock_wq;
+ }
+
mdp->rproc_handle = scp_get_rproc(mdp->scp);
dev_dbg(&pdev->dev, "MDP rproc_handle: %pK", mdp->rproc_handle);
diff --git a/drivers/media/platform/mediatek/vcodec/vdec/vdec_h264_req_if.c b/drivers/media/platform/mediatek/vcodec/vdec/vdec_h264_req_if.c
index 4bc05ab5afea..d6fa1e249dc7 100644
--- a/drivers/media/platform/mediatek/vcodec/vdec/vdec_h264_req_if.c
+++ b/drivers/media/platform/mediatek/vcodec/vdec/vdec_h264_req_if.c
@@ -352,11 +352,16 @@ static int vdec_h264_slice_decode(void *h_vdec, struct mtk_vcodec_mem *bs,
return vpu_dec_reset(vpu);
fb = inst->ctx->dev->vdec_pdata->get_cap_buffer(inst->ctx);
+ if (!fb) {
+ mtk_vcodec_err(inst, "fb buffer is NULL");
+ return -ENOMEM;
+ }
+
src_buf_info = container_of(bs, struct mtk_video_dec_buf, bs_buffer);
dst_buf_info = container_of(fb, struct mtk_video_dec_buf, frame_buffer);
- y_fb_dma = fb ? (u64)fb->base_y.dma_addr : 0;
- c_fb_dma = fb ? (u64)fb->base_c.dma_addr : 0;
+ y_fb_dma = fb->base_y.dma_addr;
+ c_fb_dma = fb->base_c.dma_addr;
mtk_vcodec_debug(inst, "+ [%d] FB y_dma=%llx c_dma=%llx va=%p",
inst->num_nalu, y_fb_dma, c_fb_dma, fb);
diff --git a/drivers/media/platform/mediatek/vcodec/venc/venc_h264_if.c b/drivers/media/platform/mediatek/vcodec/venc/venc_h264_if.c
index dc35ec7f9c04..e070bd258ce6 100644
--- a/drivers/media/platform/mediatek/vcodec/venc/venc_h264_if.c
+++ b/drivers/media/platform/mediatek/vcodec/venc/venc_h264_if.c
@@ -304,11 +304,12 @@ static void h264_enc_free_work_buf(struct venc_h264_inst *inst)
* other buffers need to be freed by AP.
*/
for (i = 0; i < VENC_H264_VPU_WORK_BUF_MAX; i++) {
- if (i != VENC_H264_VPU_WORK_BUF_SKIP_FRAME)
+ if (i != VENC_H264_VPU_WORK_BUF_SKIP_FRAME && inst->work_bufs[i].va)
mtk_vcodec_mem_free(inst->ctx, &inst->work_bufs[i]);
}
- mtk_vcodec_mem_free(inst->ctx, &inst->pps_buf);
+ if (inst->pps_buf.va)
+ mtk_vcodec_mem_free(inst->ctx, &inst->pps_buf);
mtk_vcodec_debug_leave(inst);
}
diff --git a/drivers/media/platform/verisilicon/hantro.h b/drivers/media/platform/verisilicon/hantro.h
index 2989ebc631cc..fca6ef4a1166 100644
--- a/drivers/media/platform/verisilicon/hantro.h
+++ b/drivers/media/platform/verisilicon/hantro.h
@@ -76,6 +76,7 @@ struct hantro_irq {
* @double_buffer: core needs double buffering
* @legacy_regs: core uses legacy register set
* @late_postproc: postproc must be set up at the end of the job
+ * @shared_devices: an array of device ids that cannot run concurrently
*/
struct hantro_variant {
unsigned int enc_offset;
@@ -100,6 +101,7 @@ struct hantro_variant {
unsigned int double_buffer : 1;
unsigned int legacy_regs : 1;
unsigned int late_postproc : 1;
+ const struct of_device_id *shared_devices;
};
/**
diff --git a/drivers/media/platform/verisilicon/hantro_drv.c b/drivers/media/platform/verisilicon/hantro_drv.c
index a35b6ae62d58..29baefc93628 100644
--- a/drivers/media/platform/verisilicon/hantro_drv.c
+++ b/drivers/media/platform/verisilicon/hantro_drv.c
@@ -13,6 +13,7 @@
#include <linux/clk.h>
#include <linux/module.h>
#include <linux/of.h>
+#include <linux/of_platform.h>
#include <linux/platform_device.h>
#include <linux/pm.h>
#include <linux/pm_runtime.h>
@@ -896,6 +897,84 @@ static const struct media_device_ops hantro_m2m_media_ops = {
.req_queue = v4l2_m2m_request_queue,
};
+/*
+ * Some SoCs, like RK3588 have multiple identical Hantro cores, but the
+ * kernel is currently missing support for multi-core handling. Exposing
+ * separate devices for each core to userspace is bad, since that does
+ * not allow scheduling tasks properly (and creates ABI). With this workaround
+ * the driver will only probe for the first core and early exit for the other
+ * cores. Once the driver gains multi-core support, the same technique
+ * for detecting the main core can be used to cluster all cores together.
+ */
+static int hantro_disable_multicore(struct hantro_dev *vpu)
+{
+ struct device_node *node = NULL;
+ const char *compatible;
+ bool is_main_core;
+ int ret;
+
+ /* Intentionally ignores the fallback strings */
+ ret = of_property_read_string(vpu->dev->of_node, "compatible", &compatible);
+ if (ret)
+ return ret;
+
+ /* The first compatible and available node found is considered the main core */
+ do {
+ node = of_find_compatible_node(node, NULL, compatible);
+ if (of_device_is_available(node))
+ break;
+ } while (node);
+
+ if (!node)
+ return -EINVAL;
+
+ is_main_core = (vpu->dev->of_node == node);
+
+ of_node_put(node);
+
+ if (!is_main_core) {
+ dev_info(vpu->dev, "missing multi-core support, ignoring this instance\n");
+ return -ENODEV;
+ }
+
+ return 0;
+}
+
+static struct v4l2_m2m_dev *hantro_get_v4l2_m2m_dev(struct hantro_dev *vpu)
+{
+ struct device_node *node;
+ struct hantro_dev *shared_vpu;
+
+ if (!vpu->variant || !vpu->variant->shared_devices)
+ goto init_new_m2m_dev;
+
+ for_each_matching_node(node, vpu->variant->shared_devices) {
+ struct platform_device *pdev;
+ struct v4l2_m2m_dev *m2m_dev;
+
+ pdev = of_find_device_by_node(node);
+ if (!pdev)
+ continue;
+
+ shared_vpu = platform_get_drvdata(pdev);
+ if (IS_ERR_OR_NULL(shared_vpu) || shared_vpu == vpu) {
+ platform_device_put(pdev);
+ continue;
+ }
+
+ v4l2_m2m_get(shared_vpu->m2m_dev);
+ m2m_dev = shared_vpu->m2m_dev;
+ platform_device_put(pdev);
+
+ of_node_put(node);
+
+ return m2m_dev;
+ }
+
+init_new_m2m_dev:
+ return v4l2_m2m_init(&vpu_m2m_ops);
+}
+
static int hantro_probe(struct platform_device *pdev)
{
const struct of_device_id *match;
@@ -916,6 +995,10 @@ static int hantro_probe(struct platform_device *pdev)
match = of_match_node(of_hantro_match, pdev->dev.of_node);
vpu->variant = match->data;
+ ret = hantro_disable_multicore(vpu);
+ if (ret)
+ return ret;
+
/*
* Support for nxp,imx8mq-vpu is kept for backwards compatibility
* but it's deprecated. Please update your DTS file to use
@@ -1046,7 +1129,7 @@ static int hantro_probe(struct platform_device *pdev)
}
platform_set_drvdata(pdev, vpu);
- vpu->m2m_dev = v4l2_m2m_init(&vpu_m2m_ops);
+ vpu->m2m_dev = hantro_get_v4l2_m2m_dev(vpu);
if (IS_ERR(vpu->m2m_dev)) {
v4l2_err(&vpu->v4l2_dev, "Failed to init mem2mem device\n");
ret = PTR_ERR(vpu->m2m_dev);
@@ -1087,7 +1170,7 @@ static int hantro_probe(struct platform_device *pdev)
hantro_remove_enc_func(vpu);
err_m2m_rel:
media_device_cleanup(&vpu->mdev);
- v4l2_m2m_release(vpu->m2m_dev);
+ v4l2_m2m_put(vpu->m2m_dev);
err_v4l2_unreg:
v4l2_device_unregister(&vpu->v4l2_dev);
err_clk_unprepare:
@@ -1110,7 +1193,7 @@ static int hantro_remove(struct platform_device *pdev)
hantro_remove_dec_func(vpu);
hantro_remove_enc_func(vpu);
media_device_cleanup(&vpu->mdev);
- v4l2_m2m_release(vpu->m2m_dev);
+ v4l2_m2m_put(vpu->m2m_dev);
v4l2_device_unregister(&vpu->v4l2_dev);
clk_bulk_unprepare(vpu->variant->num_clocks, vpu->clocks);
reset_control_assert(vpu->resets);
diff --git a/drivers/media/platform/verisilicon/imx8m_vpu_hw.c b/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
index d89c2c3501aa..984a42db3882 100644
--- a/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
+++ b/drivers/media/platform/verisilicon/imx8m_vpu_hw.c
@@ -359,6 +359,12 @@ const struct hantro_variant imx8mq_vpu_variant = {
.num_regs = ARRAY_SIZE(imx8mq_reg_names)
};
+static const struct of_device_id imx8mq_vpu_shared_resources[] = {
+ { .compatible = "nxp,imx8mq-vpu-g1", },
+ { .compatible = "nxp,imx8mq-vpu-g2", },
+ { /* sentinel */ }
+};
+
const struct hantro_variant imx8mq_vpu_g1_variant = {
.dec_fmts = imx8m_vpu_dec_fmts,
.num_dec_fmts = ARRAY_SIZE(imx8m_vpu_dec_fmts),
@@ -372,6 +378,7 @@ const struct hantro_variant imx8mq_vpu_g1_variant = {
.num_irqs = ARRAY_SIZE(imx8mq_irqs),
.clk_names = imx8mq_g1_clk_names,
.num_clocks = ARRAY_SIZE(imx8mq_g1_clk_names),
+ .shared_devices = imx8mq_vpu_shared_resources,
};
const struct hantro_variant imx8mq_vpu_g2_variant = {
@@ -387,6 +394,7 @@ const struct hantro_variant imx8mq_vpu_g2_variant = {
.num_irqs = ARRAY_SIZE(imx8mq_g2_irqs),
.clk_names = imx8mq_g2_clk_names,
.num_clocks = ARRAY_SIZE(imx8mq_g2_clk_names),
+ .shared_devices = imx8mq_vpu_shared_resources,
};
const struct hantro_variant imx8mm_vpu_g1_variant = {
diff --git a/drivers/media/test-drivers/vivid/vivid-core.c b/drivers/media/test-drivers/vivid/vivid-core.c
index f28440e6c9f8..c8fe0a6a86cb 100644
--- a/drivers/media/test-drivers/vivid/vivid-core.c
+++ b/drivers/media/test-drivers/vivid/vivid-core.c
@@ -2163,8 +2163,10 @@ static int __init vivid_init(void)
int ret;
ret = platform_device_register(&vivid_pdev);
- if (ret)
+ if (ret) {
+ platform_device_put(&vivid_pdev);
return ret;
+ }
ret = platform_driver_register(&vivid_pdrv);
if (ret)
diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
index 97645d6509e1..64a389aa7f81 100644
--- a/drivers/media/v4l2-core/v4l2-mem2mem.c
+++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
@@ -90,6 +90,7 @@ static const char * const m2m_entity_name[] = {
* @job_work: worker to run queued jobs.
* @job_queue_flags: flags of the queue status, %QUEUE_PAUSED.
* @m2m_ops: driver callbacks
+ * @kref: device reference count
*/
struct v4l2_m2m_dev {
struct v4l2_m2m_ctx *curr_ctx;
@@ -109,6 +110,8 @@ struct v4l2_m2m_dev {
unsigned long job_queue_flags;
const struct v4l2_m2m_ops *m2m_ops;
+
+ struct kref kref;
};
static struct v4l2_m2m_queue_ctx *get_queue_ctx(struct v4l2_m2m_ctx *m2m_ctx,
@@ -1207,6 +1210,7 @@ struct v4l2_m2m_dev *v4l2_m2m_init(const struct v4l2_m2m_ops *m2m_ops)
INIT_LIST_HEAD(&m2m_dev->job_queue);
spin_lock_init(&m2m_dev->job_spinlock);
INIT_WORK(&m2m_dev->job_work, v4l2_m2m_device_run_work);
+ kref_init(&m2m_dev->kref);
return m2m_dev;
}
@@ -1218,6 +1222,25 @@ void v4l2_m2m_release(struct v4l2_m2m_dev *m2m_dev)
}
EXPORT_SYMBOL_GPL(v4l2_m2m_release);
+void v4l2_m2m_get(struct v4l2_m2m_dev *m2m_dev)
+{
+ kref_get(&m2m_dev->kref);
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_get);
+
+static void v4l2_m2m_release_from_kref(struct kref *kref)
+{
+ struct v4l2_m2m_dev *m2m_dev = container_of(kref, struct v4l2_m2m_dev, kref);
+
+ v4l2_m2m_release(m2m_dev);
+}
+
+void v4l2_m2m_put(struct v4l2_m2m_dev *m2m_dev)
+{
+ kref_put(&m2m_dev->kref, v4l2_m2m_release_from_kref);
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_put);
+
struct v4l2_m2m_ctx *v4l2_m2m_ctx_init(struct v4l2_m2m_dev *m2m_dev,
void *drv_priv,
int (*queue_init)(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq))
diff --git a/drivers/memstick/core/memstick.c b/drivers/memstick/core/memstick.c
index fc9e6db3bb72..9a2acf5c4014 100644
--- a/drivers/memstick/core/memstick.c
+++ b/drivers/memstick/core/memstick.c
@@ -367,9 +367,7 @@ int memstick_set_rw_addr(struct memstick_dev *card)
{
card->next_request = h_memstick_set_rw_addr;
memstick_new_req(card->host);
- if (!wait_for_completion_timeout(&card->mrq_complete,
- msecs_to_jiffies(500)))
- card->current_mrq.error = -ETIMEDOUT;
+ wait_for_completion(&card->mrq_complete);
return card->current_mrq.error;
}
@@ -403,9 +401,7 @@ static struct memstick_dev *memstick_alloc_card(struct memstick_host *host)
card->next_request = h_memstick_read_dev_id;
memstick_new_req(host);
- if (!wait_for_completion_timeout(&card->mrq_complete,
- msecs_to_jiffies(500)))
- card->current_mrq.error = -ETIMEDOUT;
+ wait_for_completion(&card->mrq_complete);
if (card->current_mrq.error)
goto err_out;
diff --git a/drivers/memstick/host/rtsx_usb_ms.c b/drivers/memstick/host/rtsx_usb_ms.c
index 43ec4948daa2..4c90bf886684 100644
--- a/drivers/memstick/host/rtsx_usb_ms.c
+++ b/drivers/memstick/host/rtsx_usb_ms.c
@@ -27,7 +27,6 @@ struct rtsx_usb_ms {
struct memstick_host *msh;
struct memstick_request *req;
- struct mutex host_mutex;
struct work_struct handle_req;
struct delayed_work poll_card;
@@ -511,6 +510,13 @@ static void rtsx_usb_ms_handle_req(struct work_struct *work)
struct memstick_host *msh = host->msh;
int rc;
+ /* Fail requests after eject so their waiters are released. */
+ if (host->eject) {
+ while (!memstick_next_req(msh, &host->req))
+ host->req->error = -ENOMEDIUM;
+ return;
+ }
+
if (!host->req) {
pm_runtime_get_sync(ms_dev(host));
do {
@@ -544,8 +550,7 @@ static void rtsx_usb_ms_request(struct memstick_host *msh)
dev_dbg(ms_dev(host), "--> %s\n", __func__);
- if (!host->eject)
- schedule_work(&host->handle_req);
+ schedule_work(&host->handle_req);
}
static int rtsx_usb_ms_set_param(struct memstick_host *msh,
@@ -778,7 +783,6 @@ static int rtsx_usb_ms_drv_probe(struct platform_device *pdev)
host->power_mode = MEMSTICK_POWER_OFF;
platform_set_drvdata(pdev, host);
- mutex_init(&host->host_mutex);
INIT_WORK(&host->handle_req, rtsx_usb_ms_handle_req);
INIT_DELAYED_WORK(&host->poll_card, rtsx_usb_ms_poll_card);
@@ -809,27 +813,12 @@ static int rtsx_usb_ms_drv_remove(struct platform_device *pdev)
{
struct rtsx_usb_ms *host = platform_get_drvdata(pdev);
struct memstick_host *msh = host->msh;
- int err;
host->eject = true;
msh->removing = true;
- cancel_work_sync(&host->handle_req);
+ flush_work(&host->handle_req);
cancel_delayed_work_sync(&host->poll_card);
- mutex_lock(&host->host_mutex);
- if (host->req) {
- dev_dbg(ms_dev(host),
- "%s: Controller removed during transfer\n",
- dev_name(&msh->dev));
- host->req->error = -ENOMEDIUM;
- do {
- err = memstick_next_req(msh, &host->req);
- if (!err)
- host->req->error = -ENOMEDIUM;
- } while (!err);
- }
- mutex_unlock(&host->host_mutex);
-
/* Balance possible unbalanced usage count
* e.g. unconditional module removal
*/
@@ -838,6 +827,8 @@ static int rtsx_usb_ms_drv_remove(struct platform_device *pdev)
pm_runtime_disable(ms_dev(host));
memstick_remove_host(msh);
+ /* No card, no new requests; wait for the last failed one to finish. */
+ cancel_work_sync(&host->handle_req);
dev_dbg(ms_dev(host),
": Realtek USB Memstick controller has been removed\n");
memstick_free_host(msh);
diff --git a/drivers/mfd/sm501.c b/drivers/mfd/sm501.c
index fef0dc568488..5752099683eb 100644
--- a/drivers/mfd/sm501.c
+++ b/drivers/mfd/sm501.c
@@ -1434,8 +1434,6 @@ static int sm501_plat_probe(struct platform_device *dev)
}
-#ifdef CONFIG_PM
-
/* power management support */
static void sm501_set_power(struct sm501_devdata *sm, int on)
@@ -1511,10 +1509,6 @@ static int sm501_plat_resume(struct platform_device *pdev)
return 0;
}
-#else
-#define sm501_plat_suspend NULL
-#define sm501_plat_resume NULL
-#endif
/* Initialisation data for PCI devices */
@@ -1673,9 +1667,10 @@ static void sm501_pci_remove(struct pci_dev *dev)
release_mem_region(sm->io_res->start, 0x100);
pci_disable_device(dev);
+ kfree(sm);
}
-static int sm501_plat_remove(struct platform_device *dev)
+static void sm501_plat_remove(struct platform_device *dev)
{
struct sm501_devdata *sm = platform_get_drvdata(dev);
@@ -1683,8 +1678,7 @@ static int sm501_plat_remove(struct platform_device *dev)
iounmap(sm->regs);
release_mem_region(sm->io_res->start, 0x100);
-
- return 0;
+ kfree(sm);
}
static const struct pci_device_id sm501_pci_tbl[] = {
@@ -1715,9 +1709,9 @@ static struct platform_driver sm501_plat_driver = {
.of_match_table = of_sm501_match_tbl,
},
.probe = sm501_plat_probe,
- .remove = sm501_plat_remove,
- .suspend = sm501_plat_suspend,
- .resume = sm501_plat_resume,
+ .remove_new = sm501_plat_remove,
+ .suspend = pm_sleep_ptr(sm501_plat_suspend),
+ .resume = pm_sleep_ptr(sm501_plat_resume),
};
static int __init sm501_base_init(void)
diff --git a/drivers/misc/habanalabs/common/habanalabs.h b/drivers/misc/habanalabs/common/habanalabs.h
index e6eabcdda1c5..ca4b5ce5e6f8 100644
--- a/drivers/misc/habanalabs/common/habanalabs.h
+++ b/drivers/misc/habanalabs/common/habanalabs.h
@@ -1732,8 +1732,6 @@ struct hl_cs_counters_atomic {
* struct hl_dmabuf_priv - a dma-buf private object.
* @dmabuf: pointer to dma-buf object.
* @ctx: pointer to the dma-buf owner's context.
- * @phys_pg_pack: pointer to physical page pack if the dma-buf was exported for
- * memory allocation handle.
* @device_address: physical address of the device's memory. Relevant only
* if phys_pg_pack is NULL (dma-buf was exported from address).
* The total size can be taken from the dmabuf object.
@@ -1741,7 +1739,6 @@ struct hl_cs_counters_atomic {
struct hl_dmabuf_priv {
struct dma_buf *dmabuf;
struct hl_ctx *ctx;
- struct hl_vm_phys_pg_pack *phys_pg_pack;
uint64_t device_address;
};
@@ -2105,7 +2102,6 @@ struct hl_vm_hw_block_list_node {
* @node: used to attach to deletion list that is used when all the allocations are cleared
* at the teardown of the context.
* @mapping_cnt: number of shared mappings.
- * @exporting_cnt: number of dma-buf exporting.
* @asid: the context related to this list.
* @page_size: size of each page in the pack.
* @flags: HL_MEM_* flags related to this list.
@@ -2121,7 +2117,6 @@ struct hl_vm_phys_pg_pack {
u64 total_size;
struct list_head node;
atomic_t mapping_cnt;
- u32 exporting_cnt;
u32 asid;
u32 page_size;
u32 flags;
diff --git a/drivers/misc/habanalabs/common/memory.c b/drivers/misc/habanalabs/common/memory.c
index a49038da3f6d..c4334492543f 100644
--- a/drivers/misc/habanalabs/common/memory.c
+++ b/drivers/misc/habanalabs/common/memory.c
@@ -371,12 +371,6 @@ static int free_device_memory(struct hl_ctx *ctx, struct hl_mem_in *args)
return -EINVAL;
}
- if (phys_pg_pack->exporting_cnt) {
- spin_unlock(&vm->idr_lock);
- dev_dbg(hdev->dev, "handle %u is exported, cannot free\n", handle);
- return -EINVAL;
- }
-
/* must remove from idr before the freeing of the physical pages as the refcount of the pool
* is also the trigger of the idr destroy
*/
@@ -1700,29 +1694,19 @@ static struct sg_table *hl_map_dmabuf(struct dma_buf_attachment *attachment,
enum dma_data_direction dir)
{
struct dma_buf *dma_buf = attachment->dmabuf;
- struct hl_vm_phys_pg_pack *phys_pg_pack;
struct hl_dmabuf_priv *hl_dmabuf;
struct hl_device *hdev;
struct sg_table *sgt;
hl_dmabuf = dma_buf->priv;
hdev = hl_dmabuf->ctx->hdev;
- phys_pg_pack = hl_dmabuf->phys_pg_pack;
if (!attachment->peer2peer) {
dev_dbg(hdev->dev, "Failed to map dmabuf because p2p is disabled\n");
return ERR_PTR(-EPERM);
}
- if (phys_pg_pack)
- sgt = alloc_sgt_from_device_pages(hdev,
- phys_pg_pack->pages,
- phys_pg_pack->npages,
- phys_pg_pack->page_size,
- attachment->dev,
- dir);
- else
- sgt = alloc_sgt_from_device_pages(hdev,
+ sgt = alloc_sgt_from_device_pages(hdev,
&hl_dmabuf->device_address,
1,
hl_dmabuf->dmabuf->size,
@@ -1763,18 +1747,8 @@ static void hl_unmap_dmabuf(struct dma_buf_attachment *attachment,
static void hl_release_dmabuf(struct dma_buf *dmabuf)
{
struct hl_dmabuf_priv *hl_dmabuf = dmabuf->priv;
- struct hl_ctx *ctx = hl_dmabuf->ctx;
- struct hl_device *hdev = ctx->hdev;
- struct hl_vm *vm = &hdev->vm;
-
- if (hl_dmabuf->phys_pg_pack) {
- spin_lock(&vm->idr_lock);
- hl_dmabuf->phys_pg_pack->exporting_cnt--;
- spin_unlock(&vm->idr_lock);
- }
hl_ctx_put(hl_dmabuf->ctx);
-
kfree(hl_dmabuf);
}
@@ -1785,13 +1759,18 @@ static const struct dma_buf_ops habanalabs_dmabuf_ops = {
.release = hl_release_dmabuf,
};
-static int export_dmabuf_common(struct hl_ctx *ctx,
+static int export_dmabuf(struct hl_ctx *ctx,
struct hl_dmabuf_priv *hl_dmabuf,
u64 total_size, int flags, int *dmabuf_fd)
{
DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
struct hl_device *hdev = ctx->hdev;
- int rc, fd;
+ CLASS(get_unused_fd, fd)(flags);
+
+ if (fd < 0) {
+ dev_err(hdev->dev, "failed to get a file descriptor for a dma-buf, %d\n", fd);
+ return fd;
+ }
exp_info.ops = &habanalabs_dmabuf_ops;
exp_info.size = total_size;
@@ -1804,50 +1783,20 @@ static int export_dmabuf_common(struct hl_ctx *ctx,
return PTR_ERR(hl_dmabuf->dmabuf);
}
- fd = dma_buf_fd(hl_dmabuf->dmabuf, flags);
- if (fd < 0) {
- dev_err(hdev->dev, "failed to get a file descriptor for a dma-buf\n");
- rc = fd;
- goto err_dma_buf_put;
- }
-
hl_dmabuf->ctx = ctx;
hl_ctx_get(hl_dmabuf->ctx);
*dmabuf_fd = fd;
+ fd_install(fd, hl_dmabuf->dmabuf->file);
+ fd = -EBADF;
return 0;
-
-err_dma_buf_put:
- dma_buf_put(hl_dmabuf->dmabuf);
- return rc;
}
-/**
- * export_dmabuf_from_addr() - export a dma-buf object for the given memory
- * address and size.
- * @ctx: pointer to the context structure.
- * @device_addr: device memory physical address.
- * @size: size of device memory.
- * @flags: DMA-BUF file/FD flags.
- * @dmabuf_fd: pointer to result FD that represents the dma-buf object.
- *
- * Create and export a dma-buf object for an existing memory allocation inside
- * the device memory, and return a FD which is associated with the dma-buf
- * object.
- *
- * Return: 0 on success, non-zero for failure.
- */
-static int export_dmabuf_from_addr(struct hl_ctx *ctx, u64 device_addr,
- u64 size, int flags, int *dmabuf_fd)
+static int validate_export_params(struct hl_device *hdev, u64 device_addr, u64 size)
{
- struct hl_dmabuf_priv *hl_dmabuf;
- struct hl_device *hdev = ctx->hdev;
- struct asic_fixed_properties *prop;
+ struct asic_fixed_properties *prop = &hdev->asic_prop;
u64 bar_address;
- int rc;
-
- prop = &hdev->asic_prop;
if (!IS_ALIGNED(device_addr, PAGE_SIZE)) {
dev_dbg(hdev->dev,
@@ -1864,48 +1813,33 @@ static int export_dmabuf_from_addr(struct hl_ctx *ctx, u64 device_addr,
}
if (device_addr < prop->dram_user_base_address ||
- device_addr + size > prop->dram_end_address ||
- device_addr + size < device_addr) {
+ (device_addr + size) > prop->dram_end_address ||
+ (device_addr + size) < device_addr) {
dev_dbg(hdev->dev,
"DRAM memory range 0x%llx (+0x%llx) is outside of DRAM boundaries\n",
device_addr, size);
return -EINVAL;
}
- bar_address = hdev->dram_pci_bar_start +
- (device_addr - prop->dram_base_address);
+ bar_address = hdev->dram_pci_bar_start + (device_addr - prop->dram_base_address);
- if (bar_address + size >
- hdev->dram_pci_bar_start + prop->dram_pci_bar_size ||
- bar_address + size < bar_address) {
+ if ((bar_address + size) > (hdev->dram_pci_bar_start + prop->dram_pci_bar_size) ||
+ (bar_address + size) < bar_address) {
dev_dbg(hdev->dev,
"DRAM memory range 0x%llx (+0x%llx) is outside of PCI BAR boundaries\n",
device_addr, size);
return -EINVAL;
}
- hl_dmabuf = kzalloc(sizeof(*hl_dmabuf), GFP_KERNEL);
- if (!hl_dmabuf)
- return -ENOMEM;
-
- hl_dmabuf->device_address = device_addr;
-
- rc = export_dmabuf_common(ctx, hl_dmabuf, size, flags, dmabuf_fd);
- if (rc)
- goto err_free_dmabuf_wrapper;
-
return 0;
-
-err_free_dmabuf_wrapper:
- kfree(hl_dmabuf);
- return rc;
}
/**
- * export_dmabuf_from_handle() - export a dma-buf object for the given memory
- * handle.
+ * export_dmabuf_from_addr() - export a dma-buf object for the given memory
+ * address and size.
* @ctx: pointer to the context structure.
- * @handle: device memory allocation handle.
+ * @device_addr: device memory physical address.
+ * @size: size of device memory.
* @flags: DMA-BUF file/FD flags.
* @dmabuf_fd: pointer to result FD that represents the dma-buf object.
*
@@ -1915,74 +1849,32 @@ static int export_dmabuf_from_addr(struct hl_ctx *ctx, u64 device_addr,
*
* Return: 0 on success, non-zero for failure.
*/
-static int export_dmabuf_from_handle(struct hl_ctx *ctx, u64 handle, int flags,
- int *dmabuf_fd)
+static int export_dmabuf_from_addr(struct hl_ctx *ctx, u64 device_addr,
+ u64 size, int flags, int *dmabuf_fd)
{
- struct hl_vm_phys_pg_pack *phys_pg_pack;
struct hl_dmabuf_priv *hl_dmabuf;
struct hl_device *hdev = ctx->hdev;
struct asic_fixed_properties *prop;
- struct hl_vm *vm = &hdev->vm;
- u64 bar_address;
- int rc, i;
+ int rc;
prop = &hdev->asic_prop;
- if (upper_32_bits(handle)) {
- dev_dbg(hdev->dev, "no match for handle 0x%llx\n", handle);
- return -EINVAL;
- }
-
- spin_lock(&vm->idr_lock);
-
- phys_pg_pack = idr_find(&vm->phys_pg_pack_handles, (u32) handle);
- if (!phys_pg_pack) {
- spin_unlock(&vm->idr_lock);
- dev_dbg(hdev->dev, "no match for handle 0x%x\n", (u32) handle);
- return -EINVAL;
+ if (prop->dram_supports_virtual_memory) {
+ dev_dbg(hdev->dev, "Export not supported for devices with virtual memory\n");
+ return -EOPNOTSUPP;
}
- /* increment now to avoid freeing device memory while exporting */
- phys_pg_pack->exporting_cnt++;
-
- spin_unlock(&vm->idr_lock);
-
- if (phys_pg_pack->vm_type != VM_TYPE_PHYS_PACK) {
- dev_dbg(hdev->dev, "handle 0x%llx does not represent DRAM memory\n", handle);
- rc = -EINVAL;
- goto err_dec_exporting_cnt;
- }
-
- for (i = 0 ; i < phys_pg_pack->npages ; i++) {
-
- bar_address = hdev->dram_pci_bar_start +
- (phys_pg_pack->pages[i] -
- prop->dram_base_address);
-
- if (bar_address + phys_pg_pack->page_size >
- hdev->dram_pci_bar_start + prop->dram_pci_bar_size ||
- bar_address + phys_pg_pack->page_size < bar_address) {
-
- dev_dbg(hdev->dev,
- "DRAM memory range 0x%llx (+0x%x) is outside of PCI BAR boundaries\n",
- phys_pg_pack->pages[i],
- phys_pg_pack->page_size);
-
- rc = -EINVAL;
- goto err_dec_exporting_cnt;
- }
- }
+ rc = validate_export_params(hdev, device_addr, size);
+ if (rc)
+ return rc;
hl_dmabuf = kzalloc(sizeof(*hl_dmabuf), GFP_KERNEL);
- if (!hl_dmabuf) {
- rc = -ENOMEM;
- goto err_dec_exporting_cnt;
- }
+ if (!hl_dmabuf)
+ return -ENOMEM;
- hl_dmabuf->phys_pg_pack = phys_pg_pack;
+ hl_dmabuf->device_address = device_addr;
- rc = export_dmabuf_common(ctx, hl_dmabuf, phys_pg_pack->total_size,
- flags, dmabuf_fd);
+ rc = export_dmabuf(ctx, hl_dmabuf, size, flags, dmabuf_fd);
if (rc)
goto err_free_dmabuf_wrapper;
@@ -1990,12 +1882,6 @@ static int export_dmabuf_from_handle(struct hl_ctx *ctx, u64 handle, int flags,
err_free_dmabuf_wrapper:
kfree(hl_dmabuf);
-
-err_dec_exporting_cnt:
- spin_lock(&vm->idr_lock);
- phys_pg_pack->exporting_cnt--;
- spin_unlock(&vm->idr_lock);
-
return rc;
}
@@ -2270,13 +2156,7 @@ int hl_mem_ioctl(struct hl_fpriv *hpriv, void *data)
break;
case HL_MEM_OP_EXPORT_DMABUF_FD:
- if (hdev->asic_prop.dram_supports_virtual_memory)
- rc = export_dmabuf_from_handle(ctx,
- args->in.export_dmabuf_fd.handle,
- args->in.flags,
- &dmabuf_fd);
- else
- rc = export_dmabuf_from_addr(ctx,
+ rc = export_dmabuf_from_addr(ctx,
args->in.export_dmabuf_fd.handle,
args->in.export_dmabuf_fd.mem_size,
args->in.flags,
diff --git a/drivers/misc/habanalabs/gaudi/gaudi.c b/drivers/misc/habanalabs/gaudi/gaudi.c
index 92560414e843..7224a4636b68 100644
--- a/drivers/misc/habanalabs/gaudi/gaudi.c
+++ b/drivers/misc/habanalabs/gaudi/gaudi.c
@@ -4239,10 +4239,29 @@ static int gaudi_mmap(struct hl_device *hdev, struct vm_area_struct *vma,
vma->vm_flags |= VM_IO | VM_PFNMAP | VM_DONTEXPAND | VM_DONTDUMP |
VM_DONTCOPY | VM_NORESERVE;
+#ifdef _HAS_DMA_MMAP_COHERENT
+ /*
+ * If dma_alloc_coherent() returns a vmalloc address, set VM_MIXEDMAP
+ * so vm_insert_page() can handle it safely. Without this, the kernel
+ * may BUG_ON due to VM_PFNMAP.
+ */
+ if (is_vmalloc_addr(cpu_addr))
+ vm_flags_set(vma, VM_MIXEDMAP);
+
rc = dma_mmap_coherent(hdev->dev, vma, cpu_addr,
(dma_addr - HOST_PHYS_BASE), size);
if (rc)
dev_err(hdev->dev, "dma_mmap_coherent error %d", rc);
+#else
+
+ rc = remap_pfn_range(vma, vma->vm_start,
+ virt_to_phys(cpu_addr) >> PAGE_SHIFT,
+ size, vma->vm_page_prot);
+ if (rc)
+ dev_err(hdev->dev, "remap_pfn_range error %d", rc);
+
+ #endif
+
return rc;
}
diff --git a/drivers/misc/habanalabs/gaudi2/gaudi2.c b/drivers/misc/habanalabs/gaudi2/gaudi2.c
index 65e6cae6100a..e043f4d2343b 100644
--- a/drivers/misc/habanalabs/gaudi2/gaudi2.c
+++ b/drivers/misc/habanalabs/gaudi2/gaudi2.c
@@ -5603,6 +5603,13 @@ static int gaudi2_mmap(struct hl_device *hdev, struct vm_area_struct *vma,
VM_DONTCOPY | VM_NORESERVE;
#ifdef _HAS_DMA_MMAP_COHERENT
+ /*
+ * If dma_alloc_coherent() returns a vmalloc address, set VM_MIXEDMAP
+ * so vm_insert_page() can handle it safely. Without this, the kernel
+ * may BUG_ON due to VM_PFNMAP.
+ */
+ if (is_vmalloc_addr(cpu_addr))
+ vm_flags_set(vma, VM_MIXEDMAP);
rc = dma_mmap_coherent(hdev->dev, vma, cpu_addr, dma_addr, size);
if (rc)
diff --git a/drivers/misc/sgi-gru/grufault.c b/drivers/misc/sgi-gru/grufault.c
index b836936e9747..c2f38e650b66 100644
--- a/drivers/misc/sgi-gru/grufault.c
+++ b/drivers/misc/sgi-gru/grufault.c
@@ -166,13 +166,8 @@ static void get_clear_fault_map(struct gru_state *gru,
}
/*
- * Atomic (interrupt context) & non-atomic (user context) functions to
- * convert a vaddr into a physical address. The size of the page
- * is returned in pageshift.
- * returns:
- * 0 - successful
- * < 0 - error code
- * 1 - (atomic only) try again in non-atomic context
+ * Convert a user virtual address to a physical address in process context.
+ * The size of the page is returned in pageshift.
*/
static int non_atomic_pte_lookup(struct vm_area_struct *vma,
unsigned long vaddr, int write,
@@ -192,87 +187,25 @@ static int non_atomic_pte_lookup(struct vm_area_struct *vma,
return 0;
}
-/*
- * atomic_pte_lookup
- *
- * Convert a user virtual address to a physical address
- * Only supports Intel large pages (2MB only) on x86_64.
- * ZZZ - hugepage support is incomplete
- *
- * NOTE: mmap_lock is already held on entry to this function. This
- * guarantees existence of the page tables.
- */
-static int atomic_pte_lookup(struct vm_area_struct *vma, unsigned long vaddr,
- int write, unsigned long *paddr, int *pageshift)
-{
- pgd_t *pgdp;
- p4d_t *p4dp;
- pud_t *pudp;
- pmd_t *pmdp;
- pte_t pte;
-
- pgdp = pgd_offset(vma->vm_mm, vaddr);
- if (unlikely(pgd_none(*pgdp)))
- goto err;
-
- p4dp = p4d_offset(pgdp, vaddr);
- if (unlikely(p4d_none(*p4dp)))
- goto err;
-
- pudp = pud_offset(p4dp, vaddr);
- if (unlikely(pud_none(*pudp)))
- goto err;
-
- pmdp = pmd_offset(pudp, vaddr);
- if (unlikely(pmd_none(*pmdp)))
- goto err;
-#ifdef CONFIG_X86_64
- if (unlikely(pmd_large(*pmdp)))
- pte = *(pte_t *) pmdp;
- else
-#endif
- pte = *pte_offset_kernel(pmdp, vaddr);
-
- if (unlikely(!pte_present(pte) ||
- (write && (!pte_write(pte) || !pte_dirty(pte)))))
- return 1;
-
- *paddr = pte_pfn(pte) << PAGE_SHIFT;
-#ifdef CONFIG_HUGETLB_PAGE
- *pageshift = is_vm_hugetlb_page(vma) ? HPAGE_SHIFT : PAGE_SHIFT;
-#else
- *pageshift = PAGE_SHIFT;
-#endif
- return 0;
-
-err:
- return 1;
-}
-
static int gru_vtop(struct gru_thread_state *gts, unsigned long vaddr,
int write, int atomic, unsigned long *gpa, int *pageshift)
{
struct mm_struct *mm = gts->ts_mm;
struct vm_area_struct *vma;
unsigned long paddr;
- int ret, ps;
+ int ps;
vma = find_vma(mm, vaddr);
if (!vma)
goto inval;
- /*
- * Atomic lookup is faster & usually works even if called in non-atomic
- * context.
- */
- rmb(); /* Must/check ms_range_active before loading PTEs */
- ret = atomic_pte_lookup(vma, vaddr, write, &paddr, &ps);
- if (ret) {
- if (atomic)
- goto upm;
- if (non_atomic_pte_lookup(vma, vaddr, write, &paddr, &ps))
- goto inval;
- }
+ if (atomic)
+ goto upm;
+
+ /* Order the caller's ms_range_active check before loading PTEs. */
+ rmb();
+ if (non_atomic_pte_lookup(vma, vaddr, write, &paddr, &ps))
+ goto inval;
if (is_gru_paddr(paddr))
goto inval;
paddr = paddr & ~((1UL << ps) - 1);
@@ -569,19 +502,9 @@ static irqreturn_t gru_intr(int chiplet, int blade)
continue;
}
- /*
- * This is running in interrupt context. Trylock the mmap_lock.
- * If it fails, retry the fault in user context.
- */
+ /* Address translation may sleep, so retry the fault in user context. */
gts->ustats.fmm_tlbmiss++;
- if (!gts->ts_force_cch_reload &&
- mmap_read_trylock(gts->ts_mm)) {
- gru_try_dropin(gru, gts, tfh, NULL);
- mmap_read_unlock(gts->ts_mm);
- } else {
- tfh_user_polling_mode(tfh);
- STAT(intr_mm_lock_failed);
- }
+ tfh_user_polling_mode(tfh);
}
return IRQ_HANDLED;
}
diff --git a/drivers/misc/sgi-gru/gruprocfs.c b/drivers/misc/sgi-gru/gruprocfs.c
index 97b8b38ab47d..b8139c27bc7f 100644
--- a/drivers/misc/sgi-gru/gruprocfs.c
+++ b/drivers/misc/sgi-gru/gruprocfs.c
@@ -54,7 +54,6 @@ static int statistics_show(struct seq_file *s, void *p)
printstat(s, intr_cbr);
printstat(s, intr_tfh);
printstat(s, intr_spurious);
- printstat(s, intr_mm_lock_failed);
printstat(s, call_os);
printstat(s, call_os_wait_queue);
printstat(s, user_flush_tlb);
diff --git a/drivers/misc/sgi-gru/grutables.h b/drivers/misc/sgi-gru/grutables.h
index 640daf1994df..3348552925c6 100644
--- a/drivers/misc/sgi-gru/grutables.h
+++ b/drivers/misc/sgi-gru/grutables.h
@@ -182,7 +182,6 @@ struct gru_stats_s {
atomic_long_t intr_cbr;
atomic_long_t intr_tfh;
atomic_long_t intr_spurious;
- atomic_long_t intr_mm_lock_failed;
atomic_long_t call_os;
atomic_long_t call_os_wait_queue;
atomic_long_t user_flush_tlb;
diff --git a/drivers/mmc/host/cavium-octeon.c b/drivers/mmc/host/cavium-octeon.c
index 12dca91a8ef6..ce55689ffe5c 100644
--- a/drivers/mmc/host/cavium-octeon.c
+++ b/drivers/mmc/host/cavium-octeon.c
@@ -300,9 +300,12 @@ static int octeon_mmc_remove(struct platform_device *pdev)
u64 dma_cfg;
int i;
- for (i = 0; i < CAVIUM_MAX_MMC; i++)
+ for (i = 0; i < CAVIUM_MAX_MMC; i++) {
if (host->slot[i])
cvm_mmc_of_slot_remove(host->slot[i]);
+ if (host->slot_pdev[i])
+ of_platform_device_destroy(&host->slot_pdev[i]->dev, NULL);
+ }
dma_cfg = readq(host->dma_base + MIO_EMM_DMA_CFG(host));
dma_cfg &= ~MIO_EMM_DMA_CFG_EN;
diff --git a/drivers/mmc/host/cavium-thunderx.c b/drivers/mmc/host/cavium-thunderx.c
index 202b1d6da678..cba939b34341 100644
--- a/drivers/mmc/host/cavium-thunderx.c
+++ b/drivers/mmc/host/cavium-thunderx.c
@@ -173,9 +173,12 @@ static void thunder_mmc_remove(struct pci_dev *pdev)
u64 dma_cfg;
int i;
- for (i = 0; i < CAVIUM_MAX_MMC; i++)
+ for (i = 0; i < CAVIUM_MAX_MMC; i++) {
if (host->slot[i])
cvm_mmc_of_slot_remove(host->slot[i]);
+ if (host->slot_pdev[i])
+ of_platform_device_destroy(&host->slot_pdev[i]->dev, NULL);
+ }
dma_cfg = readq(host->dma_base + MIO_EMM_DMA_CFG(host));
dma_cfg &= ~MIO_EMM_DMA_CFG_EN;
diff --git a/drivers/mmc/host/mtk-sd.c b/drivers/mmc/host/mtk-sd.c
index a766e34402cb..68da866b8864 100644
--- a/drivers/mmc/host/mtk-sd.c
+++ b/drivers/mmc/host/mtk-sd.c
@@ -2854,6 +2854,7 @@ static int msdc_drv_remove(struct platform_device *pdev)
platform_set_drvdata(pdev, NULL);
mmc_remove_host(mmc);
+ cancel_delayed_work_sync(&host->req_timeout);
msdc_deinit_hw(host);
msdc_gate_clock(host);
diff --git a/drivers/mmc/host/sdhci-sprd.c b/drivers/mmc/host/sdhci-sprd.c
index 66c1782823d8..2d2d6d3bf6b7 100644
--- a/drivers/mmc/host/sdhci-sprd.c
+++ b/drivers/mmc/host/sdhci-sprd.c
@@ -756,6 +756,10 @@ static int sdhci_sprd_remove(struct platform_device *pdev)
struct sdhci_host *host = platform_get_drvdata(pdev);
struct sdhci_sprd_host *sprd_host = TO_SPRD_HOST(host);
+ pm_runtime_get_sync(&pdev->dev);
+ pm_runtime_disable(&pdev->dev);
+ pm_runtime_put_noidle(&pdev->dev);
+
sdhci_remove_host(host, 0);
clk_disable_unprepare(sprd_host->clk_sdio);
diff --git a/drivers/mmc/host/via-sdmmc.c b/drivers/mmc/host/via-sdmmc.c
index 5bf508b6db0e..3ae8f005f8fe 100644
--- a/drivers/mmc/host/via-sdmmc.c
+++ b/drivers/mmc/host/via-sdmmc.c
@@ -1208,6 +1208,10 @@ static void via_sd_remove(struct pci_dev *pcidev)
free_irq(pcidev->irq, sdhost);
+ cancel_work_sync(&sdhost->carddet_work);
+ /* carddet_work may re-enable the interrupt via via_reset_pcictrl(). */
+ writeb(0x0, sdhost->pcictrl_mmiobase + VIA_CRDR_PCIINTCTRL);
+
del_timer_sync(&sdhost->timer);
tasklet_kill(&sdhost->finish_tasklet);
diff --git a/drivers/mtd/devices/block2mtd.c b/drivers/mtd/devices/block2mtd.c
index 4cd37ec45762..9175196110c6 100644
--- a/drivers/mtd/devices/block2mtd.c
+++ b/drivers/mtd/devices/block2mtd.c
@@ -272,7 +272,7 @@ static struct block2mtd_dev *add_device(char *devname, int erase_size,
goto err_free_block2mtd;
}
- if ((long)dev->blkdev->bd_inode->i_size % erase_size) {
+ if (!erase_size || (long)dev->blkdev->bd_inode->i_size % erase_size) {
pr_err("erasesize must be a divisor of device size\n");
goto err_free_block2mtd;
}
diff --git a/drivers/mtd/maps/vmu-flash.c b/drivers/mtd/maps/vmu-flash.c
index f96a165fccc7..7ae5ecd01ff0 100644
--- a/drivers/mtd/maps/vmu-flash.c
+++ b/drivers/mtd/maps/vmu-flash.c
@@ -610,7 +610,7 @@ static int vmu_connect(struct maple_device *mdev)
basic_flash_data = be32_to_cpu(mdev->devinfo.function_data[c - 1]);
- card = kmalloc(sizeof(struct memcard), GFP_KERNEL);
+ card = kzalloc(sizeof(struct memcard), GFP_KERNEL);
if (!card) {
error = -ENOMEM;
goto fail_nomem;
@@ -628,15 +628,15 @@ static int vmu_connect(struct maple_device *mdev)
* Not sure there are actually any multi-partition devices in the
* real world, but the hardware supports them, so, so will we
*/
- card->parts = kmalloc_array(card->partitions, sizeof(struct vmupart),
- GFP_KERNEL);
+ card->parts = kcalloc(card->partitions, sizeof(struct vmupart),
+ GFP_KERNEL);
if (!card->parts) {
error = -ENOMEM;
goto fail_partitions;
}
- card->mtd = kmalloc_array(card->partitions, sizeof(struct mtd_info),
- GFP_KERNEL);
+ card->mtd = kcalloc(card->partitions, sizeof(struct mtd_info),
+ GFP_KERNEL);
if (!card->mtd) {
error = -ENOMEM;
goto fail_mtd_info;
@@ -720,7 +720,7 @@ static int vmu_can_unload(struct maple_device *mdev)
card = maple_get_drvdata(mdev);
for (x = 0; x < card->partitions; x++) {
mtd = &((card->mtd)[x]);
- if (mtd->usecount > 0)
+ if (kref_read(&mtd->refcnt))
return 0;
}
return 1;
diff --git a/drivers/mtd/mtdcore.c b/drivers/mtd/mtdcore.c
index ad527bdbd463..de2167952844 100644
--- a/drivers/mtd/mtdcore.c
+++ b/drivers/mtd/mtdcore.c
@@ -92,10 +92,48 @@ static void mtd_release(struct device *dev)
struct mtd_info *mtd = dev_get_drvdata(dev);
dev_t index = MTD_DEVT(mtd->index);
+ idr_remove(&mtd_idr, mtd->index);
+ of_node_put(mtd_get_of_node(mtd));
+
+ if (mtd_is_partition(mtd))
+ release_mtd_partition(mtd);
+
/* remove /dev/mtdXro node */
device_destroy(&mtd_class, index + 1);
}
+/*
+ * No-op device release used in add_mtd_device() error paths.
+ * Prevents mtd_release() from being called via device_release(),
+ * which would free the mtd_info that the caller still manages.
+ */
+static void mtd_dev_release_nop(struct device *dev)
+{
+}
+
+static void mtd_device_release(struct kref *kref)
+{
+ struct mtd_info *mtd = container_of(kref, struct mtd_info, refcnt);
+ bool is_partition = mtd_is_partition(mtd);
+
+ debugfs_remove_recursive(mtd->dbg.dfs_dir);
+
+ /* Try to remove the NVMEM provider */
+ nvmem_unregister(mtd->nvmem);
+
+ device_unregister(&mtd->dev);
+
+ /*
+ * Clear dev so mtd can be safely re-registered later if desired.
+ * Should not be done for partition,
+ * as it was already destroyed in device_unregister().
+ */
+ if (!is_partition)
+ memset(&mtd->dev, 0, sizeof(mtd->dev));
+
+ module_put(THIS_MODULE);
+}
+
#define MTD_DEVICE_ATTR_RO(name) \
static DEVICE_ATTR(name, 0444, mtd_##name##_show, NULL)
@@ -674,7 +712,7 @@ int add_mtd_device(struct mtd_info *mtd)
}
mtd->index = i;
- mtd->usecount = 0;
+ kref_init(&mtd->refcnt);
/* default value if not set by driver */
if (mtd->bitflip_threshold == 0)
@@ -723,10 +761,8 @@ int add_mtd_device(struct mtd_info *mtd)
mtd_check_of_node(mtd);
of_node_get(mtd_get_of_node(mtd));
error = device_register(&mtd->dev);
- if (error) {
- put_device(&mtd->dev);
+ if (error)
goto fail_added;
- }
/* Add the nvmem provider */
error = mtd_nvmem_add(mtd);
@@ -753,8 +789,16 @@ int add_mtd_device(struct mtd_info *mtd)
return 0;
fail_nvmem_add:
- device_unregister(&mtd->dev);
+ device_del(&mtd->dev);
fail_added:
+ /*
+ * Clear type and set nop release to prevent mtd_release() ->
+ * release_mtd_partition() -> free_partition() from freeing mtd.
+ * The caller handles cleanup on failure.
+ */
+ mtd->dev.type = NULL;
+ mtd->dev.release = mtd_dev_release_nop;
+ put_device(&mtd->dev);
of_node_put(mtd_get_of_node(mtd));
idr_remove(&mtd_idr, i);
fail_locked:
@@ -776,7 +820,6 @@ int del_mtd_device(struct mtd_info *mtd)
{
int ret;
struct mtd_notifier *not;
- struct device_node *mtd_of_node;
mutex_lock(&mtd_table_mutex);
@@ -790,28 +833,8 @@ int del_mtd_device(struct mtd_info *mtd)
list_for_each_entry(not, &mtd_notifiers, list)
not->remove(mtd);
- if (mtd->usecount) {
- printk(KERN_NOTICE "Removing MTD device #%d (%s) with use count %d\n",
- mtd->index, mtd->name, mtd->usecount);
- ret = -EBUSY;
- } else {
- mtd_of_node = mtd_get_of_node(mtd);
- debugfs_remove_recursive(mtd->dbg.dfs_dir);
-
- /* Try to remove the NVMEM provider */
- nvmem_unregister(mtd->nvmem);
-
- device_unregister(&mtd->dev);
-
- /* Clear dev so mtd can be safely re-registered later if desired */
- memset(&mtd->dev, 0, sizeof(mtd->dev));
-
- idr_remove(&mtd_idr, mtd->index);
- of_node_put(mtd_of_node);
-
- module_put(THIS_MODULE);
- ret = 0;
- }
+ kref_put(&mtd->refcnt, mtd_device_release);
+ ret = 0;
out_error:
mutex_unlock(&mtd_table_mutex);
@@ -1207,25 +1230,27 @@ int __get_mtd_device(struct mtd_info *mtd)
struct mtd_info *master = mtd_get_master(mtd);
int err;
- if (!try_module_get(master->owner))
- return -ENODEV;
-
if (master->_get_device) {
- err = master->_get_device(mtd);
-
- if (err) {
- module_put(master->owner);
+ err = master->_get_device(master);
+ if (err)
return err;
- }
}
- master->usecount++;
+ if (!try_module_get(master->owner)) {
+ if (master->_put_device)
+ master->_put_device(master);
+ return -ENODEV;
+ }
- while (mtd->parent) {
- mtd->usecount++;
+ while (mtd) {
+ if (mtd != master)
+ kref_get(&mtd->refcnt);
mtd = mtd->parent;
}
+ if (IS_ENABLED(CONFIG_MTD_PARTITIONED_MASTER))
+ kref_get(&master->refcnt);
+
return 0;
}
EXPORT_SYMBOL_GPL(__get_mtd_device);
@@ -1309,18 +1334,23 @@ void __put_mtd_device(struct mtd_info *mtd)
{
struct mtd_info *master = mtd_get_master(mtd);
- while (mtd->parent) {
- --mtd->usecount;
- BUG_ON(mtd->usecount < 0);
- mtd = mtd->parent;
+ while (mtd) {
+ /* kref_put() can relese mtd, so keep a reference mtd->parent */
+ struct mtd_info *parent = mtd->parent;
+
+ if (mtd != master)
+ kref_put(&mtd->refcnt, mtd_device_release);
+ mtd = parent;
}
- master->usecount--;
+ if (IS_ENABLED(CONFIG_MTD_PARTITIONED_MASTER))
+ kref_put(&master->refcnt, mtd_device_release);
+
+ module_put(master->owner);
+ /* must be the last as master can be freed in the _put_device */
if (master->_put_device)
master->_put_device(master);
-
- module_put(master->owner);
}
EXPORT_SYMBOL_GPL(__put_mtd_device);
diff --git a/drivers/mtd/mtdcore.h b/drivers/mtd/mtdcore.h
index b5eefeabf310..b014861a06a6 100644
--- a/drivers/mtd/mtdcore.h
+++ b/drivers/mtd/mtdcore.h
@@ -12,6 +12,7 @@ int __must_check add_mtd_device(struct mtd_info *mtd);
int del_mtd_device(struct mtd_info *mtd);
int add_mtd_partitions(struct mtd_info *, const struct mtd_partition *, int);
int del_mtd_partitions(struct mtd_info *);
+void release_mtd_partition(struct mtd_info *mtd);
struct mtd_partitions;
diff --git a/drivers/mtd/mtdpart.c b/drivers/mtd/mtdpart.c
index f4419fdf36a0..f9cb2d972040 100644
--- a/drivers/mtd/mtdpart.c
+++ b/drivers/mtd/mtdpart.c
@@ -32,6 +32,12 @@ static inline void free_partition(struct mtd_info *mtd)
kfree(mtd);
}
+void release_mtd_partition(struct mtd_info *mtd)
+{
+ WARN_ON(!list_empty(&mtd->part.node));
+ free_partition(mtd);
+}
+
static struct mtd_info *allocate_partition(struct mtd_info *parent,
const struct mtd_partition *part,
int partno, uint64_t cur_offset)
@@ -310,13 +316,11 @@ static int __mtd_del_partition(struct mtd_info *mtd)
sysfs_remove_files(&mtd->dev.kobj, mtd_partition_attrs);
+ list_del_init(&mtd->part.node);
err = del_mtd_device(mtd);
if (err)
return err;
- list_del(&mtd->part.node);
- free_partition(mtd);
-
return 0;
}
@@ -335,6 +339,7 @@ static int __del_mtd_partitions(struct mtd_info *mtd)
__del_mtd_partitions(child);
pr_info("Deleting %s MTD partition\n", child->name);
+ list_del_init(&child->part.node);
ret = del_mtd_device(child);
if (ret < 0) {
pr_err("Error when deleting partition \"%s\" (%d)\n",
@@ -342,9 +347,6 @@ static int __del_mtd_partitions(struct mtd_info *mtd)
err = ret;
continue;
}
-
- list_del(&child->part.node);
- free_partition(child);
}
return err;
diff --git a/drivers/mtd/nand/raw/cadence-nand-controller.c b/drivers/mtd/nand/raw/cadence-nand-controller.c
index 717d24d2cf53..27d38d53d659 100644
--- a/drivers/mtd/nand/raw/cadence-nand-controller.c
+++ b/drivers/mtd/nand/raw/cadence-nand-controller.c
@@ -2847,6 +2847,9 @@ static int cadence_nand_init(struct cdns_nand_ctrl *cdns_ctrl)
goto free_buf_desc;
}
+ spin_lock_init(&cdns_ctrl->irq_lock);
+ init_completion(&cdns_ctrl->complete);
+
if (devm_request_irq(cdns_ctrl->dev, cdns_ctrl->irq, cadence_nand_isr,
IRQF_SHARED, "cadence-nand-controller",
cdns_ctrl)) {
@@ -2855,9 +2858,6 @@ static int cadence_nand_init(struct cdns_nand_ctrl *cdns_ctrl)
goto free_buf;
}
- spin_lock_init(&cdns_ctrl->irq_lock);
- init_completion(&cdns_ctrl->complete);
-
ret = cadence_nand_hw_init(cdns_ctrl);
if (ret)
goto disable_irq;
diff --git a/drivers/mtd/nand/raw/vf610_nfc.c b/drivers/mtd/nand/raw/vf610_nfc.c
index a2b89b75073f..391fff635747 100644
--- a/drivers/mtd/nand/raw/vf610_nfc.c
+++ b/drivers/mtd/nand/raw/vf610_nfc.c
@@ -504,6 +504,11 @@ static int vf610_nfc_exec_op(struct nand_chip *chip,
check_only);
}
+static unsigned int vf610_nfc_spare_size(struct mtd_info *mtd)
+{
+ return min_t(unsigned int, mtd->oobsize, 64);
+}
+
static inline int vf610_nfc_correct_data(struct nand_chip *chip, uint8_t *dat,
uint8_t *oob, int page)
{
@@ -513,6 +518,7 @@ static inline int vf610_nfc_correct_data(struct nand_chip *chip, uint8_t *dat,
u8 ecc_status;
u8 ecc_count;
int flips_threshold = nfc->chip.ecc.strength / 2;
+ int ret;
ecc_status = vf610_nfc_read(nfc, ecc_status_off) & 0xff;
ecc_count = ecc_status & ECC_STATUS_ERR_COUNT;
@@ -520,16 +526,22 @@ static inline int vf610_nfc_correct_data(struct nand_chip *chip, uint8_t *dat,
if (!(ecc_status & ECC_STATUS_MASK))
return ecc_count;
+ /* The failed decode leaves a bogus correction in SRAM; re-read without ECC */
nfc->data_access = true;
- nand_read_oob_op(&nfc->chip, page, 0, oob, mtd->oobsize);
+ ret = nand_read_page_op(&nfc->chip, page, 0, dat, mtd->writesize);
+ if (!ret)
+ ret = nand_read_oob_op(&nfc->chip, page, 0, oob,
+ vf610_nfc_spare_size(mtd));
nfc->data_access = false;
+ if (ret)
+ return ret;
/*
- * On an erased page, bit count (including OOB) should be zero or
- * at least less then half of the ECC strength.
+ * Run the erased-page check with the driver's historic threshold
+ * of half the ECC strength.
*/
- return nand_check_erased_ecc_chunk(dat, nfc->chip.ecc.size, oob,
- mtd->oobsize, NULL, 0,
+ return nand_check_erased_ecc_chunk(dat, mtd->writesize, oob,
+ vf610_nfc_spare_size(mtd), NULL, 0,
flips_threshold);
}
@@ -550,7 +562,7 @@ static int vf610_nfc_read_page(struct nand_chip *chip, uint8_t *buf,
{
struct vf610_nfc *nfc = chip_to_nfc(chip);
struct mtd_info *mtd = nand_to_mtd(chip);
- int trfr_sz = mtd->writesize + mtd->oobsize;
+ int trfr_sz = mtd->writesize + vf610_nfc_spare_size(mtd);
u32 row = 0, cmd1 = 0, cmd2 = 0, code = 0;
int stat;
@@ -576,11 +588,16 @@ static int vf610_nfc_read_page(struct nand_chip *chip, uint8_t *buf,
*/
vf610_nfc_rd_from_sram(buf, nfc->regs + NFC_MAIN_AREA(0),
mtd->writesize, false);
- if (oob_required)
+ if (oob_required) {
+ unsigned int spare = vf610_nfc_spare_size(mtd);
+
vf610_nfc_rd_from_sram(chip->oob_poi,
nfc->regs + NFC_MAIN_AREA(0) +
mtd->writesize,
- mtd->oobsize, false);
+ spare, false);
+ /* Not transferred, and never written: reads back erased */
+ memset(chip->oob_poi + spare, 0xff, mtd->oobsize - spare);
+ }
stat = vf610_nfc_correct_data(chip, buf, chip->oob_poi, page);
@@ -598,7 +615,7 @@ static int vf610_nfc_write_page(struct nand_chip *chip, const uint8_t *buf,
{
struct vf610_nfc *nfc = chip_to_nfc(chip);
struct mtd_info *mtd = nand_to_mtd(chip);
- int trfr_sz = mtd->writesize + mtd->oobsize;
+ int trfr_sz = mtd->writesize + vf610_nfc_spare_size(mtd);
u32 row = 0, cmd1 = 0, cmd2 = 0, code = 0;
u8 status;
int ret;
@@ -739,6 +756,49 @@ static void vf610_nfc_init_controller(struct vf610_nfc *nfc)
}
}
+/*
+ * With 64 byte OOB chips the core's large page layout matches what
+ * U-Boot uses, and on chips with more, U-Boot and older kernels clamped
+ * mtd->oobsize to 64. Modifying the OOB size is no longer possible, the
+ * actual chip geometry must be respected, so to avoid breaking those
+ * existing setups use our own layout: the core's large page one,
+ * computed over the first 64 spare bytes only.
+ */
+static int vf610_nfc_ooblayout_ecc(struct mtd_info *mtd, int section,
+ struct mtd_oob_region *oobregion)
+{
+ struct nand_device *nand = mtd_to_nanddev(mtd);
+ unsigned int total_ecc_bytes = nand->ecc.ctx.total;
+
+ if (section || !total_ecc_bytes)
+ return -ERANGE;
+
+ oobregion->length = total_ecc_bytes;
+ oobregion->offset = vf610_nfc_spare_size(mtd) - oobregion->length;
+
+ return 0;
+}
+
+static int vf610_nfc_ooblayout_free(struct mtd_info *mtd, int section,
+ struct mtd_oob_region *oobregion)
+{
+ struct nand_device *nand = mtd_to_nanddev(mtd);
+ unsigned int total_ecc_bytes = nand->ecc.ctx.total;
+
+ if (section)
+ return -ERANGE;
+
+ oobregion->length = vf610_nfc_spare_size(mtd) - total_ecc_bytes - 2;
+ oobregion->offset = 2;
+
+ return 0;
+}
+
+static const struct mtd_ooblayout_ops vf610_nfc_ooblayout_ops = {
+ .ecc = vf610_nfc_ooblayout_ecc,
+ .free = vf610_nfc_ooblayout_free,
+};
+
static int vf610_nfc_attach_chip(struct nand_chip *chip)
{
struct mtd_info *mtd = nand_to_mtd(chip);
@@ -769,12 +829,7 @@ static int vf610_nfc_attach_chip(struct nand_chip *chip)
return -ENXIO;
}
- /* Only 64 byte ECC layouts known */
- if (mtd->oobsize > 64)
- mtd->oobsize = 64;
-
- /* Use default large page ECC layout defined in NAND core */
- mtd_set_ooblayout(mtd, nand_get_large_page_ooblayout());
+ mtd_set_ooblayout(mtd, &vf610_nfc_ooblayout_ops);
if (chip->ecc.strength == 32) {
nfc->ecc_mode = ECC_60_BYTE;
chip->ecc.bytes = 60;
diff --git a/drivers/mtd/nand/spi/core.c b/drivers/mtd/nand/spi/core.c
index 1ae5364c0c7c..17d9c5ea2670 100644
--- a/drivers/mtd/nand/spi/core.c
+++ b/drivers/mtd/nand/spi/core.c
@@ -179,7 +179,10 @@ static int spinand_init_cfg_cache(struct spinand_device *spinand)
static int spinand_init_quad_enable(struct spinand_device *spinand)
{
+ struct nand_device *nand = spinand_to_nand(spinand);
+ unsigned int target;
bool enable = false;
+ int ret;
if (!(spinand->flags & SPINAND_HAS_QE_BIT))
return 0;
@@ -189,8 +192,22 @@ static int spinand_init_quad_enable(struct spinand_device *spinand)
spinand->op_templates.update_cache->data.buswidth == 4)
enable = true;
- return spinand_upd_cfg(spinand, CFG_QUAD_ENABLE,
- enable ? CFG_QUAD_ENABLE : 0);
+ /*
+ * QE is a per-die setting on some devices. Program each target
+ * individually when enabling or disabling quad I/O mode.
+ */
+ for (target = 0; target < nand->memorg.ntargets; target++) {
+ ret = spinand_select_target(spinand, target);
+ if (ret)
+ return ret;
+
+ ret = spinand_upd_cfg(spinand, CFG_QUAD_ENABLE,
+ enable ? CFG_QUAD_ENABLE : 0);
+ if (ret)
+ return ret;
+ }
+
+ return 0;
}
static int spinand_ecc_enable(struct spinand_device *spinand,
@@ -884,7 +901,8 @@ static int spinand_create_dirmap(struct spinand_device *spinand,
spinand->dirmaps[plane].rdesc = desc;
- if (nand->ecc.engine->integration != NAND_ECC_ENGINE_INTEGRATION_PIPELINED) {
+ if (!nand->ecc.engine ||
+ nand->ecc.engine->integration != NAND_ECC_ENGINE_INTEGRATION_PIPELINED) {
spinand->dirmaps[plane].wdesc_ecc = spinand->dirmaps[plane].wdesc;
spinand->dirmaps[plane].rdesc_ecc = spinand->dirmaps[plane].rdesc;
@@ -1276,12 +1294,17 @@ static int spinand_init(struct spinand_device *spinand)
mtd->_max_bad_blocks = nanddev_mtd_max_bad_blocks;
mtd->_resume = spinand_mtd_resume;
- if (nand->ecc.engine) {
- ret = mtd_ooblayout_count_freebytes(mtd);
- if (ret < 0)
- goto err_cleanup_ecc_engine;
+ if (!nand->ecc.engine) {
+ if (spinand->eccinfo.ooblayout)
+ mtd_set_ooblayout(mtd, spinand->eccinfo.ooblayout);
+ else
+ mtd_set_ooblayout(mtd, &spinand_noecc_ooblayout);
}
+ ret = mtd_ooblayout_count_freebytes(mtd);
+ if (ret < 0)
+ goto err_cleanup_ecc_engine;
+
mtd->oobavail = ret;
/* Propagate ECC information to mtd_info */
diff --git a/drivers/net/amt.c b/drivers/net/amt.c
index 4799f8fd03c4..9bdc6c0e7bf9 100644
--- a/drivers/net/amt.c
+++ b/drivers/net/amt.c
@@ -78,15 +78,6 @@ static struct in6_addr mld2_all_node = MLD2_ALL_NODE_INIT;
static struct mld2_grec mldv2_zero_grec;
#endif
-static struct amt_skb_cb *amt_skb_cb(struct sk_buff *skb)
-{
- BUILD_BUG_ON(sizeof(struct amt_skb_cb) + sizeof(struct tc_skb_cb) >
- sizeof_field(struct sk_buff, cb));
-
- return (struct amt_skb_cb *)((void *)skb->cb +
- sizeof(struct tc_skb_cb));
-}
-
static void __amt_source_gc_work(void)
{
struct amt_source_node *snode;
@@ -787,6 +778,11 @@ static void amt_send_request(struct amt_dev *amt, bool v6)
rcu_read_unlock();
}
+static bool amt_send_membership_query(struct amt_dev *amt,
+ struct sk_buff *skb,
+ struct amt_tunnel_list *tunnel,
+ bool v6);
+
static void amt_send_igmp_gq(struct amt_dev *amt,
struct amt_tunnel_list *tunnel)
{
@@ -796,8 +792,11 @@ static void amt_send_igmp_gq(struct amt_dev *amt,
if (!skb)
return;
- amt_skb_cb(skb)->tunnel = tunnel;
- dev_queue_xmit(skb);
+ skb_pull(skb, sizeof(struct ethhdr));
+ if (amt_send_membership_query(amt, skb, tunnel, false)) {
+ amt->dev->stats.tx_dropped++;
+ kfree_skb(skb);
+ }
}
#if IS_ENABLED(CONFIG_IPV6)
@@ -881,8 +880,11 @@ static void amt_send_mld_gq(struct amt_dev *amt, struct amt_tunnel_list *tunnel)
if (!skb)
return;
- amt_skb_cb(skb)->tunnel = tunnel;
- dev_queue_xmit(skb);
+ skb_pull(skb, sizeof(struct ethhdr));
+ if (amt_send_membership_query(amt, skb, tunnel, true)) {
+ amt->dev->stats.tx_dropped++;
+ kfree_skb(skb);
+ }
}
#else
static void amt_send_mld_gq(struct amt_dev *amt, struct amt_tunnel_list *tunnel)
@@ -1178,7 +1180,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
#endif
bool report = false;
struct igmphdr *ih;
- bool query = false;
struct iphdr *iph;
bool data = false;
bool v6 = false;
@@ -1196,9 +1197,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
case IGMP_HOST_MEMBERSHIP_REPORT:
report = true;
break;
- case IGMP_HOST_MEMBERSHIP_QUERY:
- query = true;
- break;
default:
goto free;
}
@@ -1220,9 +1218,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
case ICMPV6_MLD2_REPORT:
report = true;
break;
- case ICMPV6_MGM_QUERY:
- query = true;
- break;
default:
goto free;
}
@@ -1253,19 +1248,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev)
goto free;
goto unlock;
} else if (amt->mode == AMT_MODE_RELAY) {
- if (query) {
- tunnel = amt_skb_cb(skb)->tunnel;
- if (!tunnel) {
- WARN_ON(1);
- goto free;
- }
-
- /* Do not forward unexpected query */
- if (amt_send_membership_query(amt, skb, tunnel, v6))
- goto free;
- goto unlock;
- }
-
if (!data)
goto free;
list_for_each_entry_rcu(tunnel, &amt->tunnel_list, list) {
@@ -3213,6 +3195,9 @@ static int amt_newlink(struct net *net, struct net_device *dev,
struct amt_dev *amt = netdev_priv(dev);
int err = -EINVAL;
+ if (!net_eq(net, dev_net(dev)))
+ return err;
+
amt->net = net;
amt->mode = nla_get_u32(data[IFLA_AMT_MODE]);
diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c
index 5b8770118c4f..cd148bd014fa 100644
--- a/drivers/net/bonding/bond_main.c
+++ b/drivers/net/bonding/bond_main.c
@@ -1543,6 +1543,50 @@ static void bond_compute_features(struct bonding *bond)
netdev_change_features(bond_dev);
}
+static int bond_header_create(struct sk_buff *skb, struct net_device *bond_dev,
+ unsigned short type, const void *daddr,
+ const void *saddr, unsigned int len)
+{
+ struct bonding *bond = netdev_priv(bond_dev);
+ const struct header_ops *slave_ops;
+ struct slave *slave;
+ int ret = 0;
+
+ rcu_read_lock();
+ slave = rcu_dereference(bond->curr_active_slave);
+ if (slave) {
+ slave_ops = READ_ONCE(slave->dev->header_ops);
+ if (slave_ops && slave_ops->create)
+ ret = slave_ops->create(skb, slave->dev,
+ type, daddr, saddr, len);
+ }
+ rcu_read_unlock();
+ return ret;
+}
+
+static int bond_header_parse(const struct sk_buff *skb, unsigned char *haddr)
+{
+ struct bonding *bond = netdev_priv(skb->dev);
+ const struct header_ops *slave_ops;
+ struct slave *slave;
+ int ret = 0;
+
+ rcu_read_lock();
+ slave = rcu_dereference(bond->curr_active_slave);
+ if (slave) {
+ slave_ops = READ_ONCE(slave->dev->header_ops);
+ if (slave_ops && slave_ops->parse)
+ ret = slave_ops->parse(skb, haddr);
+ }
+ rcu_read_unlock();
+ return ret;
+}
+
+static const struct header_ops bond_header_ops = {
+ .create = bond_header_create,
+ .parse = bond_header_parse,
+};
+
static void bond_setup_by_slave(struct net_device *bond_dev,
struct net_device *slave_dev)
{
@@ -1550,7 +1594,8 @@ static void bond_setup_by_slave(struct net_device *bond_dev,
dev_close(bond_dev);
- bond_dev->header_ops = slave_dev->header_ops;
+ bond_dev->header_ops = slave_dev->header_ops ?
+ &bond_header_ops : NULL;
bond_dev->type = slave_dev->type;
bond_dev->hard_header_len = slave_dev->hard_header_len;
@@ -2269,11 +2314,6 @@ int bond_enslave(struct net_device *bond_dev, struct net_device *slave_dev,
unblock_netpoll_tx();
}
- /* broadcast mode uses the all_slaves to loop through slaves. */
- if (bond_mode_can_use_xmit_hash(bond) ||
- BOND_MODE(bond) == BOND_MODE_BROADCAST)
- bond_update_slave_arr(bond, NULL);
-
if (!slave_dev->netdev_ops->ndo_bpf ||
!slave_dev->netdev_ops->ndo_xdp_xmit) {
if (bond->xdp_prog) {
@@ -2307,6 +2347,11 @@ int bond_enslave(struct net_device *bond_dev, struct net_device *slave_dev,
bpf_prog_inc(bond->xdp_prog);
}
+ /* broadcast mode uses the all_slaves to loop through slaves. */
+ if (bond_mode_can_use_xmit_hash(bond) ||
+ BOND_MODE(bond) == BOND_MODE_BROADCAST)
+ bond_update_slave_arr(bond, NULL);
+
slave_info(bond_dev, slave_dev, "Enslaving as %s interface with %s link\n",
bond_is_active_slave(new_slave) ? "an active" : "a backup",
new_slave->link != BOND_LINK_DOWN ? "an up" : "a down");
@@ -5128,13 +5173,18 @@ static void bond_set_slave_arr(struct bonding *bond,
{
struct bond_up_slave *usable, *all;
- usable = rtnl_dereference(bond->usable_slaves);
- rcu_assign_pointer(bond->usable_slaves, usable_slaves);
- kfree_rcu(usable, rcu);
-
all = rtnl_dereference(bond->all_slaves);
rcu_assign_pointer(bond->all_slaves, all_slaves);
kfree_rcu(all, rcu);
+
+ if (BOND_MODE(bond) == BOND_MODE_BROADCAST) {
+ kfree_rcu(usable_slaves, rcu);
+ return;
+ }
+
+ usable = rtnl_dereference(bond->usable_slaves);
+ rcu_assign_pointer(bond->usable_slaves, usable_slaves);
+ kfree_rcu(usable, rcu);
}
static void bond_reset_slave_arr(struct bonding *bond)
diff --git a/drivers/net/can/dev/skb.c b/drivers/net/can/dev/skb.c
index 0da615afa04d..3ebd4f779b9b 100644
--- a/drivers/net/can/dev/skb.c
+++ b/drivers/net/can/dev/skb.c
@@ -202,6 +202,7 @@ static void init_can_skb_reserve(struct sk_buff *skb)
skb_reset_transport_header(skb);
can_skb_reserve(skb);
+ can_skb_prv(skb)->skbcnt = 0;
}
struct sk_buff *alloc_can_skb(struct net_device *dev, struct can_frame **cf)
@@ -311,6 +312,7 @@ static bool can_skb_headroom_valid(struct net_device *dev, struct sk_buff *skb)
if (skb->ip_summed == CHECKSUM_NONE) {
/* init headroom */
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skb->ip_summed = CHECKSUM_UNNECESSARY;
diff --git a/drivers/net/can/usb/gs_usb.c b/drivers/net/can/usb/gs_usb.c
index 3ce8cf4ecb2e..1f021366afab 100644
--- a/drivers/net/can/usb/gs_usb.c
+++ b/drivers/net/can/usb/gs_usb.c
@@ -251,6 +251,11 @@ struct canfd_quirk {
u8 quirk;
} __packed;
+/* struct gs_host_frame::echo_id == GS_HOST_FRAME_ECHO_ID_RX indicates
+ * a regular RX'ed CAN frame
+ */
+#define GS_HOST_FRAME_ECHO_ID_RX 0xffffffff
+
struct gs_host_frame {
struct_group(header,
u32 echo_id;
@@ -297,12 +302,6 @@ struct gs_can {
struct can_bittiming_const bt_const, data_bt_const;
unsigned int channel; /* channel number */
- /* time counter for hardware timestamps */
- struct cyclecounter cc;
- struct timecounter tc;
- spinlock_t tc_lock; /* spinlock to guard access tc->cycle_last */
- struct delayed_work timestamp;
-
u32 feature;
unsigned int hf_size_tx;
@@ -318,6 +317,13 @@ struct gs_can {
struct gs_usb {
struct usb_anchor rx_submitted;
struct usb_device *udev;
+
+ /* time counter for hardware timestamps */
+ struct cyclecounter cc;
+ struct timecounter tc;
+ spinlock_t tc_lock; /* spinlock to guard access tc->cycle_last */
+ struct delayed_work timestamp;
+
unsigned int hf_size_rx;
u8 active_channels;
u8 channel_cnt;
@@ -387,16 +393,15 @@ static int gs_cmd_reset(struct gs_can *dev)
GFP_KERNEL);
}
-static inline int gs_usb_get_timestamp(const struct gs_can *dev,
+static inline int gs_usb_get_timestamp(const struct gs_usb *parent,
u32 *timestamp_p)
{
__le32 timestamp;
int rc;
- rc = usb_control_msg_recv(interface_to_usbdev(dev->iface), 0,
- GS_USB_BREQ_TIMESTAMP,
+ rc = usb_control_msg_recv(parent->udev, 0, GS_USB_BREQ_TIMESTAMP,
USB_DIR_IN | USB_TYPE_VENDOR | USB_RECIP_INTERFACE,
- dev->channel, 0,
+ 0, 0,
×tamp, sizeof(timestamp),
USB_CTRL_GET_TIMEOUT,
GFP_KERNEL);
@@ -410,20 +415,20 @@ static inline int gs_usb_get_timestamp(const struct gs_can *dev,
static u64 gs_usb_timestamp_read(const struct cyclecounter *cc) __must_hold(&dev->tc_lock)
{
- struct gs_can *dev = container_of(cc, struct gs_can, cc);
+ struct gs_usb *parent = container_of(cc, struct gs_usb, cc);
u32 timestamp = 0;
int err;
- lockdep_assert_held(&dev->tc_lock);
+ lockdep_assert_held(&parent->tc_lock);
/* drop lock for synchronous USB transfer */
- spin_unlock_bh(&dev->tc_lock);
- err = gs_usb_get_timestamp(dev, ×tamp);
- spin_lock_bh(&dev->tc_lock);
+ spin_unlock_bh(&parent->tc_lock);
+ err = gs_usb_get_timestamp(parent, ×tamp);
+ spin_lock_bh(&parent->tc_lock);
if (err)
- netdev_err(dev->netdev,
- "Error %d while reading timestamp. HW timestamps may be inaccurate.",
- err);
+ dev_err(&parent->udev->dev,
+ "Error %d while reading timestamp. HW timestamps may be inaccurate.",
+ err);
return timestamp;
}
@@ -431,14 +436,14 @@ static u64 gs_usb_timestamp_read(const struct cyclecounter *cc) __must_hold(&dev
static void gs_usb_timestamp_work(struct work_struct *work)
{
struct delayed_work *delayed_work = to_delayed_work(work);
- struct gs_can *dev;
+ struct gs_usb *parent;
- dev = container_of(delayed_work, struct gs_can, timestamp);
- spin_lock_bh(&dev->tc_lock);
- timecounter_read(&dev->tc);
- spin_unlock_bh(&dev->tc_lock);
+ parent = container_of(delayed_work, struct gs_usb, timestamp);
+ spin_lock_bh(&parent->tc_lock);
+ timecounter_read(&parent->tc);
+ spin_unlock_bh(&parent->tc_lock);
- schedule_delayed_work(&dev->timestamp,
+ schedule_delayed_work(&parent->timestamp,
GS_USB_TIMESTAMP_WORK_DELAY_SEC * HZ);
}
@@ -446,37 +451,38 @@ static void gs_usb_skb_set_timestamp(struct gs_can *dev,
struct sk_buff *skb, u32 timestamp)
{
struct skb_shared_hwtstamps *hwtstamps = skb_hwtstamps(skb);
+ struct gs_usb *parent = dev->parent;
u64 ns;
- spin_lock_bh(&dev->tc_lock);
- ns = timecounter_cyc2time(&dev->tc, timestamp);
- spin_unlock_bh(&dev->tc_lock);
+ spin_lock_bh(&parent->tc_lock);
+ ns = timecounter_cyc2time(&parent->tc, timestamp);
+ spin_unlock_bh(&parent->tc_lock);
hwtstamps->hwtstamp = ns_to_ktime(ns);
}
-static void gs_usb_timestamp_init(struct gs_can *dev)
+static void gs_usb_timestamp_init(struct gs_usb *parent)
{
- struct cyclecounter *cc = &dev->cc;
+ struct cyclecounter *cc = &parent->cc;
cc->read = gs_usb_timestamp_read;
cc->mask = CYCLECOUNTER_MASK(32);
cc->shift = 32 - bits_per(NSEC_PER_SEC / GS_USB_TIMESTAMP_TIMER_HZ);
cc->mult = clocksource_hz2mult(GS_USB_TIMESTAMP_TIMER_HZ, cc->shift);
- spin_lock_init(&dev->tc_lock);
- spin_lock_bh(&dev->tc_lock);
- timecounter_init(&dev->tc, &dev->cc, ktime_get_real_ns());
- spin_unlock_bh(&dev->tc_lock);
+ spin_lock_init(&parent->tc_lock);
+ spin_lock_bh(&parent->tc_lock);
+ timecounter_init(&parent->tc, &parent->cc, ktime_get_real_ns());
+ spin_unlock_bh(&parent->tc_lock);
- INIT_DELAYED_WORK(&dev->timestamp, gs_usb_timestamp_work);
- schedule_delayed_work(&dev->timestamp,
+ INIT_DELAYED_WORK(&parent->timestamp, gs_usb_timestamp_work);
+ schedule_delayed_work(&parent->timestamp,
GS_USB_TIMESTAMP_WORK_DELAY_SEC * HZ);
}
-static void gs_usb_timestamp_stop(struct gs_can *dev)
+static void gs_usb_timestamp_stop(struct gs_usb *parent)
{
- cancel_delayed_work_sync(&dev->timestamp);
+ cancel_delayed_work_sync(&parent->timestamp);
}
static void gs_update_state(struct gs_can *dev, struct can_frame *cf)
@@ -522,6 +528,37 @@ static void gs_usb_set_timestamp(struct gs_can *dev, struct sk_buff *skb,
return;
}
+static unsigned int
+gs_usb_get_minimum_rx_length(const struct gs_can *dev, const struct gs_host_frame *hf,
+ unsigned int *data_length_p)
+{
+ unsigned int minimum_length, data_length = 0;
+
+ if (hf->flags & GS_CAN_FLAG_FD) {
+ if (hf->echo_id == GS_HOST_FRAME_ECHO_ID_RX)
+ data_length = can_fd_dlc2len(hf->can_dlc);
+
+ if (dev->feature & GS_CAN_FEATURE_HW_TIMESTAMP)
+ /* timestamp follows data field of max size */
+ minimum_length = struct_size(hf, canfd_ts, 1);
+ else
+ minimum_length = sizeof(hf->header) + data_length;
+ } else {
+ if (hf->echo_id == GS_HOST_FRAME_ECHO_ID_RX &&
+ !(hf->can_id & cpu_to_le32(CAN_RTR_FLAG)))
+ data_length = can_cc_dlc2len(hf->can_dlc);
+
+ if (dev->feature & GS_CAN_FEATURE_HW_TIMESTAMP)
+ /* timestamp follows data field of max size */
+ minimum_length = struct_size(hf, classic_can_ts, 1);
+ else
+ minimum_length = sizeof(hf->header) + data_length;
+ }
+
+ *data_length_p = data_length;
+ return minimum_length;
+}
+
static void gs_usb_receive_bulk_callback(struct urb *urb)
{
struct gs_usb *parent = urb->context;
@@ -530,7 +567,7 @@ static void gs_usb_receive_bulk_callback(struct urb *urb)
int rc;
struct net_device_stats *stats;
struct gs_host_frame *hf = urb->transfer_buffer;
- unsigned int minimum_length;
+ unsigned int minimum_length, data_length;
struct gs_tx_context *txc;
struct can_frame *cf;
struct canfd_frame *cfd;
@@ -570,20 +607,36 @@ static void gs_usb_receive_bulk_callback(struct urb *urb)
if (!netif_device_present(netdev))
return;
- if (hf->echo_id == -1) { /* normal rx */
+ if (!netif_running(netdev))
+ goto resubmit_urb;
+
+ minimum_length = gs_usb_get_minimum_rx_length(dev, hf, &data_length);
+ if (urb->actual_length < minimum_length) {
+ stats->rx_errors++;
+ stats->rx_length_errors++;
+
+ if (net_ratelimit())
+ netdev_err(netdev,
+ "short read (actual_length=%u, minimum_length=%u)\n",
+ urb->actual_length, minimum_length);
+
+ goto resubmit_urb;
+ }
+
+ if (hf->echo_id == GS_HOST_FRAME_ECHO_ID_RX) { /* normal rx */
if (hf->flags & GS_CAN_FLAG_FD) {
skb = alloc_canfd_skb(dev->netdev, &cfd);
if (!skb)
goto resubmit_urb;
cfd->can_id = le32_to_cpu(hf->can_id);
- cfd->len = can_fd_dlc2len(hf->can_dlc);
+ cfd->len = data_length;
if (hf->flags & GS_CAN_FLAG_BRS)
cfd->flags |= CANFD_BRS;
if (hf->flags & GS_CAN_FLAG_ESI)
cfd->flags |= CANFD_ESI;
- memcpy(cfd->data, hf->canfd->data, cfd->len);
+ memcpy(cfd->data, hf->canfd->data, data_length);
} else {
skb = alloc_can_skb(dev->netdev, &cf);
if (!skb)
@@ -592,7 +645,7 @@ static void gs_usb_receive_bulk_callback(struct urb *urb)
cf->can_id = le32_to_cpu(hf->can_id);
can_frame_set_cc_len(cf, hf->can_dlc, dev->can.ctrlmode);
- memcpy(cf->data, hf->classic_can->data, 8);
+ memcpy(cf->data, hf->classic_can->data, data_length);
/* ERROR frames tell us information about the controller */
if (le32_to_cpu(hf->can_id) & CAN_ERR_FLAG)
@@ -891,6 +944,9 @@ static int gs_can_open(struct net_device *netdev)
}
if (!parent->active_channels) {
+ if (dev->feature & GS_CAN_FEATURE_HW_TIMESTAMP)
+ gs_usb_timestamp_init(parent);
+
for (i = 0; i < GS_MAX_RX_URBS; i++) {
u8 *buf;
@@ -973,10 +1029,6 @@ static int gs_can_open(struct net_device *netdev)
}
}
- /* start polling timestamp */
- if (dev->feature & GS_CAN_FEATURE_HW_TIMESTAMP)
- gs_usb_timestamp_init(dev);
-
/* finally start device */
dev->can.state = CAN_STATE_ERROR_ACTIVE;
dm.flags = cpu_to_le32(flags);
@@ -987,8 +1039,6 @@ static int gs_can_open(struct net_device *netdev)
GFP_KERNEL);
if (rc) {
netdev_err(netdev, "Couldn't start device (err=%d)\n", rc);
- if (dev->feature & GS_CAN_FEATURE_HW_TIMESTAMP)
- gs_usb_timestamp_stop(dev);
dev->can.state = CAN_STATE_STOPPED;
goto out_usb_kill_anchored_urbs;
@@ -1005,9 +1055,13 @@ static int gs_can_open(struct net_device *netdev)
out_usb_free_urb:
usb_free_urb(urb);
out_usb_kill_anchored_urbs:
- if (!parent->active_channels)
+ if (!parent->active_channels) {
usb_kill_anchored_urbs(&parent->rx_submitted);
+ if (dev->feature & GS_CAN_FEATURE_HW_TIMESTAMP)
+ gs_usb_timestamp_stop(parent);
+ }
+
close_candev(netdev);
return rc;
@@ -1021,14 +1075,13 @@ static int gs_can_close(struct net_device *netdev)
netif_stop_queue(netdev);
- /* stop polling timestamp */
- if (dev->feature & GS_CAN_FEATURE_HW_TIMESTAMP)
- gs_usb_timestamp_stop(dev);
-
/* Stop polling */
parent->active_channels--;
if (!parent->active_channels) {
usb_kill_anchored_urbs(&parent->rx_submitted);
+
+ if (dev->feature & GS_CAN_FEATURE_HW_TIMESTAMP)
+ gs_usb_timestamp_stop(parent);
}
/* Stop sending URBs */
diff --git a/drivers/net/dsa/b53/b53_common.c b/drivers/net/dsa/b53/b53_common.c
index aaf9fbaefd6b..7777d4cef351 100644
--- a/drivers/net/dsa/b53/b53_common.c
+++ b/drivers/net/dsa/b53/b53_common.c
@@ -2237,8 +2237,12 @@ EXPORT_SYMBOL(b53_eee_enable_set);
*/
int b53_eee_init(struct dsa_switch *ds, int port, struct phy_device *phy)
{
+ struct b53_device *dev = ds->priv;
int ret;
+ if (is5325(dev) || is5365(dev) || is63xx(dev))
+ return 0;
+
ret = phy_init_eee(phy, false);
if (ret)
return 0;
@@ -2255,7 +2259,7 @@ int b53_get_mac_eee(struct dsa_switch *ds, int port, struct ethtool_eee *e)
struct ethtool_eee *p = &dev->ports[port].eee;
u16 reg;
- if (is5325(dev) || is5365(dev))
+ if (is5325(dev) || is5365(dev) || is63xx(dev))
return -EOPNOTSUPP;
b53_read16(dev, B53_EEE_PAGE, B53_EEE_LPI_INDICATE, ®);
@@ -2271,7 +2275,7 @@ int b53_set_mac_eee(struct dsa_switch *ds, int port, struct ethtool_eee *e)
struct b53_device *dev = ds->priv;
struct ethtool_eee *p = &dev->ports[port].eee;
- if (is5325(dev) || is5365(dev))
+ if (is5325(dev) || is5365(dev) || is63xx(dev))
return -EOPNOTSUPP;
p->eee_enabled = e->eee_enabled;
diff --git a/drivers/net/dsa/microchip/ksz_common.h b/drivers/net/dsa/microchip/ksz_common.h
index b35c98adbfcb..2eaa8732d3a8 100644
--- a/drivers/net/dsa/microchip/ksz_common.h
+++ b/drivers/net/dsa/microchip/ksz_common.h
@@ -552,7 +552,7 @@ static inline int is_lan937x(struct ksz_device *dev)
#define KSZ87_FAMILY_ID 0x87
#define KSZ88_FAMILY_ID 0x88
-#define KSZ8_PORT_STATUS_0 0x08
+#define KSZ8_PORT_STATUS_0 0x18
#define KSZ8_PORT_FIBER_MODE BIT(7)
#define SW_CHIP_ID_M GENMASK(7, 4)
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c
index 276f6a8631fb..a7b98c0b5f28 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_com.c
@@ -763,25 +763,16 @@ static int ena_com_wait_and_process_admin_cq_interrupts(struct ena_comp_ctx *com
if (comp_ctx->status == ENA_CMD_COMPLETED) {
netdev_err(admin_queue->ena_dev->net_device,
- "The ena device sent a completion but the driver didn't receive a MSI-X interrupt (cmd %d), autopolling mode is %s\n",
- comp_ctx->cmd_opcode, admin_queue->auto_polling ? "ON" : "OFF");
- /* Check if fallback to polling is enabled */
- if (admin_queue->auto_polling)
- admin_queue->polling = true;
+ "The ena device sent a completion but the driver didn't receive a MSI-X interrupt (cmd %d)\n",
+ comp_ctx->cmd_opcode);
} else {
netdev_err(admin_queue->ena_dev->net_device,
"The ena device didn't send a completion for the admin cmd %d status %d\n",
comp_ctx->cmd_opcode, comp_ctx->status);
}
- /* Check if shifted to polling mode.
- * This will happen if there is a completion without an interrupt
- * and autopolling mode is enabled. Continuing normal execution in such case
- */
- if (!admin_queue->polling) {
- admin_queue->running_state = false;
- ret = -ETIME;
- goto err;
- }
+ admin_queue->running_state = false;
+ ret = -ETIME;
+ goto err;
}
ret = ena_com_comp_status_to_errno(admin_queue, comp_ctx->comp_status);
@@ -1658,12 +1649,6 @@ void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling)
ena_dev->admin_queue.polling = polling;
}
-void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev,
- bool polling)
-{
- ena_dev->admin_queue.auto_polling = polling;
-}
-
int ena_com_mmio_reg_read_request_init(struct ena_com_dev *ena_dev)
{
struct ena_com_mmio_read *mmio_read = &ena_dev->mmio_read;
diff --git a/drivers/net/ethernet/amazon/ena/ena_com.h b/drivers/net/ethernet/amazon/ena/ena_com.h
index 3c5081d9d25d..bb33fe209eec 100644
--- a/drivers/net/ethernet/amazon/ena/ena_com.h
+++ b/drivers/net/ethernet/amazon/ena/ena_com.h
@@ -227,9 +227,6 @@ struct ena_com_admin_queue {
/* Indicate if the admin queue should poll for completion */
bool polling;
- /* Define if fallback to polling mode should occur */
- bool auto_polling;
-
u16 curr_cmd_id;
/* Indicate that the ena was initialized and can
@@ -484,17 +481,6 @@ bool ena_com_get_admin_running_state(struct ena_com_dev *ena_dev);
*/
void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling);
-/* ena_com_set_admin_auto_polling_mode - Enable autoswitch to polling mode
- * @ena_dev: ENA communication layer struct
- * @polling: Enable/Disable polling mode
- *
- * Set the autopolling mode.
- * If autopolling is on:
- * In case of missing interrupt when data is available switch to polling.
- */
-void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev,
- bool polling);
-
/* ena_com_admin_q_comp_intr_handler - admin queue interrupt handler
* @ena_dev: ENA communication layer struct
*
diff --git a/drivers/net/ethernet/amazon/ena/ena_eth_com.c b/drivers/net/ethernet/amazon/ena/ena_eth_com.c
index 933e619b3a31..4c6e07aa4bbb 100644
--- a/drivers/net/ethernet/amazon/ena/ena_eth_com.c
+++ b/drivers/net/ethernet/amazon/ena/ena_eth_com.c
@@ -229,30 +229,43 @@ static struct ena_eth_io_rx_cdesc_base *
idx * io_cq->cdesc_entry_size_in_bytes);
}
-static u16 ena_com_cdesc_rx_pkt_get(struct ena_com_io_cq *io_cq,
- u16 *first_cdesc_idx)
+static int ena_com_cdesc_rx_pkt_get(struct ena_com_io_cq *io_cq,
+ u16 *first_cdesc_idx,
+ u16 *num_descs)
{
+ u16 count = io_cq->cur_rx_pkt_cdesc_count, head_masked;
struct ena_eth_io_rx_cdesc_base *cdesc;
- u16 count = 0, head_masked;
u32 last = 0;
do {
+ u32 status;
+
cdesc = ena_com_get_next_rx_cdesc(io_cq);
if (!cdesc)
break;
+ status = READ_ONCE(cdesc->status);
ena_com_cq_inc_head(io_cq);
+ if (unlikely((status & ENA_ETH_IO_RX_CDESC_BASE_FIRST_MASK) >>
+ ENA_ETH_IO_RX_CDESC_BASE_FIRST_SHIFT && count != 0)) {
+ struct ena_com_dev *dev = ena_com_io_cq_to_ena_dev(io_cq);
+
+ netdev_err(dev->net_device,
+ "First bit is on in descriptor #%d on q_id: %d, req_id: %u\n",
+ count, io_cq->qid, cdesc->req_id);
+ return -EFAULT;
+ }
count++;
- last = (READ_ONCE(cdesc->status) & ENA_ETH_IO_RX_CDESC_BASE_LAST_MASK) >>
- ENA_ETH_IO_RX_CDESC_BASE_LAST_SHIFT;
+ last = (status & ENA_ETH_IO_RX_CDESC_BASE_LAST_MASK) >>
+ ENA_ETH_IO_RX_CDESC_BASE_LAST_SHIFT;
} while (!last);
if (last) {
*first_cdesc_idx = io_cq->cur_rx_pkt_cdesc_start_idx;
- count += io_cq->cur_rx_pkt_cdesc_count;
head_masked = io_cq->head & (io_cq->q_depth - 1);
+ *num_descs = count;
io_cq->cur_rx_pkt_cdesc_count = 0;
io_cq->cur_rx_pkt_cdesc_start_idx = head_masked;
@@ -260,11 +273,11 @@ static u16 ena_com_cdesc_rx_pkt_get(struct ena_com_io_cq *io_cq,
"ENA q_id: %d packets were completed. first desc idx %u descs# %d\n",
io_cq->qid, *first_cdesc_idx, count);
} else {
- io_cq->cur_rx_pkt_cdesc_count += count;
- count = 0;
+ io_cq->cur_rx_pkt_cdesc_count = count;
+ *num_descs = 0;
}
- return count;
+ return 0;
}
static int ena_com_create_meta(struct ena_com_io_sq *io_sq,
@@ -539,10 +552,14 @@ int ena_com_rx_pkt(struct ena_com_io_cq *io_cq,
u16 cdesc_idx = 0;
u16 nb_hw_desc;
u16 i = 0;
+ int rc;
WARN(io_cq->direction != ENA_COM_IO_QUEUE_DIRECTION_RX, "wrong Q type");
- nb_hw_desc = ena_com_cdesc_rx_pkt_get(io_cq, &cdesc_idx);
+ rc = ena_com_cdesc_rx_pkt_get(io_cq, &cdesc_idx, &nb_hw_desc);
+ if (unlikely(rc != 0))
+ return -EFAULT;
+
if (nb_hw_desc == 0) {
ena_rx_ctx->descs = nb_hw_desc;
return 0;
diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c
index 5159fb9fbe66..b46bf8bf0660 100644
--- a/drivers/net/ethernet/amazon/ena/ena_netdev.c
+++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c
@@ -1853,6 +1853,8 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi,
if (rc == -ENOSPC) {
ena_increase_stat(&rx_ring->rx_stats.bad_desc_num, 1, &rx_ring->syncp);
ena_reset_device(adapter, ENA_REGS_RESET_TOO_MANY_RX_DESCS);
+ } else if (rc == -EFAULT) {
+ ena_reset_device(adapter, ENA_REGS_RESET_RX_DESCRIPTOR_MALFORMED);
} else {
ena_increase_stat(&rx_ring->rx_stats.bad_req_id, 1,
&rx_ring->syncp);
@@ -4497,6 +4499,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
err_device_destroy:
ena_com_delete_host_info(ena_dev);
ena_com_admin_destroy(ena_dev);
+ ena_com_mmio_reg_read_request_destroy(ena_dev);
err_netdev_destroy:
free_netdev(netdev);
err_free_region:
diff --git a/drivers/net/ethernet/amazon/ena/ena_regs_defs.h b/drivers/net/ethernet/amazon/ena/ena_regs_defs.h
index 1e007a41a525..23466d62fbc5 100644
--- a/drivers/net/ethernet/amazon/ena/ena_regs_defs.h
+++ b/drivers/net/ethernet/amazon/ena/ena_regs_defs.h
@@ -21,6 +21,7 @@ enum ena_regs_reset_reason_types {
ENA_REGS_RESET_USER_TRIGGER = 12,
ENA_REGS_RESET_GENERIC = 13,
ENA_REGS_RESET_MISS_INTERRUPT = 14,
+ ENA_REGS_RESET_RX_DESCRIPTOR_MALFORMED = 16,
};
/* ena_registers offsets */
diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c
index e53ab9b5482e..c9e6a3583f2a 100644
--- a/drivers/net/ethernet/broadcom/bcmsysport.c
+++ b/drivers/net/ethernet/broadcom/bcmsysport.c
@@ -422,8 +422,10 @@ static void bcm_sysport_update_mib_counters(struct bcm_sysport_priv *priv)
if (priv->is_lite)
continue;
- if (s->type != BCM_SYSPORT_STAT_MIB_RX)
+ if (s->type == BCM_SYSPORT_STAT_MIB_TX)
offset = UMAC_MIB_STAT_OFFSET;
+ else if (s->type == BCM_SYSPORT_STAT_RUNT)
+ offset = 2 * UMAC_MIB_STAT_OFFSET;
val = umac_readl(priv, UMAC_MIB_START + j + offset);
break;
case BCM_SYSPORT_STAT_RXCHK:
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
index 8d7eda5cafe2..7328697c016c 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c
@@ -392,7 +392,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
struct netdev_queue *txq;
int i;
dma_addr_t mapping;
- unsigned int length, pad = 0;
+ unsigned int length;
u32 len, free_size, vlan_tag_flags, cfa_action, flags;
u16 prod, last_frag;
struct pci_dev *pdev = bp->pdev;
@@ -411,6 +411,14 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
txr = &bp->tx_ring[bp->tx_ring_map[i]];
prod = txr->tx_prod;
+ /* Must be done before skb_shinfo(skb)->nr_frags is sampled,
+ * because skb_put_padto() might linearize the skb.
+ */
+ if (skb_put_padto(skb, BNXT_MIN_PKT_SIZE)) {
+ /* SKB already freed. */
+ goto tx_kick_pending;
+ }
+
free_size = bnxt_tx_avail(bp, txr);
if (unlikely(free_size < skb_shinfo(skb)->nr_frags + 2)) {
/* We must have raced with NAPI cleanup */
@@ -539,14 +547,6 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
}
normal_tx:
- if (length < BNXT_MIN_PKT_SIZE) {
- pad = BNXT_MIN_PKT_SIZE - length;
- if (skb_pad(skb, pad))
- /* SKB already freed. */
- goto tx_kick_pending;
- length = BNXT_MIN_PKT_SIZE;
- }
-
mapping = dma_map_single(&pdev->dev, skb->data, len, DMA_TO_DEVICE);
if (unlikely(dma_mapping_error(&pdev->dev, mapping)))
@@ -618,10 +618,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev)
txbd->tx_bd_len_flags_type = cpu_to_le32(flags);
}
- flags &= ~TX_BD_LEN;
- txbd->tx_bd_len_flags_type =
- cpu_to_le32(((len + pad) << TX_BD_LEN_SHIFT) | flags |
- TX_BD_FLAGS_PACKET_END);
+ txbd->tx_bd_len_flags_type |= cpu_to_le32(TX_BD_FLAGS_PACKET_END);
netdev_tx_sent_queue(txq, skb->len);
@@ -819,6 +816,7 @@ int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
mapping += bp->rx_dma_offset;
rx_buf->data = page;
rx_buf->data_ptr = page_address(page) + offset + bp->rx_offset;
+ rx_buf->offset = offset;
} else {
u8 *data = __bnxt_alloc_rx_frag(bp, &mapping, gfp);
@@ -827,6 +825,7 @@ int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr,
rx_buf->data = data;
rx_buf->data_ptr = data + bp->rx_offset;
+ rx_buf->offset = 0;
}
rx_buf->mapping = mapping;
@@ -847,6 +846,7 @@ void bnxt_reuse_rx_data(struct bnxt_rx_ring_info *rxr, u16 cons, void *data)
prod_rx_buf->data_ptr = cons_rx_buf->data_ptr;
prod_rx_buf->mapping = cons_rx_buf->mapping;
+ prod_rx_buf->offset = cons_rx_buf->offset;
prod_bd = &rxr->rx_desc_ring[RX_RING(prod)][RX_IDX(prod)];
cons_bd = &rxr->rx_desc_ring[RX_RING(cons)][RX_IDX(cons)];
@@ -1017,8 +1017,11 @@ static struct sk_buff *bnxt_rx_multi_page_skb(struct bnxt *bp,
struct page *page = data;
u16 prod = rxr->rx_prod;
struct sk_buff *skb;
+ void *frag_start;
int err;
+ frag_start = page_address(page) + rxr->rx_buf_ring[cons].offset;
+
err = bnxt_alloc_rx_data(bp, rxr, prod, GFP_ATOMIC);
if (unlikely(err)) {
bnxt_reuse_rx_data(rxr, cons, data);
@@ -1027,13 +1030,13 @@ static struct sk_buff *bnxt_rx_multi_page_skb(struct bnxt *bp,
dma_addr -= bp->rx_dma_offset;
dma_unmap_page_attrs(&bp->pdev->dev, dma_addr, BNXT_RX_PAGE_SIZE,
bp->rx_dir, DMA_ATTR_WEAK_ORDERING);
- skb = build_skb(data_ptr - bp->rx_offset, BNXT_RX_PAGE_SIZE);
+ skb = build_skb(frag_start, BNXT_RX_PAGE_SIZE);
if (!skb) {
page_pool_recycle_direct(rxr->page_pool, page);
return NULL;
}
skb_mark_for_recycle(skb);
- skb_reserve(skb, bp->rx_offset);
+ skb_reserve(skb, data_ptr - (u8 *)frag_start);
__skb_put(skb, len);
return skb;
@@ -1836,6 +1839,7 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
struct rx_cmp_ext *rxcmp1;
u32 tmp_raw_cons = *raw_cons;
u16 cfa_code, cons, prod, cp_cons = RING_CMP(tmp_raw_cons);
+ struct skb_shared_info *sinfo;
struct bnxt_sw_rx_bd *rx_buf;
unsigned int len;
u8 *data_ptr, agg_bufs, cmp_type;
@@ -1960,6 +1964,7 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
false);
if (!frag_len)
goto oom_next_rx;
+
}
xdp_active = true;
}
@@ -1969,6 +1974,12 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
rc = 1;
goto next_rx;
}
+ if (xdp_buff_has_frags(&xdp)) {
+ sinfo = xdp_get_shared_info_from_buff(&xdp);
+ agg_bufs = sinfo->nr_frags;
+ } else {
+ agg_bufs = 0;
+ }
}
if (len <= bp->rx_copy_thresh) {
@@ -2538,6 +2549,7 @@ static int __bnxt_poll_work(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
{
struct bnxt_napi *bnapi = cpr->bnapi;
u32 raw_cons = cpr->cp_raw_cons;
+ bool flush_xdp = false;
u32 cons;
int tx_pkts = 0;
int rx_pkts = 0;
@@ -2575,6 +2587,8 @@ static int __bnxt_poll_work(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
else
rc = bnxt_force_rx_discard(bp, cpr, &raw_cons,
&event);
+ if (event & BNXT_REDIRECT_EVENT)
+ flush_xdp = true;
if (likely(rc >= 0))
rx_pkts += rc;
/* Increment rx_pkts when rc is -ENOMEM to count towards
@@ -2602,7 +2616,7 @@ static int __bnxt_poll_work(struct bnxt *bp, struct bnxt_cp_ring_info *cpr,
}
}
- if (event & BNXT_REDIRECT_EVENT)
+ if (flush_xdp)
xdp_do_flush();
if (event & BNXT_TX_EVENT) {
@@ -10512,6 +10526,20 @@ static int bnxt_reinit_after_abort(struct bnxt *bp)
return rc;
}
+static int bnxt_tx_nr_rings(struct bnxt *bp)
+{
+ int tcs = netdev_get_num_tc(bp->dev);
+
+ return tcs ? bp->tx_nr_rings_per_tc * tcs : bp->tx_nr_rings_per_tc;
+}
+
+static int bnxt_tx_nr_rings_per_tc(struct bnxt *bp)
+{
+ int tcs = netdev_get_num_tc(bp->dev);
+
+ return tcs ? bp->tx_nr_rings / tcs : bp->tx_nr_rings;
+}
+
static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init)
{
int rc = 0;
@@ -13463,7 +13491,7 @@ static void bnxt_trim_dflt_sh_rings(struct bnxt *bp)
bp->cp_nr_rings = min_t(int, bp->tx_nr_rings_per_tc, bp->rx_nr_rings);
bp->rx_nr_rings = bp->cp_nr_rings;
bp->tx_nr_rings_per_tc = bp->cp_nr_rings;
- bp->tx_nr_rings = bp->tx_nr_rings_per_tc;
+ bp->tx_nr_rings = bnxt_tx_nr_rings(bp);
}
static int bnxt_set_dflt_rings(struct bnxt *bp, bool sh)
@@ -13494,12 +13522,12 @@ static int bnxt_set_dflt_rings(struct bnxt *bp, bool sh)
bnxt_trim_dflt_sh_rings(bp);
else
bp->cp_nr_rings = bp->tx_nr_rings_per_tc + bp->rx_nr_rings;
- bp->tx_nr_rings = bp->tx_nr_rings_per_tc;
+ bp->tx_nr_rings = bnxt_tx_nr_rings(bp);
rc = __bnxt_reserve_rings(bp);
if (rc && rc != -ENODEV)
netdev_warn(bp->dev, "Unable to reserve tx rings\n");
- bp->tx_nr_rings_per_tc = bp->tx_nr_rings;
+ bp->tx_nr_rings_per_tc = bnxt_tx_nr_rings_per_tc(bp);
if (sh)
bnxt_trim_dflt_sh_rings(bp);
@@ -13508,7 +13536,7 @@ static int bnxt_set_dflt_rings(struct bnxt *bp, bool sh)
rc = __bnxt_reserve_rings(bp);
if (rc && rc != -ENODEV)
netdev_warn(bp->dev, "2nd rings reservation failed.\n");
- bp->tx_nr_rings_per_tc = bp->tx_nr_rings;
+ bp->tx_nr_rings_per_tc = bnxt_tx_nr_rings_per_tc(bp);
}
if (BNXT_CHIP_TYPE_NITRO_A0(bp)) {
bp->rx_nr_rings++;
@@ -13542,7 +13570,7 @@ static int bnxt_init_dflt_ring_mode(struct bnxt *bp)
if (rc)
goto init_dflt_ring_err;
- bp->tx_nr_rings_per_tc = bp->tx_nr_rings;
+ bp->tx_nr_rings_per_tc = bnxt_tx_nr_rings_per_tc(bp);
bnxt_set_dflt_rfs(bp);
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
index 006987d3caa1..86d8d837fb10 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h
@@ -728,6 +728,7 @@ struct bnxt_sw_rx_bd {
void *data;
u8 *data_ptr;
dma_addr_t mapping;
+ unsigned int offset;
};
struct bnxt_sw_rx_agg_bd {
diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
index 1b35805627c5..0d787e105b6a 100644
--- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
+++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c
@@ -464,7 +464,7 @@ bnxt_xdp_build_skb(struct bnxt *bp, struct sk_buff *skb, u8 num_frags,
xdp_update_skb_shared_info(skb, num_frags,
sinfo->xdp_frags_size,
- BNXT_RX_PAGE_SIZE * sinfo->nr_frags,
+ BNXT_RX_PAGE_SIZE * num_frags,
xdp_buff_is_frag_pfmemalloc(xdp));
return skb;
}
diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c
index c729da84a19b..8abdff23521d 100644
--- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c
+++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c
@@ -2099,6 +2099,16 @@ static netdev_tx_t bcmgenet_xmit(struct sk_buff *skb, struct net_device *dev)
goto out;
}
+ /* We store Tx state in the control block, so the skb must not be
+ * shared, but ether_setup() leaves IFF_TX_SKB_SHARING set.
+ */
+ skb = skb_share_check(skb, GFP_ATOMIC);
+ if (!skb) {
+ BCMGENET_STATS64_INC((&ring->stats64), dropped);
+ ret = NETDEV_TX_OK;
+ goto out;
+ }
+
/* Retain how many bytes will be sent on the wire, without TSB inserted
* by transmit checksum offload
*/
@@ -4079,8 +4089,11 @@ static int bcmgenet_probe(struct platform_device *pdev)
netif_set_real_num_rx_queues(priv->dev, priv->hw_params->rx_queues + 1);
/* Set default coalescing parameters */
- for (i = 0; i <= priv->hw_params->rx_queues; i++)
+ for (i = 0; i <= priv->hw_params->rx_queues; i++) {
+ priv->rx_rings[i].priv = priv;
+ priv->rx_rings[i].index = i;
priv->rx_rings[i].rx_max_coalesced_frames = 1;
+ }
/* Initialize u64 stats seq counter for 32bit machines */
for (i = 0; i <= GENET_MAX_MQ_CNT; i++) {
diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c
index aecdb98f8a9c..1733ccb89311 100644
--- a/drivers/net/ethernet/brocade/bna/bnad.c
+++ b/drivers/net/ethernet/brocade/bna/bnad.c
@@ -3728,9 +3728,11 @@ bnad_pci_probe(struct pci_dev *pdev,
bnad_res_free(bnad, &bnad->mod_res_info[0], BNA_MOD_RES_T_MAX);
disable_ioceth:
bnad_ioceth_disable(bnad);
- del_timer_sync(&bnad->bna.ioceth.ioc.ioc_timer);
- del_timer_sync(&bnad->bna.ioceth.ioc.sem_timer);
- del_timer_sync(&bnad->bna.ioceth.ioc.hb_timer);
+ /* The IOC timers rearm one another. Shut down all of them. */
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.ioc_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.sem_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.hb_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.iocpf_timer);
spin_lock_irqsave(&bnad->bna_lock, flags);
bna_uninit(bna);
spin_unlock_irqrestore(&bnad->bna_lock, flags);
@@ -3771,9 +3773,11 @@ bnad_pci_remove(struct pci_dev *pdev)
mutex_lock(&bnad->conf_mutex);
bnad_ioceth_disable(bnad);
- del_timer_sync(&bnad->bna.ioceth.ioc.ioc_timer);
- del_timer_sync(&bnad->bna.ioceth.ioc.sem_timer);
- del_timer_sync(&bnad->bna.ioceth.ioc.hb_timer);
+ /* The IOC timers rearm one another. Shut down all of them. */
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.ioc_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.sem_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.hb_timer);
+ timer_shutdown_sync(&bnad->bna.ioceth.ioc.iocpf_timer);
spin_lock_irqsave(&bnad->bna_lock, flags);
bna_uninit(bna);
spin_unlock_irqrestore(&bnad->bna_lock, flags);
diff --git a/drivers/net/ethernet/cavium/liquidio/lio_main.c b/drivers/net/ethernet/cavium/liquidio/lio_main.c
index 131996fd5406..f64c6dfafeb4 100644
--- a/drivers/net/ethernet/cavium/liquidio/lio_main.c
+++ b/drivers/net/ethernet/cavium/liquidio/lio_main.c
@@ -3801,9 +3801,7 @@ static int setup_nic_devices(struct octeon_device *octeon_dev)
static int octeon_enable_sriov(struct octeon_device *oct)
{
unsigned int num_vfs_alloced = oct->sriov_info.num_vfs_alloced;
- struct pci_dev *vfdev;
int err;
- u32 u;
if (OCTEON_CN23XX_PF(oct) && num_vfs_alloced) {
err = pci_enable_sriov(oct->pci_dev,
@@ -3816,23 +3814,6 @@ static int octeon_enable_sriov(struct octeon_device *oct)
return err;
}
oct->sriov_info.sriov_enabled = 1;
-
- /* init lookup table that maps DPI ring number to VF pci_dev
- * struct pointer
- */
- u = 0;
- vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
- OCTEON_CN23XX_VF_VID, NULL);
- while (vfdev) {
- if (vfdev->is_virtfn &&
- (vfdev->physfn == oct->pci_dev)) {
- oct->sriov_info.dpiring_to_vfpcidev_lut[u] =
- vfdev;
- u += oct->sriov_info.rings_per_vf;
- }
- vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
- OCTEON_CN23XX_VF_VID, vfdev);
- }
}
return num_vfs_alloced;
@@ -3840,8 +3821,6 @@ static int octeon_enable_sriov(struct octeon_device *oct)
static int lio_pci_sriov_disable(struct octeon_device *oct)
{
- int u;
-
if (pci_vfs_assigned(oct->pci_dev)) {
dev_err(&oct->pci_dev->dev, "VFs are still assigned to VMs.\n");
return -EPERM;
@@ -3849,12 +3828,6 @@ static int lio_pci_sriov_disable(struct octeon_device *oct)
pci_disable_sriov(oct->pci_dev);
- u = 0;
- while (u < MAX_POSSIBLE_VFS) {
- oct->sriov_info.dpiring_to_vfpcidev_lut[u] = NULL;
- u += oct->sriov_info.rings_per_vf;
- }
-
oct->sriov_info.num_vfs_alloced = 0;
dev_info(&oct->pci_dev->dev, "oct->pf_num:%d disabled VFs\n",
oct->pf_num);
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_device.h b/drivers/net/ethernet/cavium/liquidio/octeon_device.h
index fb380b4f3e02..cfa3b769227f 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_device.h
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_device.h
@@ -390,9 +390,6 @@ struct octeon_sriov_info {
struct lio_trusted_vf trusted_vf;
- /*lookup table that maps DPI ring number to VF pci_dev struct pointer*/
- struct pci_dev *dpiring_to_vfpcidev_lut[MAX_POSSIBLE_VFS];
-
u64 vf_macaddr[MAX_POSSIBLE_VFS];
u16 vf_vlantci[MAX_POSSIBLE_VFS];
diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
index ad685f5d0a13..697fcdc41e3c 100644
--- a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
+++ b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c
@@ -26,6 +26,31 @@
#include "octeon_mailbox.h"
#include "cn23xx_pf_device.h"
+static struct pci_dev *lio_vf_pci_dev_by_qno(struct octeon_device *oct, u32 q_no)
+{
+ struct pci_dev *vfdev = NULL;
+ int vfidx;
+
+ if (!oct->sriov_info.rings_per_vf)
+ return NULL;
+
+ if (q_no % oct->sriov_info.rings_per_vf)
+ return NULL;
+
+ vfidx = q_no / oct->sriov_info.rings_per_vf;
+ if (vfidx >= oct->sriov_info.num_vfs_alloced)
+ return NULL;
+
+ while ((vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM,
+ OCTEON_CN23XX_VF_VID, vfdev))) {
+ if (pci_physfn(vfdev) && pci_physfn(vfdev) == oct->pci_dev &&
+ pci_iov_vf_id(vfdev) == vfidx)
+ return vfdev;
+ }
+
+ return NULL;
+}
+
/**
* octeon_mbox_read:
* @mbox: Pointer mailbox
@@ -237,6 +262,7 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox,
struct octeon_mbox_cmd *mbox_cmd)
{
struct octeon_device *oct = mbox->oct_dev;
+ struct pci_dev *vfdev;
switch (mbox_cmd->msg.s.cmd) {
case OCTEON_VF_ACTIVE:
@@ -260,7 +286,12 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox,
dev_info(&oct->pci_dev->dev,
"got a request for FLR from VF that owns DPI ring %u\n",
mbox->q_no);
- pcie_flr(oct->sriov_info.dpiring_to_vfpcidev_lut[mbox->q_no]);
+ vfdev = lio_vf_pci_dev_by_qno(oct, mbox->q_no);
+ if (!vfdev)
+ break;
+
+ pcie_flr(vfdev);
+ pci_dev_put(vfdev);
break;
case OCTEON_PF_CHANGED_VF_MACADDR:
diff --git a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
index da9973b711f4..b2adb1581236 100644
--- a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
+++ b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c
@@ -390,7 +390,7 @@ static void chcr_ktls_dev_del(struct net_device *netdev,
cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan,
tx_info->tid, tx_info->ip_family);
- xa_erase(&u_ctx->tid_list, tx_info->tid);
+ xa_erase_bh(&u_ctx->tid_list, tx_info->tid);
}
port_stats = &tx_info->adap->ch_ktls_stats.ktls_port[tx_info->port_id];
@@ -581,7 +581,7 @@ static int chcr_ktls_dev_add(struct net_device *netdev, struct sock *sk,
cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan,
tx_info->tid, tx_info->ip_family);
- xa_erase(&u_ctx->tid_list, tx_info->tid);
+ xa_erase_bh(&u_ctx->tid_list, tx_info->tid);
put_module:
/* release module refcount */
diff --git a/drivers/net/ethernet/freescale/enetc/enetc.c b/drivers/net/ethernet/freescale/enetc/enetc.c
index 80eec93f9987..ce1503423439 100644
--- a/drivers/net/ethernet/freescale/enetc/enetc.c
+++ b/drivers/net/ethernet/freescale/enetc/enetc.c
@@ -2713,6 +2713,9 @@ static int enetc_hwtstamp_set(struct net_device *ndev, struct ifreq *ifr)
priv->active_offloads |= ENETC_F_TX_TSTAMP;
break;
case HWTSTAMP_TX_ONESTEP_SYNC:
+ if (!enetc_si_is_pf(priv->si))
+ return -EOPNOTSUPP;
+
priv->active_offloads &= ~ENETC_F_TX_TSTAMP_MASK;
priv->active_offloads |= ENETC_F_TX_ONESTEP_SYNC_TSTAMP;
break;
diff --git a/drivers/net/ethernet/freescale/enetc/enetc_ethtool.c b/drivers/net/ethernet/freescale/enetc/enetc_ethtool.c
index d4623a41f013..71c64317c5b8 100644
--- a/drivers/net/ethernet/freescale/enetc/enetc_ethtool.c
+++ b/drivers/net/ethernet/freescale/enetc/enetc_ethtool.c
@@ -772,6 +772,9 @@ static int enetc_set_coalesce(struct net_device *ndev,
static int enetc_get_ts_info(struct net_device *ndev,
struct ethtool_ts_info *info)
{
+#ifdef CONFIG_FSL_ENETC_PTP_CLOCK
+ struct enetc_ndev_priv *priv = netdev_priv(ndev);
+#endif
int *phc_idx;
phc_idx = symbol_get(enetc_phc_index);
@@ -791,8 +794,11 @@ static int enetc_get_ts_info(struct net_device *ndev,
SOF_TIMESTAMPING_SOFTWARE;
info->tx_types = (1 << HWTSTAMP_TX_OFF) |
- (1 << HWTSTAMP_TX_ON) |
- (1 << HWTSTAMP_TX_ONESTEP_SYNC);
+ (1 << HWTSTAMP_TX_ON);
+
+ if (enetc_si_is_pf(priv->si))
+ info->tx_types |= (1 << HWTSTAMP_TX_ONESTEP_SYNC);
+
info->rx_filters = (1 << HWTSTAMP_FILTER_NONE) |
(1 << HWTSTAMP_FILTER_ALL);
#else
diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c
index eec089acc519..69e8233920a1 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -465,10 +465,15 @@ static void ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter)
static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx)
{
+ void *ptr = adapter->tx_ltb_ptr[idx];
+
+ if (!ptr)
+ return;
+
+ adapter->tx_ltb_ptr[idx] = NULL;
dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx],
adapter->tx_ltb_size, DMA_TO_DEVICE);
- kfree(adapter->tx_ltb_ptr[idx]);
- adapter->tx_ltb_ptr[idx] = NULL;
+ kfree(ptr);
}
static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx)
@@ -579,7 +584,7 @@ static int ibmveth_open(struct net_device *netdev)
for (i = 0; i < netdev->real_num_tx_queues; i++) {
if (ibmveth_allocate_tx_ltb(adapter, i))
- goto out_free_tx_ltb;
+ goto out_unmap_filter_list;
}
adapter->rx_queue.index = 0;
@@ -630,10 +635,6 @@ static int ibmveth_open(struct net_device *netdev)
if (rc != 0) {
netdev_err(netdev, "unable to request irq 0x%x, rc %d\n",
netdev->irq, rc);
- do {
- lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
- } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
-
goto out_free_buffer_pools;
}
@@ -649,6 +650,9 @@ static int ibmveth_open(struct net_device *netdev)
return 0;
out_free_buffer_pools:
+ do {
+ lpar_rc = h_free_logical_lan(adapter->vdev->unit_address);
+ } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY));
while (--i >= 0) {
if (adapter->rx_buff_pool[i].active)
ibmveth_free_buffer_pool(adapter,
@@ -658,10 +662,8 @@ static int ibmveth_open(struct net_device *netdev)
dma_unmap_single(dev, adapter->filter_list_dma, 4096,
DMA_BIDIRECTIONAL);
-out_free_tx_ltb:
- while (--i >= 0) {
+ for (i = netdev->real_num_tx_queues - 1; i >= 0; i--)
ibmveth_free_tx_ltb(adapter, i);
- }
out_unmap_buffer_list:
dma_unmap_single(dev, adapter->buffer_list_dma, 4096,
@@ -690,8 +692,7 @@ static int ibmveth_close(struct net_device *netdev)
napi_disable(&adapter->napi);
- if (!adapter->pool_config)
- netif_tx_stop_all_queues(netdev);
+ netif_tx_stop_all_queues(netdev);
h_vio_signal(adapter->vdev->unit_address, VIO_IRQ_DISABLE);
@@ -799,9 +800,7 @@ static int ibmveth_set_csum_offload(struct net_device *dev, u32 data)
if (netif_running(dev)) {
restart = 1;
- adapter->pool_config = 1;
ibmveth_close(dev);
- adapter->pool_config = 0;
}
set_attr = 0;
@@ -883,9 +882,7 @@ static int ibmveth_set_tso(struct net_device *dev, u32 data)
if (netif_running(dev)) {
restart = 1;
- adapter->pool_config = 1;
ibmveth_close(dev);
- adapter->pool_config = 0;
}
set_attr = 0;
@@ -1534,9 +1531,7 @@ static int ibmveth_change_mtu(struct net_device *dev, int new_mtu)
only the buffer pools necessary to hold the new MTU */
if (netif_running(adapter->netdev)) {
need_restart = 1;
- adapter->pool_config = 1;
ibmveth_close(adapter->netdev);
- adapter->pool_config = 0;
}
/* Look for an active buffer pool that can hold the new MTU */
@@ -1722,7 +1717,6 @@ static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
adapter->vdev = dev;
adapter->netdev = netdev;
adapter->mcastFilterSize = be32_to_cpu(*mcastFilterSize_p);
- adapter->pool_config = 0;
ibmveth_init_link_settings(netdev);
netif_napi_add_weight(netdev, &adapter->napi, ibmveth_poll, 16);
@@ -1854,20 +1848,22 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
long value = simple_strtol(buf, NULL, 10);
long rc;
+ rtnl_lock();
+
if (attr == &veth_active_attr) {
if (value && !pool->active) {
if (netif_running(netdev)) {
if (ibmveth_alloc_buffer_pool(pool)) {
netdev_err(netdev,
"unable to alloc pool\n");
- return -ENOMEM;
+ rc = -ENOMEM;
+ goto unlock_err;
}
pool->active = 1;
- adapter->pool_config = 1;
ibmveth_close(netdev);
- adapter->pool_config = 0;
- if ((rc = ibmveth_open(netdev)))
- return rc;
+ rc = ibmveth_open(netdev);
+ if (rc)
+ goto unlock_err;
} else {
pool->active = 1;
}
@@ -1887,54 +1883,59 @@ static ssize_t veth_pool_store(struct kobject *kobj, struct attribute *attr,
if (i == IBMVETH_NUM_BUFF_POOLS) {
netdev_err(netdev, "no active pool >= MTU\n");
- return -EPERM;
+ rc = -EPERM;
+ goto unlock_err;
}
if (netif_running(netdev)) {
- adapter->pool_config = 1;
ibmveth_close(netdev);
pool->active = 0;
- adapter->pool_config = 0;
- if ((rc = ibmveth_open(netdev)))
- return rc;
+ rc = ibmveth_open(netdev);
+ if (rc)
+ goto unlock_err;
}
pool->active = 0;
}
} else if (attr == &veth_num_attr) {
if (value <= 0 || value > IBMVETH_MAX_POOL_COUNT) {
- return -EINVAL;
+ rc = -EINVAL;
+ goto unlock_err;
} else {
if (netif_running(netdev)) {
- adapter->pool_config = 1;
ibmveth_close(netdev);
- adapter->pool_config = 0;
pool->size = value;
- if ((rc = ibmveth_open(netdev)))
- return rc;
+ rc = ibmveth_open(netdev);
+ if (rc)
+ goto unlock_err;
} else {
pool->size = value;
}
}
} else if (attr == &veth_size_attr) {
if (value <= IBMVETH_BUFF_OH || value > IBMVETH_MAX_BUF_SIZE) {
- return -EINVAL;
+ rc = -EINVAL;
+ goto unlock_err;
} else {
if (netif_running(netdev)) {
- adapter->pool_config = 1;
ibmveth_close(netdev);
- adapter->pool_config = 0;
pool->buff_size = value;
- if ((rc = ibmveth_open(netdev)))
- return rc;
+ rc = ibmveth_open(netdev);
+ if (rc)
+ goto unlock_err;
} else {
pool->buff_size = value;
}
}
}
+ rtnl_unlock();
/* kick the interrupt handler to allocate/deallocate pools */
ibmveth_interrupt(netdev->irq, netdev);
return count;
+
+unlock_err:
+ rtnl_unlock();
+ return rc;
}
diff --git a/drivers/net/ethernet/ibm/ibmveth.h b/drivers/net/ethernet/ibm/ibmveth.h
index 0e1dff102444..bc1c1bb83c40 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -148,7 +148,6 @@ struct ibmveth_adapter {
dma_addr_t filter_list_dma;
struct ibmveth_buff_pool rx_buff_pool[IBMVETH_NUM_BUFF_POOLS];
struct ibmveth_rx_q rx_queue;
- int pool_config;
int rx_csum;
int large_send;
bool is_active_trunk;
diff --git a/drivers/net/ethernet/intel/iavf/iavf_ethtool.c b/drivers/net/ethernet/intel/iavf/iavf_ethtool.c
index f4ac2b164b3e..630d93e0512d 100644
--- a/drivers/net/ethernet/intel/iavf/iavf_ethtool.c
+++ b/drivers/net/ethernet/intel/iavf/iavf_ethtool.c
@@ -333,14 +333,13 @@ static int iavf_get_sset_count(struct net_device *netdev, int sset)
{
/* Report the maximum number queues, even if not every queue is
* currently configured. Since allocation of queues is in pairs,
- * use netdev->real_num_tx_queues * 2. The real_num_tx_queues is set
- * at device creation and never changes.
+ * use netdev->num_tx_queues * 2. The num_tx_queues is set at
+ * device creation and never changes.
*/
if (sset == ETH_SS_STATS)
return IAVF_STATS_LEN +
- (IAVF_QUEUE_STATS_LEN * 2 *
- netdev->real_num_tx_queues);
+ (IAVF_QUEUE_STATS_LEN * 2 * netdev->num_tx_queues);
else if (sset == ETH_SS_PRIV_FLAGS)
return IAVF_PRIV_FLAGS_STR_LEN;
else
@@ -367,19 +366,19 @@ static void iavf_get_ethtool_stats(struct net_device *netdev,
iavf_add_ethtool_stats(&data, adapter, iavf_gstrings_stats);
rcu_read_lock();
- /* As num_active_queues describe both tx and rx queues, we can use
- * it to iterate over rings' stats.
+ /* Use num_tx_queues to report stats for the maximum number of queues.
+ * Queues beyond num_active_queues will report zero.
*/
- for (i = 0; i < adapter->num_active_queues; i++) {
- struct iavf_ring *ring;
+ for (i = 0; i < netdev->num_tx_queues; i++) {
+ struct iavf_ring *tx_ring = NULL, *rx_ring = NULL;
- /* Tx rings stats */
- ring = &adapter->tx_rings[i];
- iavf_add_queue_stats(&data, ring);
+ if (i < adapter->num_active_queues) {
+ tx_ring = &adapter->tx_rings[i];
+ rx_ring = &adapter->rx_rings[i];
+ }
- /* Rx rings stats */
- ring = &adapter->rx_rings[i];
- iavf_add_queue_stats(&data, ring);
+ iavf_add_queue_stats(&data, tx_ring);
+ iavf_add_queue_stats(&data, rx_ring);
}
rcu_read_unlock();
}
@@ -416,9 +415,9 @@ static void iavf_get_stat_strings(struct net_device *netdev, u8 *data)
iavf_add_stat_strings(&data, iavf_gstrings_stats);
/* Queues are always allocated in pairs, so we just use
- * real_num_tx_queues for both Tx and Rx queues.
+ * num_tx_queues for both Tx and Rx queues.
*/
- for (i = 0; i < netdev->real_num_tx_queues; i++) {
+ for (i = 0; i < netdev->num_tx_queues; i++) {
iavf_add_stat_strings(&data, iavf_gstrings_queue_stats,
"tx", i);
iavf_add_stat_strings(&data, iavf_gstrings_queue_stats,
diff --git a/drivers/net/ethernet/intel/ice/ice.h b/drivers/net/ethernet/intel/ice/ice.h
index 1fe9cccf18d2..874d6133cb17 100644
--- a/drivers/net/ethernet/intel/ice/ice.h
+++ b/drivers/net/ethernet/intel/ice/ice.h
@@ -302,6 +302,7 @@ enum ice_vsi_state {
ICE_VSI_UMAC_FLTR_CHANGED,
ICE_VSI_MMAC_FLTR_CHANGED,
ICE_VSI_PROMISC_CHANGED,
+ ICE_VSI_REBUILD_PENDING,
ICE_VSI_STATE_NBITS /* must be last */
};
@@ -390,6 +391,7 @@ struct ice_vsi {
struct ice_tx_ring **xdp_rings; /* XDP ring array */
u16 num_xdp_txq; /* Used XDP queues */
u8 xdp_mapping_mode; /* ICE_MAP_MODE_[CONTIG|SCATTER] */
+ struct mutex xdp_state_lock;
struct net_device **target_netdevs;
diff --git a/drivers/net/ethernet/intel/ice/ice_eswitch.c b/drivers/net/ethernet/intel/ice/ice_eswitch.c
index 7de4a8a4b563..569d1b95ec1e 100644
--- a/drivers/net/ethernet/intel/ice/ice_eswitch.c
+++ b/drivers/net/ethernet/intel/ice/ice_eswitch.c
@@ -235,7 +235,7 @@ ice_eswitch_release_reprs(struct ice_pf *pf, struct ice_vsi *ctrl_vsi)
continue;
ice_vsi_update_security(vsi, ice_vsi_ctx_set_antispoof);
- metadata_dst_free(vf->repr->dst);
+ dst_release(&vf->repr->dst->dst);
vf->repr->dst = NULL;
ice_fltr_add_mac_and_broadcast(vsi, vf->hw_lan_addr.addr,
ICE_FWD_TO_VSI);
diff --git a/drivers/net/ethernet/intel/ice/ice_lib.c b/drivers/net/ethernet/intel/ice/ice_lib.c
index 347c6c23bfc1..441e8a6c26a1 100644
--- a/drivers/net/ethernet/intel/ice/ice_lib.c
+++ b/drivers/net/ethernet/intel/ice/ice_lib.c
@@ -365,6 +365,7 @@ int ice_vsi_clear(struct ice_vsi *vsi)
pf->next_vsi = vsi->idx;
ice_vsi_free_arrays(vsi);
+ mutex_destroy(&vsi->xdp_state_lock);
mutex_unlock(&pf->sw_mutex);
devm_kfree(dev, vsi);
@@ -540,6 +541,8 @@ ice_vsi_alloc(struct ice_pf *pf, enum ice_vsi_type vsi_type,
if (vsi->type == ICE_VSI_CTRL && vf)
vf->ctrl_vsi_idx = vsi->idx;
+
+ mutex_init(&vsi->xdp_state_lock);
goto unlock_pf;
err_rings:
@@ -3204,10 +3207,14 @@ int ice_vsi_rebuild(struct ice_vsi *vsi, bool init_vsi)
ice_vsi_init_vlan_ops(vsi);
+ mutex_lock(&vsi->xdp_state_lock);
+
coalesce = kcalloc(vsi->num_q_vectors,
sizeof(struct ice_coalesce_stored), GFP_KERNEL);
- if (!coalesce)
+ if (!coalesce) {
+ mutex_unlock(&vsi->xdp_state_lock);
return -ENOMEM;
+ }
prev_num_q_vectors = ice_vsi_rebuild_get_coalesce(vsi, coalesce);
@@ -3230,7 +3237,7 @@ int ice_vsi_rebuild(struct ice_vsi *vsi, bool init_vsi)
vsi->base_vector = 0;
}
- if (ice_is_xdp_ena_vsi(vsi))
+ if (vsi->xdp_rings)
/* return value check can be skipped here, it always returns
* 0 if reset is in progress
*/
@@ -3358,24 +3365,31 @@ int ice_vsi_rebuild(struct ice_vsi *vsi, bool init_vsi)
ret = -EIO;
goto err_vectors;
} else {
+ mutex_unlock(&vsi->xdp_state_lock);
return ice_schedule_reset(pf, ICE_RESET_PFR);
}
}
ice_vsi_rebuild_set_coalesce(vsi, coalesce, prev_num_q_vectors);
+ clear_bit(ICE_VSI_REBUILD_PENDING, vsi->state);
kfree(coalesce);
+ mutex_unlock(&vsi->xdp_state_lock);
return 0;
err_vectors:
ice_vsi_free_q_vectors(vsi);
err_rings:
+ mutex_unlock(&vsi->xdp_state_lock);
if (vsi->netdev) {
vsi->current_netdev_flags = 0;
unregister_netdev(vsi->netdev);
free_netdev(vsi->netdev);
vsi->netdev = NULL;
}
+ goto err_clear;
err_vsi:
+ mutex_unlock(&vsi->xdp_state_lock);
+err_clear:
ice_vsi_clear(vsi);
set_bit(ICE_RESET_FAILED, pf->state);
kfree(coalesce);
diff --git a/drivers/net/ethernet/intel/ice/ice_main.c b/drivers/net/ethernet/intel/ice/ice_main.c
index e923b64f2a66..8db6e47decdb 100644
--- a/drivers/net/ethernet/intel/ice/ice_main.c
+++ b/drivers/net/ethernet/intel/ice/ice_main.c
@@ -620,6 +620,7 @@ ice_prepare_for_reset(struct ice_pf *pf, enum ice_reset_req reset_type)
/* clear SW filtering DB */
ice_clear_hw_tbls(hw);
/* disable the VSIs and their queues that are not already DOWN */
+ set_bit(ICE_VSI_REBUILD_PENDING, ice_get_main_vsi(pf)->state);
ice_pf_dis_all_vsi(pf, false);
if (test_bit(ICE_FLAG_PTP_SUPPORTED, pf->flags))
@@ -2641,6 +2642,27 @@ static void ice_vsi_assign_bpf_prog(struct ice_vsi *vsi, struct bpf_prog *prog)
bpf_prog_put(old_prog);
}
+/**
+ * ice_unmap_xdp_rings - Unmap XDP rings from interrupt vectors
+ * @vsi: the VSI with XDP rings being unmapped
+ */
+static void ice_unmap_xdp_rings(struct ice_vsi *vsi)
+{
+ int v_idx;
+
+ ice_for_each_q_vector(vsi, v_idx) {
+ struct ice_q_vector *q_vector = vsi->q_vectors[v_idx];
+ struct ice_tx_ring *ring;
+
+ ice_for_each_tx_ring(ring, q_vector->tx)
+ if (!ring->tx_buf || !ice_ring_is_xdp(ring))
+ break;
+
+ /* restore the value of last node prior to XDP setup */
+ q_vector->tx.tx_ring = ring;
+ }
+}
+
/**
* ice_prepare_xdp_rings - Allocate, configure and setup Tx rings for XDP
* @vsi: VSI to bring up Tx rings used by XDP
@@ -2738,7 +2760,7 @@ int ice_prepare_xdp_rings(struct ice_vsi *vsi, struct bpf_prog *prog)
if (status) {
dev_err(dev, "Failed VSI LAN queue config for XDP, error: %d\n",
status);
- goto clear_xdp_rings;
+ goto unmap_xdp_rings;
}
/* assign the prog only when it's not already present on VSI;
@@ -2754,6 +2776,8 @@ int ice_prepare_xdp_rings(struct ice_vsi *vsi, struct bpf_prog *prog)
ice_vsi_assign_bpf_prog(vsi, prog);
return 0;
+unmap_xdp_rings:
+ ice_unmap_xdp_rings(vsi);
clear_xdp_rings:
ice_for_each_xdp_txq(vsi, i)
if (vsi->xdp_rings[i]) {
@@ -2770,6 +2794,8 @@ int ice_prepare_xdp_rings(struct ice_vsi *vsi, struct bpf_prog *prog)
mutex_unlock(&pf->avail_q_mutex);
devm_kfree(dev, vsi->xdp_rings);
+ vsi->xdp_rings = NULL;
+
return -ENOMEM;
}
@@ -2784,7 +2810,7 @@ int ice_destroy_xdp_rings(struct ice_vsi *vsi)
{
u16 max_txqs[ICE_MAX_TRAFFIC_CLASS] = { 0 };
struct ice_pf *pf = vsi->back;
- int i, v_idx;
+ int i;
/* q_vectors are freed in reset path so there's no point in detaching
* rings; in case of rebuild being triggered not from reset bits
@@ -2794,17 +2820,7 @@ int ice_destroy_xdp_rings(struct ice_vsi *vsi)
if (ice_is_reset_in_progress(pf->state) || !vsi->q_vectors[0])
goto free_qmap;
- ice_for_each_q_vector(vsi, v_idx) {
- struct ice_q_vector *q_vector = vsi->q_vectors[v_idx];
- struct ice_tx_ring *ring;
-
- ice_for_each_tx_ring(ring, q_vector->tx)
- if (!ring->tx_buf || !ice_ring_is_xdp(ring))
- break;
-
- /* restore the value of last node prior to XDP setup */
- q_vector->tx.tx_ring = ring;
- }
+ ice_unmap_xdp_rings(vsi);
free_qmap:
mutex_lock(&pf->avail_q_mutex);
@@ -2919,7 +2935,8 @@ ice_xdp_setup_prog(struct ice_vsi *vsi, struct bpf_prog *prog,
}
/* hot swap progs and avoid toggling link */
- if (ice_is_xdp_ena_vsi(vsi) == !!prog) {
+ if (ice_is_xdp_ena_vsi(vsi) == !!prog ||
+ test_bit(ICE_VSI_REBUILD_PENDING, vsi->state)) {
ice_vsi_assign_bpf_prog(vsi, prog);
return 0;
}
@@ -2940,10 +2957,13 @@ ice_xdp_setup_prog(struct ice_vsi *vsi, struct bpf_prog *prog,
xdp_ring_err = ice_vsi_determine_xdp_res(vsi);
if (xdp_ring_err) {
NL_SET_ERR_MSG_MOD(extack, "Not enough Tx resources for XDP");
+ goto resume_if;
} else {
xdp_ring_err = ice_prepare_xdp_rings(vsi, prog);
- if (xdp_ring_err)
+ if (xdp_ring_err) {
NL_SET_ERR_MSG_MOD(extack, "Setting up XDP Tx resources failed");
+ goto resume_if;
+ }
}
/* reallocate Rx queues that are used for zero-copy */
xdp_ring_err = ice_realloc_zc_buf(vsi, true);
@@ -2959,6 +2979,7 @@ ice_xdp_setup_prog(struct ice_vsi *vsi, struct bpf_prog *prog,
NL_SET_ERR_MSG_MOD(extack, "Freeing XDP Rx resources failed");
}
+resume_if:
if (if_running)
ret = ice_up(vsi);
@@ -2991,21 +3012,28 @@ static int ice_xdp(struct net_device *dev, struct netdev_bpf *xdp)
{
struct ice_netdev_priv *np = netdev_priv(dev);
struct ice_vsi *vsi = np->vsi;
+ int ret;
if (vsi->type != ICE_VSI_PF) {
NL_SET_ERR_MSG_MOD(xdp->extack, "XDP can be loaded only on PF VSI");
return -EINVAL;
}
+ mutex_lock(&vsi->xdp_state_lock);
+
switch (xdp->command) {
case XDP_SETUP_PROG:
- return ice_xdp_setup_prog(vsi, xdp->prog, xdp->extack);
+ ret = ice_xdp_setup_prog(vsi, xdp->prog, xdp->extack);
+ break;
case XDP_SETUP_XSK_POOL:
- return ice_xsk_pool_setup(vsi, xdp->xsk.pool,
- xdp->xsk.queue_id);
+ ret = ice_xsk_pool_setup(vsi, xdp->xsk.pool, xdp->xsk.queue_id);
+ break;
default:
- return -EINVAL;
+ ret = -EINVAL;
}
+
+ mutex_unlock(&vsi->xdp_state_lock);
+ return ret;
}
/**
diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.c b/drivers/net/ethernet/intel/ice/ice_ptp.c
index 06c5a165e066..f2f1078166fc 100644
--- a/drivers/net/ethernet/intel/ice/ice_ptp.c
+++ b/drivers/net/ethernet/intel/ice/ice_ptp.c
@@ -1488,6 +1488,10 @@ void ice_ptp_extts_event(struct ice_pf *pf)
u8 chan, tmr_idx;
u32 hi, lo;
+ /* Don't process timestamp events if PTP is not ready */
+ if (!test_bit(ICE_FLAG_PTP, pf->flags))
+ return;
+
tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
/* Event time is captured by one of the two matched registers
* GLTSYN_EVNT_L: 32 LSB of sampled time event
@@ -1513,27 +1517,24 @@ void ice_ptp_extts_event(struct ice_pf *pf)
/**
* ice_ptp_cfg_extts - Configure EXTTS pin and channel
* @pf: Board private structure
- * @ena: true to enable; false to disable
* @chan: GPIO channel (0-3)
- * @gpio_pin: GPIO pin
- * @extts_flags: request flags from the ptp_extts_request.flags
+ * @config: desired EXTTS configuration.
+ * @store: If set to true, the values will be stored
+ *
+ * Configure an external timestamp event on the requested channel.
*/
-static int
-ice_ptp_cfg_extts(struct ice_pf *pf, bool ena, unsigned int chan, u32 gpio_pin,
- unsigned int extts_flags)
+static void ice_ptp_cfg_extts(struct ice_pf *pf, unsigned int chan,
+ struct ice_extts_channel *config, bool store)
{
u32 func, aux_reg, gpio_reg, irq_reg;
struct ice_hw *hw = &pf->hw;
u8 tmr_idx;
- if (chan > (unsigned int)pf->ptp.info.n_ext_ts)
- return -EINVAL;
-
tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned;
irq_reg = rd32(hw, PFINT_OICR_ENA);
- if (ena) {
+ if (config->ena) {
/* Enable the interrupt */
irq_reg |= PFINT_OICR_TSYN_EVNT_M;
aux_reg = GLTSYN_AUX_IN_0_INT_ENA_M;
@@ -1542,9 +1543,9 @@ ice_ptp_cfg_extts(struct ice_pf *pf, bool ena, unsigned int chan, u32 gpio_pin,
#define GLTSYN_AUX_IN_0_EVNTLVL_FALLING_EDGE BIT(1)
/* set event level to requested edge */
- if (extts_flags & PTP_FALLING_EDGE)
+ if (config->flags & PTP_FALLING_EDGE)
aux_reg |= GLTSYN_AUX_IN_0_EVNTLVL_FALLING_EDGE;
- if (extts_flags & PTP_RISING_EDGE)
+ if (config->flags & PTP_RISING_EDGE)
aux_reg |= GLTSYN_AUX_IN_0_EVNTLVL_RISING_EDGE;
/* Write GPIO CTL reg.
@@ -1566,9 +1567,30 @@ ice_ptp_cfg_extts(struct ice_pf *pf, bool ena, unsigned int chan, u32 gpio_pin,
wr32(hw, PFINT_OICR_ENA, irq_reg);
wr32(hw, GLTSYN_AUX_IN(chan, tmr_idx), aux_reg);
- wr32(hw, GLGEN_GPIO_CTL(gpio_pin), gpio_reg);
+ wr32(hw, GLGEN_GPIO_CTL(config->gpio_pin), gpio_reg);
- return 0;
+ if (store)
+ memcpy(&pf->ptp.extts_channels[chan], config, sizeof(*config));
+}
+
+/**
+ * ice_ptp_disable_all_extts - Disable all EXTTS channels
+ * @pf: Board private structure
+ */
+static void ice_ptp_disable_all_extts(struct ice_pf *pf)
+{
+ struct ice_extts_channel extts_cfg = {};
+ int i;
+
+ for (i = 0; i < pf->ptp.info.n_ext_ts; i++) {
+ if (pf->ptp.extts_channels[i].ena) {
+ extts_cfg.gpio_pin = pf->ptp.extts_channels[i].gpio_pin;
+ extts_cfg.ena = false;
+ ice_ptp_cfg_extts(pf, i, &extts_cfg, false);
+ }
+ }
+
+ synchronize_irq(pf->msix_entries[pf->oicr_idx].vector);
}
/**
@@ -1725,7 +1747,6 @@ ice_ptp_gpio_enable_e810(struct ptp_clock_info *info,
struct ptp_clock_request *rq, int on)
{
struct ice_pf *pf = ptp_info_to_pf(info);
- struct ice_perout_channel clk_cfg = {0};
bool sma_pres = false;
unsigned int chan;
u32 gpio_pin;
@@ -1736,6 +1757,9 @@ ice_ptp_gpio_enable_e810(struct ptp_clock_info *info,
switch (rq->type) {
case PTP_CLK_REQ_PEROUT:
+ {
+ struct ice_perout_channel clk_cfg = {};
+
chan = rq->perout.index;
if (sma_pres) {
if (chan == ice_pin_desc_e810t[SMA1].chan)
@@ -1763,7 +1787,11 @@ ice_ptp_gpio_enable_e810(struct ptp_clock_info *info,
err = ice_ptp_cfg_clkout(pf, chan, &clk_cfg, true);
break;
+ }
case PTP_CLK_REQ_EXTTS:
+ {
+ struct ice_extts_channel extts_cfg = {};
+
chan = rq->extts.index;
if (sma_pres) {
if (chan < ice_pin_desc_e810t[SMA2].chan)
@@ -1779,9 +1807,13 @@ ice_ptp_gpio_enable_e810(struct ptp_clock_info *info,
gpio_pin = chan;
}
- err = ice_ptp_cfg_extts(pf, !!on, chan, gpio_pin,
- rq->extts.flags);
- break;
+ extts_cfg.flags = rq->extts.flags;
+ extts_cfg.gpio_pin = gpio_pin;
+ extts_cfg.ena = !!on;
+
+ ice_ptp_cfg_extts(pf, chan, &extts_cfg, true);
+ return 0;
+ }
default:
return -EOPNOTSUPP;
}
@@ -2711,6 +2743,8 @@ void ice_ptp_release(struct ice_pf *pf)
clear_bit(ICE_FLAG_PTP, pf->flags);
+ ice_ptp_disable_all_extts(pf);
+
kthread_cancel_delayed_work_sync(&pf->ptp.work);
ice_ptp_port_phy_stop(&pf->ptp.port);
diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.h b/drivers/net/ethernet/intel/ice/ice_ptp.h
index e689c05bb001..146df66bef6d 100644
--- a/drivers/net/ethernet/intel/ice/ice_ptp.h
+++ b/drivers/net/ethernet/intel/ice/ice_ptp.h
@@ -33,6 +33,12 @@ struct ice_perout_channel {
u64 start_time;
};
+struct ice_extts_channel {
+ bool ena;
+ u32 gpio_pin;
+ u32 flags;
+};
+
/* The ice hardware captures Tx hardware timestamps in the PHY. The timestamp
* is stored in a buffer of registers. Depending on the specific hardware,
* this buffer might be shared across multiple PHY ports.
@@ -165,6 +171,7 @@ struct ice_ptp_port {
* @ext_ts_irq: the external timestamp IRQ in use
* @kworker: kwork thread for handling periodic work
* @perout_channels: periodic output data
+ * @extts_channels: channels for external timestamps
* @info: structure defining PTP hardware capabilities
* @clock: pointer to registered PTP clock device
* @tstamp_config: hardware timestamping configuration
@@ -185,6 +192,7 @@ struct ice_ptp {
u8 ext_ts_irq;
struct kthread_worker *kworker;
struct ice_perout_channel perout_channels[GLTSYN_TGT_H_IDX_MAX];
+ struct ice_extts_channel extts_channels[GLTSYN_TGT_H_IDX_MAX];
struct ptp_clock_info info;
struct ptp_clock *clock;
struct hwtstamp_config tstamp_config;
diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c
index 1d64966da1e8..73c974747ac4 100644
--- a/drivers/net/ethernet/intel/ice/ice_txrx.c
+++ b/drivers/net/ethernet/intel/ice/ice_txrx.c
@@ -2277,6 +2277,9 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
ice_trace(xmit_frame_ring, tx_ring, skb);
+ /* record the location of the first descriptor for this packet */
+ first = &tx_ring->tx_buf[tx_ring->next_to_use];
+
count = ice_xmit_desc_count(skb);
if (ice_chk_linearize(skb, count)) {
if (__skb_linearize(skb))
@@ -2302,8 +2305,6 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
offload.tx_ring = tx_ring;
- /* record the location of the first descriptor for this packet */
- first = &tx_ring->tx_buf[tx_ring->next_to_use];
first->skb = skb;
first->bytecount = max_t(unsigned int, skb->len, ETH_ZLEN);
first->gso_segs = 1;
@@ -2365,6 +2366,7 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring)
out_drop:
ice_trace(xmit_frame_ring_drop, tx_ring, skb);
dev_kfree_skb_any(skb);
+ first->skb = NULL;
return NETDEV_TX_OK;
}
diff --git a/drivers/net/ethernet/intel/ice/ice_xsk.c b/drivers/net/ethernet/intel/ice/ice_xsk.c
index 2677d7c86a6d..98d827db8484 100644
--- a/drivers/net/ethernet/intel/ice/ice_xsk.c
+++ b/drivers/net/ethernet/intel/ice/ice_xsk.c
@@ -388,7 +388,8 @@ int ice_xsk_pool_setup(struct ice_vsi *vsi, struct xsk_buff_pool *pool, u16 qid)
goto failure;
}
- if_running = netif_running(vsi->netdev) && ice_is_xdp_ena_vsi(vsi);
+ if_running = !test_bit(ICE_VSI_DOWN, vsi->state) &&
+ ice_is_xdp_ena_vsi(vsi);
if (if_running) {
struct ice_rx_ring *rx_ring = vsi->rx_rings[qid];
diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
index 086cc2573033..fd5a2ce4862b 100644
--- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
+++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c
@@ -3522,7 +3522,8 @@ void ixgbe_configure_tx_ring(struct ixgbe_adapter *adapter,
}
/* initialize XPS */
- if (!test_and_set_bit(__IXGBE_TX_XPS_INIT_DONE, &ring->state)) {
+ if (!ring_is_xdp(ring) &&
+ !test_and_set_bit(__IXGBE_TX_XPS_INIT_DONE, &ring->state)) {
struct ixgbe_q_vector *q_vector = ring->q_vector;
if (q_vector)
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
index d28dfe11227b..c6e94b6a1085 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c
@@ -495,6 +495,18 @@ int cgx_set_pkind(void *cgxd, u8 lmac_id, int pkind)
return 0;
}
+int cgx_get_pkind(void *cgxd, u8 lmac_id, int *pkind)
+{
+ struct cgx *cgx = cgxd;
+
+ if (!is_lmac_valid(cgx, lmac_id))
+ return -ENODEV;
+
+ *pkind = cgx_read(cgx, lmac_id, CGXX_CMRX_RX_ID_MAP);
+ *pkind = *pkind & 0x3F;
+ return 0;
+}
+
static u8 cgx_get_lmac_type(void *cgxd, int lmac_id)
{
struct cgx *cgx = cgxd;
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.h b/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
index 09ddb00f63cc..4ac86fa3a6d0 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.h
@@ -136,6 +136,7 @@ int cgx_get_cgxid(void *cgxd);
int cgx_get_lmac_cnt(void *cgxd);
void *cgx_get_pdata(int cgx_id);
int cgx_set_pkind(void *cgxd, u8 lmac_id, int pkind);
+int cgx_get_pkind(void *cgxd, u8 lmac_id, int *pkind);
int cgx_lmac_evh_register(struct cgx_event_cb *cb, void *cgxd, int lmac_id);
int cgx_lmac_evh_unregister(void *cgxd, int lmac_id);
int cgx_get_tx_stats(void *cgxd, int lmac_id, int idx, u64 *tx_stat);
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/mcs.c b/drivers/net/ethernet/marvell/octeontx2/af/mcs.c
index 211c10aa5880..30c672d526a0 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/mcs.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/mcs.c
@@ -314,7 +314,7 @@ int mcs_clear_all_stats(struct mcs *mcs, u16 pcifunc, int dir)
}
/* Clear SC stats */
- for (id = 0; id < map->secy.max; id++) {
+ for (id = 0; id < map->sc.max; id++) {
if (map->sc2pf_map[id] != pcifunc)
continue;
mcs_clear_stats(mcs, MCS_SC_STATS, id, dir);
@@ -738,7 +738,7 @@ int mcs_free_all_rsrc(struct mcs *mcs, int dir, u16 pcifunc)
}
/* free sc entries */
- for (id = 0; id < map->secy.max; id++) {
+ for (id = 0; id < map->sc.max; id++) {
if (map->sc2pf_map[id] != pcifunc)
continue;
mcs_free_rsrc(&map->sc, map->sc2pf_map, id, pcifunc);
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
index c0a2f4ca5c17..354b28acff0c 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h
@@ -868,6 +868,8 @@ void npc_read_mcam_entry(struct rvu *rvu, struct npc_mcam *mcam,
int blkaddr, u16 src, struct mcam_entry *entry,
u8 *intf, u8 *ena);
bool is_cgx_config_permitted(struct rvu *rvu, u16 pcifunc);
+bool rvu_cgx_check_permission_and_set_pkind(struct rvu *rvu, u16 pcifunc, int pkind);
+bool rvu_cgx_is_pkind_config_permitted(struct rvu *rvu, u16 pcifunc);
bool is_mac_feature_supported(struct rvu *rvu, int pf, int feature);
u32 rvu_cgx_get_fifolen(struct rvu *rvu);
void *rvu_first_cgx_pdata(struct rvu *rvu);
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
index cc6d6c94f400..f1ef9ecad1bd 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c
@@ -1260,3 +1260,82 @@ int rvu_mbox_handler_cgx_prio_flow_ctrl_cfg(struct rvu *rvu,
mac_ops->mac_get_pfc_frm_cfg(cgxd, lmac_id, &rsp->tx_pause, &rsp->rx_pause);
return err;
}
+
+/* Do not allow CGX-mapped VFs to overwrite PKIND when special parse kinds
+ * (HiGig, EDSA, etc.) are in use on the shared LMAC. VFs must not program
+ * NPC_TX_DEF_PKIND on NIX_AF_LFX_TX_PARSE_CFG in that case: the PF owns
+ * parse mode and no separate NPC_TX_HIGIG_PKIND is installed on the VF LF.
+ * TX-parse callers skip the write when denied; rvu_lf_reset() clears each LF
+ * before alloc so the next permitted owner programs NPC_TX_DEF_PKIND.
+ */
+bool rvu_cgx_is_pkind_config_permitted(struct rvu *rvu, u16 pcifunc)
+{
+ int pf, err, rxpkind;
+ u8 cgx_id, lmac_id;
+ void *cgxd;
+
+ pf = rvu_get_pf(pcifunc);
+
+ if (!(pcifunc & RVU_PFVF_FUNC_MASK))
+ return true;
+
+ if (!is_pf_cgxmapped(rvu, pf))
+ return true;
+
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ err = cgx_get_pkind(cgxd, lmac_id, &rxpkind);
+ if (err)
+ return false;
+
+ switch (rxpkind) {
+ case NPC_RX_HIGIG_PKIND:
+ case NPC_RX_EDSA_PKIND:
+ return false;
+ default:
+ return true;
+ }
+}
+
+/* Do not allow CGX-mapped VFs to overwrite PKIND when special parse kinds
+ * (HiGig, EDSA, etc.) are in use on the shared LMAC.
+ */
+bool rvu_cgx_check_permission_and_set_pkind(struct rvu *rvu, u16 pcifunc, int pkind)
+{
+ int pf, err, rxpkind;
+ u8 cgx_id, lmac_id;
+ struct cgx *cgxd;
+
+ pf = rvu_get_pf(pcifunc);
+
+ if (!is_pf_cgxmapped(rvu, pf))
+ return false;
+
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+
+ mutex_lock(&cgxd->lock);
+ if (!is_vf(pcifunc))
+ goto set;
+
+ err = cgx_get_pkind(cgxd, lmac_id, &rxpkind);
+ if (err)
+ goto err;
+
+ switch (rxpkind) {
+ case NPC_RX_HIGIG_PKIND:
+ case NPC_RX_EDSA_PKIND:
+ goto err;
+ default:
+ break;
+ }
+
+set:
+ cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, pkind);
+ mutex_unlock(&cgxd->lock);
+ return true;
+
+err:
+ mutex_unlock(&cgxd->lock);
+ return false;
+}
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
index 3d4985e2f9db..8b2e59e864d3 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c
@@ -324,8 +324,8 @@ static int nix_interface_init(struct rvu *rvu, u16 pcifunc, int type, int nixlf,
pfvf->tx_chan_cnt = 1;
rsp->tx_link = cgx_id * hw->lmac_per_cgx + lmac_id;
- cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, pkind);
- rvu_npc_set_pkind(rvu, pkind, pfvf);
+ if (rvu_cgx_check_permission_and_set_pkind(rvu, pcifunc, pkind))
+ rvu_npc_set_pkind(rvu, pkind, pfvf);
break;
case NIX_INTF_TYPE_LBK:
@@ -1314,11 +1314,13 @@ int rvu_mbox_handler_nix_lf_alloc(struct rvu *rvu,
struct nix_lf_alloc_req *req,
struct nix_lf_alloc_rsp *rsp)
{
- int nixlf, qints, hwctx_size, intf, err, rc = 0;
+ int nixlf, qints, hwctx_size, intf, err, rc = 0, pf;
struct rvu_hwinfo *hw = rvu->hw;
u16 pcifunc = req->hdr.pcifunc;
+ u8 cgx_id = 0, lmac_id = 0;
struct rvu_block *block;
struct rvu_pfvf *pfvf;
+ struct cgx *cgxd;
u64 cfg, ctx_cfg;
int blkaddr;
@@ -1486,8 +1488,20 @@ int rvu_mbox_handler_nix_lf_alloc(struct rvu *rvu,
rvu_write64(rvu, blkaddr, NIX_AF_LFX_RX_CFG(nixlf), req->rx_cfg);
/* Configure pkind for TX parse config */
+
+ pf = rvu_get_pf(pcifunc);
cfg = NPC_TX_DEF_PKIND;
- rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+
+ if (is_pf_cgxmapped(rvu, pf) && is_vf(pcifunc)) {
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ mutex_lock(&cgxd->lock);
+ if (rvu_cgx_is_pkind_config_permitted(rvu, pcifunc))
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+ mutex_unlock(&cgxd->lock);
+ } else {
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg);
+ }
intf = is_afvf(pcifunc) ? NIX_INTF_TYPE_LBK : NIX_INTF_TYPE_CGX;
if (is_sdp_pfvf(pcifunc))
diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
index 65ad7476e609..c8146e3afb1b 100644
--- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
+++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c
@@ -16,6 +16,7 @@
#include "cgx.h"
#include "npc_profile.h"
#include "rvu_npc_hash.h"
+#include "lmac_common.h"
#define RSVD_MCAM_ENTRIES_PER_PF 3 /* Broadcast, Promisc and AllMulticast */
#define RSVD_MCAM_ENTRIES_PER_NIXLF 1 /* Ucast for LFs */
@@ -1933,7 +1934,7 @@ static int npc_mcam_rsrcs_init(struct rvu *rvu, int blkaddr)
/* Alloc memory for MCAM entry to counter mapping and for tracking
* counter's reference count.
*/
- mcam->entry2cntr_map = devm_kcalloc(rvu->dev, mcam->bmap_entries,
+ mcam->entry2cntr_map = devm_kcalloc(rvu->dev, mcam->total_entries,
sizeof(u16), GFP_KERNEL);
if (!mcam->entry2cntr_map)
goto free_mem;
@@ -1949,10 +1950,11 @@ static int npc_mcam_rsrcs_init(struct rvu *rvu, int blkaddr)
if (!mcam->entry2target_pffunc)
goto free_mem;
- for (index = 0; index < mcam->bmap_entries; index++) {
+ for (index = 0; index < mcam->bmap_entries; index++)
mcam->entry2pfvf_map[index] = NPC_MCAM_INVALID_MAP;
+
+ for (index = 0; index < mcam->total_entries; index++)
mcam->entry2cntr_map[index] = NPC_MCAM_INVALID_MAP;
- }
for (cntr = 0; cntr < mcam->counters.max; cntr++)
mcam->cntr2pfvf_map[cntr] = NPC_MCAM_INVALID_MAP;
@@ -3039,7 +3041,7 @@ int rvu_mbox_handler_npc_mcam_free_counter(struct rvu *rvu,
struct npc_mcam_oper_counter_req *req, struct msg_rsp *rsp)
{
struct npc_mcam *mcam = &rvu->hw->mcam;
- u16 index, entry = 0;
+ u16 index;
int blkaddr, err;
blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
@@ -3057,20 +3059,16 @@ int rvu_mbox_handler_npc_mcam_free_counter(struct rvu *rvu,
mcam->cntr2pfvf_map[req->cntr] = NPC_MCAM_INVALID_MAP;
rvu_free_rsrc(&mcam->counters, req->cntr);
- /* Disable all MCAM entry's stats which are using this counter */
- while (entry < mcam->bmap_entries) {
+ /* Disable all MCAM entry's stats which are using this counter.
+ * Scan the full MCAM index range: AF-reserved rules (e.g. CPT pass-2)
+ */
+ for (index = 0; index < mcam->total_entries; index++) {
if (!mcam->cntr_refcnt[req->cntr])
break;
-
- index = find_next_bit(mcam->bmap, mcam->bmap_entries, entry);
- if (index >= mcam->bmap_entries)
- break;
- entry = index + 1;
if (mcam->entry2cntr_map[index] != req->cntr)
continue;
-
- npc_unmap_mcam_entry_and_cntr(rvu, mcam, blkaddr,
- index, req->cntr);
+ npc_unmap_mcam_entry_and_cntr(rvu, mcam, blkaddr, index,
+ req->cntr);
}
mutex_unlock(&mcam->lock);
@@ -3081,7 +3079,7 @@ int rvu_mbox_handler_npc_mcam_unmap_counter(struct rvu *rvu,
struct npc_mcam_unmap_counter_req *req, struct msg_rsp *rsp)
{
struct npc_mcam *mcam = &rvu->hw->mcam;
- u16 index, entry = 0;
+ u16 index;
int blkaddr, rc;
blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0);
@@ -3104,20 +3102,13 @@ int rvu_mbox_handler_npc_mcam_unmap_counter(struct rvu *rvu,
}
/* Disable all MCAM entry's stats which are using this counter */
- while (entry < mcam->bmap_entries) {
+ for (index = 0; index < mcam->total_entries; index++) {
if (!mcam->cntr_refcnt[req->cntr])
break;
-
- index = find_next_bit(mcam->bmap, mcam->bmap_entries, entry);
- if (index >= mcam->bmap_entries)
- break;
- entry = index + 1;
-
if (mcam->entry2cntr_map[index] != req->cntr)
continue;
-
- npc_unmap_mcam_entry_and_cntr(rvu, mcam, blkaddr,
- index, req->cntr);
+ npc_unmap_mcam_entry_and_cntr(rvu, mcam, blkaddr, index,
+ req->cntr);
}
exit:
mutex_unlock(&mcam->lock);
@@ -3337,10 +3328,11 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
{
struct rvu_pfvf *pfvf = rvu_get_pfvf(rvu, pcifunc);
- int blkaddr, nixlf, rc, intf_mode;
int pf = rvu_get_pf(pcifunc);
+ int blkaddr, nixlf, rc, intf_mode;
+ u8 cgx_id = 0, lmac_id = 0;
u64 rxpkind, txpkind;
- u8 cgx_id, lmac_id;
+ struct cgx *cgxd;
/* use default pkind to disable edsa/higig */
rxpkind = rvu_npc_get_pkind(rvu, pf);
@@ -3364,12 +3356,8 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
/* rx pkind set req valid only for cgx mapped PFs */
if (!is_cgx_config_permitted(rvu, pcifunc))
return 0;
- rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id);
-
- rc = cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id,
- rxpkind);
- if (rc)
- return rc;
+ if (!rvu_cgx_check_permission_and_set_pkind(rvu, pcifunc, rxpkind))
+ return -EINVAL;
}
if (dir & PKIND_TX) {
@@ -3378,8 +3366,19 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir,
if (rc)
return rc;
- rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
- txpkind);
+ if (is_pf_cgxmapped(rvu, pf) && is_vf(pcifunc)) {
+ rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id,
+ &lmac_id);
+ cgxd = rvu_cgx_pdata(cgx_id, rvu);
+ mutex_lock(&cgxd->lock);
+ if (rvu_cgx_is_pkind_config_permitted(rvu, pcifunc))
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
+ txpkind);
+ mutex_unlock(&cgxd->lock);
+ } else {
+ rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf),
+ txpkind);
+ }
}
pfvf->intf_mode = intf_mode;
diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c
index af5d69e6d267..f1ae4bb15b8c 100644
--- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c
+++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c
@@ -685,16 +685,16 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id)
snprintf(netdev->name, sizeof(netdev->name), "lbk%d", n);
}
+ err = otx2_wq_init(vf);
+ if (err)
+ goto err_ptp_destroy;
+
err = register_netdev(netdev);
if (err) {
dev_err(dev, "Failed to register netdevice\n");
- goto err_ptp_destroy;
+ goto err_wq_destroy;
}
- err = otx2_wq_init(vf);
- if (err)
- goto err_unreg_netdev;
-
otx2vf_set_ethtool_ops(netdev);
err = otx2vf_mcam_flow_init(vf);
@@ -722,6 +722,10 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id)
otx2_shutdown_tc(vf);
err_unreg_netdev:
unregister_netdev(netdev);
+err_wq_destroy:
+ cancel_work_sync(&vf->reset_task);
+ cancel_work_sync(&vf->rx_mode_work);
+ destroy_workqueue(vf->otx2_wq);
err_ptp_destroy:
otx2_ptp_destroy(vf);
err_detach_rsrc:
@@ -769,11 +773,13 @@ static void otx2vf_remove(struct pci_dev *pdev)
}
#endif
- cancel_work_sync(&vf->reset_task);
otx2_unregister_dl(vf);
unregister_netdev(netdev);
- if (vf->otx2_wq)
+ if (vf->otx2_wq) {
+ cancel_work_sync(&vf->reset_task);
+ cancel_work_sync(&vf->rx_mode_work);
destroy_workqueue(vf->otx2_wq);
+ }
otx2_ptp_destroy(vf);
otx2_mcam_flow_del(vf);
otx2_shutdown_tc(vf);
diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
index 20d14e3ae6ef..a65d6b131777 100644
--- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c
+++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c
@@ -2916,7 +2916,7 @@ static void mtk_poll_controller(struct net_device *dev)
mtk_tx_irq_disable(eth, MTK_TX_DONE_INT);
mtk_rx_irq_disable(eth, eth->soc->txrx.rx_irq_done_mask);
- mtk_handle_irq_rx(eth->irq[2], dev);
+ mtk_handle_irq_rx(eth->irq[2], eth);
mtk_tx_irq_enable(eth, MTK_TX_DONE_INT);
mtk_rx_irq_enable(eth, eth->soc->txrx.rx_irq_done_mask);
}
diff --git a/drivers/net/ethernet/mellanox/mlx4/srq.c b/drivers/net/ethernet/mellanox/mlx4/srq.c
index dd890f5d7b72..39a01aee3bef 100644
--- a/drivers/net/ethernet/mellanox/mlx4/srq.c
+++ b/drivers/net/ethernet/mellanox/mlx4/srq.c
@@ -44,13 +44,14 @@ void mlx4_srq_event(struct mlx4_dev *dev, u32 srqn, int event_type)
{
struct mlx4_srq_table *srq_table = &mlx4_priv(dev)->srq_table;
struct mlx4_srq *srq;
+ unsigned long flags;
- rcu_read_lock();
+ spin_lock_irqsave(&srq_table->lock, flags);
srq = radix_tree_lookup(&srq_table->tree, srqn & (dev->caps.num_srqs - 1));
- rcu_read_unlock();
- if (srq)
- refcount_inc(&srq->refcount);
- else {
+ if (!srq || !refcount_inc_not_zero(&srq->refcount))
+ srq = NULL;
+ spin_unlock_irqrestore(&srq_table->lock, flags);
+ if (!srq) {
mlx4_warn(dev, "Async event for bogus SRQ %08x\n", srqn);
return;
}
@@ -203,8 +204,9 @@ int mlx4_srq_alloc(struct mlx4_dev *dev, u32 pdn, u32 cqn, u16 xrcd,
if (err)
goto err_radix;
- refcount_set(&srq->refcount, 1);
init_completion(&srq->free);
+ smp_wmb();
+ refcount_set(&srq->refcount, 1);
return 0;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cq.c b/drivers/net/ethernet/mellanox/mlx5/core/cq.c
index 4caa1b6f40ba..d21c14bb3ecf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/cq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/cq.c
@@ -66,8 +66,8 @@ void mlx5_cq_tasklet_cb(struct tasklet_struct *t)
tasklet_schedule(&ctx->task);
}
-static void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq,
- struct mlx5_eqe *eqe)
+void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq,
+ struct mlx5_eqe *eqe)
{
unsigned long flags;
struct mlx5_eq_tasklet *tasklet_ctx = cq->tasklet_ctx.priv;
@@ -84,7 +84,15 @@ static void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq,
}
spin_unlock_irqrestore(&tasklet_ctx->lock, flags);
}
+EXPORT_SYMBOL(mlx5_add_cq_to_tasklet);
+static void mlx5_core_cq_dummy_cb(struct mlx5_core_cq *cq, struct mlx5_eqe *eqe)
+{
+ mlx5_core_err(cq->eq->core.dev,
+ "CQ default completion callback, CQ #%u\n", cq->cqn);
+}
+
+#define MLX5_CQ_INIT_CMD_SN cpu_to_be32(2 << 28)
/* Callers must verify outbox status in case of err */
int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
u32 *in, int inlen, u32 *out, int outlen)
@@ -110,10 +118,19 @@ int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
cq->arm_sn = 0;
cq->eq = eq;
cq->uid = MLX5_GET(create_cq_in, in, uid);
+
+ /* Kernel CQs must set the arm_db address prior to calling
+ * this function, allowing for the proper value to be
+ * initialized. User CQs are responsible for their own
+ * initialization since they do not use the arm_db field.
+ */
+ if (cq->arm_db)
+ *cq->arm_db = MLX5_CQ_INIT_CMD_SN;
+
refcount_set(&cq->refcount, 1);
init_completion(&cq->free);
if (!cq->comp)
- cq->comp = mlx5_add_cq_to_tasklet;
+ cq->comp = mlx5_core_cq_dummy_cb;
/* assuming CQ will be deleted before the EQ */
cq->tasklet_ctx.priv = &eq->tasklet_ctx;
INIT_LIST_HEAD(&cq->tasklet_ctx.list);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h
index 906bf327cdef..d9b03859684a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h
@@ -958,6 +958,18 @@ struct mlx5e_priv {
struct mlx5e_mqprio_rl *mqprio_rl;
};
+static inline u16 mlx5e_stats_nch_read(const struct mlx5e_priv *priv)
+{
+ /* Pairs with smp_store_release in mlx5e_stats_nch_write(). */
+ return smp_load_acquire(&priv->stats_nch);
+}
+
+static inline void mlx5e_stats_nch_write(struct mlx5e_priv *priv, u16 n)
+{
+ /* Pairs with smp_load_acquire in mlx5e_stats_nch_read(). */
+ smp_store_release(&priv->stats_nch, n);
+}
+
struct mlx5e_rx_handlers {
mlx5e_fp_handle_rx_cqe handle_rx_cqe;
mlx5e_fp_handle_rx_cqe handle_rx_cqe_mpwqe;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c
index 1856d211339f..adfb49ff42aa 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c
@@ -33,9 +33,10 @@ mlx5e_hv_vhca_fill_ring_stats(struct mlx5e_priv *priv, int ch,
static void mlx5e_hv_vhca_fill_stats(struct mlx5e_priv *priv, void *data,
int buf_len)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
int ch, i = 0;
- for (ch = 0; ch < priv->stats_nch; ch++) {
+ for (ch = 0; ch < nch; ch++) {
void *buf = data + i;
if (WARN_ON_ONCE(buf +
@@ -50,8 +51,9 @@ static void mlx5e_hv_vhca_fill_stats(struct mlx5e_priv *priv, void *data,
static int mlx5e_hv_vhca_stats_buf_size(struct mlx5e_priv *priv)
{
- return (sizeof(struct mlx5e_hv_vhca_per_ring_stats) *
- priv->stats_nch);
+ u16 nch = mlx5e_stats_nch_read(priv);
+
+ return sizeof(struct mlx5e_hv_vhca_per_ring_stats) * nch;
}
static int mlx5e_hv_vhca_stats_buf_max_size(struct mlx5e_priv *priv)
@@ -106,7 +108,7 @@ static void mlx5e_hv_vhca_stats_control(struct mlx5_hv_vhca_agent *agent,
sagent = &priv->stats_agent;
block->version = MLX5_HV_VHCA_STATS_VERSION;
- block->rings = priv->stats_nch;
+ block->rings = mlx5e_stats_nch_read(priv);
if (!block->command) {
cancel_delayed_work_sync(&priv->stats_agent.work);
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_priv.h b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_priv.h
index 2e42d7c5451e..b474fdbd6eb5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/tc_priv.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/tc_priv.h
@@ -30,6 +30,7 @@ enum {
MLX5E_TC_FLOW_FLAG_TUN_RX = MLX5E_TC_FLOW_BASE + 9,
MLX5E_TC_FLOW_FLAG_FAILED = MLX5E_TC_FLOW_BASE + 10,
MLX5E_TC_FLOW_FLAG_SAMPLE = MLX5E_TC_FLOW_BASE + 11,
+ MLX5E_TC_FLOW_FLAG_PEER = MLX5E_TC_FLOW_BASE + 12,
};
struct mlx5e_tc_flow_parse_attr {
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
index 20507ef2f956..98e9df752c4b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c
@@ -100,8 +100,12 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq,
xdpi.frame.dma_addr = dma_addr;
if (unlikely(!INDIRECT_CALL_2(sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe,
- mlx5e_xmit_xdp_frame, sq, &xdptxd, NULL, 0)))
+ mlx5e_xmit_xdp_frame, sq, &xdptxd, NULL, 0))) {
+ dma_unmap_single(sq->pdev, dma_addr, xdptxd.len,
+ DMA_TO_DEVICE);
+ xdp_return_frame(xdpf);
return false;
+ }
mlx5e_xdpi_fifo_push(&sq->db.xdpi_fifo, &xdpi);
return true;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_rxtx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_rxtx.c
index c4a84f0a3b73..9646abc630ae 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_rxtx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec_rxtx.c
@@ -314,23 +314,27 @@ void mlx5e_ipsec_offload_handle_rx_skb(struct net_device *netdev,
struct mlx5_cqe64 *cqe)
{
u32 ipsec_meta_data = be32_to_cpu(cqe->ft_metadata);
- struct mlx5e_priv *priv;
+ struct mlx5e_priv *priv = netdev_priv(netdev);
+ struct mlx5e_ipsec *ipsec = priv->ipsec;
struct xfrm_offload *xo;
struct xfrm_state *xs;
struct sec_path *sp;
u32 sa_handle;
sa_handle = MLX5_IPSEC_METADATA_HANDLE(ipsec_meta_data);
- priv = netdev_priv(netdev);
sp = secpath_set(skb);
if (unlikely(!sp)) {
- atomic64_inc(&priv->ipsec->sw_stats.ipsec_rx_drop_sp_alloc);
+ atomic64_inc(&ipsec->sw_stats.ipsec_rx_drop_sp_alloc);
return;
}
- xs = mlx5e_ipsec_sadb_rx_lookup(priv->ipsec, sa_handle);
+ xs = mlx5e_ipsec_sadb_rx_lookup(ipsec, sa_handle);
if (unlikely(!xs)) {
- atomic64_inc(&priv->ipsec->sw_stats.ipsec_rx_drop_sadb_miss);
+ atomic64_inc(&ipsec->sw_stats.ipsec_rx_drop_sadb_miss);
+ /* Clear secpath to prevent invalid dereference
+ * in downstream XFRM policy checks.
+ */
+ secpath_reset(skb);
return;
}
@@ -351,6 +355,6 @@ void mlx5e_ipsec_offload_handle_rx_skb(struct net_device *netdev,
xo->status = CRYPTO_INVALID_PACKET_SYNTAX;
break;
default:
- atomic64_inc(&priv->ipsec->sw_stats.ipsec_rx_drop_syndrome);
+ atomic64_inc(&ipsec->sw_stats.ipsec_rx_drop_syndrome);
}
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
index 2d0b57583ea3..fb126c7e40c5 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c
@@ -989,9 +989,11 @@ static int mlx5e_modify_rq_state(struct mlx5e_rq *rq, int curr_state, int next_s
return err;
}
-static int mlx5e_rq_to_ready(struct mlx5e_rq *rq, int curr_state)
+int mlx5e_flush_rq(struct mlx5e_rq *rq, int curr_state)
{
+ struct mlx5_cqwq *cqwq = &rq->cq.wq;
struct net_device *dev = rq->netdev;
+ struct mlx5_cqe64 *cqe;
int err;
err = mlx5e_modify_rq_state(rq, curr_state, MLX5_RQC_STATE_RST);
@@ -999,6 +1001,14 @@ static int mlx5e_rq_to_ready(struct mlx5e_rq *rq, int curr_state)
netdev_err(dev, "Failed to move rq 0x%x to reset\n", rq->rqn);
return err;
}
+
+ mlx5e_free_rx_descs(rq);
+
+ while ((cqe = mlx5_cqwq_get_cqe(cqwq)))
+ mlx5_cqwq_pop(cqwq);
+
+ mlx5_cqwq_update_db_record(cqwq);
+
err = mlx5e_modify_rq_state(rq, MLX5_RQC_STATE_RST, MLX5_RQC_STATE_RDY);
if (err) {
netdev_err(dev, "Failed to move rq 0x%x to ready\n", rq->rqn);
@@ -1008,13 +1018,6 @@ static int mlx5e_rq_to_ready(struct mlx5e_rq *rq, int curr_state)
return 0;
}
-int mlx5e_flush_rq(struct mlx5e_rq *rq, int curr_state)
-{
- mlx5e_free_rx_descs(rq);
-
- return mlx5e_rq_to_ready(rq, curr_state);
-}
-
static int mlx5e_modify_rq_vsd(struct mlx5e_rq *rq, bool vsd)
{
struct mlx5_core_dev *mdev = rq->mdev;
@@ -1859,7 +1862,6 @@ static int mlx5e_alloc_cq_common(struct mlx5e_priv *priv,
mcq->set_ci_db = cq->wq_ctrl.db.db;
mcq->arm_db = cq->wq_ctrl.db.db + 1;
*mcq->set_ci_db = 0;
- *mcq->arm_db = 0;
mcq->vector = param->eq_ix;
mcq->comp = mlx5e_completion_event;
mcq->event = mlx5e_cq_error_event;
@@ -2344,7 +2346,7 @@ static int mlx5e_channel_stats_alloc(struct mlx5e_priv *priv, int ix, int cpu)
GFP_KERNEL, cpu_to_node(cpu));
if (!priv->channel_stats[ix])
return -ENOMEM;
- priv->stats_nch++;
+ mlx5e_stats_nch_write(priv, priv->stats_nch + 1);
return 0;
}
@@ -3593,9 +3595,10 @@ static int mlx5e_setup_tc(struct net_device *dev, enum tc_setup_type type,
void mlx5e_fold_sw_stats64(struct mlx5e_priv *priv, struct rtnl_link_stats64 *s)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
int i;
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i];
struct mlx5e_rq_stats *xskrq_stats = &channel_stats->xskrq;
struct mlx5e_rq_stats *rq_stats = &channel_stats->rq;
@@ -5896,13 +5899,14 @@ static int mlx5e_resume(struct auxiliary_device *adev)
return 0;
}
-static int mlx5e_suspend(struct auxiliary_device *adev, pm_message_t state)
+static int mlx5e_suspend_common(struct auxiliary_device *adev,
+ bool pre_netdev_reg)
{
struct mlx5e_priv *priv = auxiliary_get_drvdata(adev);
struct net_device *netdev = priv->netdev;
struct mlx5_core_dev *mdev = priv->mdev;
- if (!netif_device_present(netdev)) {
+ if (!pre_netdev_reg && !netif_device_present(netdev)) {
if (test_bit(MLX5E_STATE_DESTROYING, &priv->state))
mlx5e_destroy_mdev_resources(mdev);
return -ENODEV;
@@ -5913,6 +5917,11 @@ static int mlx5e_suspend(struct auxiliary_device *adev, pm_message_t state)
return 0;
}
+static int mlx5e_suspend(struct auxiliary_device *adev, pm_message_t state)
+{
+ return mlx5e_suspend_common(adev, false);
+}
+
static int mlx5e_probe(struct auxiliary_device *adev,
const struct auxiliary_device_id *id)
{
@@ -5920,7 +5929,6 @@ static int mlx5e_probe(struct auxiliary_device *adev,
const struct mlx5e_profile *profile = &mlx5e_nic_profile;
struct mlx5_core_dev *mdev = edev->mdev;
struct net_device *netdev;
- pm_message_t state = {};
struct mlx5e_priv *priv;
int err;
@@ -5969,7 +5977,7 @@ static int mlx5e_probe(struct auxiliary_device *adev,
return 0;
err_resume:
- mlx5e_suspend(adev, state);
+ mlx5e_suspend_common(adev, true);
err_profile_cleanup:
profile->cleanup(priv);
err_devlink_cleanup:
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
index 75934d0f144c..346db86c5948 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c
@@ -931,10 +931,12 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq)
} while (!last_wqe);
} while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq)));
- sq->cc = sqcc;
-
mlx5_cqwq_update_db_record(&cq->wq);
+ /* ensure cq space is freed before enabling more cqes */
+ dma_wmb();
+
+ sq->cc = sqcc;
return i;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
index 5db4fd40fe8b..f8f013bd9a0d 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
@@ -517,6 +517,7 @@ static void mlx5e_stats_update_stats_rq_page_pool(struct mlx5e_channel *c)
static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
{
struct mlx5e_sw_stats *s = &priv->stats.sw;
+ u16 nch = mlx5e_stats_nch_read(priv);
int i;
memset(s, 0, sizeof(*s));
@@ -524,7 +525,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw)
for (i = 0; i < priv->channels.num; i++) /* for active channels only */
mlx5e_stats_update_stats_rq_page_pool(priv->channels.c[i]);
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats =
priv->channel_stats[i];
@@ -2328,7 +2329,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(ptp) { return; }
static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS(channels)
{
- int max_nch = priv->stats_nch;
+ int max_nch = mlx5e_stats_nch_read(priv);
return (NUM_RQ_STATS * max_nch) +
(NUM_CH_STATS * max_nch) +
@@ -2341,8 +2342,8 @@ static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS(channels)
static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(channels)
{
+ int max_nch = mlx5e_stats_nch_read(priv);
bool is_xsk = priv->xsk.ever_used;
- int max_nch = priv->stats_nch;
int i, j, tc;
for (i = 0; i < max_nch; i++)
@@ -2383,8 +2384,8 @@ static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(channels)
static MLX5E_DECLARE_STATS_GRP_OP_FILL_STATS(channels)
{
+ int max_nch = mlx5e_stats_nch_read(priv);
bool is_xsk = priv->xsk.ever_used;
- int max_nch = priv->stats_nch;
int i, j, tc;
for (i = 0; i < max_nch; i++)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
index d2f226a09a0c..11df2761c139 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
@@ -2005,7 +2005,8 @@ static void mlx5e_tc_del_flow(struct mlx5e_priv *priv,
struct mlx5e_tc_flow *flow)
{
if (mlx5e_is_eswitch_flow(flow)) {
- mlx5e_tc_del_fdb_peer_flow(flow);
+ if (!flow_flag_test(flow, PEER))
+ mlx5e_tc_del_fdb_peer_flow(flow);
mlx5e_tc_del_fdb_flow(priv, flow);
} else {
mlx5e_tc_del_nic_flow(priv, flow);
@@ -4389,6 +4390,7 @@ static int mlx5e_tc_add_fdb_peer_flow(struct flow_cls_offload *f,
else
in_mdev = priv->mdev;
+ flow_flags |= BIT(MLX5E_TC_FLOW_FLAG_PEER);
parse_attr = flow->attr->parse_attr;
peer_flow = __mlx5e_add_fdb_flow(peer_priv, f, flow_flags,
parse_attr->filter_dev,
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
index bf9869ee7523..67cdf8cb4886 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.c
@@ -516,23 +516,16 @@ static void esw_update_vport_addr_list(struct mlx5_eswitch *esw,
struct mlx5_vport *vport, int list_type)
{
bool is_uc = list_type == MLX5_NVPRT_LIST_TYPE_UC;
- u8 (*mac_list)[ETH_ALEN];
+ u8 (*mac_list)[ETH_ALEN] = NULL;
struct l2addr_node *node;
struct vport_addr *addr;
struct hlist_head *hash;
struct hlist_node *tmp;
- int size;
+ int size = 0;
int err;
int hi;
int i;
- size = is_uc ? MLX5_MAX_UC_PER_VPORT(esw->dev) :
- MLX5_MAX_MC_PER_VPORT(esw->dev);
-
- mac_list = kcalloc(size, ETH_ALEN, GFP_KERNEL);
- if (!mac_list)
- return;
-
hash = is_uc ? vport->uc_list : vport->mc_list;
for_each_l2hash_node(node, tmp, hash, hi) {
@@ -544,7 +537,7 @@ static void esw_update_vport_addr_list(struct mlx5_eswitch *esw,
goto out;
err = mlx5_query_nic_vport_mac_list(esw->dev, vport->vport, list_type,
- mac_list, &size);
+ &mac_list, &size);
if (err)
goto out;
esw_debug(esw->dev, "vport[%d] context update %s list size (%d)\n",
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
index 00d169a11a0a..b410e836169f 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h
@@ -746,7 +746,7 @@ int mlx5_eswitch_offloads_config_single_fdb(struct mlx5_eswitch *master_esw,
struct mlx5_eswitch *slave_esw);
void mlx5_eswitch_offloads_destroy_single_fdb(struct mlx5_eswitch *master_esw,
struct mlx5_eswitch *slave_esw);
-int mlx5_eswitch_reload_reps(struct mlx5_eswitch *esw);
+int mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw);
static inline int mlx5_eswitch_num_vfs(struct mlx5_eswitch *esw)
{
@@ -798,7 +798,7 @@ mlx5_eswitch_offloads_destroy_single_fdb(struct mlx5_eswitch *master_esw,
struct mlx5_eswitch *slave_esw) {}
static inline int
-mlx5_eswitch_reload_reps(struct mlx5_eswitch *esw)
+mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw)
{
return 0;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
index 2a64d0fd2fe5..004f869661f3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch_offloads.c
@@ -2405,6 +2405,16 @@ int esw_offloads_init_reps(struct mlx5_eswitch *esw)
return err;
}
+static int __esw_offloads_load_rep(struct mlx5_eswitch *esw,
+ struct mlx5_eswitch_rep *rep, u8 rep_type)
+{
+ if (atomic_cmpxchg(&rep->rep_data[rep_type].state,
+ REP_REGISTERED, REP_LOADED) == REP_REGISTERED)
+ return esw->offloads.rep_ops[rep_type]->load(esw->dev, rep);
+
+ return 0;
+}
+
static void __esw_offloads_unload_rep(struct mlx5_eswitch *esw,
struct mlx5_eswitch_rep *rep, u8 rep_type)
{
@@ -2453,13 +2463,11 @@ int mlx5_esw_offloads_rep_load(struct mlx5_eswitch *esw, u16 vport_num)
int err;
rep = mlx5_eswitch_get_rep(esw, vport_num);
- for (rep_type = 0; rep_type < NUM_REP_TYPES; rep_type++)
- if (atomic_cmpxchg(&rep->rep_data[rep_type].state,
- REP_REGISTERED, REP_LOADED) == REP_REGISTERED) {
- err = esw->offloads.rep_ops[rep_type]->load(esw->dev, rep);
- if (err)
- goto err_reps;
- }
+ for (rep_type = 0; rep_type < NUM_REP_TYPES; rep_type++) {
+ err = __esw_offloads_load_rep(esw, rep, rep_type);
+ if (err)
+ goto err_reps;
+ }
return 0;
@@ -3102,7 +3110,7 @@ static void esw_destroy_uplink_offloads_acl_tables(struct mlx5_eswitch *esw)
esw_vport_destroy_offloads_acl_tables(esw, vport);
}
-int mlx5_eswitch_reload_reps(struct mlx5_eswitch *esw)
+int mlx5_eswitch_reload_ib_reps(struct mlx5_eswitch *esw)
{
struct mlx5_eswitch_rep *rep;
unsigned long i;
@@ -3115,13 +3123,13 @@ int mlx5_eswitch_reload_reps(struct mlx5_eswitch *esw)
if (atomic_read(&rep->rep_data[REP_ETH].state) != REP_LOADED)
return 0;
- ret = mlx5_esw_offloads_rep_load(esw, MLX5_VPORT_UPLINK);
+ ret = __esw_offloads_load_rep(esw, rep, REP_IB);
if (ret)
return ret;
mlx5_esw_for_each_rep(esw, i, rep) {
if (atomic_read(&rep->rep_data[REP_ETH].state) == REP_LOADED)
- mlx5_esw_offloads_rep_load(esw, rep->vport);
+ __esw_offloads_load_rep(esw, rep, REP_IB);
}
return 0;
@@ -3897,7 +3905,7 @@ static int mlx5_esw_query_vport_vhca_id(struct mlx5_eswitch *esw, u16 vport_num,
if (!query_ctx)
return -ENOMEM;
- err = mlx5_vport_get_other_func_cap(esw->dev, vport_num, query_ctx);
+ err = mlx5_vport_get_other_func_general_cap(esw->dev, vport_num, query_ctx);
if (err)
goto out_free;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
index 12abe991583a..23eadac8c79b 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c
@@ -437,6 +437,14 @@ static int mlx5_fpga_conn_create_cq(struct mlx5_fpga_conn *conn, int cq_size)
cqe->op_own = MLX5_CQE_INVALID << 4 | MLX5_CQE_OWNER_MASK;
}
+ conn->cq.mcq.cqe_sz = 64;
+ conn->cq.mcq.set_ci_db = conn->cq.wq_ctrl.db.db;
+ conn->cq.mcq.arm_db = conn->cq.wq_ctrl.db.db + 1;
+ *conn->cq.mcq.set_ci_db = 0;
+ conn->cq.mcq.vector = 0;
+ conn->cq.mcq.comp = mlx5_fpga_conn_cq_complete;
+ conn->cq.mcq.uar = fdev->conn_res.uar;
+
inlen = MLX5_ST_SZ_BYTES(create_cq_in) +
sizeof(u64) * conn->cq.wq_ctrl.buf.npages;
in = kvzalloc(inlen, GFP_KERNEL);
@@ -468,16 +476,7 @@ static int mlx5_fpga_conn_create_cq(struct mlx5_fpga_conn *conn, int cq_size)
if (err)
goto err_cqwq;
- conn->cq.mcq.cqe_sz = 64;
- conn->cq.mcq.set_ci_db = conn->cq.wq_ctrl.db.db;
- conn->cq.mcq.arm_db = conn->cq.wq_ctrl.db.db + 1;
- *conn->cq.mcq.set_ci_db = 0;
- *conn->cq.mcq.arm_db = 0;
- conn->cq.mcq.vector = 0;
- conn->cq.mcq.comp = mlx5_fpga_conn_cq_complete;
- conn->cq.mcq.uar = fdev->conn_res.uar;
tasklet_setup(&conn->cq.tasklet, mlx5_fpga_conn_cq_tasklet);
-
mlx5_fpga_dbg(fdev, "Created CQ #0x%x\n", conn->cq.mcq.cqn);
goto out;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
index aed4e896179a..b0123d1e24e3 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c
@@ -114,10 +114,11 @@ void mlx5i_cleanup(struct mlx5e_priv *priv)
static void mlx5i_grp_sw_update_stats(struct mlx5e_priv *priv)
{
+ u16 nch = mlx5e_stats_nch_read(priv);
struct rtnl_link_stats64 s = {};
int i, j;
- for (i = 0; i < priv->stats_nch; i++) {
+ for (i = 0; i < nch; i++) {
struct mlx5e_channel_stats *channel_stats;
struct mlx5e_rq_stats *rq_stats;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
index 4b4d76108111..e127cbdf2bbf 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/lag/lag.c
@@ -796,9 +796,9 @@ void mlx5_disable_lag(struct mlx5_lag *ldev)
if (shared_fdb) {
if (!(dev0->priv.flags & MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV))
- mlx5_eswitch_reload_reps(dev0->priv.eswitch);
+ mlx5_eswitch_reload_ib_reps(dev0->priv.eswitch);
if (!(dev1->priv.flags & MLX5_PRIV_FLAGS_DISABLE_ALL_ADEV))
- mlx5_eswitch_reload_reps(dev1->priv.eswitch);
+ mlx5_eswitch_reload_ib_reps(dev1->priv.eswitch);
}
}
@@ -898,17 +898,17 @@ static void mlx5_do_bond(struct mlx5_lag *ldev)
dev0->priv.flags &= ~MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
mlx5_rescan_drivers_locked(dev0);
- err = mlx5_eswitch_reload_reps(dev0->priv.eswitch);
+ err = mlx5_eswitch_reload_ib_reps(dev0->priv.eswitch);
if (!err)
- err = mlx5_eswitch_reload_reps(dev1->priv.eswitch);
+ err = mlx5_eswitch_reload_ib_reps(dev1->priv.eswitch);
if (err) {
dev0->priv.flags |= MLX5_PRIV_FLAGS_DISABLE_IB_ADEV;
mlx5_rescan_drivers_locked(dev0);
mlx5_deactivate_lag(ldev);
mlx5_lag_add_devices(ldev);
- mlx5_eswitch_reload_reps(dev0->priv.eswitch);
- mlx5_eswitch_reload_reps(dev1->priv.eswitch);
+ mlx5_eswitch_reload_ib_reps(dev0->priv.eswitch);
+ mlx5_eswitch_reload_ib_reps(dev1->priv.eswitch);
mlx5_core_err(dev0, "Failed to enable lag\n");
return;
}
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
index 67c1afc1df8f..33b546c0eda6 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
+++ b/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.h
@@ -315,7 +315,8 @@ void mlx5_unload_one_devl_locked(struct mlx5_core_dev *dev, bool suspend);
int mlx5_load_one(struct mlx5_core_dev *dev, bool recovery);
int mlx5_load_one_devl_locked(struct mlx5_core_dev *dev, bool recovery);
-int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 function_id, void *out);
+#define mlx5_vport_get_other_func_general_cap(dev, fid, out) \
+ mlx5_vport_get_other_func_cap(dev, fid, out, MLX5_CAP_GENERAL)
void mlx5_events_work_enqueue(struct mlx5_core_dev *dev, struct work_struct *work);
static inline u32 mlx5_sriov_get_vf_total_msix(struct pci_dev *pdev)
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c b/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
index 10b9dc2aaf06..cf2ab6dacdd7 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/pci_irq.c
@@ -4,6 +4,7 @@
#include <linux/interrupt.h>
#include <linux/notifier.h>
#include <linux/mlx5/driver.h>
+#include <linux/mlx5/vport.h>
#include "mlx5_core.h"
#include "mlx5_irq.h"
#include "pci_irq.h"
@@ -101,7 +102,7 @@ int mlx5_set_msix_vec_count(struct mlx5_core_dev *dev, int function_id,
goto out;
}
- ret = mlx5_vport_get_other_func_cap(dev, function_id, query_cap);
+ ret = mlx5_vport_get_other_func_general_cap(dev, function_id, query_cap);
if (ret)
goto out;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/port.c b/drivers/net/ethernet/mellanox/mlx5/core/port.c
index 28ec31722ec2..4493ee0cb8ea 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/port.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/port.c
@@ -316,7 +316,7 @@ static int mlx5_query_module_id(struct mlx5_core_dev *dev, int module_num,
status);
return -EIO;
}
- ptr = MLX5_ADDR_OF(mcia_reg, out, dword_0);
+ ptr = MLX5_ADDR_OF(mcia_reg, out, dwords);
*module_id = ptr[0];
@@ -401,7 +401,7 @@ static int mlx5_query_mcia(struct mlx5_core_dev *dev,
return -EIO;
}
- ptr = MLX5_ADDR_OF(mcia_reg, out, dword_0);
+ ptr = MLX5_ADDR_OF(mcia_reg, out, dwords);
memcpy(data, ptr, size);
return size;
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c
index ef19a66f5233..5a9fb24856ab 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c
@@ -748,12 +748,6 @@ static int dr_prepare_qp_to_rts(struct mlx5dr_domain *dmn)
return 0;
}
-static void dr_cq_complete(struct mlx5_core_cq *mcq,
- struct mlx5_eqe *eqe)
-{
- pr_err("CQ completion CQ: #%u\n", mcq->cqn);
-}
-
static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev,
struct mlx5_uars_page *uar,
size_t ncqe)
@@ -789,6 +783,13 @@ static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev,
cqe->op_own = MLX5_CQE_INVALID << 4 | MLX5_CQE_OWNER_MASK;
}
+ cq->mcq.cqe_sz = 64;
+ cq->mcq.set_ci_db = cq->wq_ctrl.db.db;
+ cq->mcq.arm_db = cq->wq_ctrl.db.db + 1;
+ *cq->mcq.set_ci_db = 0;
+ cq->mcq.vector = 0;
+ cq->mcq.uar = uar;
+
inlen = MLX5_ST_SZ_BYTES(create_cq_in) +
sizeof(u64) * cq->wq_ctrl.buf.npages;
in = kvzalloc(inlen, GFP_KERNEL);
@@ -813,27 +814,12 @@ static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev,
pas = (__be64 *)MLX5_ADDR_OF(create_cq_in, in, pas);
mlx5_fill_page_frag_array(&cq->wq_ctrl.buf, pas);
- cq->mcq.comp = dr_cq_complete;
-
err = mlx5_core_create_cq(mdev, &cq->mcq, in, inlen, out, sizeof(out));
kvfree(in);
if (err)
goto err_cqwq;
- cq->mcq.cqe_sz = 64;
- cq->mcq.set_ci_db = cq->wq_ctrl.db.db;
- cq->mcq.arm_db = cq->wq_ctrl.db.db + 1;
- *cq->mcq.set_ci_db = 0;
-
- /* set no-zero value, in order to avoid the HW to run db-recovery on
- * CQ that used in polling mode.
- */
- *cq->mcq.arm_db = cpu_to_be32(2 << 28);
-
- cq->mcq.vector = 0;
- cq->mcq.uar = uar;
-
return cq;
err_cqwq:
diff --git a/drivers/net/ethernet/mellanox/mlx5/core/vport.c b/drivers/net/ethernet/mellanox/mlx5/core/vport.c
index 2fbad258520f..c00fa186701a 100644
--- a/drivers/net/ethernet/mellanox/mlx5/core/vport.c
+++ b/drivers/net/ethernet/mellanox/mlx5/core/vport.c
@@ -250,35 +250,63 @@ int mlx5_modify_nic_vport_mtu(struct mlx5_core_dev *mdev, u16 mtu)
}
EXPORT_SYMBOL_GPL(mlx5_modify_nic_vport_mtu);
+static int mlx5_vport_max_mac_list_size(struct mlx5_core_dev *dev, u16 vport,
+ enum mlx5_list_type list_type)
+{
+ void *query_ctx, *hca_caps;
+ int ret = 0;
+
+ if (!vport && !mlx5_core_is_ecpf(dev))
+ return list_type == MLX5_NVPRT_LIST_TYPE_UC ?
+ 1 << MLX5_CAP_GEN(dev, log_max_current_uc_list) :
+ 1 << MLX5_CAP_GEN(dev, log_max_current_mc_list);
+
+ query_ctx = kzalloc(MLX5_ST_SZ_BYTES(query_hca_cap_out), GFP_KERNEL);
+ if (!query_ctx)
+ return -ENOMEM;
+
+ ret = mlx5_vport_get_other_func_general_cap(dev, vport, query_ctx);
+ if (ret)
+ goto out;
+
+ hca_caps = MLX5_ADDR_OF(query_hca_cap_out, query_ctx, capability);
+ ret = list_type == MLX5_NVPRT_LIST_TYPE_UC ?
+ 1 << MLX5_GET(cmd_hca_cap, hca_caps, log_max_current_uc_list) :
+ 1 << MLX5_GET(cmd_hca_cap, hca_caps, log_max_current_mc_list);
+
+out:
+ kfree(query_ctx);
+
+ return ret;
+}
+
int mlx5_query_nic_vport_mac_list(struct mlx5_core_dev *dev,
u16 vport,
enum mlx5_list_type list_type,
- u8 addr_list[][ETH_ALEN],
- int *list_size)
+ u8 (**addr_list)[ETH_ALEN],
+ int *addr_list_size)
{
u32 in[MLX5_ST_SZ_DW(query_nic_vport_context_in)] = {0};
+ int allowed_list_size;
void *nic_vport_ctx;
int max_list_size;
- int req_list_size;
int out_sz;
void *out;
int err;
int i;
- req_list_size = *list_size;
+ if (!addr_list || !addr_list_size)
+ return -EINVAL;
- max_list_size = list_type == MLX5_NVPRT_LIST_TYPE_UC ?
- 1 << MLX5_CAP_GEN(dev, log_max_current_uc_list) :
- 1 << MLX5_CAP_GEN(dev, log_max_current_mc_list);
+ *addr_list = NULL;
+ *addr_list_size = 0;
- if (req_list_size > max_list_size) {
- mlx5_core_warn(dev, "Requested list size (%d) > (%d) max_list_size\n",
- req_list_size, max_list_size);
- req_list_size = max_list_size;
- }
+ max_list_size = mlx5_vport_max_mac_list_size(dev, vport, list_type);
+ if (max_list_size < 0)
+ return max_list_size;
out_sz = MLX5_ST_SZ_BYTES(query_nic_vport_context_out) +
- req_list_size * MLX5_ST_SZ_BYTES(mac_address_layout);
+ max_list_size * MLX5_ST_SZ_BYTES(mac_address_layout);
out = kvzalloc(out_sz, GFP_KERNEL);
if (!out)
@@ -296,16 +324,24 @@ int mlx5_query_nic_vport_mac_list(struct mlx5_core_dev *dev,
nic_vport_ctx = MLX5_ADDR_OF(query_nic_vport_context_out, out,
nic_vport_context);
- req_list_size = MLX5_GET(nic_vport_context, nic_vport_ctx,
- allowed_list_size);
+ allowed_list_size = MLX5_GET(nic_vport_context, nic_vport_ctx,
+ allowed_list_size);
+ if (!allowed_list_size)
+ goto out;
+
+ *addr_list = kcalloc(allowed_list_size, ETH_ALEN, GFP_KERNEL);
+ if (!*addr_list) {
+ err = -ENOMEM;
+ goto out;
+ }
- *list_size = req_list_size;
- for (i = 0; i < req_list_size; i++) {
+ for (i = 0; i < allowed_list_size; i++) {
u8 *mac_addr = MLX5_ADDR_OF(nic_vport_context,
nic_vport_ctx,
current_uc_mac_address[i]) + 2;
- ether_addr_copy(addr_list[i], mac_addr);
+ ether_addr_copy((*addr_list)[i], mac_addr);
}
+ *addr_list_size = allowed_list_size;
out:
kvfree(out);
return err;
@@ -1166,14 +1202,16 @@ u64 mlx5_query_nic_system_image_guid(struct mlx5_core_dev *mdev)
}
EXPORT_SYMBOL_GPL(mlx5_query_nic_system_image_guid);
-int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 function_id, void *out)
+int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 function_id, void *out,
+ u16 opmod)
{
- u16 opmod = (MLX5_CAP_GENERAL << 1) | (HCA_CAP_OPMOD_GET_MAX & 0x01);
u8 in[MLX5_ST_SZ_BYTES(query_hca_cap_in)] = {};
+ opmod = (opmod << 1) | (HCA_CAP_OPMOD_GET_MAX & 0x01);
MLX5_SET(query_hca_cap_in, in, opcode, MLX5_CMD_OP_QUERY_HCA_CAP);
MLX5_SET(query_hca_cap_in, in, op_mod, opmod);
MLX5_SET(query_hca_cap_in, in, function_id, function_id);
MLX5_SET(query_hca_cap_in, in, other_function, true);
return mlx5_cmd_exec_inout(dev, query_hca_cap, in, out);
}
+EXPORT_SYMBOL_GPL(mlx5_vport_get_other_func_cap);
diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_ethtool.c b/drivers/net/ethernet/microchip/lan966x/lan966x_ethtool.c
index 06811c60d598..df10a0b68a08 100644
--- a/drivers/net/ethernet/microchip/lan966x/lan966x_ethtool.c
+++ b/drivers/net/ethernet/microchip/lan966x/lan966x_ethtool.c
@@ -294,7 +294,7 @@ static void lan966x_stats_update(struct lan966x *lan966x)
{
int i, j;
- mutex_lock(&lan966x->stats_lock);
+ spin_lock(&lan966x->stats_lock);
for (i = 0; i < lan966x->num_phys_ports; i++) {
uint idx = i * lan966x->num_stats;
@@ -310,7 +310,7 @@ static void lan966x_stats_update(struct lan966x *lan966x)
}
}
- mutex_unlock(&lan966x->stats_lock);
+ spin_unlock(&lan966x->stats_lock);
}
static int lan966x_get_sset_count(struct net_device *dev, int sset)
@@ -365,7 +365,7 @@ static void lan966x_get_eth_mac_stats(struct net_device *dev,
idx = port->chip_port * lan966x->num_stats;
- mutex_lock(&lan966x->stats_lock);
+ spin_lock(&lan966x->stats_lock);
mac_stats->FramesTransmittedOK =
lan966x->stats[idx + SYS_COUNT_TX_UC] +
@@ -424,7 +424,7 @@ static void lan966x_get_eth_mac_stats(struct net_device *dev,
lan966x->stats[idx + SYS_COUNT_RX_LONG] +
lan966x->stats[idx + SYS_COUNT_RX_PMAC_LONG];
- mutex_unlock(&lan966x->stats_lock);
+ spin_unlock(&lan966x->stats_lock);
}
static const struct ethtool_rmon_hist_range lan966x_rmon_ranges[] = {
@@ -450,7 +450,7 @@ static void lan966x_get_eth_rmon_stats(struct net_device *dev,
idx = port->chip_port * lan966x->num_stats;
- mutex_lock(&lan966x->stats_lock);
+ spin_lock(&lan966x->stats_lock);
rmon_stats->undersize_pkts =
lan966x->stats[idx + SYS_COUNT_RX_SHORT] +
@@ -508,7 +508,7 @@ static void lan966x_get_eth_rmon_stats(struct net_device *dev,
lan966x->stats[idx + SYS_COUNT_TX_SZ_1024_1526] +
lan966x->stats[idx + SYS_COUNT_TX_PMAC_SZ_1024_1526];
- mutex_unlock(&lan966x->stats_lock);
+ spin_unlock(&lan966x->stats_lock);
*ranges = lan966x_rmon_ranges;
}
@@ -614,7 +614,7 @@ void lan966x_stats_get(struct net_device *dev,
idx = port->chip_port * lan966x->num_stats;
- mutex_lock(&lan966x->stats_lock);
+ spin_lock(&lan966x->stats_lock);
stats->rx_bytes = lan966x->stats[idx + SYS_COUNT_RX_OCT] +
lan966x->stats[idx + SYS_COUNT_RX_PMAC_OCT];
@@ -696,7 +696,7 @@ void lan966x_stats_get(struct net_device *dev,
stats->collisions = lan966x->stats[idx + SYS_COUNT_TX_COL];
- mutex_unlock(&lan966x->stats_lock);
+ spin_unlock(&lan966x->stats_lock);
}
int lan966x_stats_init(struct lan966x *lan966x)
@@ -712,7 +712,7 @@ int lan966x_stats_init(struct lan966x *lan966x)
return -ENOMEM;
/* Init stats worker */
- mutex_init(&lan966x->stats_lock);
+ spin_lock_init(&lan966x->stats_lock);
snprintf(queue_name, sizeof(queue_name), "%s-stats",
dev_name(lan966x->dev));
lan966x->stats_queue = create_singlethread_workqueue(queue_name);
diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_main.c b/drivers/net/ethernet/microchip/lan966x/lan966x_main.c
index 8347def40f9d..4a85cd5ebcce 100644
--- a/drivers/net/ethernet/microchip/lan966x/lan966x_main.c
+++ b/drivers/net/ethernet/microchip/lan966x/lan966x_main.c
@@ -1182,7 +1182,6 @@ static int lan966x_probe(struct platform_device *pdev)
cancel_delayed_work_sync(&lan966x->stats_work);
destroy_workqueue(lan966x->stats_queue);
- mutex_destroy(&lan966x->stats_lock);
return err;
}
@@ -1197,7 +1196,6 @@ static int lan966x_remove(struct platform_device *pdev)
cancel_delayed_work_sync(&lan966x->stats_work);
destroy_workqueue(lan966x->stats_queue);
- mutex_destroy(&lan966x->stats_lock);
lan966x_mac_purge_entries(lan966x);
lan966x_mdb_deinit(lan966x);
diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_main.h b/drivers/net/ethernet/microchip/lan966x/lan966x_main.h
index ff5736d2c7a6..3f8b724eac47 100644
--- a/drivers/net/ethernet/microchip/lan966x/lan966x_main.h
+++ b/drivers/net/ethernet/microchip/lan966x/lan966x_main.h
@@ -231,8 +231,8 @@ struct lan966x {
const struct lan966x_stat_layout *stats_layout;
u32 num_stats;
- /* workqueue for reading stats */
- struct mutex stats_lock;
+ /* lock for reading stats */
+ spinlock_t stats_lock;
u64 *stats;
struct delayed_work stats_work;
struct workqueue_struct *stats_queue;
diff --git a/drivers/net/ethernet/microchip/sparx5/sparx5_main.c b/drivers/net/ethernet/microchip/sparx5/sparx5_main.c
index 1ed69e77b895..fbff85b52472 100644
--- a/drivers/net/ethernet/microchip/sparx5/sparx5_main.c
+++ b/drivers/net/ethernet/microchip/sparx5/sparx5_main.c
@@ -828,6 +828,7 @@ static int mchp_sparx5_probe(struct platform_device *pdev)
dev_info(sparx5->dev, "MAC addr was not set, use random MAC\n");
eth_random_addr(sparx5->base_mac);
sparx5->base_mac[5] = 0;
+ sparx5->base_mac_assign_type = NET_ADDR_RANDOM;
}
sparx5->fdma_irq = platform_get_irq_byname(sparx5->pdev, "fdma");
diff --git a/drivers/net/ethernet/microchip/sparx5/sparx5_main.h b/drivers/net/ethernet/microchip/sparx5/sparx5_main.h
index cb3173d2b0e8..6d01585dbedf 100644
--- a/drivers/net/ethernet/microchip/sparx5/sparx5_main.h
+++ b/drivers/net/ethernet/microchip/sparx5/sparx5_main.h
@@ -255,6 +255,7 @@ struct sparx5 {
struct notifier_block switchdev_blocking_nb;
/* Switch state */
u8 base_mac[ETH_ALEN];
+ u8 base_mac_assign_type;
/* Associated bridge device (when bridged) */
struct net_device *hw_bridge_dev;
/* Bridged interfaces */
diff --git a/drivers/net/ethernet/microchip/sparx5/sparx5_netdev.c b/drivers/net/ethernet/microchip/sparx5/sparx5_netdev.c
index d078156581d5..090aa6c14a0f 100644
--- a/drivers/net/ethernet/microchip/sparx5/sparx5_netdev.c
+++ b/drivers/net/ethernet/microchip/sparx5/sparx5_netdev.c
@@ -270,6 +270,7 @@ struct net_device *sparx5_create_netdev(struct sparx5 *sparx5, u32 portno)
ndev->ethtool_ops = &sparx5_ethtool_ops;
eth_hw_addr_gen(ndev, sparx5->base_mac, portno + 1);
+ ndev->addr_assign_type = sparx5->base_mac_assign_type;
return ndev;
}
diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c
index 4afcb202e350..6509d020fcd2 100644
--- a/drivers/net/ethernet/microsoft/mana/mana_en.c
+++ b/drivers/net/ethernet/microsoft/mana/mana_en.c
@@ -1662,6 +1662,10 @@ static int mana_alloc_rx_wqe(struct mana_port_context *apc,
*cq_size += COMP_ENTRY_SIZE;
}
+ /* Reserve an extra slot for Fence completion
+ * event (CQE_RX_OBJECT_FENCE) in case RX CQ is full.
+ */
+ *cq_size += COMP_ENTRY_SIZE;
return 0;
}
@@ -1715,7 +1719,7 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc,
goto out;
rq_size = PAGE_ALIGN(rq_size);
- cq_size = PAGE_ALIGN(cq_size);
+ cq_size = PAGE_ALIGN(roundup_pow_of_two(cq_size));
/* Create RQ */
memset(&spec, 0, sizeof(spec));
@@ -2272,11 +2276,16 @@ void mana_remove(struct gdma_dev *gd, bool suspending)
{
struct gdma_context *gc = gd->gdma_context;
struct mana_context *ac = gd->driver_data;
- struct device *dev = gc->dev;
+ struct device *dev;
struct net_device *ndev;
int err;
int i;
+ if (!gc || !ac)
+ return;
+
+ dev = gc->dev;
+
for (i = 0; i < ac->num_ports; i++) {
ndev = ac->ports[i];
if (!ndev) {
diff --git a/drivers/net/ethernet/mscc/ocelot_stats.c b/drivers/net/ethernet/mscc/ocelot_stats.c
index 6b95262dad90..30b978eaf403 100644
--- a/drivers/net/ethernet/mscc/ocelot_stats.c
+++ b/drivers/net/ethernet/mscc/ocelot_stats.c
@@ -454,6 +454,6 @@ int ocelot_stats_init(struct ocelot *ocelot)
void ocelot_stats_deinit(struct ocelot *ocelot)
{
- cancel_delayed_work(&ocelot->stats_work);
+ cancel_delayed_work_sync(&ocelot->stats_work);
destroy_workqueue(ocelot->stats_queue);
}
diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c
index 09f20c794754..d6cc6288436d 100644
--- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c
+++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c
@@ -6,6 +6,7 @@
#include "qlcnic.h"
#include "qlcnic_hw.h"
+#include <asm/unaligned.h>
struct crb_addr_pair {
u32 addr;
@@ -740,159 +741,212 @@ qlcnic_has_mn(struct qlcnic_adapter *adapter)
return 0;
}
-static
-struct uni_table_desc *qlcnic_get_table_desc(const u8 *unirom, int section)
-{
- u32 i, entries;
- struct uni_table_desc *directory = (struct uni_table_desc *) &unirom[0];
- entries = le32_to_cpu(directory->num_entries);
-
- for (i = 0; i < entries; i++) {
+#define FILEHEADER_SIZE (14 * 4)
+#define QLCNIC_UNI_DIR_TYPE_OFF (8 * sizeof(__le32))
+#define QLCNIC_UNI_DIR_ENTRY_MIN_SIZE (9 * sizeof(__le32))
+#define QLCNIC_UNI_PRODUCT_ENTRY_MIN_SIZE \
+ ((QLCNIC_UNI_FIRMWARE_IDX_OFF + 1) * sizeof(__le32))
+#define QLCNIC_UNI_VERSION_TAIL_SIZE 17
+#define QLCNIC_UNI_BOOTLD_SIZE \
+ (QLCNIC_IMAGE_START - QLCNIC_BOOTLD_START)
+
+struct qlcnic_uni_data {
+ u32 offset;
+ u32 size;
+};
- u32 offs = le32_to_cpu(directory->findex) +
- i * le32_to_cpu(directory->entry_size);
- u32 tab_type = le32_to_cpu(*((__le32 *)&unirom[offs] + 8));
+static bool qlcnic_rom_range_valid(size_t size, size_t offset, size_t len)
+{
+ return offset <= size && len <= size - offset;
+}
- if (tab_type == section)
- return (struct uni_table_desc *) &unirom[offs];
- }
+static bool qlcnic_rom_table_valid(size_t size, u32 offset, u32 entries,
+ u32 entry_size, u32 min_entry_size)
+{
+ if (entry_size < min_entry_size || offset > size)
+ return false;
- return NULL;
+ return entries <= (size - offset) / entry_size;
}
-#define FILEHEADER_SIZE (14 * 4)
-
-static int
-qlcnic_validate_header(struct qlcnic_adapter *adapter)
+static int qlcnic_get_directory(struct qlcnic_adapter *adapter,
+ size_t *offset, u32 *entries, u32 *entry_size)
{
- const u8 *unirom = adapter->fw->data;
- struct uni_table_desc *directory = (struct uni_table_desc *) &unirom[0];
- u32 entries, entry_size, tab_size, fw_file_size;
-
- fw_file_size = adapter->fw->size;
+ const struct firmware *fw = adapter->fw;
+ const u8 *directory = fw->data;
- if (fw_file_size < FILEHEADER_SIZE)
+ if (fw->size < FILEHEADER_SIZE)
return -EINVAL;
- entries = le32_to_cpu(directory->num_entries);
- entry_size = le32_to_cpu(directory->entry_size);
- tab_size = le32_to_cpu(directory->findex) + (entries * entry_size);
+ *offset = get_unaligned_le32(directory +
+ offsetof(struct uni_table_desc, findex));
+ *entries = get_unaligned_le32(directory +
+ offsetof(struct uni_table_desc, num_entries));
+ *entry_size = get_unaligned_le32(directory +
+ offsetof(struct uni_table_desc, entry_size));
- if (fw_file_size < tab_size)
+ if (!qlcnic_rom_table_valid(fw->size, *offset, *entries, *entry_size,
+ QLCNIC_UNI_DIR_ENTRY_MIN_SIZE))
return -EINVAL;
return 0;
}
-static int
-qlcnic_validate_bootld(struct qlcnic_adapter *adapter)
+static int qlcnic_get_table_desc(struct qlcnic_adapter *adapter, int section,
+ size_t *desc_offset)
{
- struct uni_table_desc *tab_desc;
- struct uni_data_desc *descr;
- u32 offs, tab_size, data_size, idx;
const u8 *unirom = adapter->fw->data;
- __le32 temp;
-
- temp = *((__le32 *)&unirom[adapter->file_prd_off] +
- QLCNIC_UNI_BOOTLD_IDX_OFF);
- idx = le32_to_cpu(temp);
- tab_desc = qlcnic_get_table_desc(unirom, QLCNIC_UNI_DIR_SECT_BOOTLD);
-
- if (!tab_desc)
- return -EINVAL;
-
- tab_size = le32_to_cpu(tab_desc->findex) +
- le32_to_cpu(tab_desc->entry_size) * (idx + 1);
-
- if (adapter->fw->size < tab_size)
- return -EINVAL;
+ size_t directory_offset;
+ u32 entries, entry_size;
+ size_t i;
+ int ret;
- offs = le32_to_cpu(tab_desc->findex) +
- le32_to_cpu(tab_desc->entry_size) * idx;
- descr = (struct uni_data_desc *)&unirom[offs];
+ ret = qlcnic_get_directory(adapter, &directory_offset, &entries,
+ &entry_size);
+ if (ret)
+ return ret;
- data_size = le32_to_cpu(descr->findex) + le32_to_cpu(descr->size);
+ for (i = 0; i < entries; i++) {
+ size_t offset = directory_offset + i * entry_size;
+ u32 table_type;
- if (adapter->fw->size < data_size)
- return -EINVAL;
+ table_type = get_unaligned_le32(unirom + offset +
+ QLCNIC_UNI_DIR_TYPE_OFF);
+ if (table_type == section) {
+ *desc_offset = offset;
+ return 0;
+ }
+ }
- return 0;
+ return -ENOENT;
}
static int
-qlcnic_validate_fw(struct qlcnic_adapter *adapter)
+qlcnic_validate_header(struct qlcnic_adapter *adapter)
{
- struct uni_table_desc *tab_desc;
- struct uni_data_desc *descr;
- const u8 *unirom = adapter->fw->data;
- u32 offs, tab_size, data_size, idx;
- __le32 temp;
+ u32 entries, entry_size;
+ size_t offset;
+
+ return qlcnic_get_directory(adapter, &offset, &entries, &entry_size);
+}
- temp = *((__le32 *)&unirom[adapter->file_prd_off] +
- QLCNIC_UNI_FIRMWARE_IDX_OFF);
- idx = le32_to_cpu(temp);
- tab_desc = qlcnic_get_table_desc(unirom, QLCNIC_UNI_DIR_SECT_FW);
+static int qlcnic_get_data_desc(struct qlcnic_adapter *adapter, u32 section,
+ u32 index_offset, struct qlcnic_uni_data *data)
+{
+ size_t table_desc_offset, table_offset, desc_offset;
+ const struct firmware *fw = adapter->fw;
+ const u8 *unirom = fw->data;
+ size_t product_index_offset;
+ u32 entries, entry_size, idx;
+ int ret;
- if (!tab_desc)
+ product_index_offset = adapter->file_prd_off +
+ (size_t)index_offset * sizeof(__le32);
+ if (!qlcnic_rom_range_valid(fw->size, product_index_offset,
+ sizeof(__le32)))
return -EINVAL;
- tab_size = le32_to_cpu(tab_desc->findex) +
- le32_to_cpu(tab_desc->entry_size) * (idx + 1);
+ idx = get_unaligned_le32(unirom + product_index_offset);
+ ret = qlcnic_get_table_desc(adapter, section, &table_desc_offset);
+ if (ret)
+ return ret;
- if (adapter->fw->size < tab_size)
+ table_offset = get_unaligned_le32(unirom + table_desc_offset +
+ offsetof(struct uni_table_desc, findex));
+ entries = get_unaligned_le32(unirom + table_desc_offset +
+ offsetof(struct uni_table_desc, num_entries));
+ entry_size = get_unaligned_le32(unirom + table_desc_offset +
+ offsetof(struct uni_table_desc, entry_size));
+ if (!qlcnic_rom_table_valid(fw->size, table_offset, entries,
+ entry_size, sizeof(struct uni_data_desc)) ||
+ idx >= entries)
return -EINVAL;
- offs = le32_to_cpu(tab_desc->findex) +
- le32_to_cpu(tab_desc->entry_size) * idx;
- descr = (struct uni_data_desc *)&unirom[offs];
- data_size = le32_to_cpu(descr->findex) + le32_to_cpu(descr->size);
+ desc_offset = table_offset + (size_t)idx * entry_size;
+ data->offset = get_unaligned_le32(unirom + desc_offset +
+ offsetof(struct uni_data_desc, findex));
+ data->size = get_unaligned_le32(unirom + desc_offset +
+ offsetof(struct uni_data_desc, size));
- if (adapter->fw->size < data_size)
+ if (!qlcnic_rom_range_valid(fw->size, data->offset, data->size))
return -EINVAL;
return 0;
}
+static int qlcnic_validate_bootld(struct qlcnic_adapter *adapter)
+{
+ struct qlcnic_uni_data data;
+ int ret;
+
+ ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD,
+ QLCNIC_UNI_BOOTLD_IDX_OFF, &data);
+ if (ret)
+ return ret;
+
+ return data.size < QLCNIC_UNI_BOOTLD_SIZE ? -EINVAL : 0;
+}
+
+static int qlcnic_validate_fw(struct qlcnic_adapter *adapter)
+{
+ struct qlcnic_uni_data data;
+ int ret;
+
+ ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW,
+ QLCNIC_UNI_FIRMWARE_IDX_OFF, &data);
+ if (ret)
+ return ret;
+
+ return data.size < QLCNIC_UNI_VERSION_TAIL_SIZE ? -EINVAL : 0;
+}
+
static int
qlcnic_validate_product_offs(struct qlcnic_adapter *adapter)
{
- struct uni_table_desc *ptab_descr;
+ size_t table_desc_offset, table_offset;
const u8 *unirom = adapter->fw->data;
int mn_present = qlcnic_has_mn(adapter);
- u32 entries, entry_size, tab_size, i;
- __le32 temp;
-
- ptab_descr = qlcnic_get_table_desc(unirom,
- QLCNIC_UNI_DIR_SECT_PRODUCT_TBL);
- if (!ptab_descr)
- return -EINVAL;
+ u32 entries, entry_size;
+ size_t i;
+ int ret;
- entries = le32_to_cpu(ptab_descr->num_entries);
- entry_size = le32_to_cpu(ptab_descr->entry_size);
- tab_size = le32_to_cpu(ptab_descr->findex) + (entries * entry_size);
+ ret = qlcnic_get_table_desc(adapter, QLCNIC_UNI_DIR_SECT_PRODUCT_TBL,
+ &table_desc_offset);
+ if (ret)
+ return ret;
- if (adapter->fw->size < tab_size)
+ table_offset = get_unaligned_le32(unirom + table_desc_offset +
+ offsetof(struct uni_table_desc, findex));
+ entries = get_unaligned_le32(unirom + table_desc_offset +
+ offsetof(struct uni_table_desc, num_entries));
+ entry_size = get_unaligned_le32(unirom + table_desc_offset +
+ offsetof(struct uni_table_desc, entry_size));
+ if (!qlcnic_rom_table_valid(adapter->fw->size, table_offset, entries,
+ entry_size,
+ QLCNIC_UNI_PRODUCT_ENTRY_MIN_SIZE))
return -EINVAL;
nomn:
for (i = 0; i < entries; i++) {
-
- u32 flags, file_chiprev, offs;
+ size_t offset = table_offset + i * entry_size;
u8 chiprev = adapter->ahw->revision_id;
+ u32 flags, file_chiprev;
u32 flagbit;
- offs = le32_to_cpu(ptab_descr->findex) +
- i * le32_to_cpu(ptab_descr->entry_size);
- temp = *((__le32 *)&unirom[offs] + QLCNIC_UNI_FLAGS_OFF);
- flags = le32_to_cpu(temp);
- temp = *((__le32 *)&unirom[offs] + QLCNIC_UNI_CHIP_REV_OFF);
- file_chiprev = le32_to_cpu(temp);
+ flags = get_unaligned_le32(unirom + offset +
+ QLCNIC_UNI_FLAGS_OFF * sizeof(__le32));
+ file_chiprev = get_unaligned_le32(unirom + offset +
+ QLCNIC_UNI_CHIP_REV_OFF *
+ sizeof(__le32));
flagbit = mn_present ? 1 : 2;
if ((chiprev == file_chiprev) &&
((1ULL << flagbit) & flags)) {
- adapter->file_prd_off = offs;
+ if (offset > U32_MAX)
+ return -EINVAL;
+
+ adapter->file_prd_off = offset;
return 0;
}
}
@@ -933,93 +987,81 @@ qlcnic_validate_unified_romimage(struct qlcnic_adapter *adapter)
return 0;
}
-static
-struct uni_data_desc *qlcnic_get_data_desc(struct qlcnic_adapter *adapter,
- u32 section, u32 idx_offset)
-{
- const u8 *unirom = adapter->fw->data;
- struct uni_table_desc *tab_desc;
- u32 offs, idx;
- __le32 temp;
-
- temp = *((__le32 *)&unirom[adapter->file_prd_off] + idx_offset);
- idx = le32_to_cpu(temp);
-
- tab_desc = qlcnic_get_table_desc(unirom, section);
-
- if (tab_desc == NULL)
- return NULL;
-
- offs = le32_to_cpu(tab_desc->findex) +
- le32_to_cpu(tab_desc->entry_size) * idx;
-
- return (struct uni_data_desc *)&unirom[offs];
-}
-
-static u8 *
-qlcnic_get_bootld_offs(struct qlcnic_adapter *adapter)
+static int qlcnic_get_bootld_data(struct qlcnic_adapter *adapter,
+ const u8 **bootld)
{
u32 offs = QLCNIC_BOOTLD_START;
- struct uni_data_desc *data_desc;
-
- data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD,
- QLCNIC_UNI_BOOTLD_IDX_OFF);
+ struct qlcnic_uni_data data;
+ int ret;
- if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE)
- offs = le32_to_cpu(data_desc->findex);
+ if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) {
+ ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD,
+ QLCNIC_UNI_BOOTLD_IDX_OFF, &data);
+ if (ret || data.size < QLCNIC_UNI_BOOTLD_SIZE)
+ return -EINVAL;
+ offs = data.offset;
+ } else if (!qlcnic_rom_range_valid(adapter->fw->size, offs,
+ QLCNIC_UNI_BOOTLD_SIZE)) {
+ return -EINVAL;
+ }
- return (u8 *)&adapter->fw->data[offs];
+ *bootld = adapter->fw->data + offs;
+ return 0;
}
-static u8 *
-qlcnic_get_fw_offs(struct qlcnic_adapter *adapter)
+static int qlcnic_get_fw_data(struct qlcnic_adapter *adapter,
+ const u8 **image, u32 *image_size)
{
u32 offs = QLCNIC_IMAGE_START;
- struct uni_data_desc *data_desc;
-
- data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW,
- QLCNIC_UNI_FIRMWARE_IDX_OFF);
- if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE)
- offs = le32_to_cpu(data_desc->findex);
-
- return (u8 *)&adapter->fw->data[offs];
-}
+ struct qlcnic_uni_data data;
+ int ret;
-static u32 qlcnic_get_fw_size(struct qlcnic_adapter *adapter)
-{
- struct uni_data_desc *data_desc;
- const u8 *unirom = adapter->fw->data;
+ if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) {
+ ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW,
+ QLCNIC_UNI_FIRMWARE_IDX_OFF, &data);
+ if (ret)
+ return ret;
+ offs = data.offset;
+ *image_size = data.size;
+ } else {
+ if (!qlcnic_rom_range_valid(adapter->fw->size,
+ QLCNIC_FW_SIZE_OFFSET,
+ sizeof(__le32)))
+ return -EINVAL;
+ *image_size = get_unaligned_le32(adapter->fw->data +
+ QLCNIC_FW_SIZE_OFFSET);
+ }
- data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW,
- QLCNIC_UNI_FIRMWARE_IDX_OFF);
+ if (!qlcnic_rom_range_valid(adapter->fw->size, offs, *image_size))
+ return -EINVAL;
- if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE)
- return le32_to_cpu(data_desc->size);
- else
- return le32_to_cpu(*(__le32 *)&unirom[QLCNIC_FW_SIZE_OFFSET]);
+ *image = adapter->fw->data + offs;
+ return 0;
}
static u32 qlcnic_get_fw_version(struct qlcnic_adapter *adapter)
{
- struct uni_data_desc *fw_data_desc;
+ char ver_str[QLCNIC_UNI_VERSION_TAIL_SIZE + 1];
const struct firmware *fw = adapter->fw;
+ struct qlcnic_uni_data data;
u32 major, minor, sub;
- __le32 version_offset;
- const u8 *ver_str;
int i, ret;
- if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) {
- version_offset = *(__le32 *)&fw->data[QLCNIC_FW_VERSION_OFFSET];
- return le32_to_cpu(version_offset);
- }
+ if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE)
+ return get_unaligned_le32(fw->data + QLCNIC_FW_VERSION_OFFSET);
+
+ ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW,
+ QLCNIC_UNI_FIRMWARE_IDX_OFF, &data);
+ if (ret || data.size < QLCNIC_UNI_VERSION_TAIL_SIZE)
+ return 0;
- fw_data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW,
- QLCNIC_UNI_FIRMWARE_IDX_OFF);
- ver_str = fw->data + le32_to_cpu(fw_data_desc->findex) +
- le32_to_cpu(fw_data_desc->size) - 17;
+ memcpy(ver_str, fw->data + data.offset + data.size -
+ QLCNIC_UNI_VERSION_TAIL_SIZE,
+ QLCNIC_UNI_VERSION_TAIL_SIZE);
+ ver_str[QLCNIC_UNI_VERSION_TAIL_SIZE] = '\0';
for (i = 0; i < 12; i++) {
- if (!strncmp(&ver_str[i], "REV=", 4)) {
+ if (!strncmp(ver_str + i, "REV=", 4)) {
ret = sscanf(&ver_str[i+4], "%u.%u.%u ",
&major, &minor, &sub);
if (ret != 3)
@@ -1034,18 +1076,15 @@ static u32 qlcnic_get_fw_version(struct qlcnic_adapter *adapter)
static u32 qlcnic_get_bios_version(struct qlcnic_adapter *adapter)
{
- const struct firmware *fw = adapter->fw;
u32 bios_ver, prd_off = adapter->file_prd_off;
- u8 *version_offset;
- __le32 temp;
+ const struct firmware *fw = adapter->fw;
- if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) {
- version_offset = (u8 *)&fw->data[QLCNIC_BIOS_VERSION_OFFSET];
- return le32_to_cpu(*(__le32 *)version_offset);
- }
+ if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE)
+ return get_unaligned_le32(fw->data + QLCNIC_BIOS_VERSION_OFFSET);
- temp = *((__le32 *)(&fw->data[prd_off]) + QLCNIC_UNI_BIOS_VERSION_OFF);
- bios_ver = le32_to_cpu(temp);
+ bios_ver = get_unaligned_le32(fw->data + prd_off +
+ QLCNIC_UNI_BIOS_VERSION_OFF *
+ sizeof(__le32));
return (bios_ver << 16) + ((bios_ver >> 8) & 0xff00) + (bios_ver >> 24);
}
@@ -1106,24 +1145,26 @@ static const char *fw_name[] = {
int
qlcnic_load_firmware(struct qlcnic_adapter *adapter)
{
- __le64 *ptr64;
- u32 i, flashaddr, size;
const struct firmware *fw = adapter->fw;
struct pci_dev *pdev = adapter->pdev;
+ const u8 *bootld, *image;
+ u32 i, flashaddr, image_size;
+ int ret;
dev_info(&pdev->dev, "loading firmware from %s\n",
fw_name[adapter->ahw->fw_type]);
if (fw) {
+ u32 words, remainder;
u64 data;
- size = (QLCNIC_IMAGE_START - QLCNIC_BOOTLD_START) / 8;
-
- ptr64 = (__le64 *)qlcnic_get_bootld_offs(adapter);
+ ret = qlcnic_get_bootld_data(adapter, &bootld);
+ if (ret)
+ return ret;
flashaddr = QLCNIC_BOOTLD_START;
- for (i = 0; i < size; i++) {
- data = le64_to_cpu(ptr64[i]);
+ for (i = 0; i < QLCNIC_UNI_BOOTLD_SIZE / sizeof(u64); i++) {
+ data = get_unaligned_le64(bootld + i * sizeof(u64));
if (qlcnic_pci_mem_write_2M(adapter, flashaddr, data))
return -EIO;
@@ -1131,13 +1172,15 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter)
flashaddr += 8;
}
- size = qlcnic_get_fw_size(adapter) / 8;
-
- ptr64 = (__le64 *)qlcnic_get_fw_offs(adapter);
+ ret = qlcnic_get_fw_data(adapter, &image, &image_size);
+ if (ret)
+ return ret;
+ words = image_size / sizeof(u64);
+ remainder = image_size % sizeof(u64);
flashaddr = QLCNIC_IMAGE_START;
- for (i = 0; i < size; i++) {
- data = le64_to_cpu(ptr64[i]);
+ for (i = 0; i < words; i++) {
+ data = get_unaligned_le64(image + i * sizeof(u64));
if (qlcnic_pci_mem_write_2M(adapter,
flashaddr, data))
@@ -1146,9 +1189,11 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter)
flashaddr += 8;
}
- size = qlcnic_get_fw_size(adapter) % 8;
- if (size) {
- data = le64_to_cpu(ptr64[i]);
+ if (remainder) {
+ __le64 tail = 0;
+
+ memcpy(&tail, image + words * sizeof(u64), remainder);
+ data = le64_to_cpu(tail);
if (qlcnic_pci_mem_write_2M(adapter,
flashaddr, data))
@@ -1156,10 +1201,9 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter)
}
} else {
- u64 data;
- u32 hi, lo;
- int ret;
struct qlcnic_flt_entry bootld_entry;
+ u32 hi, lo, size;
+ u64 data;
ret = qlcnic_get_flt_entry(adapter, QLCNIC_BOOTLD_REGION,
&bootld_entry);
@@ -1200,26 +1244,33 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter)
static int
qlcnic_validate_firmware(struct qlcnic_adapter *adapter)
{
- u32 val;
- u32 ver, bios, min_size;
- struct pci_dev *pdev = adapter->pdev;
const struct firmware *fw = adapter->fw;
+ struct pci_dev *pdev = adapter->pdev;
u8 fw_type = adapter->ahw->fw_type;
+ u32 ver, bios, min_size;
+ const u8 *data;
+ u32 data_size;
+ u32 val;
+
+ if (fw_type == QLCNIC_UNIFIED_ROMIMAGE)
+ min_size = QLCNIC_UNI_FW_MIN_SIZE;
+ else
+ min_size = QLCNIC_FW_MIN_SIZE;
+
+ if (fw->size < min_size)
+ return -EINVAL;
if (fw_type == QLCNIC_UNIFIED_ROMIMAGE) {
if (qlcnic_validate_unified_romimage(adapter))
return -EINVAL;
-
- min_size = QLCNIC_UNI_FW_MIN_SIZE;
} else {
- val = le32_to_cpu(*(__le32 *)&fw->data[QLCNIC_FW_MAGIC_OFFSET]);
+ val = get_unaligned_le32(fw->data + QLCNIC_FW_MAGIC_OFFSET);
if (val != QLCNIC_BDINFO_MAGIC)
return -EINVAL;
-
- min_size = QLCNIC_FW_MIN_SIZE;
}
- if (fw->size < min_size)
+ if (qlcnic_get_bootld_data(adapter, &data) ||
+ qlcnic_get_fw_data(adapter, &data, &data_size))
return -EINVAL;
val = qlcnic_get_fw_version(adapter);
diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c
index dcacd691b2ef..40b82f00e6e5 100644
--- a/drivers/net/ethernet/renesas/ravb_main.c
+++ b/drivers/net/ethernet/renesas/ravb_main.c
@@ -3028,10 +3028,12 @@ static int __maybe_unused ravb_suspend(struct device *dev)
netif_device_detach(ndev);
+ rtnl_lock();
if (priv->wol_enabled)
ret = ravb_wol_setup(ndev);
else
ret = ravb_close(ndev);
+ rtnl_unlock();
if (priv->info->ccc_gac)
ravb_ptp_stop(ndev);
@@ -3083,12 +3085,16 @@ static int __maybe_unused ravb_resume(struct device *dev)
ravb_ptp_init(ndev, priv->pdev);
if (netif_running(ndev)) {
+ rtnl_lock();
if (priv->wol_enabled) {
ret = ravb_wol_restore(ndev);
- if (ret)
+ if (ret) {
+ rtnl_unlock();
return ret;
+ }
}
ret = ravb_open(ndev);
+ rtnl_unlock();
if (ret < 0)
return ret;
ravb_set_rx_mode(ndev);
diff --git a/drivers/net/ethernet/renesas/ravb_ptp.c b/drivers/net/ethernet/renesas/ravb_ptp.c
index 97267a0c45a6..3285739dc351 100644
--- a/drivers/net/ethernet/renesas/ravb_ptp.c
+++ b/drivers/net/ethernet/renesas/ravb_ptp.c
@@ -306,16 +306,17 @@ static const struct ptp_clock_info ravb_ptp_info = {
void ravb_ptp_interrupt(struct net_device *ndev)
{
struct ravb_private *priv = netdev_priv(ndev);
+ struct ptp_clock *clock = READ_ONCE(priv->ptp.clock);
u32 gis = ravb_read(ndev, GIS);
gis &= ravb_read(ndev, GIC);
- if (gis & GIS_PTCF) {
+ if ((gis & GIS_PTCF) && clock) {
struct ptp_clock_event event;
event.type = PTP_CLOCK_EXTTS;
event.index = 0;
event.timestamp = ravb_read(ndev, GCPT);
- ptp_clock_event(priv->ptp.clock, &event);
+ ptp_clock_event(clock, &event);
}
if (gis & GIS_PTMF) {
struct ravb_ptp_perout *perout = priv->ptp.perout;
@@ -351,19 +352,39 @@ void ravb_ptp_init(struct net_device *ndev, struct platform_device *pdev)
clock = NULL;
}
- priv->ptp.clock = clock;
+ WRITE_ONCE(priv->ptp.clock, clock);
if (clock)
WRITE_ONCE(priv->ptp.phc_index, ptp_clock_index(clock));
}
-void ravb_ptp_stop(struct net_device *ndev)
+static void ravb_ptp_disable(struct net_device *ndev)
{
- struct ravb_private *priv = netdev_priv(ndev);
-
ravb_write(ndev, 0, GIC);
ravb_write(ndev, 0, GIS);
+}
+
+static void ravb_ptp_sync_irqs(struct net_device *ndev)
+{
+ struct ravb_private *priv = netdev_priv(ndev);
+
+ synchronize_irq(ndev->irq);
+ if (priv->info->err_mgmt_irqs) {
+ synchronize_irq(priv->erra_irq);
+ synchronize_irq(priv->mgmta_irq);
+ }
+}
+
+void ravb_ptp_stop(struct net_device *ndev)
+{
+ struct ravb_private *priv = netdev_priv(ndev);
+ struct ptp_clock *clock;
WRITE_ONCE(priv->ptp.phc_index, -1);
- if (priv->ptp.clock)
- ptp_clock_unregister(priv->ptp.clock);
+ clock = xchg(&priv->ptp.clock, NULL);
+
+ ravb_ptp_disable(ndev);
+ ravb_ptp_sync_irqs(ndev);
+
+ if (clock)
+ ptp_clock_unregister(clock);
}
diff --git a/drivers/net/ethernet/sfc/ef100_netdev.c b/drivers/net/ethernet/sfc/ef100_netdev.c
index c6b9ba6803c8..258e31ef7f2c 100644
--- a/drivers/net/ethernet/sfc/ef100_netdev.c
+++ b/drivers/net/ethernet/sfc/ef100_netdev.c
@@ -354,6 +354,7 @@ int ef100_probe_netdev(struct efx_probe_data *probe_data)
{
struct efx_nic *efx = &probe_data->efx;
struct efx_probe_data **probe_ptr;
+ struct ef100_nic_data *nic_data;
struct net_device *net_dev;
int rc;
@@ -379,8 +380,9 @@ int ef100_probe_netdev(struct efx_probe_data *probe_data)
net_dev->hw_enc_features |= efx->type->offload_features;
net_dev->vlan_features |= NETIF_F_HW_CSUM | NETIF_F_SG |
NETIF_F_HIGHDMA | NETIF_F_ALL_TSO;
- netif_set_tso_max_segs(net_dev,
- ESE_EF100_DP_GZ_TSO_MAX_HDR_NUM_SEGS_DEFAULT);
+ nic_data = efx->nic_data;
+ netif_set_tso_max_size(efx->net_dev, nic_data->tso_max_payload_len);
+ netif_set_tso_max_segs(efx->net_dev, nic_data->tso_max_payload_num_segs);
efx->mdio.dev = net_dev;
rc = efx_ef100_init_datapath_caps(efx);
diff --git a/drivers/net/ethernet/sfc/ef100_nic.c b/drivers/net/ethernet/sfc/ef100_nic.c
index fa1f7039a8e2..5bf264414b9e 100644
--- a/drivers/net/ethernet/sfc/ef100_nic.c
+++ b/drivers/net/ethernet/sfc/ef100_nic.c
@@ -866,8 +866,7 @@ static int ef100_process_design_param(struct efx_nic *efx,
case ESE_EF100_DP_GZ_TSO_MAX_HDR_NUM_SEGS:
/* We always put HDR_NUM_SEGS=1 in our TSO descriptors */
if (!reader->value) {
- netif_err(efx, probe, efx->net_dev,
- "TSO_MAX_HDR_NUM_SEGS < 1\n");
+ pci_err(efx->pci_dev, "TSO_MAX_HDR_NUM_SEGS < 1\n");
return -EOPNOTSUPP;
}
return 0;
@@ -880,32 +879,28 @@ static int ef100_process_design_param(struct efx_nic *efx,
*/
if (!reader->value || reader->value > EFX_MIN_DMAQ_SIZE ||
EFX_MIN_DMAQ_SIZE % (u32)reader->value) {
- netif_err(efx, probe, efx->net_dev,
- "%s size granularity is %llu, can't guarantee safety\n",
- reader->type == ESE_EF100_DP_GZ_RXQ_SIZE_GRANULARITY ? "RXQ" : "TXQ",
- reader->value);
+ pci_err(efx->pci_dev,
+ "%s size granularity is %llu, can't guarantee safety\n",
+ reader->type == ESE_EF100_DP_GZ_RXQ_SIZE_GRANULARITY ? "RXQ" : "TXQ",
+ reader->value);
return -EOPNOTSUPP;
}
return 0;
case ESE_EF100_DP_GZ_TSO_MAX_PAYLOAD_LEN:
nic_data->tso_max_payload_len = min_t(u64, reader->value,
GSO_LEGACY_MAX_SIZE);
- netif_set_tso_max_size(efx->net_dev,
- nic_data->tso_max_payload_len);
return 0;
case ESE_EF100_DP_GZ_TSO_MAX_PAYLOAD_NUM_SEGS:
nic_data->tso_max_payload_num_segs = min_t(u64, reader->value, 0xffff);
- netif_set_tso_max_segs(efx->net_dev,
- nic_data->tso_max_payload_num_segs);
return 0;
case ESE_EF100_DP_GZ_TSO_MAX_NUM_FRAMES:
nic_data->tso_max_frames = min_t(u64, reader->value, 0xffff);
return 0;
case ESE_EF100_DP_GZ_COMPAT:
if (reader->value) {
- netif_err(efx, probe, efx->net_dev,
- "DP_COMPAT has unknown bits %#llx, driver not compatible with this hw\n",
- reader->value);
+ pci_err(efx->pci_dev,
+ "DP_COMPAT has unknown bits %#llx, driver not compatible with this hw\n",
+ reader->value);
return -EOPNOTSUPP;
}
return 0;
@@ -925,10 +920,10 @@ static int ef100_process_design_param(struct efx_nic *efx,
* So the value of this shouldn't matter.
*/
if (reader->value != ESE_EF100_DP_GZ_VI_STRIDES_DEFAULT)
- netif_dbg(efx, probe, efx->net_dev,
- "NIC has other than default VI_STRIDES (mask "
- "%#llx), early probing might use wrong one\n",
- reader->value);
+ pci_dbg(efx->pci_dev,
+ "NIC has other than default VI_STRIDES (mask "
+ "%#llx), early probing might use wrong one\n",
+ reader->value);
return 0;
case ESE_EF100_DP_GZ_RX_MAX_RUNT:
/* Driver doesn't look at L2_STATUS:LEN_ERR bit, so we don't
@@ -940,9 +935,9 @@ static int ef100_process_design_param(struct efx_nic *efx,
/* Host interface says "Drivers should ignore design parameters
* that they do not recognise."
*/
- netif_dbg(efx, probe, efx->net_dev,
- "Ignoring unrecognised design parameter %u\n",
- reader->type);
+ pci_dbg(efx->pci_dev,
+ "Ignoring unrecognised design parameter %u\n",
+ reader->type);
return 0;
}
}
@@ -978,13 +973,13 @@ static int ef100_check_design_params(struct efx_nic *efx)
*/
if (reader.state != EF100_TLV_TYPE) {
if (reader.state == EF100_TLV_TYPE_CONT)
- netif_err(efx, probe, efx->net_dev,
- "truncated design parameter (incomplete type %u)\n",
- reader.type);
+ pci_err(efx->pci_dev,
+ "truncated design parameter (incomplete type %u)\n",
+ reader.type);
else
- netif_err(efx, probe, efx->net_dev,
- "truncated design parameter %u\n",
- reader.type);
+ pci_err(efx->pci_dev,
+ "truncated design parameter %u\n",
+ reader.type);
rc = -EIO;
}
out:
diff --git a/drivers/net/ethernet/stmicro/stmmac/common.h b/drivers/net/ethernet/stmicro/stmmac/common.h
index 671ada52bf66..df4ece196654 100644
--- a/drivers/net/ethernet/stmicro/stmmac/common.h
+++ b/drivers/net/ethernet/stmicro/stmmac/common.h
@@ -58,28 +58,51 @@
#undef FRAME_FILTER_DEBUG
/* #define FRAME_FILTER_DEBUG */
+struct stmmac_q_tx_stats {
+ u64_stats_t tx_bytes;
+ u64_stats_t tx_set_ic_bit;
+ u64_stats_t tx_tso_frames;
+ u64_stats_t tx_tso_nfrags;
+};
+
+struct stmmac_napi_tx_stats {
+ u64_stats_t tx_packets;
+ u64_stats_t tx_pkt_n;
+ u64_stats_t poll;
+ u64_stats_t tx_clean;
+ u64_stats_t tx_set_ic_bit;
+};
+
struct stmmac_txq_stats {
- u64 tx_bytes;
- u64 tx_packets;
- u64 tx_pkt_n;
- u64 tx_normal_irq_n;
- u64 napi_poll;
- u64 tx_clean;
- u64 tx_set_ic_bit;
- u64 tx_tso_frames;
- u64 tx_tso_nfrags;
- struct u64_stats_sync syncp;
+ /* Updates protected by tx queue lock. */
+ struct u64_stats_sync q_syncp;
+ struct stmmac_q_tx_stats q;
+
+ /* Updates protected by NAPI poll logic. */
+ struct u64_stats_sync napi_syncp;
+ struct stmmac_napi_tx_stats napi;
} ____cacheline_aligned_in_smp;
+struct stmmac_napi_rx_stats {
+ u64_stats_t rx_bytes;
+ u64_stats_t rx_packets;
+ u64_stats_t rx_pkt_n;
+ u64_stats_t poll;
+};
+
struct stmmac_rxq_stats {
- u64 rx_bytes;
- u64 rx_packets;
- u64 rx_pkt_n;
- u64 rx_normal_irq_n;
- u64 napi_poll;
- struct u64_stats_sync syncp;
+ /* Updates protected by NAPI poll logic. */
+ struct u64_stats_sync napi_syncp;
+ struct stmmac_napi_rx_stats napi;
} ____cacheline_aligned_in_smp;
+/* Updates on each CPU protected by not allowing nested irqs. */
+struct stmmac_pcpu_stats {
+ struct u64_stats_sync syncp;
+ u64_stats_t rx_normal_irq_n[MTL_MAX_TX_QUEUES];
+ u64_stats_t tx_normal_irq_n[MTL_MAX_RX_QUEUES];
+};
+
/* Extra statistic and debug information exposed by ethtool */
struct stmmac_extra_stats {
/* Transmit errors */
@@ -204,6 +227,7 @@ struct stmmac_extra_stats {
/* per queue statistics */
struct stmmac_txq_stats txq_stats[MTL_MAX_TX_QUEUES];
struct stmmac_rxq_stats rxq_stats[MTL_MAX_RX_QUEUES];
+ struct stmmac_pcpu_stats __percpu *pcpu_stats;
unsigned long rx_dropped;
unsigned long rx_errors;
unsigned long tx_dropped;
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac-sun8i.c b/drivers/net/ethernet/stmicro/stmmac/dwmac-sun8i.c
index ddf6fa00cedd..b09ea8564141 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac-sun8i.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac-sun8i.c
@@ -440,8 +440,7 @@ static int sun8i_dwmac_dma_interrupt(struct stmmac_priv *priv,
struct stmmac_extra_stats *x, u32 chan,
u32 dir)
{
- struct stmmac_rxq_stats *rxq_stats = &priv->xstats.rxq_stats[chan];
- struct stmmac_txq_stats *txq_stats = &priv->xstats.txq_stats[chan];
+ struct stmmac_pcpu_stats *stats = this_cpu_ptr(priv->xstats.pcpu_stats);
int ret = 0;
u32 v;
@@ -454,9 +453,9 @@ static int sun8i_dwmac_dma_interrupt(struct stmmac_priv *priv,
if (v & EMAC_TX_INT) {
ret |= handle_tx;
- u64_stats_update_begin(&txq_stats->syncp);
- txq_stats->tx_normal_irq_n++;
- u64_stats_update_end(&txq_stats->syncp);
+ u64_stats_update_begin(&stats->syncp);
+ u64_stats_inc(&stats->tx_normal_irq_n[chan]);
+ u64_stats_update_end(&stats->syncp);
}
if (v & EMAC_TX_DMA_STOP_INT)
@@ -478,9 +477,9 @@ static int sun8i_dwmac_dma_interrupt(struct stmmac_priv *priv,
if (v & EMAC_RX_INT) {
ret |= handle_rx;
- u64_stats_update_begin(&rxq_stats->syncp);
- rxq_stats->rx_normal_irq_n++;
- u64_stats_update_end(&rxq_stats->syncp);
+ u64_stats_update_begin(&stats->syncp);
+ u64_stats_inc(&stats->rx_normal_irq_n[chan]);
+ u64_stats_update_end(&stats->syncp);
}
if (v & EMAC_RX_BUF_UA_INT)
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c b/drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c
index 9470d3fd2ded..0d185e54eb7e 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac4_lib.c
@@ -171,8 +171,7 @@ int dwmac4_dma_interrupt(struct stmmac_priv *priv, void __iomem *ioaddr,
const struct dwmac4_addrs *dwmac4_addrs = priv->plat->dwmac4_addrs;
u32 intr_status = readl(ioaddr + DMA_CHAN_STATUS(dwmac4_addrs, chan));
u32 intr_en = readl(ioaddr + DMA_CHAN_INTR_ENA(dwmac4_addrs, chan));
- struct stmmac_rxq_stats *rxq_stats = &priv->xstats.rxq_stats[chan];
- struct stmmac_txq_stats *txq_stats = &priv->xstats.txq_stats[chan];
+ struct stmmac_pcpu_stats *stats = this_cpu_ptr(priv->xstats.pcpu_stats);
int ret = 0;
if (dir == DMA_DIR_RX)
@@ -201,15 +200,15 @@ int dwmac4_dma_interrupt(struct stmmac_priv *priv, void __iomem *ioaddr,
}
/* TX/RX NORMAL interrupts */
if (likely(intr_status & DMA_CHAN_STATUS_RI)) {
- u64_stats_update_begin(&rxq_stats->syncp);
- rxq_stats->rx_normal_irq_n++;
- u64_stats_update_end(&rxq_stats->syncp);
+ u64_stats_update_begin(&stats->syncp);
+ u64_stats_inc(&stats->rx_normal_irq_n[chan]);
+ u64_stats_update_end(&stats->syncp);
ret |= handle_rx;
}
if (likely(intr_status & DMA_CHAN_STATUS_TI)) {
- u64_stats_update_begin(&txq_stats->syncp);
- txq_stats->tx_normal_irq_n++;
- u64_stats_update_end(&txq_stats->syncp);
+ u64_stats_update_begin(&stats->syncp);
+ u64_stats_inc(&stats->tx_normal_irq_n[chan]);
+ u64_stats_update_end(&stats->syncp);
ret |= handle_tx;
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c b/drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c
index 7907d62d3437..85e18f9a22f9 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwmac_lib.c
@@ -162,8 +162,7 @@ static void show_rx_process_state(unsigned int status)
int dwmac_dma_interrupt(struct stmmac_priv *priv, void __iomem *ioaddr,
struct stmmac_extra_stats *x, u32 chan, u32 dir)
{
- struct stmmac_rxq_stats *rxq_stats = &priv->xstats.rxq_stats[chan];
- struct stmmac_txq_stats *txq_stats = &priv->xstats.txq_stats[chan];
+ struct stmmac_pcpu_stats *stats = this_cpu_ptr(priv->xstats.pcpu_stats);
int ret = 0;
/* read the status register (CSR5) */
u32 intr_status = readl(ioaddr + DMA_STATUS);
@@ -215,16 +214,16 @@ int dwmac_dma_interrupt(struct stmmac_priv *priv, void __iomem *ioaddr,
u32 value = readl(ioaddr + DMA_INTR_ENA);
/* to schedule NAPI on real RIE event. */
if (likely(value & DMA_INTR_ENA_RIE)) {
- u64_stats_update_begin(&rxq_stats->syncp);
- rxq_stats->rx_normal_irq_n++;
- u64_stats_update_end(&rxq_stats->syncp);
+ u64_stats_update_begin(&stats->syncp);
+ u64_stats_inc(&stats->rx_normal_irq_n[chan]);
+ u64_stats_update_end(&stats->syncp);
ret |= handle_rx;
}
}
if (likely(intr_status & DMA_STATUS_TI)) {
- u64_stats_update_begin(&txq_stats->syncp);
- txq_stats->tx_normal_irq_n++;
- u64_stats_update_end(&txq_stats->syncp);
+ u64_stats_update_begin(&stats->syncp);
+ u64_stats_inc(&stats->tx_normal_irq_n[chan]);
+ u64_stats_update_end(&stats->syncp);
ret |= handle_tx;
}
if (unlikely(intr_status & DMA_STATUS_ERI))
diff --git a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c
index 1e5ffc0d81c3..8d11c6472d8d 100644
--- a/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c
+++ b/drivers/net/ethernet/stmicro/stmmac/dwxgmac2_dma.c
@@ -333,8 +333,7 @@ static int dwxgmac2_dma_interrupt(struct stmmac_priv *priv,
struct stmmac_extra_stats *x, u32 chan,
u32 dir)
{
- struct stmmac_rxq_stats *rxq_stats = &priv->xstats.rxq_stats[chan];
- struct stmmac_txq_stats *txq_stats = &priv->xstats.txq_stats[chan];
+ struct stmmac_pcpu_stats *stats = this_cpu_ptr(priv->xstats.pcpu_stats);
u32 intr_status = readl(ioaddr + XGMAC_DMA_CH_STATUS(chan));
u32 intr_en = readl(ioaddr + XGMAC_DMA_CH_INT_EN(chan));
int ret = 0;
@@ -363,15 +362,15 @@ static int dwxgmac2_dma_interrupt(struct stmmac_priv *priv,
/* TX/RX NORMAL interrupts */
if (likely(intr_status & XGMAC_NIS)) {
if (likely(intr_status & XGMAC_RI)) {
- u64_stats_update_begin(&rxq_stats->syncp);
- rxq_stats->rx_normal_irq_n++;
- u64_stats_update_end(&rxq_stats->syncp);
+ u64_stats_update_begin(&stats->syncp);
+ u64_stats_inc(&stats->rx_normal_irq_n[chan]);
+ u64_stats_update_end(&stats->syncp);
ret |= handle_rx;
}
if (likely(intr_status & (XGMAC_TI | XGMAC_TBU))) {
- u64_stats_update_begin(&txq_stats->syncp);
- txq_stats->tx_normal_irq_n++;
- u64_stats_update_end(&txq_stats->syncp);
+ u64_stats_update_begin(&stats->syncp);
+ u64_stats_inc(&stats->tx_normal_irq_n[chan]);
+ u64_stats_update_end(&stats->syncp);
ret |= handle_tx;
}
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac.h b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
index 0b9cc3fbf5d6..7c7ed92094d1 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac.h
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac.h
@@ -243,6 +243,8 @@ struct stmmac_priv {
struct stmmac_extra_stats xstats ____cacheline_aligned_in_smp;
struct stmmac_safety_stats sstats;
struct plat_stmmacenet_data *plat;
+ /* Protect est parameters */
+ struct mutex est_lock;
struct dma_features dma_cap;
struct stmmac_counters mmc;
int hw_cap_support;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
index 2e67e9abe1b7..4d40e0c8c1b3 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_ethtool.c
@@ -546,44 +546,79 @@ stmmac_set_pauseparam(struct net_device *netdev,
}
}
+static u64 stmmac_get_rx_normal_irq_n(struct stmmac_priv *priv, int q)
+{
+ u64 total;
+ int cpu;
+
+ total = 0;
+ for_each_possible_cpu(cpu) {
+ struct stmmac_pcpu_stats *pcpu;
+ unsigned int start;
+ u64 irq_n;
+
+ pcpu = per_cpu_ptr(priv->xstats.pcpu_stats, cpu);
+ do {
+ start = u64_stats_fetch_begin(&pcpu->syncp);
+ irq_n = u64_stats_read(&pcpu->rx_normal_irq_n[q]);
+ } while (u64_stats_fetch_retry(&pcpu->syncp, start));
+ total += irq_n;
+ }
+ return total;
+}
+
+static u64 stmmac_get_tx_normal_irq_n(struct stmmac_priv *priv, int q)
+{
+ u64 total;
+ int cpu;
+
+ total = 0;
+ for_each_possible_cpu(cpu) {
+ struct stmmac_pcpu_stats *pcpu;
+ unsigned int start;
+ u64 irq_n;
+
+ pcpu = per_cpu_ptr(priv->xstats.pcpu_stats, cpu);
+ do {
+ start = u64_stats_fetch_begin(&pcpu->syncp);
+ irq_n = u64_stats_read(&pcpu->tx_normal_irq_n[q]);
+ } while (u64_stats_fetch_retry(&pcpu->syncp, start));
+ total += irq_n;
+ }
+ return total;
+}
+
static void stmmac_get_per_qstats(struct stmmac_priv *priv, u64 *data)
{
u32 tx_cnt = priv->plat->tx_queues_to_use;
u32 rx_cnt = priv->plat->rx_queues_to_use;
unsigned int start;
- int q, stat;
- char *p;
+ int q;
for (q = 0; q < tx_cnt; q++) {
struct stmmac_txq_stats *txq_stats = &priv->xstats.txq_stats[q];
- struct stmmac_txq_stats snapshot;
+ u64 pkt_n;
do {
- start = u64_stats_fetch_begin(&txq_stats->syncp);
- snapshot = *txq_stats;
- } while (u64_stats_fetch_retry(&txq_stats->syncp, start));
+ start = u64_stats_fetch_begin(&txq_stats->napi_syncp);
+ pkt_n = u64_stats_read(&txq_stats->napi.tx_pkt_n);
+ } while (u64_stats_fetch_retry(&txq_stats->napi_syncp, start));
- p = (char *)&snapshot + offsetof(struct stmmac_txq_stats, tx_pkt_n);
- for (stat = 0; stat < STMMAC_TXQ_STATS; stat++) {
- *data++ = (*(u64 *)p);
- p += sizeof(u64);
- }
+ *data++ = pkt_n;
+ *data++ = stmmac_get_tx_normal_irq_n(priv, q);
}
for (q = 0; q < rx_cnt; q++) {
struct stmmac_rxq_stats *rxq_stats = &priv->xstats.rxq_stats[q];
- struct stmmac_rxq_stats snapshot;
+ u64 pkt_n;
do {
- start = u64_stats_fetch_begin(&rxq_stats->syncp);
- snapshot = *rxq_stats;
- } while (u64_stats_fetch_retry(&rxq_stats->syncp, start));
+ start = u64_stats_fetch_begin(&rxq_stats->napi_syncp);
+ pkt_n = u64_stats_read(&rxq_stats->napi.rx_pkt_n);
+ } while (u64_stats_fetch_retry(&rxq_stats->napi_syncp, start));
- p = (char *)&snapshot + offsetof(struct stmmac_rxq_stats, rx_pkt_n);
- for (stat = 0; stat < STMMAC_RXQ_STATS; stat++) {
- *data++ = (*(u64 *)p);
- p += sizeof(u64);
- }
+ *data++ = pkt_n;
+ *data++ = stmmac_get_rx_normal_irq_n(priv, q);
}
}
@@ -642,39 +677,49 @@ static void stmmac_get_ethtool_stats(struct net_device *dev,
pos = j;
for (i = 0; i < rx_queues_count; i++) {
struct stmmac_rxq_stats *rxq_stats = &priv->xstats.rxq_stats[i];
- struct stmmac_rxq_stats snapshot;
+ struct stmmac_napi_rx_stats snapshot;
+ u64 n_irq;
j = pos;
do {
- start = u64_stats_fetch_begin(&rxq_stats->syncp);
- snapshot = *rxq_stats;
- } while (u64_stats_fetch_retry(&rxq_stats->syncp, start));
-
- data[j++] += snapshot.rx_pkt_n;
- data[j++] += snapshot.rx_normal_irq_n;
- normal_irq_n += snapshot.rx_normal_irq_n;
- napi_poll += snapshot.napi_poll;
+ start = u64_stats_fetch_begin(&rxq_stats->napi_syncp);
+ snapshot = rxq_stats->napi;
+ } while (u64_stats_fetch_retry(&rxq_stats->napi_syncp, start));
+
+ data[j++] += u64_stats_read(&snapshot.rx_pkt_n);
+ n_irq = stmmac_get_rx_normal_irq_n(priv, i);
+ data[j++] += n_irq;
+ normal_irq_n += n_irq;
+ napi_poll += u64_stats_read(&snapshot.poll);
}
pos = j;
for (i = 0; i < tx_queues_count; i++) {
struct stmmac_txq_stats *txq_stats = &priv->xstats.txq_stats[i];
- struct stmmac_txq_stats snapshot;
+ struct stmmac_napi_tx_stats napi_snapshot;
+ struct stmmac_q_tx_stats q_snapshot;
+ u64 n_irq;
j = pos;
do {
- start = u64_stats_fetch_begin(&txq_stats->syncp);
- snapshot = *txq_stats;
- } while (u64_stats_fetch_retry(&txq_stats->syncp, start));
-
- data[j++] += snapshot.tx_pkt_n;
- data[j++] += snapshot.tx_normal_irq_n;
- normal_irq_n += snapshot.tx_normal_irq_n;
- data[j++] += snapshot.tx_clean;
- data[j++] += snapshot.tx_set_ic_bit;
- data[j++] += snapshot.tx_tso_frames;
- data[j++] += snapshot.tx_tso_nfrags;
- napi_poll += snapshot.napi_poll;
+ start = u64_stats_fetch_begin(&txq_stats->q_syncp);
+ q_snapshot = txq_stats->q;
+ } while (u64_stats_fetch_retry(&txq_stats->q_syncp, start));
+ do {
+ start = u64_stats_fetch_begin(&txq_stats->napi_syncp);
+ napi_snapshot = txq_stats->napi;
+ } while (u64_stats_fetch_retry(&txq_stats->napi_syncp, start));
+
+ data[j++] += u64_stats_read(&napi_snapshot.tx_pkt_n);
+ n_irq = stmmac_get_tx_normal_irq_n(priv, i);
+ data[j++] += n_irq;
+ normal_irq_n += n_irq;
+ data[j++] += u64_stats_read(&napi_snapshot.tx_clean);
+ data[j++] += u64_stats_read(&q_snapshot.tx_set_ic_bit) +
+ u64_stats_read(&napi_snapshot.tx_set_ic_bit);
+ data[j++] += u64_stats_read(&q_snapshot.tx_tso_frames);
+ data[j++] += u64_stats_read(&q_snapshot.tx_tso_nfrags);
+ napi_poll += u64_stats_read(&napi_snapshot.poll);
}
normal_irq_n += priv->xstats.rx_early_irq;
data[j++] = normal_irq_n;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
index 5ca615720886..4e427a2f0a7c 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c
@@ -2433,7 +2433,6 @@ static bool stmmac_xdp_xmit_zc(struct stmmac_priv *priv, u32 queue, u32 budget)
struct xdp_desc xdp_desc;
bool work_done = true;
u32 tx_set_ic_bit = 0;
- unsigned long flags;
/* Avoids TX time-out as we are sharing with slow path */
txq_trans_cond_update(nq);
@@ -2506,9 +2505,9 @@ static bool stmmac_xdp_xmit_zc(struct stmmac_priv *priv, u32 queue, u32 budget)
tx_q->cur_tx = STMMAC_NEXT_ENTRY(tx_q->cur_tx, priv->dma_conf.dma_tx_size);
entry = tx_q->cur_tx;
}
- flags = u64_stats_update_begin_irqsave(&txq_stats->syncp);
- txq_stats->tx_set_ic_bit += tx_set_ic_bit;
- u64_stats_update_end_irqrestore(&txq_stats->syncp, flags);
+ u64_stats_update_begin(&txq_stats->napi_syncp);
+ u64_stats_add(&txq_stats->napi.tx_set_ic_bit, tx_set_ic_bit);
+ u64_stats_update_end(&txq_stats->napi_syncp);
if (tx_desc) {
stmmac_flush_tx_descriptors(priv, queue);
@@ -2552,7 +2551,6 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
unsigned int bytes_compl = 0, pkts_compl = 0;
unsigned int entry, xmits = 0, count = 0;
u32 tx_packets = 0, tx_errors = 0;
- unsigned long flags;
__netif_tx_lock_bh(netdev_get_tx_queue(priv->dev, queue));
@@ -2708,11 +2706,11 @@ static int stmmac_tx_clean(struct stmmac_priv *priv, int budget, u32 queue)
if (tx_q->dirty_tx != tx_q->cur_tx)
stmmac_tx_timer_arm(priv, queue);
- flags = u64_stats_update_begin_irqsave(&txq_stats->syncp);
- txq_stats->tx_packets += tx_packets;
- txq_stats->tx_pkt_n += tx_packets;
- txq_stats->tx_clean++;
- u64_stats_update_end_irqrestore(&txq_stats->syncp, flags);
+ u64_stats_update_begin(&txq_stats->napi_syncp);
+ u64_stats_add(&txq_stats->napi.tx_packets, tx_packets);
+ u64_stats_add(&txq_stats->napi.tx_pkt_n, tx_packets);
+ u64_stats_inc(&txq_stats->napi.tx_clean);
+ u64_stats_update_end(&txq_stats->napi_syncp);
priv->xstats.tx_errors += tx_errors;
@@ -4101,7 +4099,6 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
bool has_vlan, set_ic;
dma_addr_t tso_des, des;
u8 proto_hdr_len, hdr;
- unsigned long flags;
u32 pay_len, mss;
int i;
@@ -4278,13 +4275,13 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
netif_tx_stop_queue(netdev_get_tx_queue(priv->dev, queue));
}
- flags = u64_stats_update_begin_irqsave(&txq_stats->syncp);
- txq_stats->tx_bytes += skb->len;
- txq_stats->tx_tso_frames++;
- txq_stats->tx_tso_nfrags += nfrags;
+ u64_stats_update_begin(&txq_stats->q_syncp);
+ u64_stats_add(&txq_stats->q.tx_bytes, skb->len);
+ u64_stats_inc(&txq_stats->q.tx_tso_frames);
+ u64_stats_add(&txq_stats->q.tx_tso_nfrags, nfrags);
if (set_ic)
- txq_stats->tx_set_ic_bit++;
- u64_stats_update_end_irqrestore(&txq_stats->syncp, flags);
+ u64_stats_inc(&txq_stats->q.tx_set_ic_bit);
+ u64_stats_update_end(&txq_stats->q_syncp);
if (priv->sarc_type)
stmmac_set_desc_sarc(priv, first, priv->sarc_type);
@@ -4338,6 +4335,28 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev)
return NETDEV_TX_OK;
}
+/**
+ * stmmac_has_ip_ethertype() - Check if packet has IP ethertype
+ * @skb: socket buffer to check
+ *
+ * Check if a packet has an ethertype that will trigger the IP header checks
+ * and IP/TCP checksum engine of the stmmac core.
+ *
+ * Return: true if the ethertype can trigger the checksum engine, false
+ * otherwise
+ */
+static bool stmmac_has_ip_ethertype(struct sk_buff *skb)
+{
+ int depth = 0;
+ __be16 proto;
+
+ proto = __vlan_get_protocol(skb, eth_header_parse_protocol(skb),
+ &depth);
+
+ return (depth <= ETH_HLEN) &&
+ (proto == htons(ETH_P_IP) || proto == htons(ETH_P_IPV6));
+}
+
/**
* stmmac_xmit - Tx entry point of the driver
* @skb : the socket buffer
@@ -4361,7 +4380,6 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
struct stmmac_tx_queue *tx_q;
bool has_vlan, set_ic;
int entry, first_tx;
- unsigned long flags;
dma_addr_t des;
tx_q = &priv->dma_conf.tx_queue[queue];
@@ -4399,6 +4417,17 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
WARN_ON(tx_q->tx_skbuff[first_entry]);
csum_insertion = (skb->ip_summed == CHECKSUM_PARTIAL);
+ /* Packets that won't trigger the COE e.g. most DSA-tagged packets will
+ * have to be checksummed in software.
+ */
+ if (csum_insertion && !stmmac_has_ip_ethertype(skb)) {
+ if (unlikely(skb_checksum_help(skb)))
+ goto dma_map_err;
+ csum_insertion = !csum_insertion;
+ /* skb_checksum_help() may have linearized the skb */
+ nopaged_len = skb_headlen(skb);
+ nfrags = skb_shinfo(skb)->nr_frags;
+ }
if (likely(priv->extend_desc))
desc = (struct dma_desc *)(tx_q->dma_etx + entry);
@@ -4517,11 +4546,11 @@ static netdev_tx_t stmmac_xmit(struct sk_buff *skb, struct net_device *dev)
netif_tx_stop_queue(netdev_get_tx_queue(priv->dev, queue));
}
- flags = u64_stats_update_begin_irqsave(&txq_stats->syncp);
- txq_stats->tx_bytes += skb->len;
+ u64_stats_update_begin(&txq_stats->q_syncp);
+ u64_stats_add(&txq_stats->q.tx_bytes, skb->len);
if (set_ic)
- txq_stats->tx_set_ic_bit++;
- u64_stats_update_end_irqrestore(&txq_stats->syncp, flags);
+ u64_stats_inc(&txq_stats->q.tx_set_ic_bit);
+ u64_stats_update_end(&txq_stats->q_syncp);
if (priv->sarc_type)
stmmac_set_desc_sarc(priv, first, priv->sarc_type);
@@ -4715,13 +4744,27 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv,
if (!priv->sph)
return 0;
- /* Not last descriptor */
- if (status & rx_not_ls)
+ /* For GMAC4, when split header is enabled, in some rare cases, the
+ * hardware does not fill buf2 of the first descriptor with payload.
+ * Thus we cannot assume buf2 is always fully filled if it is not
+ * the last descriptor. Otherwise, the length of buf2 of the second
+ * descriptor will be calculated wrong and cause an oops.
+ *
+ * If this is the last descriptor, 'plen' is the length of the
+ * received packet that was transferred to system memory.
+ * Otherwise, it is the accumulated number of bytes that have been
+ * transferred for the current packet.
+ *
+ * Thus 'plen - len' always gives the correct length of buf2.
+ */
+
+ /* Not GMAC4 and not last descriptor */
+ if (!priv->plat->has_gmac4 && (status & rx_not_ls))
return priv->dma_conf.dma_buf_sz;
+ /* GMAC4 or last descriptor */
plen = stmmac_get_rx_frame_len(priv, p, coe);
- /* Last descriptor */
return plen - len;
}
@@ -4785,12 +4828,11 @@ static int stmmac_xdp_xmit_xdpf(struct stmmac_priv *priv, int queue,
set_ic = false;
if (set_ic) {
- unsigned long flags;
tx_q->tx_count_frames = 0;
stmmac_set_tx_ic(priv, tx_desc);
- flags = u64_stats_update_begin_irqsave(&txq_stats->syncp);
- txq_stats->tx_set_ic_bit++;
- u64_stats_update_end_irqrestore(&txq_stats->syncp, flags);
+ u64_stats_update_begin(&txq_stats->q_syncp);
+ u64_stats_inc(&txq_stats->q.tx_set_ic_bit);
+ u64_stats_update_end(&txq_stats->q_syncp);
}
stmmac_enable_dma_transmission(priv, priv->ioaddr);
@@ -4950,7 +4992,6 @@ static void stmmac_dispatch_skb_zc(struct stmmac_priv *priv, u32 queue,
unsigned int len = xdp->data_end - xdp->data;
enum pkt_hash_types hash_type;
int coe = priv->hw->rx_csum;
- unsigned long flags;
struct sk_buff *skb;
u32 hash;
@@ -4964,7 +5005,7 @@ static void stmmac_dispatch_skb_zc(struct stmmac_priv *priv, u32 queue,
stmmac_rx_vlan(priv->dev, skb);
skb->protocol = eth_type_trans(skb, priv->dev);
- if (unlikely(!coe))
+ if (unlikely(!coe) || !stmmac_has_ip_ethertype(skb))
skb_checksum_none_assert(skb);
else
skb->ip_summed = CHECKSUM_UNNECESSARY;
@@ -4975,10 +5016,10 @@ static void stmmac_dispatch_skb_zc(struct stmmac_priv *priv, u32 queue,
skb_record_rx_queue(skb, queue);
napi_gro_receive(&ch->rxtx_napi, skb);
- flags = u64_stats_update_begin_irqsave(&rxq_stats->syncp);
- rxq_stats->rx_pkt_n++;
- rxq_stats->rx_bytes += len;
- u64_stats_update_end_irqrestore(&rxq_stats->syncp, flags);
+ u64_stats_update_begin(&rxq_stats->napi_syncp);
+ u64_stats_inc(&rxq_stats->napi.rx_pkt_n);
+ u64_stats_add(&rxq_stats->napi.rx_bytes, len);
+ u64_stats_update_end(&rxq_stats->napi_syncp);
}
static bool stmmac_rx_refill_zc(struct stmmac_priv *priv, u32 queue, u32 budget)
@@ -5050,7 +5091,6 @@ static int stmmac_rx_zc(struct stmmac_priv *priv, int limit, u32 queue)
unsigned int desc_size;
struct bpf_prog *prog;
bool failure = false;
- unsigned long flags;
int xdp_status = 0;
int status = 0;
@@ -5204,9 +5244,9 @@ static int stmmac_rx_zc(struct stmmac_priv *priv, int limit, u32 queue)
stmmac_finalize_xdp_rx(priv, xdp_status);
- flags = u64_stats_update_begin_irqsave(&rxq_stats->syncp);
- rxq_stats->rx_pkt_n += count;
- u64_stats_update_end_irqrestore(&rxq_stats->syncp, flags);
+ u64_stats_update_begin(&rxq_stats->napi_syncp);
+ u64_stats_add(&rxq_stats->napi.rx_pkt_n, count);
+ u64_stats_update_end(&rxq_stats->napi_syncp);
priv->xstats.rx_dropped += rx_dropped;
priv->xstats.rx_errors += rx_errors;
@@ -5244,7 +5284,6 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
unsigned int desc_size;
struct sk_buff *skb = NULL;
struct stmmac_xdp_buff ctx;
- unsigned long flags;
int xdp_status = 0;
int bufsz;
@@ -5466,7 +5505,8 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
stmmac_rx_vlan(priv->dev, skb);
skb->protocol = eth_type_trans(skb, priv->dev);
- if (unlikely(!coe))
+ if (unlikely(!coe) || !stmmac_has_ip_ethertype(skb) ||
+ (status & csum_none))
skb_checksum_none_assert(skb);
else
skb->ip_summed = CHECKSUM_UNNECESSARY;
@@ -5494,11 +5534,11 @@ static int stmmac_rx(struct stmmac_priv *priv, int limit, u32 queue)
stmmac_rx_refill(priv, queue);
- flags = u64_stats_update_begin_irqsave(&rxq_stats->syncp);
- rxq_stats->rx_packets += rx_packets;
- rxq_stats->rx_bytes += rx_bytes;
- rxq_stats->rx_pkt_n += count;
- u64_stats_update_end_irqrestore(&rxq_stats->syncp, flags);
+ u64_stats_update_begin(&rxq_stats->napi_syncp);
+ u64_stats_add(&rxq_stats->napi.rx_packets, rx_packets);
+ u64_stats_add(&rxq_stats->napi.rx_bytes, rx_bytes);
+ u64_stats_add(&rxq_stats->napi.rx_pkt_n, count);
+ u64_stats_update_end(&rxq_stats->napi_syncp);
priv->xstats.rx_dropped += rx_dropped;
priv->xstats.rx_errors += rx_errors;
@@ -5513,13 +5553,12 @@ static int stmmac_napi_poll_rx(struct napi_struct *napi, int budget)
struct stmmac_priv *priv = ch->priv_data;
struct stmmac_rxq_stats *rxq_stats;
u32 chan = ch->index;
- unsigned long flags;
int work_done;
rxq_stats = &priv->xstats.rxq_stats[chan];
- flags = u64_stats_update_begin_irqsave(&rxq_stats->syncp);
- rxq_stats->napi_poll++;
- u64_stats_update_end_irqrestore(&rxq_stats->syncp, flags);
+ u64_stats_update_begin(&rxq_stats->napi_syncp);
+ u64_stats_inc(&rxq_stats->napi.poll);
+ u64_stats_update_end(&rxq_stats->napi_syncp);
work_done = stmmac_rx(priv, budget, chan);
if (work_done < budget && napi_complete_done(napi, work_done)) {
@@ -5540,13 +5579,12 @@ static int stmmac_napi_poll_tx(struct napi_struct *napi, int budget)
struct stmmac_priv *priv = ch->priv_data;
struct stmmac_txq_stats *txq_stats;
u32 chan = ch->index;
- unsigned long flags;
int work_done;
txq_stats = &priv->xstats.txq_stats[chan];
- flags = u64_stats_update_begin_irqsave(&txq_stats->syncp);
- txq_stats->napi_poll++;
- u64_stats_update_end_irqrestore(&txq_stats->syncp, flags);
+ u64_stats_update_begin(&txq_stats->napi_syncp);
+ u64_stats_inc(&txq_stats->napi.poll);
+ u64_stats_update_end(&txq_stats->napi_syncp);
work_done = stmmac_tx_clean(priv, budget, chan);
work_done = min(work_done, budget);
@@ -5571,17 +5609,16 @@ static int stmmac_napi_poll_rxtx(struct napi_struct *napi, int budget)
struct stmmac_rxq_stats *rxq_stats;
struct stmmac_txq_stats *txq_stats;
u32 chan = ch->index;
- unsigned long flags;
rxq_stats = &priv->xstats.rxq_stats[chan];
- flags = u64_stats_update_begin_irqsave(&rxq_stats->syncp);
- rxq_stats->napi_poll++;
- u64_stats_update_end_irqrestore(&rxq_stats->syncp, flags);
+ u64_stats_update_begin(&rxq_stats->napi_syncp);
+ u64_stats_inc(&rxq_stats->napi.poll);
+ u64_stats_update_end(&rxq_stats->napi_syncp);
txq_stats = &priv->xstats.txq_stats[chan];
- flags = u64_stats_update_begin_irqsave(&txq_stats->syncp);
- txq_stats->napi_poll++;
- u64_stats_update_end_irqrestore(&txq_stats->syncp, flags);
+ u64_stats_update_begin(&txq_stats->napi_syncp);
+ u64_stats_inc(&txq_stats->napi.poll);
+ u64_stats_update_end(&txq_stats->napi_syncp);
tx_done = stmmac_tx_clean(priv, budget, chan);
tx_done = min(tx_done, budget);
@@ -6843,10 +6880,13 @@ static void stmmac_get_stats64(struct net_device *dev, struct rtnl_link_stats64
u64 tx_bytes;
do {
- start = u64_stats_fetch_begin(&txq_stats->syncp);
- tx_packets = txq_stats->tx_packets;
- tx_bytes = txq_stats->tx_bytes;
- } while (u64_stats_fetch_retry(&txq_stats->syncp, start));
+ start = u64_stats_fetch_begin(&txq_stats->q_syncp);
+ tx_bytes = u64_stats_read(&txq_stats->q.tx_bytes);
+ } while (u64_stats_fetch_retry(&txq_stats->q_syncp, start));
+ do {
+ start = u64_stats_fetch_begin(&txq_stats->napi_syncp);
+ tx_packets = u64_stats_read(&txq_stats->napi.tx_packets);
+ } while (u64_stats_fetch_retry(&txq_stats->napi_syncp, start));
stats->tx_packets += tx_packets;
stats->tx_bytes += tx_bytes;
@@ -6858,10 +6898,10 @@ static void stmmac_get_stats64(struct net_device *dev, struct rtnl_link_stats64
u64 rx_bytes;
do {
- start = u64_stats_fetch_begin(&rxq_stats->syncp);
- rx_packets = rxq_stats->rx_packets;
- rx_bytes = rxq_stats->rx_bytes;
- } while (u64_stats_fetch_retry(&rxq_stats->syncp, start));
+ start = u64_stats_fetch_begin(&rxq_stats->napi_syncp);
+ rx_packets = u64_stats_read(&rxq_stats->napi.rx_packets);
+ rx_bytes = u64_stats_read(&rxq_stats->napi.rx_bytes);
+ } while (u64_stats_fetch_retry(&rxq_stats->napi_syncp, start));
stats->rx_packets += rx_packets;
stats->rx_bytes += rx_bytes;
@@ -7225,9 +7265,16 @@ int stmmac_dvr_probe(struct device *device,
priv->dev = ndev;
for (i = 0; i < MTL_MAX_RX_QUEUES; i++)
- u64_stats_init(&priv->xstats.rxq_stats[i].syncp);
- for (i = 0; i < MTL_MAX_TX_QUEUES; i++)
- u64_stats_init(&priv->xstats.txq_stats[i].syncp);
+ u64_stats_init(&priv->xstats.rxq_stats[i].napi_syncp);
+ for (i = 0; i < MTL_MAX_TX_QUEUES; i++) {
+ u64_stats_init(&priv->xstats.txq_stats[i].q_syncp);
+ u64_stats_init(&priv->xstats.txq_stats[i].napi_syncp);
+ }
+
+ priv->xstats.pcpu_stats =
+ devm_netdev_alloc_pcpu_stats(device, struct stmmac_pcpu_stats);
+ if (!priv->xstats.pcpu_stats)
+ return -ENOMEM;
stmmac_set_ethtool_ops(ndev);
priv->pause = pause;
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_ptp.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_ptp.c
index d1ad9161de3a..db5b03d48afe 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_ptp.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_ptp.c
@@ -79,11 +79,11 @@ static int stmmac_adjust_time(struct ptp_clock_info *ptp, s64 delta)
/* If EST is enabled, disabled it before adjust ptp time. */
if (priv->plat->est && priv->plat->est->enable) {
est_rst = true;
- mutex_lock(&priv->plat->est->lock);
+ mutex_lock(&priv->est_lock);
priv->plat->est->enable = false;
stmmac_est_configure(priv, priv->ioaddr, priv->plat->est,
priv->plat->clk_ptp_rate);
- mutex_unlock(&priv->plat->est->lock);
+ mutex_unlock(&priv->est_lock);
}
write_lock_irqsave(&priv->ptp_lock, flags);
@@ -96,7 +96,7 @@ static int stmmac_adjust_time(struct ptp_clock_info *ptp, s64 delta)
ktime_t current_time_ns, basetime;
u64 cycle_time;
- mutex_lock(&priv->plat->est->lock);
+ mutex_lock(&priv->est_lock);
priv->ptp_clock_ops.gettime64(&priv->ptp_clock_ops, ¤t_time);
current_time_ns = timespec64_to_ktime(current_time);
time.tv_nsec = priv->plat->est->btr_reserve[0];
@@ -113,7 +113,7 @@ static int stmmac_adjust_time(struct ptp_clock_info *ptp, s64 delta)
priv->plat->est->enable = true;
ret = stmmac_est_configure(priv, priv->ioaddr, priv->plat->est,
priv->plat->clk_ptp_rate);
- mutex_unlock(&priv->plat->est->lock);
+ mutex_unlock(&priv->est_lock);
if (ret)
netdev_err(priv->dev, "failed to configure EST\n");
}
diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c
index e9aac51c1b9b..1ff53ffb05a5 100644
--- a/drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c
+++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_tc.c
@@ -1013,17 +1013,19 @@ static int tc_setup_taprio(struct stmmac_priv *priv,
if (!plat->est)
return -ENOMEM;
- mutex_init(&priv->plat->est->lock);
+ mutex_init(&priv->est_lock);
} else {
+ mutex_lock(&priv->est_lock);
memset(plat->est, 0, sizeof(*plat->est));
+ mutex_unlock(&priv->est_lock);
}
size = qopt->num_entries;
- mutex_lock(&priv->plat->est->lock);
+ mutex_lock(&priv->est_lock);
priv->plat->est->gcl_size = size;
priv->plat->est->enable = qopt->enable;
- mutex_unlock(&priv->plat->est->lock);
+ mutex_unlock(&priv->est_lock);
for (i = 0; i < size; i++) {
s64 delta_ns = qopt->entries[i].interval;
@@ -1054,7 +1056,7 @@ static int tc_setup_taprio(struct stmmac_priv *priv,
priv->plat->est->gcl[i] = delta_ns | (gates << wid);
}
- mutex_lock(&priv->plat->est->lock);
+ mutex_lock(&priv->est_lock);
/* Adjust for real system time */
priv->ptp_clock_ops.gettime64(&priv->ptp_clock_ops, ¤t_time);
current_time_ns = timespec64_to_ktime(current_time);
@@ -1073,7 +1075,7 @@ static int tc_setup_taprio(struct stmmac_priv *priv,
priv->plat->est->ctr[1] = (u32)ctr;
if (fpe && !priv->dma_cap.fpesel) {
- mutex_unlock(&priv->plat->est->lock);
+ mutex_unlock(&priv->est_lock);
return -EOPNOTSUPP;
}
@@ -1084,7 +1086,7 @@ static int tc_setup_taprio(struct stmmac_priv *priv,
ret = stmmac_est_configure(priv, priv->ioaddr, priv->plat->est,
priv->plat->clk_ptp_rate);
- mutex_unlock(&priv->plat->est->lock);
+ mutex_unlock(&priv->est_lock);
if (ret) {
netdev_err(priv->dev, "failed to configure EST\n");
goto disable;
@@ -1101,11 +1103,11 @@ static int tc_setup_taprio(struct stmmac_priv *priv,
disable:
if (priv->plat->est) {
- mutex_lock(&priv->plat->est->lock);
+ mutex_lock(&priv->est_lock);
priv->plat->est->enable = false;
stmmac_est_configure(priv, priv->ioaddr, priv->plat->est,
priv->plat->clk_ptp_rate);
- mutex_unlock(&priv->plat->est->lock);
+ mutex_unlock(&priv->est_lock);
}
priv->plat->fpe_cfg->enable = false;
diff --git a/drivers/net/ethernet/ti/am65-cpsw-nuss.c b/drivers/net/ethernet/ti/am65-cpsw-nuss.c
index 450fd4fb58d1..a4184e9dddf1 100644
--- a/drivers/net/ethernet/ti/am65-cpsw-nuss.c
+++ b/drivers/net/ethernet/ti/am65-cpsw-nuss.c
@@ -514,7 +514,7 @@ static int am65_cpsw_nuss_common_stop(struct am65_cpsw_common *common)
for (i = 0; i < AM65_CPSW_MAX_RX_FLOWS; i++)
k3_udma_glue_reset_rx_chn(common->rx_chns.rx_chn, i,
&common->rx_chns,
- am65_cpsw_nuss_rx_cleanup, !!i);
+ am65_cpsw_nuss_rx_cleanup);
k3_udma_glue_disable_rx_chn(common->rx_chns.rx_chn);
@@ -2583,7 +2583,7 @@ static int am65_cpsw_nuss_register_ndevs(struct am65_cpsw_common *common)
for (i = 0; i < AM65_CPSW_MAX_RX_FLOWS; i++)
k3_udma_glue_reset_rx_chn(rx_chan->rx_chn, i, rx_chan,
- am65_cpsw_nuss_rx_cleanup, !!i);
+ am65_cpsw_nuss_rx_cleanup);
k3_udma_glue_disable_rx_chn(rx_chan->rx_chn);
diff --git a/drivers/net/gtp.c b/drivers/net/gtp.c
index 0817ff58b8da..501dbfa91740 100644
--- a/drivers/net/gtp.c
+++ b/drivers/net/gtp.c
@@ -12,6 +12,7 @@
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <linux/module.h>
+#include <linux/mutex.h>
#include <linux/skbuff.h>
#include <linux/udp.h>
#include <linux/rculist.h>
@@ -92,6 +93,7 @@ struct gtp_net {
};
static u32 gtp_h_initval;
+static DEFINE_MUTEX(gtp_pdp_lock);
static struct genl_family gtp_genl_family;
@@ -129,7 +131,8 @@ static struct pdp_ctx *gtp0_pdp_find(struct gtp_dev *gtp, u64 tid)
head = >p->tid_hash[gtp0_hashfn(tid) % gtp->hash_size];
- hlist_for_each_entry_rcu(pdp, head, hlist_tid) {
+ hlist_for_each_entry_rcu(pdp, head, hlist_tid,
+ lockdep_is_held(>p_pdp_lock)) {
if (pdp->gtp_version == GTP_V0 &&
pdp->u.v0.tid == tid)
return pdp;
@@ -145,7 +148,8 @@ static struct pdp_ctx *gtp1_pdp_find(struct gtp_dev *gtp, u32 tid)
head = >p->tid_hash[gtp1u_hashfn(tid) % gtp->hash_size];
- hlist_for_each_entry_rcu(pdp, head, hlist_tid) {
+ hlist_for_each_entry_rcu(pdp, head, hlist_tid,
+ lockdep_is_held(>p_pdp_lock)) {
if (pdp->gtp_version == GTP_V1 &&
pdp->u.v1.i_tei == tid)
return pdp;
@@ -161,7 +165,8 @@ static struct pdp_ctx *ipv4_pdp_find(struct gtp_dev *gtp, __be32 ms_addr)
head = >p->addr_hash[ipv4_hashfn(ms_addr) % gtp->hash_size];
- hlist_for_each_entry_rcu(pdp, head, hlist_addr) {
+ hlist_for_each_entry_rcu(pdp, head, hlist_addr,
+ lockdep_is_held(>p_pdp_lock)) {
if (pdp->af == AF_INET &&
pdp->ms_addr_ip4.s_addr == ms_addr)
return pdp;
@@ -1120,9 +1125,11 @@ static void gtp_dellink(struct net_device *dev, struct list_head *head)
struct pdp_ctx *pctx;
int i;
+ mutex_lock(>p_pdp_lock);
for (i = 0; i < gtp->hash_size; i++)
hlist_for_each_entry_safe(pctx, next, >p->tid_hash[i], hlist_tid)
pdp_context_delete(pctx);
+ mutex_unlock(>p_pdp_lock);
list_del(>p->list);
unregister_netdevice_queue(dev, head);
@@ -1507,6 +1514,7 @@ static int gtp_genl_new_pdp(struct sk_buff *skb, struct genl_info *info)
goto out_unlock;
}
+ mutex_lock(>p_pdp_lock);
pctx = gtp_pdp_add(gtp, sk, info);
if (IS_ERR(pctx)) {
err = PTR_ERR(pctx);
@@ -1514,6 +1522,7 @@ static int gtp_genl_new_pdp(struct sk_buff *skb, struct genl_info *info)
gtp_tunnel_notify(pctx, GTP_CMD_NEWPDP, GFP_KERNEL);
err = 0;
}
+ mutex_unlock(>p_pdp_lock);
out_unlock:
rtnl_unlock();
@@ -1568,6 +1577,8 @@ static int gtp_genl_del_pdp(struct sk_buff *skb, struct genl_info *info)
if (!info->attrs[GTPA_VERSION])
return -EINVAL;
+ mutex_lock(>p_pdp_lock);
+
rcu_read_lock();
pctx = gtp_find_pdp(sock_net(skb->sk), info->attrs);
@@ -1588,6 +1599,7 @@ static int gtp_genl_del_pdp(struct sk_buff *skb, struct genl_info *info)
out_unlock:
rcu_read_unlock();
+ mutex_unlock(>p_pdp_lock);
return err;
}
diff --git a/drivers/net/hyperv/netvsc.c b/drivers/net/hyperv/netvsc.c
index 077595b42d4a..e17c6dadfef8 100644
--- a/drivers/net/hyperv/netvsc.c
+++ b/drivers/net/hyperv/netvsc.c
@@ -156,15 +156,19 @@ static void free_netvsc_device(struct rcu_head *head)
kfree(nvdev->extension);
- if (nvdev->recv_original_buf)
- vfree(nvdev->recv_original_buf);
- else
- vfree(nvdev->recv_buf);
+ if (!nvdev->recv_buf_gpadl_handle.decrypted) {
+ if (nvdev->recv_original_buf)
+ vfree(nvdev->recv_original_buf);
+ else
+ vfree(nvdev->recv_buf);
+ }
- if (nvdev->send_original_buf)
- vfree(nvdev->send_original_buf);
- else
- vfree(nvdev->send_buf);
+ if (!nvdev->send_buf_gpadl_handle.decrypted) {
+ if (nvdev->send_original_buf)
+ vfree(nvdev->send_original_buf);
+ else
+ vfree(nvdev->send_buf);
+ }
bitmap_free(nvdev->send_section_map);
diff --git a/drivers/net/ieee802154/mac802154_hwsim.c b/drivers/net/ieee802154/mac802154_hwsim.c
index 9b3da61840a8..1b00efc44dcb 100644
--- a/drivers/net/ieee802154/mac802154_hwsim.c
+++ b/drivers/net/ieee802154/mac802154_hwsim.c
@@ -69,6 +69,8 @@ struct hwsim_phy {
struct ieee802154_hw *hw;
u32 idx;
+ /* Serializes phy->pib_updates. */
+ spinlock_t pib_lock;
struct hwsim_pib __rcu *pib;
bool suspended;
@@ -100,8 +102,10 @@ static int hwsim_hw_channel(struct ieee802154_hw *hw, u8 page, u8 channel)
pib->page = page;
pib->channel = channel;
- pib_old = rtnl_dereference(phy->pib);
- rcu_assign_pointer(phy->pib, pib);
+ spin_lock_bh(&phy->pib_lock);
+ pib_old = rcu_replace_pointer(phy->pib, pib,
+ lockdep_is_held(&phy->pib_lock));
+ spin_unlock_bh(&phy->pib_lock);
kfree_rcu(pib_old, rcu);
return 0;
}
@@ -789,6 +793,7 @@ static int hwsim_add_one(struct genl_info *info, struct device *dev,
goto err_pib;
}
+ spin_lock_init(&phy->pib_lock);
pib->channel = 13;
rcu_assign_pointer(phy->pib, pib);
phy->idx = idx;
diff --git a/drivers/net/phy/phy_device.c b/drivers/net/phy/phy_device.c
index 9b0c88c19795..b6161175191a 100644
--- a/drivers/net/phy/phy_device.c
+++ b/drivers/net/phy/phy_device.c
@@ -1399,11 +1399,19 @@ int phy_sfp_probe(struct phy_device *phydev,
ret = sfp_bus_add_upstream(bus, phydev, ops);
sfp_bus_put(bus);
+
+ if (ret)
+ phydev->sfp_bus = NULL;
}
return ret;
}
EXPORT_SYMBOL(phy_sfp_probe);
+static bool phy_drv_supports_irq(struct phy_driver *phydrv)
+{
+ return phydrv->config_intr && phydrv->handle_interrupt;
+}
+
/**
* phy_attach_direct - attach a network device to a given PHY device pointer
* @dev: network device to attach
@@ -1513,6 +1521,9 @@ int phy_attach_direct(struct net_device *dev, struct phy_device *phydev,
phydev->interrupts = PHY_INTERRUPT_DISABLED;
+ if (!phy_drv_supports_irq(phydev->drv) && phy_interrupt_is_valid(phydev))
+ phydev->irq = PHY_POLL;
+
/* Port is set to PORT_TP by default and the actual PHY driver will set
* it to different value depending on the PHY configuration. If we have
* the generic PHY driver we can't figure it out, thus set the old
@@ -2951,11 +2962,6 @@ s32 phy_get_internal_delay(struct phy_device *phydev, struct device *dev,
}
EXPORT_SYMBOL(phy_get_internal_delay);
-static bool phy_drv_supports_irq(struct phy_driver *phydrv)
-{
- return phydrv->config_intr && phydrv->handle_interrupt;
-}
-
/**
* fwnode_mdio_find_device - Given a fwnode, find the mdio_device
* @fwnode: pointer to the mdio_device's fwnode
@@ -3140,6 +3146,9 @@ static int phy_probe(struct device *dev)
return 0;
out:
+ sfp_bus_del_upstream(phydev->sfp_bus);
+ phydev->sfp_bus = NULL;
+
/* Re-assert the reset signal on error */
phy_device_reset(phydev, 1);
diff --git a/drivers/net/tap.c b/drivers/net/tap.c
index 15ab71f5288a..5454b242a8f9 100644
--- a/drivers/net/tap.c
+++ b/drivers/net/tap.c
@@ -638,6 +638,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control,
int err;
struct virtio_net_hdr vnet_hdr = { 0 };
int vnet_hdr_len = 0;
+ int hdr_len = 0;
int copylen = 0;
int depth;
bool zerocopy = false;
@@ -656,13 +657,13 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control,
if (!copy_from_iter_full(&vnet_hdr, sizeof(vnet_hdr), from))
goto err;
iov_iter_advance(from, vnet_hdr_len - sizeof(vnet_hdr));
- if ((vnet_hdr.flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) &&
- tap16_to_cpu(q, vnet_hdr.csum_start) +
- tap16_to_cpu(q, vnet_hdr.csum_offset) + 2 >
- tap16_to_cpu(q, vnet_hdr.hdr_len))
- vnet_hdr.hdr_len = cpu_to_tap16(q,
- tap16_to_cpu(q, vnet_hdr.csum_start) +
- tap16_to_cpu(q, vnet_hdr.csum_offset) + 2);
+ hdr_len = tap16_to_cpu(q, vnet_hdr.hdr_len);
+ if (vnet_hdr.flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) {
+ hdr_len = max(tap16_to_cpu(q, vnet_hdr.csum_start) +
+ tap16_to_cpu(q, vnet_hdr.csum_offset) + 2,
+ hdr_len);
+ vnet_hdr.hdr_len = cpu_to_tap16(q, hdr_len);
+ }
err = -EINVAL;
if (tap16_to_cpu(q, vnet_hdr.hdr_len) > len)
goto err;
@@ -675,12 +676,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control,
if (msg_control && sock_flag(&q->sk, SOCK_ZEROCOPY)) {
struct iov_iter i;
- copylen = vnet_hdr.hdr_len ?
- tap16_to_cpu(q, vnet_hdr.hdr_len) : GOODCOPY_LEN;
- if (copylen > good_linear)
- copylen = good_linear;
- else if (copylen < ETH_HLEN)
- copylen = ETH_HLEN;
+ copylen = clamp(hdr_len ?: GOODCOPY_LEN, ETH_HLEN, good_linear);
linear = copylen;
i = *from;
iov_iter_advance(&i, copylen);
@@ -690,11 +686,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control,
if (!zerocopy) {
copylen = len;
- linear = tap16_to_cpu(q, vnet_hdr.hdr_len);
- if (linear > good_linear)
- linear = good_linear;
- else if (linear < ETH_HLEN)
- linear = ETH_HLEN;
+ linear = clamp(hdr_len, ETH_HLEN, good_linear);
}
skb = tap_alloc_skb(&q->sk, TAP_RESERVE, copylen,
@@ -995,7 +987,7 @@ static long tap_ioctl(struct file *file, unsigned int cmd,
unsigned int __user *up = argp;
unsigned short u;
int __user *sp = argp;
- struct sockaddr sa;
+ struct sockaddr sa = {};
int s;
int ret;
@@ -1179,26 +1171,31 @@ static int tap_get_user_xdp(struct tap_queue *q, struct xdp_buff *xdp)
skb_reset_mac_header(skb);
skb->protocol = eth_hdr(skb)->h_proto;
+ rcu_read_lock();
+ tap = rcu_dereference(q->tap);
+ if (!tap) {
+ kfree_skb(skb);
+ rcu_read_unlock();
+ return 0;
+ }
+ skb->dev = tap->dev;
+
if (vnet_hdr_len) {
err = virtio_net_hdr_to_skb(skb, gso, tap_is_little_endian(q));
- if (err)
+ if (err) {
+ rcu_read_unlock();
goto err_kfree;
+ }
}
+ skb_probe_transport_header(skb);
+
/* Move network header to the right position for VLAN tagged packets */
if (eth_type_vlan(skb->protocol) &&
vlan_get_protocol_and_depth(skb, skb->protocol, &depth) != 0)
skb_set_network_header(skb, depth);
- rcu_read_lock();
- tap = rcu_dereference(q->tap);
- if (tap) {
- skb->dev = tap->dev;
- skb_probe_transport_header(skb);
- dev_queue_xmit(skb);
- } else {
- kfree_skb(skb);
- }
+ dev_queue_xmit(skb);
rcu_read_unlock();
return 0;
diff --git a/drivers/net/team/team.c b/drivers/net/team/team.c
index ea2a7cea3f6a..9d5f7026adec 100644
--- a/drivers/net/team/team.c
+++ b/drivers/net/team/team.c
@@ -535,21 +535,23 @@ static void team_adjust_ops(struct team *team)
if (!team->en_port_count || !team_is_mode_set(team) ||
!team->mode->ops->transmit)
- team->ops.transmit = team_dummy_transmit;
+ WRITE_ONCE(team->ops.transmit, team_dummy_transmit);
else
- team->ops.transmit = team->mode->ops->transmit;
+ WRITE_ONCE(team->ops.transmit, team->mode->ops->transmit);
if (!team->en_port_count || !team_is_mode_set(team) ||
!team->mode->ops->receive)
- team->ops.receive = team_dummy_receive;
+ WRITE_ONCE(team->ops.receive, team_dummy_receive);
else
- team->ops.receive = team->mode->ops->receive;
+ WRITE_ONCE(team->ops.receive, team->mode->ops->receive);
}
/*
- * We can benefit from the fact that it's ensured no port is present
- * at the time of mode change. Therefore no packets are in fly so there's no
- * need to set mode operations in any special way.
+ * team_change_mode() ensures no ports are present during mode change,
+ * but lockless readers can still reach team_xmit(). Avoid touching
+ * transmit/receive -- they are already set to dummies by
+ * team_adjust_ops() since no ports are enabled. synchronize_net()
+ * drains in-flight readers before destroying old mode state.
*/
static int __team_change_mode(struct team *team,
const struct team_mode *new_mode)
@@ -558,9 +560,21 @@ static int __team_change_mode(struct team *team,
if (team_is_mode_set(team)) {
void (*exit_op)(struct team *team) = team->ops.exit;
- /* Clear ops area so no callback is called any longer */
- memset(&team->ops, 0, sizeof(struct team_mode_ops));
- team_adjust_ops(team);
+ /* Clear cold-path ops used only under RTNL. transmit and
+ * receive are already dummies (no ports) so leave them
+ * alone -- overwriting them is the source of the race.
+ */
+ team->ops.init = NULL;
+ team->ops.exit = NULL;
+ team->ops.port_enter = NULL;
+ team->ops.port_leave = NULL;
+ team->ops.port_change_dev_addr = NULL;
+ team->ops.port_tx_disabled = NULL;
+
+ /* Wait for in-flight readers before tearing down mode
+ * state they may reference.
+ */
+ synchronize_net();
if (exit_op)
exit_op(team);
@@ -583,7 +597,12 @@ static int __team_change_mode(struct team *team,
}
team->mode = new_mode;
- memcpy(&team->ops, new_mode->ops, sizeof(struct team_mode_ops));
+ team->ops.init = new_mode->ops->init;
+ team->ops.exit = new_mode->ops->exit;
+ team->ops.port_enter = new_mode->ops->port_enter;
+ team->ops.port_leave = new_mode->ops->port_leave;
+ team->ops.port_change_dev_addr = new_mode->ops->port_change_dev_addr;
+ team->ops.port_tx_disabled = new_mode->ops->port_tx_disabled;
team_adjust_ops(team);
return 0;
@@ -744,7 +763,7 @@ static rx_handler_result_t team_handle_frame(struct sk_buff **pskb)
/* allow exact match delivery for disabled ports */
res = RX_HANDLER_EXACT;
} else {
- res = team->ops.receive(team, port, skb);
+ res = READ_ONCE(team->ops.receive)(team, port, skb);
}
if (res == RX_HANDLER_ANOTHER) {
struct team_pcpu_stats *pcpu_stats;
@@ -803,7 +822,8 @@ static struct list_head *__team_get_qom_list(struct team *team, u16 queue_id)
/*
* note: already called with rcu_read_lock
*/
-static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb)
+static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb,
+ bool *tx_success)
{
struct list_head *qom_list;
struct team_port *port;
@@ -811,11 +831,11 @@ static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb)
if (!team->queue_override_enabled || !skb->queue_mapping)
return false;
qom_list = __team_get_qom_list(team, skb->queue_mapping);
- list_for_each_entry_rcu(port, qom_list, qom_list) {
- if (!team_dev_queue_xmit(team, port, skb))
- return true;
- }
- return false;
+ port = list_first_or_null_rcu(qom_list, struct team_port, qom_list);
+ if (!port)
+ return false;
+ *tx_success = !team_dev_queue_xmit(team, port, skb);
+ return true;
}
static void __team_queue_override_port_del(struct team *team,
@@ -945,8 +965,6 @@ static void team_port_enable(struct team *team,
team_port_index_hash(team, port->index));
team_adjust_ops(team);
team_queue_override_port_add(team, port);
- if (team->ops.port_enabled)
- team->ops.port_enabled(team, port);
team_notify_peers(team);
team_mcast_rejoin(team);
team_lower_state_changed(port);
@@ -971,8 +989,8 @@ static void team_port_disable(struct team *team,
{
if (!team_port_enabled(port))
return;
- if (team->ops.port_disabled)
- team->ops.port_disabled(team, port);
+ if (team->ops.port_tx_disabled)
+ team->ops.port_tx_disabled(team, port);
hlist_del_rcu(&port->hlist);
__reconstruct_port_hlist(team, port->index);
port->index = -1;
@@ -1343,7 +1361,7 @@ static int team_port_add(struct team *team, struct net_device *port_dev,
static void __team_port_change_port_removed(struct team_port *port);
-static int team_port_del(struct team *team, struct net_device *port_dev)
+static int team_port_del(struct team *team, struct net_device *port_dev, bool unregister)
{
struct net_device *dev = team->dev;
struct team_port *port;
@@ -1381,7 +1399,10 @@ static int team_port_del(struct team *team, struct net_device *port_dev)
__team_port_change_port_removed(port);
team_port_set_orig_dev_addr(port);
- dev_set_mtu(port_dev, port->orig.mtu);
+ if (unregister)
+ __dev_set_mtu(port_dev, port->orig.mtu);
+ else
+ dev_set_mtu(port_dev, port->orig.mtu);
kfree_rcu(port, rcu);
netdev_info(dev, "Port device %s removed\n", portname);
__team_compute_features(team);
@@ -1696,7 +1717,7 @@ static void team_uninit(struct net_device *dev)
mutex_lock(&team->lock);
list_for_each_entry_safe(port, tmp, &team->port_list, list)
- team_port_del(team, port->dev);
+ team_port_del(team, port->dev, false);
__team_change_mode(team, NULL); /* cleanup */
__team_options_unregister(team, team_options, ARRAY_SIZE(team_options));
@@ -1742,9 +1763,8 @@ static netdev_tx_t team_xmit(struct sk_buff *skb, struct net_device *dev)
bool tx_success;
unsigned int len = skb->len;
- tx_success = team_queue_override_transmit(team, skb);
- if (!tx_success)
- tx_success = team->ops.transmit(team, skb);
+ if (!team_queue_override_transmit(team, skb, &tx_success))
+ tx_success = READ_ONCE(team->ops.transmit)(team, skb);
if (tx_success) {
struct team_pcpu_stats *pcpu_stats;
@@ -2008,13 +2028,14 @@ static int team_add_slave(struct net_device *dev, struct net_device *port_dev,
return err;
}
-static int team_del_slave(struct net_device *dev, struct net_device *port_dev)
+static int __team_del_slave(struct net_device *dev, struct net_device *port_dev,
+ bool unregister)
{
struct team *team = netdev_priv(dev);
int err;
mutex_lock(&team->lock);
- err = team_port_del(team, port_dev);
+ err = team_port_del(team, port_dev, unregister);
mutex_unlock(&team->lock);
if (err)
@@ -2030,6 +2051,11 @@ static int team_del_slave(struct net_device *dev, struct net_device *port_dev)
return err;
}
+static int team_del_slave(struct net_device *dev, struct net_device *port_dev)
+{
+ return __team_del_slave(dev, port_dev, false);
+}
+
static netdev_features_t team_fix_features(struct net_device *dev,
netdev_features_t features)
{
@@ -2141,6 +2167,76 @@ static const struct ethtool_ops team_ethtool_ops = {
* rt netlink interface
***********************/
+/* For tx path we need a linkup && enabled port and for parse any port
+ * suffices.
+ */
+static struct team_port *team_header_port_get_rcu(struct team *team,
+ bool txable)
+{
+ struct team_port *port;
+
+ list_for_each_entry_rcu(port, &team->port_list, list) {
+ if (!txable || team_port_txable(port))
+ return port;
+ }
+
+ return NULL;
+}
+
+static int team_header_create(struct sk_buff *skb, struct net_device *team_dev,
+ unsigned short type, const void *daddr,
+ const void *saddr, unsigned int len)
+{
+ struct team *team = netdev_priv(team_dev);
+ const struct header_ops *port_ops;
+ struct team_port *port;
+ int ret = 0;
+
+ rcu_read_lock();
+ port = team_header_port_get_rcu(team, true);
+ if (port) {
+ port_ops = READ_ONCE(port->dev->header_ops);
+ if (port_ops && port_ops->create)
+ ret = port_ops->create(skb, port->dev,
+ type, daddr, saddr, len);
+ }
+ rcu_read_unlock();
+ return ret;
+}
+
+static int team_header_parse(const struct sk_buff *skb, unsigned char *haddr)
+{
+ const struct net_device *dev = skb->dev;
+ const struct header_ops *port_ops;
+ struct team_port *port;
+ int ret = 0;
+
+ rcu_read_lock();
+ for (;;) {
+ if (!(dev->priv_flags & IFF_TEAM))
+ break;
+ port = team_header_port_get_rcu(netdev_priv(dev), false);
+ if (!port)
+ break;
+ port_ops = READ_ONCE(port->dev->header_ops);
+ if (!port_ops || !port_ops->parse)
+ break;
+ if (port_ops->parse == team_header_parse) {
+ dev = port->dev;
+ continue;
+ }
+ ret = port_ops->parse(skb, haddr);
+ break;
+ }
+ rcu_read_unlock();
+ return ret;
+}
+
+static const struct header_ops team_header_ops = {
+ .create = team_header_create,
+ .parse = team_header_parse,
+};
+
static void team_setup_by_port(struct net_device *dev,
struct net_device *port_dev)
{
@@ -2149,7 +2245,8 @@ static void team_setup_by_port(struct net_device *dev,
if (port_dev->type == ARPHRD_ETHER)
dev->header_ops = team->header_ops_cache;
else
- dev->header_ops = port_dev->header_ops;
+ dev->header_ops = port_dev->header_ops ?
+ &team_header_ops : NULL;
dev->type = port_dev->type;
dev->hard_header_len = port_dev->hard_header_len;
dev->needed_headroom = port_dev->needed_headroom;
@@ -3058,7 +3155,7 @@ static int team_device_event(struct notifier_block *unused,
!!netif_oper_up(port->dev));
break;
case NETDEV_UNREGISTER:
- team_del_slave(port->team->dev, dev);
+ __team_del_slave(port->team->dev, dev, true);
break;
case NETDEV_FEAT_CHANGE:
if (!port->team->notifier_ctx) {
diff --git a/drivers/net/team/team_mode_loadbalance.c b/drivers/net/team/team_mode_loadbalance.c
index b095a4b4957b..89ee6bee049c 100644
--- a/drivers/net/team/team_mode_loadbalance.c
+++ b/drivers/net/team/team_mode_loadbalance.c
@@ -672,7 +672,7 @@ static void lb_port_leave(struct team *team, struct team_port *port)
free_percpu(lb_port_priv->pcpu_stats);
}
-static void lb_port_disabled(struct team *team, struct team_port *port)
+static void lb_port_tx_disabled(struct team *team, struct team_port *port)
{
lb_tx_hash_to_port_mapping_null_port(team, port);
}
@@ -682,7 +682,7 @@ static const struct team_mode_ops lb_mode_ops = {
.exit = lb_exit,
.port_enter = lb_port_enter,
.port_leave = lb_port_leave,
- .port_disabled = lb_port_disabled,
+ .port_tx_disabled = lb_port_tx_disabled,
.receive = lb_receive,
.transmit = lb_transmit,
};
diff --git a/drivers/net/thunderbolt.c b/drivers/net/thunderbolt.c
index f044a65f5e23..5f0d3f38742b 100644
--- a/drivers/net/thunderbolt.c
+++ b/drivers/net/thunderbolt.c
@@ -606,6 +606,14 @@ static int tbnet_alloc_tx_buffers(struct tbnet *net)
return 0;
}
+static void tbnet_connect_failed(struct tbnet *net)
+{
+ /* Leave login_received set: only the peer can make it true again. */
+ mutex_lock(&net->connection_lock);
+ net->login_sent = false;
+ mutex_unlock(&net->connection_lock);
+}
+
static void tbnet_connected_work(struct work_struct *work)
{
struct tbnet *net = container_of(work, typeof(*net), connected_work);
@@ -625,6 +633,9 @@ static void tbnet_connected_work(struct work_struct *work)
ret = tb_xdomain_alloc_in_hopid(net->xd, net->remote_transmit_path);
if (ret != net->remote_transmit_path) {
netdev_err(net->dev, "failed to allocate Rx HopID\n");
+ if (ret >= 0)
+ tb_xdomain_release_in_hopid(net->xd, ret);
+ tbnet_connect_failed(net);
return;
}
@@ -667,6 +678,7 @@ static void tbnet_connected_work(struct work_struct *work)
tb_ring_stop(net->rx_ring.ring);
tb_ring_stop(net->tx_ring.ring);
tb_xdomain_release_in_hopid(net->xd, net->remote_transmit_path);
+ tbnet_connect_failed(net);
}
static void tbnet_login_work(struct work_struct *work)
diff --git a/drivers/net/usb/ax88179_178a.c b/drivers/net/usb/ax88179_178a.c
index 619093d33373..63cf49bd3d03 100644
--- a/drivers/net/usb/ax88179_178a.c
+++ b/drivers/net/usb/ax88179_178a.c
@@ -1461,11 +1461,11 @@ static int ax88179_rx_fixup(struct usbnet *dev, struct sk_buff *skb)
return 1;
}
- ax_skb = netdev_alloc_skb_ip_align(dev->net, pkt_len);
+ ax_skb = netdev_alloc_skb_ip_align(dev->net, pkt_len - 2);
if (!ax_skb)
return 0;
- skb_put(ax_skb, pkt_len);
- memcpy(ax_skb->data, skb->data + 2, pkt_len);
+ skb_put(ax_skb, pkt_len - 2);
+ memcpy(ax_skb->data, skb->data + 2, pkt_len - 2);
ax88179_rx_checksum(ax_skb, pkt_hdr);
usbnet_skb_return(dev, ax_skb);
diff --git a/drivers/net/usb/qmi_wwan.c b/drivers/net/usb/qmi_wwan.c
index 8ed86bdefb8d..69dba0af5888 100644
--- a/drivers/net/usb/qmi_wwan.c
+++ b/drivers/net/usb/qmi_wwan.c
@@ -1092,6 +1092,7 @@ static const struct usb_device_id products[] = {
{QMI_MATCH_FF_FF_FF(0x2c7c, 0x0122)}, /* Quectel RG650V */
{QMI_MATCH_FF_FF_FF(0x2c7c, 0x0125)}, /* Quectel EC25, EC20 R2.0 Mini PCIe */
{QMI_MATCH_FF_FF_FF(0x2c7c, 0x0306)}, /* Quectel EP06/EG06/EM06 */
+ {QMI_MATCH_FF_FF_FF(0x2c7c, 0x030b)}, /* Quectel EM060K/EG120K-EA */
{QMI_MATCH_FF_FF_FF(0x2c7c, 0x0512)}, /* Quectel EG12/EM12 */
{QMI_MATCH_FF_FF_FF(0x2c7c, 0x0620)}, /* Quectel EM160R-GL */
{QMI_MATCH_FF_FF_FF(0x2c7c, 0x0800)}, /* Quectel RM500Q-GL */
diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c
index 591a2875dfe2..893212fad6b4 100644
--- a/drivers/net/vxlan/vxlan_core.c
+++ b/drivers/net/vxlan/vxlan_core.c
@@ -1522,6 +1522,10 @@ static bool vxlan_snoop(struct net_device *dev,
if (likely(f)) {
struct vxlan_rdst *rdst = first_remote_rcu(f);
+ /* Don't override an fdb with nexthop with a learnt entry */
+ if (rcu_access_pointer(f->nh))
+ return true;
+
if (likely(vxlan_addr_equal(&rdst->remote_ip, src_ip) &&
rdst->remote_ifindex == ifindex))
return false;
@@ -1530,10 +1534,6 @@ static bool vxlan_snoop(struct net_device *dev,
if (f->state & (NUD_PERMANENT | NUD_NOARP))
return true;
- /* Don't override an fdb with nexthop with a learnt entry */
- if (rcu_access_pointer(f->nh))
- return true;
-
if (net_ratelimit())
netdev_info(dev,
"%pM migrated from %pIS to %pIS\n",
@@ -1943,6 +1943,8 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
n = neigh_lookup(&arp_tbl, &tip, dev);
if (n) {
+ struct vxlan_rdst *rdst = NULL;
+ u8 ha[ETH_ALEN] __aligned(2);
struct vxlan_fdb *f;
struct sk_buff *reply;
@@ -1951,15 +1953,19 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
goto out;
}
- f = vxlan_find_mac(vxlan, n->ha, vni);
- if (f && vxlan_addr_any(&(first_remote_rcu(f)->remote_ip))) {
+ neigh_ha_snapshot(ha, n, n->dev);
+
+ f = vxlan_find_mac(vxlan, ha, vni);
+ if (f)
+ rdst = first_remote_rcu(f);
+ if (rdst && vxlan_addr_any(&rdst->remote_ip)) {
/* bridge-local neighbor */
neigh_release(n);
goto out;
}
reply = arp_create(ARPOP_REPLY, ETH_P_ARP, sip, dev, tip, sha,
- n->ha, sha);
+ ha, sha);
neigh_release(n);
@@ -1992,7 +1998,8 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
#if IS_ENABLED(CONFIG_IPV6)
static struct sk_buff *vxlan_na_create(struct sk_buff *request,
- struct neighbour *n, bool isrouter)
+ struct neighbour *n, u8 *ha,
+ bool isrouter)
{
struct net_device *dev = request->dev;
struct sk_buff *reply;
@@ -2000,13 +2007,15 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
struct ipv6hdr *pip6;
u8 *daddr;
int na_olen = 8; /* opt hdr + ETH_ALEN for target */
+ int headroom;
int ns_olen;
int i, len;
if (dev == NULL || !pskb_may_pull(request, request->len))
return NULL;
- len = LL_RESERVED_SPACE(dev) + sizeof(struct ipv6hdr) +
+ headroom = LL_RESERVED_SPACE(dev);
+ len = headroom + sizeof(struct ipv6hdr) +
sizeof(*na) + na_olen + dev->needed_tailroom;
reply = alloc_skb(len, GFP_ATOMIC);
if (reply == NULL)
@@ -2014,7 +2023,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
reply->protocol = htons(ETH_P_IPV6);
reply->dev = dev;
- skb_reserve(reply, LL_RESERVED_SPACE(request->dev));
+ skb_reserve(reply, headroom);
skb_push(reply, sizeof(struct ethhdr));
skb_reset_mac_header(reply);
@@ -2038,7 +2047,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
/* Ethernet header */
ether_addr_copy(eth_hdr(reply)->h_dest, daddr);
- ether_addr_copy(eth_hdr(reply)->h_source, n->ha);
+ ether_addr_copy(eth_hdr(reply)->h_source, ha);
eth_hdr(reply)->h_proto = htons(ETH_P_IPV6);
reply->protocol = htons(ETH_P_IPV6);
@@ -2067,7 +2076,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request,
na->icmph.icmp6_override = 1;
na->icmph.icmp6_solicited = 1;
na->target = ns->target;
- ether_addr_copy(&na->opt[2], n->ha);
+ ether_addr_copy(&na->opt[2], ha);
na->opt[0] = ND_OPT_TARGET_LL_ADDR;
na->opt[1] = na_olen >> 3;
@@ -2109,6 +2118,8 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
n = neigh_lookup(ipv6_stub->nd_tbl, &msg->target, dev);
if (n) {
+ struct vxlan_rdst *rdst = NULL;
+ u8 ha[ETH_ALEN] __aligned(2);
struct vxlan_fdb *f;
struct sk_buff *reply;
@@ -2117,14 +2128,17 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni)
goto out;
}
- f = vxlan_find_mac(vxlan, n->ha, vni);
- if (f && vxlan_addr_any(&(first_remote_rcu(f)->remote_ip))) {
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = vxlan_find_mac(vxlan, ha, vni);
+ if (f)
+ rdst = first_remote_rcu(f);
+ if (rdst && vxlan_addr_any(&rdst->remote_ip)) {
/* bridge-local neighbor */
neigh_release(n);
goto out;
}
- reply = vxlan_na_create(skb, n,
+ reply = vxlan_na_create(skb, n, ha,
!!(f ? f->flags & NTF_ROUTER : 0));
neigh_release(n);
diff --git a/drivers/net/vxlan/vxlan_private.h b/drivers/net/vxlan/vxlan_private.h
index b0e8c6d29057..9e1cc18ca81a 100644
--- a/drivers/net/vxlan/vxlan_private.h
+++ b/drivers/net/vxlan/vxlan_private.h
@@ -56,9 +56,7 @@ static inline struct hlist_head *vs_head(struct net *net, __be16 port)
return &vn->sock_list[hash_32(ntohs(port), PORT_HASH_BITS)];
}
-/* First remote destination for a forwarding entry.
- * Guaranteed to be non-NULL because remotes are never deleted.
- */
+/* First remote destination for a forwarding entry. */
static inline struct vxlan_rdst *first_remote_rcu(struct vxlan_fdb *fdb)
{
if (rcu_access_pointer(fdb->nh))
diff --git a/drivers/net/wireguard/noise.c b/drivers/net/wireguard/noise.c
index 720952b92e78..bccaaca66e67 100644
--- a/drivers/net/wireguard/noise.c
+++ b/drivers/net/wireguard/noise.c
@@ -784,10 +784,11 @@ wg_noise_handshake_consume_response(struct message_handshake_response *src,
/* Success! Copy everything to peer */
down_write(&handshake->lock);
- /* It's important to check that the state is still the same, while we
- * have an exclusive lock.
+ /* Check that the state is the same and that this is still the
+ * initiation we started with, while we have an exclusive lock.
*/
- if (handshake->state != state) {
+ if (handshake->state != state ||
+ crypto_memneq(handshake->ephemeral_private, ephemeral_private, NOISE_PUBLIC_KEY_LEN)) {
up_write(&handshake->lock);
goto fail;
}
diff --git a/drivers/net/wireguard/queueing.h b/drivers/net/wireguard/queueing.h
index 3ce70db9dd3f..50ceff8d107e 100644
--- a/drivers/net/wireguard/queueing.h
+++ b/drivers/net/wireguard/queueing.h
@@ -78,12 +78,14 @@ static inline void wg_reset_packet(struct sk_buff *skb, bool encapsulating)
u8 l4_hash = skb->l4_hash;
u8 sw_hash = skb->sw_hash;
u32 hash = skb->hash;
+ u8 mono_delivery_time = skb->mono_delivery_time;
skb_scrub_packet(skb, true);
memset(&skb->headers, 0, sizeof(skb->headers));
if (encapsulating) {
skb->l4_hash = l4_hash;
skb->sw_hash = sw_hash;
skb->hash = hash;
+ skb->mono_delivery_time = mono_delivery_time;
}
skb->queue_mapping = 0;
skb->nohdr = 0;
diff --git a/drivers/net/wireless/ath/ath11k/dp_rx.c b/drivers/net/wireless/ath/ath11k/dp_rx.c
index 7e59260a30b8..413b24275a84 100644
--- a/drivers/net/wireless/ath/ath11k/dp_rx.c
+++ b/drivers/net/wireless/ath/ath11k/dp_rx.c
@@ -5123,7 +5123,7 @@ static void ath11k_dp_rx_mon_dest_process(struct ath11k *ar, int mac_id,
struct ath11k_mon_data *pmon = (struct ath11k_mon_data *)&dp->mon_data;
const struct ath11k_hw_hal_params *hal_params;
void *ring_entry;
- void *mon_dst_srng;
+ struct hal_srng *mon_dst_srng;
u32 ppdu_id;
u32 rx_bufs_used;
u32 ring_id;
@@ -5147,6 +5147,7 @@ static void ath11k_dp_rx_mon_dest_process(struct ath11k *ar, int mac_id,
spin_lock_bh(&pmon->mon_lock);
+ spin_lock_bh(&mon_dst_srng->lock);
ath11k_hal_srng_access_begin(ar->ab, mon_dst_srng);
ppdu_id = pmon->mon_ppdu_info.ppdu_id;
@@ -5205,6 +5206,7 @@ static void ath11k_dp_rx_mon_dest_process(struct ath11k *ar, int mac_id,
mon_dst_srng);
}
ath11k_hal_srng_access_end(ar->ab, mon_dst_srng);
+ spin_unlock_bh(&mon_dst_srng->lock);
spin_unlock_bh(&pmon->mon_lock);
@@ -5594,7 +5596,7 @@ static int ath11k_dp_full_mon_process_rx(struct ath11k_base *ab, int mac_id,
struct hal_sw_mon_ring_entries *sw_mon_entries;
struct ath11k_pdev_mon_stats *rx_mon_stats;
struct sk_buff *head_msdu, *tail_msdu;
- void *mon_dst_srng = &ar->ab->hal.srng_list[dp->rxdma_mon_dst_ring.ring_id];
+ struct hal_srng *mon_dst_srng;
void *ring_entry;
u32 rx_bufs_used = 0, mpdu_rx_bufs_used;
int quota = 0, ret;
@@ -5610,6 +5612,9 @@ static int ath11k_dp_full_mon_process_rx(struct ath11k_base *ab, int mac_id,
goto reap_status_ring;
}
+ mon_dst_srng = &ar->ab->hal.srng_list[dp->rxdma_mon_dst_ring.ring_id];
+ spin_lock_bh(&mon_dst_srng->lock);
+
ath11k_hal_srng_access_begin(ar->ab, mon_dst_srng);
while ((ring_entry = ath11k_hal_srng_dst_peek(ar->ab, mon_dst_srng))) {
head_msdu = NULL;
@@ -5653,6 +5658,7 @@ static int ath11k_dp_full_mon_process_rx(struct ath11k_base *ab, int mac_id,
}
ath11k_hal_srng_access_end(ar->ab, mon_dst_srng);
+ spin_unlock_bh(&mon_dst_srng->lock);
spin_unlock_bh(&pmon->mon_lock);
if (rx_bufs_used) {
diff --git a/drivers/net/wireless/ath/ath11k/mac.c b/drivers/net/wireless/ath/ath11k/mac.c
index 2ef03ad1c905..8565a7031fae 100644
--- a/drivers/net/wireless/ath/ath11k/mac.c
+++ b/drivers/net/wireless/ath/ath11k/mac.c
@@ -5879,6 +5879,7 @@ static int ath11k_mac_op_start(struct ieee80211_hw *hw)
ar->num_started_vdevs = 0;
ar->num_created_vdevs = 0;
ar->num_peers = 0;
+ ar->num_stations = 0;
ar->allocated_vdev_map = 0;
/* Configure monitor status ring with default rx_filter to get rx status
diff --git a/drivers/net/wireless/ath/ath11k/mhi.c b/drivers/net/wireless/ath/ath11k/mhi.c
index 4bea36cc7108..7d53baa28f4f 100644
--- a/drivers/net/wireless/ath/ath11k/mhi.c
+++ b/drivers/net/wireless/ath/ath11k/mhi.c
@@ -60,7 +60,6 @@ static struct mhi_channel_config ath11k_mhi_channels_qca6390[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
},
{
.num = 21,
@@ -74,7 +73,6 @@ static struct mhi_channel_config ath11k_mhi_channels_qca6390[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = true,
},
};
@@ -153,7 +151,6 @@ static struct mhi_channel_config ath11k_mhi_channels_qcn9074[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = false,
},
{
.num = 21,
@@ -167,7 +164,6 @@ static struct mhi_channel_config ath11k_mhi_channels_qcn9074[] = {
.lpm_notify = false,
.offload_channel = false,
.doorbell_mode_switch = false,
- .auto_queue = true,
},
};
@@ -333,8 +329,10 @@ static void ath11k_mhi_op_status_cb(struct mhi_controller *mhi_cntrl,
ath11k_warn(ab, "firmware crashed: MHI_CB_SYS_ERROR\n");
break;
case MHI_CB_EE_RDDM:
+ spin_lock_bh(&ab->base_lock);
if (!(test_bit(ATH11K_FLAG_UNREGISTERING, &ab->dev_flags)))
queue_work(ab->workqueue_aux, &ab->reset_work);
+ spin_unlock_bh(&ab->base_lock);
break;
default:
break;
diff --git a/drivers/net/wireless/ath/ath11k/pci.c b/drivers/net/wireless/ath/ath11k/pci.c
index b50361490b9b..3a92ac80d8ae 100644
--- a/drivers/net/wireless/ath/ath11k/pci.c
+++ b/drivers/net/wireless/ath/ath11k/pci.c
@@ -972,6 +972,13 @@ static void ath11k_pci_shutdown(struct pci_dev *pdev)
struct ath11k_pci *ab_pci = ath11k_pci_priv(ab);
ath11k_pci_set_irq_affinity_hint(ab_pci, NULL);
+
+ spin_lock_bh(&ab->base_lock);
+ set_bit(ATH11K_FLAG_UNREGISTERING, &ab->dev_flags);
+ spin_unlock_bh(&ab->base_lock);
+
+ cancel_work_sync(&ab->reset_work);
+
ath11k_pci_power_down(ab);
}
diff --git a/drivers/net/wireless/ath/ath11k/wmi.c b/drivers/net/wireless/ath/ath11k/wmi.c
index c3f440cae0bf..579cd7a2d427 100644
--- a/drivers/net/wireless/ath/ath11k/wmi.c
+++ b/drivers/net/wireless/ath/ath11k/wmi.c
@@ -145,6 +145,8 @@ static const struct wmi_tlv_policy wmi_tlv_policies[] = {
.min_len = sizeof(struct wmi_per_chain_rssi_stats) },
[WMI_TAG_TWT_ADD_DIALOG_COMPLETE_EVENT] = {
.min_len = sizeof(struct wmi_twt_add_dialog_event) },
+ [WMI_TAG_PDEV_CSA_SWITCH_COUNT_STATUS_EVENT] = {
+ .min_len = sizeof(struct wmi_pdev_csa_switch_ev) },
};
#define PRIMAP(_hw_mode_) \
@@ -249,6 +251,13 @@ ath11k_wmi_tlv_parse_alloc(struct ath11k_base *ab, const void *ptr,
return tb;
}
+static u32 ath11k_wmi_tlv_data_len(const void *data)
+{
+ const struct wmi_tlv *tlv = (const struct wmi_tlv *)data - 1;
+
+ return FIELD_GET(WMI_TLV_LEN, tlv->header);
+}
+
static int ath11k_wmi_cmd_send_nowait(struct ath11k_pdev_wmi *wmi, struct sk_buff *skb,
u32 cmd_id)
{
@@ -4513,14 +4522,16 @@ static int ath11k_wmi_tlv_mac_phy_caps_parse(struct ath11k_base *soc,
if (svc_rdy_ext->n_mac_phy_caps >= svc_rdy_ext->tot_phy_id)
return -ENOBUFS;
- len = min_t(u16, len, sizeof(struct wmi_mac_phy_capabilities));
if (!svc_rdy_ext->n_mac_phy_caps) {
- svc_rdy_ext->mac_phy_caps = kcalloc(svc_rdy_ext->tot_phy_id,
- len, GFP_ATOMIC);
+ svc_rdy_ext->mac_phy_caps =
+ kcalloc(svc_rdy_ext->tot_phy_id,
+ sizeof(*svc_rdy_ext->mac_phy_caps),
+ GFP_ATOMIC);
if (!svc_rdy_ext->mac_phy_caps)
return -ENOMEM;
}
+ len = min_t(u16, len, sizeof(struct wmi_mac_phy_capabilities));
memcpy(svc_rdy_ext->mac_phy_caps + svc_rdy_ext->n_mac_phy_caps, ptr, len);
svc_rdy_ext->n_mac_phy_caps++;
return 0;
@@ -7654,15 +7665,23 @@ ath11k_wmi_process_csa_switch_count_event(struct ath11k_base *ab,
const struct wmi_pdev_csa_switch_ev *ev,
const u32 *vdev_ids)
{
- int i;
+ u32 vdev_ids_len = ath11k_wmi_tlv_data_len(vdev_ids);
+ u32 num_vdevs = ev->num_vdevs;
struct ath11k_vif *arvif;
+ int i;
/* Finish CSA once the switch count becomes NULL */
if (ev->current_switch_count)
return;
+ if (num_vdevs > vdev_ids_len / sizeof(*vdev_ids)) {
+ ath11k_warn(ab, "csa switch count num_vdevs %u exceeds tlv array length %u\n",
+ num_vdevs, vdev_ids_len);
+ return;
+ }
+
rcu_read_lock();
- for (i = 0; i < ev->num_vdevs; i++) {
+ for (i = 0; i < num_vdevs; i++) {
arvif = ath11k_mac_get_arvif_by_vdev_id(ab, vdev_ids[i]);
if (!arvif) {
diff --git a/drivers/net/wireless/ath/wcn36xx/dxe.c b/drivers/net/wireless/ath/wcn36xx/dxe.c
index 4e9e13941c8f..b168db800e07 100644
--- a/drivers/net/wireless/ath/wcn36xx/dxe.c
+++ b/drivers/net/wireless/ath/wcn36xx/dxe.c
@@ -1054,7 +1054,7 @@ void wcn36xx_dxe_deinit(struct wcn36xx *wcn)
free_irq(wcn->tx_irq, wcn);
free_irq(wcn->rx_irq, wcn);
- del_timer(&wcn->tx_ack_timer);
+ timer_shutdown_sync(&wcn->tx_ack_timer);
if (wcn->tx_ack_skb) {
ieee80211_tx_status_irqsafe(wcn->hw, wcn->tx_ack_skb);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
index dfb00d454831..44f40daf1a21 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c
@@ -1471,8 +1471,31 @@ void brcmf_detach(struct device *dev)
brcmf_bus_change_state(bus_if, BRCMF_BUS_DOWN);
/* make sure primary interface removed last */
for (i = BRCMF_MAX_IFS - 1; i > -1; i--) {
- if (drvr->iflist[i])
- brcmf_remove_interface(drvr->iflist[i], false);
+ struct brcmf_if *ifp = drvr->iflist[i];
+
+ if (!ifp)
+ continue;
+
+ if (ifp->ndev) {
+ brcmf_remove_interface(ifp, false);
+ continue;
+ }
+
+ /* The P2P device interface has no netdev. Its removal can
+ * race with NL80211_CMD_DEL_INTERFACE issued by a user space
+ * process that is exiting (e.g. wpa_supplicant), which ends
+ * in brcmf_p2p_del_vif() under RTNL and the wiphy mutex. If
+ * both paths remove the interface, the wdev is unregistered
+ * twice and the vif is used after being freed. Take the same
+ * locks and re-read iflist, so only one path removes it.
+ */
+ rtnl_lock();
+ wiphy_lock(drvr->wiphy);
+ ifp = drvr->iflist[i];
+ if (ifp)
+ brcmf_remove_interface(ifp, true);
+ wiphy_unlock(drvr->wiphy);
+ rtnl_unlock();
}
brcmf_bus_stop(drvr->bus_if);
diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
index ac6b70ff1535..b3aeaf8a476a 100644
--- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
+++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c
@@ -1281,6 +1281,9 @@ static s32 brcmf_p2p_abort_action_frame(struct brcmf_cfg80211_info *cfg)
brcmf_dbg(TRACE, "Enter\n");
vif = p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif;
+ if (!vif)
+ vif = p2p->bss_idx[P2PAPI_BSSCFG_PRIMARY].vif;
+
err = brcmf_fil_bsscfg_data_set(vif->ifp, "actframe_abort", &int_val,
sizeof(s32));
if (err)
@@ -1817,6 +1820,7 @@ bool brcmf_p2p_send_action_frame(struct brcmf_if *ifp,
/* validate channel and p2p ies */
if (config_af_params.search_channel &&
IS_P2P_SOCIAL_CHANNEL(le32_to_cpu(af_params->channel)) &&
+ p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif &&
p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif->saved_ie.probe_req_ie_len) {
afx_hdl = &p2p->afx_hdl;
afx_hdl->peer_listen_chan = le32_to_cpu(af_params->channel);
diff --git a/drivers/net/wireless/intel/iwlegacy/3945-mac.c b/drivers/net/wireless/intel/iwlegacy/3945-mac.c
index 32dd217529a6..f1362bd4a5ed 100644
--- a/drivers/net/wireless/intel/iwlegacy/3945-mac.c
+++ b/drivers/net/wireless/intel/iwlegacy/3945-mac.c
@@ -3845,6 +3845,7 @@ il3945_pci_remove(struct pci_dev *pdev)
il_free_channel_map(il);
il_free_geos(il);
+ il_eeprom_free(il);
kfree(il->scan_cmd);
dev_kfree_skb(il->beacon_skb);
ieee80211_free_hw(il->hw);
diff --git a/drivers/net/wireless/intel/iwlwifi/dvm/main.c b/drivers/net/wireless/intel/iwlwifi/dvm/main.c
index b490a88b97ca..68b253d9928c 100644
--- a/drivers/net/wireless/intel/iwlwifi/dvm/main.c
+++ b/drivers/net/wireless/intel/iwlwifi/dvm/main.c
@@ -1502,10 +1502,10 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans,
priv->workqueue = NULL;
out_uninit_drv:
iwl_uninit_drv(priv);
-out_free_eeprom_blob:
- kfree(priv->eeprom_blob);
out_free_eeprom:
kfree(priv->nvm_data);
+out_free_eeprom_blob:
+ kfree(priv->eeprom_blob);
out_free_hw:
ieee80211_free_hw(priv->hw);
out:
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c b/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c
index 677c9e0b46f1..1e1d2ab871ff 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-dbg-tlv.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2018-2022 Intel Corporation
+ * Copyright (C) 2018-2026 Intel Corporation
*/
#include <linux/firmware.h>
#include "iwl-drv.h"
@@ -597,6 +597,9 @@ static int iwl_dbg_tlv_alloc_fragments(struct iwl_fw_runtime *fwrt,
cpu_to_le32(IWL_FW_INI_LOCATION_DRAM_PATH))
return 0;
+ if (!fw_mon_cfg->req_size)
+ return -EIO;
+
num_frags = le32_to_cpu(fw_mon_cfg->max_frags_num);
if (fwrt->trans->trans_cfg->device_family < IWL_DEVICE_FAMILY_AX210) {
if (alloc_id != IWL_FW_INI_ALLOCATION_ID_DBGC1)
@@ -604,6 +607,9 @@ static int iwl_dbg_tlv_alloc_fragments(struct iwl_fw_runtime *fwrt,
num_frags = 1;
}
+ if (!num_frags)
+ return -EIO;
+
remain_pages = DIV_ROUND_UP(le32_to_cpu(fw_mon_cfg->req_size),
PAGE_SIZE);
num_frags = min_t(u32, num_frags, BUF_ALLOC_MAX_NUM_FRAGS);
@@ -797,8 +803,7 @@ static void iwl_dbg_tlv_update_drams(struct iwl_fw_runtime *fwrt)
IWL_UCODE_TLV_CAPA_DRAM_FRAG_SUPPORT))
return;
- dram_info->first_word = cpu_to_le32(DRAM_INFO_FIRST_MAGIC_WORD);
- dram_info->second_word = cpu_to_le32(DRAM_INFO_SECOND_MAGIC_WORD);
+ memset(dram_info, 0, sizeof(*dram_info));
for (i = IWL_FW_INI_ALLOCATION_ID_DBGC1;
i <= IWL_FW_INI_ALLOCATION_ID_DBGC3; i++) {
@@ -811,11 +816,10 @@ static void iwl_dbg_tlv_update_drams(struct iwl_fw_runtime *fwrt)
i, ret);
}
- if (dram_alloc)
- IWL_DEBUG_FW(fwrt, "block data after %08x\n",
- dram_info->first_word);
- else
- memset(frags->block, 0, sizeof(*dram_info));
+ if (dram_alloc) {
+ dram_info->first_word = cpu_to_le32(DRAM_INFO_FIRST_MAGIC_WORD);
+ dram_info->second_word = cpu_to_le32(DRAM_INFO_SECOND_MAGIC_WORD);
+ }
}
static void iwl_dbg_tlv_send_hcmds(struct iwl_fw_runtime *fwrt,
@@ -1363,15 +1367,15 @@ void _iwl_dbg_tlv_time_point(struct iwl_fw_runtime *fwrt,
switch (tp_id) {
case IWL_FW_INI_TIME_POINT_EARLY:
iwl_dbg_tlv_init_cfg(fwrt);
- iwl_dbg_tlv_apply_config(fwrt, conf_list);
iwl_dbg_tlv_update_drams(fwrt);
iwl_dbg_tlv_tp_trigger(fwrt, sync, trig_list, tp_data, NULL);
+ iwl_dbg_tlv_apply_config(fwrt, conf_list);
break;
case IWL_FW_INI_TIME_POINT_AFTER_ALIVE:
iwl_dbg_tlv_apply_buffers(fwrt);
iwl_dbg_tlv_send_hcmds(fwrt, hcmd_list);
- iwl_dbg_tlv_apply_config(fwrt, conf_list);
iwl_dbg_tlv_tp_trigger(fwrt, sync, trig_list, tp_data, NULL);
+ iwl_dbg_tlv_apply_config(fwrt, conf_list);
break;
case IWL_FW_INI_TIME_POINT_PERIODIC:
iwl_dbg_tlv_set_periodic_trigs(fwrt);
@@ -1381,14 +1385,14 @@ void _iwl_dbg_tlv_time_point(struct iwl_fw_runtime *fwrt,
case IWL_FW_INI_TIME_POINT_MISSED_BEACONS:
case IWL_FW_INI_TIME_POINT_FW_DHC_NOTIFICATION:
iwl_dbg_tlv_send_hcmds(fwrt, hcmd_list);
- iwl_dbg_tlv_apply_config(fwrt, conf_list);
iwl_dbg_tlv_tp_trigger(fwrt, sync, trig_list, tp_data,
iwl_dbg_tlv_check_fw_pkt);
+ iwl_dbg_tlv_apply_config(fwrt, conf_list);
break;
default:
iwl_dbg_tlv_send_hcmds(fwrt, hcmd_list);
- iwl_dbg_tlv_apply_config(fwrt, conf_list);
iwl_dbg_tlv_tp_trigger(fwrt, sync, trig_list, tp_data, NULL);
+ iwl_dbg_tlv_apply_config(fwrt, conf_list);
break;
}
}
diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h
index ad29663a356b..9380fdd707af 100644
--- a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h
+++ b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h
@@ -1194,6 +1194,9 @@ static inline void iwl_trans_reclaim(struct iwl_trans *trans, int queue,
int ssn, struct sk_buff_head *skbs,
bool is_flush)
{
+ if (unlikely(test_bit(STATUS_FW_ERROR, &trans->status)))
+ return;
+
if (WARN_ON_ONCE(trans->state != IWL_TRANS_FW_ALIVE)) {
IWL_ERR(trans, "%s bad state = %d\n", __func__, trans->state);
return;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/d3.c b/drivers/net/wireless/intel/iwlwifi/mvm/d3.c
index d7b2781a1eea..b28c9627d5b0 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/d3.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/d3.c
@@ -1508,6 +1508,10 @@ static void iwl_mvm_report_wakeup_reasons(struct iwl_mvm *mvm,
if (WARN_ON_ONCE(truncated < 0))
truncated = 0;
+ /* this would be a firmware bug */
+ if (WARN_ON_ONCE(pktsize < sizeof(*hdr)))
+ return;
+
if (ieee80211_is_data(hdr->frame_control)) {
int hdrlen = ieee80211_hdrlen(hdr->frame_control);
int ivlen = 0, icvlen = 4; /* also FCS */
@@ -1516,10 +1520,6 @@ static void iwl_mvm_report_wakeup_reasons(struct iwl_mvm *mvm,
if (!pkt)
goto report;
- skb_put_data(pkt, pktdata, hdrlen);
- pktdata += hdrlen;
- pktsize -= hdrlen;
-
if (ieee80211_has_protected(hdr->frame_control)) {
/*
* This is unlocked and using gtk_i(c)vlen,
@@ -1545,6 +1545,16 @@ static void iwl_mvm_report_wakeup_reasons(struct iwl_mvm *mvm,
truncated = 0;
}
+ if (pktsize <= hdrlen + ivlen + icvlen) {
+ IWL_WARN(mvm, "pktsize is too small %d\n",
+ pktsize);
+ kfree_skb(pkt);
+ return;
+ }
+
+ skb_put_data(pkt, pktdata, hdrlen);
+ pktdata += hdrlen;
+ pktsize -= hdrlen;
pktsize -= ivlen + icvlen;
pktdata += ivlen;
diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
index 43a732b0c168..a8036e14c059 100644
--- a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
+++ b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c
@@ -1550,6 +1550,17 @@ static inline u32 iwl_mvm_get_scd_ssn(struct iwl_mvm *mvm,
tx_resp->frame_count) & 0xfff;
}
+static inline size_t iwl_mvm_tx_resp_min_len(struct iwl_mvm *mvm,
+ struct iwl_mvm_tx_resp *tx_resp)
+{
+ struct agg_tx_status *agg_status =
+ iwl_mvm_get_agg_status(mvm, tx_resp);
+
+ /* The aggregate response ends with a trailing SCD SSN __le32 word. */
+ return (u8 *)(agg_status + tx_resp->frame_count) - (u8 *)tx_resp +
+ sizeof(__le32);
+}
+
static void iwl_mvm_rx_tx_cmd_single(struct iwl_mvm *mvm,
struct iwl_rx_packet *pkt)
{
@@ -1854,6 +1865,9 @@ static void iwl_mvm_rx_tx_cmd_agg(struct iwl_mvm *mvm,
int queue = SEQ_TO_QUEUE(sequence);
struct ieee80211_sta *sta;
+ if (WARN_ON_ONCE(iwl_mvm_has_new_tx_api(mvm)))
+ return;
+
if (WARN_ON_ONCE(queue < IWL_MVM_DQA_MIN_DATA_QUEUE &&
(queue != IWL_MVM_DQA_BSS_CLIENT_QUEUE)))
return;
@@ -1888,6 +1902,27 @@ void iwl_mvm_rx_tx_cmd(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb)
{
struct iwl_rx_packet *pkt = rxb_addr(rxb);
struct iwl_mvm_tx_resp *tx_resp = (void *)pkt->data;
+ size_t min_len;
+
+ if (!tx_resp->frame_count) {
+ IWL_WARN(mvm, "invalid TX_CMD frame_count %u\n",
+ tx_resp->frame_count);
+ return;
+ }
+
+ if (iwl_mvm_has_new_tx_api(mvm) && tx_resp->frame_count != 1) {
+ IWL_WARN(mvm, "invalid TX_CMD frame_count %u for new TX API\n",
+ tx_resp->frame_count);
+ return;
+ }
+
+ min_len = iwl_mvm_tx_resp_min_len(mvm, tx_resp);
+ if (iwl_rx_packet_payload_len(pkt) < min_len) {
+ IWL_WARN(mvm, "invalid TX_CMD len %u (frame_count %u, min %zu)\n",
+ iwl_rx_packet_payload_len(pkt), tx_resp->frame_count,
+ min_len);
+ return;
+ }
if (tx_resp->frame_count == 1)
iwl_mvm_rx_tx_cmd_single(mvm, pkt);
@@ -2094,8 +2129,14 @@ void iwl_mvm_rx_ba_notif(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb)
if (tid == IWL_MGMT_TID)
tid = IWL_MAX_TID_COUNT;
+ if (tid >= ARRAY_SIZE(mvmsta->tid_data)) {
+ IWL_WARN(mvm, "invalid TID %d in compressed BA\n",
+ tid);
+ continue;
+ }
+
if (mvmsta)
- mvmsta->tid_data[i].lq_color = lq_color;
+ mvmsta->tid_data[tid].lq_color = lq_color;
iwl_mvm_tx_reclaim(mvm, sta_id, tid,
(int)(le16_to_cpu(ba_tfd->q_num)),
@@ -2120,6 +2161,10 @@ void iwl_mvm_rx_ba_notif(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb)
ba_notif = (void *)pkt->data;
sta_id = ba_notif->sta_id;
tid = ba_notif->tid;
+ if (tid >= IWL_MAX_TID_COUNT) {
+ IWL_WARN(mvm, "invalid TID %d in BA notif\n", tid);
+ return;
+ }
/* "flow" corresponds to Tx queue */
txq = le16_to_cpu(ba_notif->scd_flow);
/* "ssn" is start of block-ack Tx window, corresponds to index
diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
index 91b73e7a4113..d4b57fe25eb1 100644
--- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
+++ b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause
/*
- * Copyright (C) 2003-2014, 2018-2023 Intel Corporation
+ * Copyright (C) 2003-2014, 2018-2023, 2026 Intel Corporation
* Copyright (C) 2013-2015 Intel Mobile Communications GmbH
* Copyright (C) 2016-2017 Intel Deutschland GmbH
*/
@@ -1232,11 +1232,16 @@ void iwl_pcie_rx_free(struct iwl_trans *trans)
}
}
kfree(trans_pcie->rx_pool);
+ trans_pcie->rx_pool = NULL;
kfree(trans_pcie->global_table);
+ trans_pcie->global_table = NULL;
kfree(trans_pcie->rxq);
+ trans_pcie->rxq = NULL;
- if (trans_pcie->alloc_page)
+ if (trans_pcie->alloc_page) {
__free_pages(trans_pcie->alloc_page, trans_pcie->rx_page_order);
+ trans_pcie->alloc_page = NULL;
+ }
}
static void iwl_pcie_rx_move_to_allocator(struct iwl_rxq *rxq,
diff --git a/drivers/net/wireless/intersil/p54/fwio.c b/drivers/net/wireless/intersil/p54/fwio.c
index 06e5df90b43e..34988cba5753 100644
--- a/drivers/net/wireless/intersil/p54/fwio.c
+++ b/drivers/net/wireless/intersil/p54/fwio.c
@@ -52,6 +52,13 @@ int p54_parse_firmware(struct ieee80211_hw *dev, const struct firmware *fw)
u32 code = le32_to_cpu(bootrec->code);
switch (code) {
case BR_CODE_COMPONENT_ID:
+ if (len < sizeof(struct bootrec_comp_id) /
+ sizeof(*bootrec->data)) {
+ wiphy_err(priv->hw->wiphy,
+ "firmware component ID is too short\n");
+ return -EINVAL;
+ }
+
priv->fw_interface = be32_to_cpup((__be32 *)
bootrec->data);
switch (priv->fw_interface) {
@@ -72,16 +79,41 @@ int p54_parse_firmware(struct ieee80211_hw *dev, const struct firmware *fw)
}
break;
case BR_CODE_COMPONENT_VERSION:
+ if (len < DIV_ROUND_UP(sizeof(struct bootrec_comp_ver),
+ sizeof(*bootrec->data))) {
+ wiphy_err(priv->hw->wiphy,
+ "firmware component version is too short\n");
+ return -EINVAL;
+ }
+
/* 24 bytes should be enough for all firmwares */
- if (strnlen((unsigned char *) bootrec->data, 24) < 24)
- fw_version = (unsigned char *) bootrec->data;
+ if (strnlen((unsigned char *)bootrec->data,
+ sizeof(struct bootrec_comp_ver)) <
+ sizeof(struct bootrec_comp_ver))
+ fw_version = (unsigned char *)bootrec->data;
break;
case BR_CODE_DESCR: {
struct bootrec_desc *desc =
(struct bootrec_desc *)bootrec->data;
- priv->rx_start = le32_to_cpu(desc->rx_start);
- /* FIXME add sanity checking */
- priv->rx_end = le32_to_cpu(desc->rx_end) - 0x3500;
+ u32 rx_start, rx_end;
+
+ /* 0xa is the shortest descriptor in supported firmware. */
+ if (len < 0xa) {
+ wiphy_err(priv->hw->wiphy,
+ "firmware descriptor is too short\n");
+ return -EINVAL;
+ }
+
+ rx_start = le32_to_cpu(desc->rx_start);
+ rx_end = le32_to_cpu(desc->rx_end);
+ if (rx_end < 0x3500 || rx_end - 0x3500 <= rx_start) {
+ wiphy_err(priv->hw->wiphy,
+ "firmware descriptor has invalid RX range\n");
+ return -EINVAL;
+ }
+
+ priv->rx_start = rx_start;
+ priv->rx_end = rx_end - 0x3500;
priv->headroom = desc->headroom;
priv->tailroom = desc->tailroom;
priv->privacy_caps = desc->privacy_caps;
diff --git a/drivers/net/wireless/marvell/libertas/tx.c b/drivers/net/wireless/marvell/libertas/tx.c
index 13d08022e414..27304a98787d 100644
--- a/drivers/net/wireless/marvell/libertas/tx.c
+++ b/drivers/net/wireless/marvell/libertas/tx.c
@@ -117,13 +117,6 @@ netdev_tx_t lbs_hard_start_xmit(struct sk_buff *skb, struct net_device *dev)
if (priv->wdev->iftype == NL80211_IFTYPE_MONITOR) {
struct tx_radiotap_hdr *rtap_hdr = (void *)skb->data;
- if (skb->len < sizeof(*rtap_hdr) + 4 + ETH_ALEN) {
- lbs_deb_tx("tx err: short monitor frame %u\n", skb->len);
- dev->stats.tx_dropped++;
- dev->stats.tx_errors++;
- goto free;
- }
-
/* set txpd fields from the radiotap header */
txpd->tx_control = cpu_to_le32(convert_radiotap_rate_to_mv(rtap_hdr->rate));
diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
index 6f4b1a784a07..e6ca72363e0f 100644
--- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
+++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c
@@ -2652,8 +2652,10 @@ int mt76_connac_mcu_add_key(struct mt76_dev *dev, struct ieee80211_vif *vif,
return PTR_ERR(skb);
ret = mt76_connac_mcu_sta_key_tlv(sta_key_conf, skb, key, cmd);
- if (ret)
+ if (ret) {
+ dev_kfree_skb(skb);
return ret;
+ }
return mt76_mcu_skb_send_msg(dev, skb, mcu_cmd, true);
}
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c b/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c
index 80f0488871df..b6fe6da68feb 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c
@@ -452,6 +452,11 @@ mt7915_rdd_monitor(struct seq_file *s, void *data)
mutex_lock(&dev->mt76.mutex);
+ if (!mt7915_eeprom_has_background_radar(dev)) {
+ seq_puts(s, "no background radar capability\n");
+ goto out;
+ }
+
if (!cfg80211_chandef_valid(chandef)) {
ret = -EINVAL;
goto out;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c
index f0ec000d46cf..8b43fb641808 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c
@@ -123,7 +123,7 @@ static int mt7915_eeprom_load(struct mt7915_dev *dev)
/* read eeprom data from efuse */
block_num = DIV_ROUND_UP(eeprom_size, eeprom_blk_size);
for (i = 0; i < block_num; i++) {
- ret = mt7915_mcu_get_eeprom(dev, i * eeprom_blk_size);
+ ret = mt7915_mcu_get_eeprom(dev, i * eeprom_blk_size, NULL);
if (ret < 0)
return ret;
}
@@ -326,6 +326,37 @@ s8 mt7915_eeprom_get_power_delta(struct mt7915_dev *dev, int band)
return val & MT_EE_RATE_DELTA_SIGN ? delta : -delta;
}
+bool
+mt7915_eeprom_has_background_radar(struct mt7915_dev *dev)
+{
+ u8 val, buf[MT7915_EEPROM_BLOCK_SIZE];
+ u8 band_sel, tx_path, rx_path;
+ int offs = MT_EE_WIFI_CONF + 1;
+
+ switch (mt76_chip(&dev->mt76)) {
+ case 0x7915:
+ return true;
+ case 0x7906:
+ /* read efuse to check background radar capability */
+ if (mt7915_mcu_get_eeprom(dev, offs, buf))
+ break;
+
+ val = buf[offs % MT7915_EEPROM_BLOCK_SIZE];
+ band_sel = u8_get_bits(val, MT_EE_WIFI_CONF0_BAND_SEL);
+ tx_path = u8_get_bits(val, MT_EE_WIFI_CONF0_TX_PATH);
+ rx_path = u8_get_bits(val, MT_EE_WIFI_CONF0_RX_PATH);
+
+ return (band_sel == MT_EE_V2_BAND_SEL_5GHZ &&
+ tx_path == rx_path && rx_path == 2);
+ case 0x7981:
+ case 0x7986:
+ default:
+ break;
+ }
+
+ return false;
+}
+
const u8 mt7915_sku_group_len[] = {
[SKU_CCK] = 4,
[SKU_OFDM] = 8,
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h
index f3e56817d36e..ddb771ba45d2 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h
@@ -45,6 +45,7 @@ enum mt7915_eeprom_field {
#define MT_EE_CAL_DPD_SIZE (54 * MT_EE_CAL_UNIT)
#define MT_EE_WIFI_CONF0_TX_PATH GENMASK(2, 0)
+#define MT_EE_WIFI_CONF0_RX_PATH GENMASK(5, 3)
#define MT_EE_WIFI_CONF0_BAND_SEL GENMASK(7, 6)
#define MT_EE_WIFI_CONF1_BAND_SEL GENMASK(7, 6)
#define MT_EE_WIFI_CONF_STREAM_NUM GENMASK(7, 5)
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/init.c b/drivers/net/wireless/mediatek/mt76/mt7915/init.c
index 7d1e2aa73289..35a8052cf1e3 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/init.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/init.c
@@ -353,9 +353,10 @@ mt7915_init_wiphy(struct ieee80211_hw *hw)
wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_UNSOL_BCAST_PROBE_RESP);
wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_FILS_DISCOVERY);
- if (!mdev->dev->of_node ||
- !of_property_read_bool(mdev->dev->of_node,
- "mediatek,disable-radar-background"))
+ if (mt7915_eeprom_has_background_radar(phy->dev) &&
+ (!mdev->dev->of_node ||
+ !of_property_read_bool(mdev->dev->of_node,
+ "mediatek,disable-radar-background")))
wiphy_ext_feature_set(wiphy,
NL80211_EXT_FEATURE_RADAR_BACKGROUND);
@@ -1151,6 +1152,8 @@ int mt7915_register_device(struct mt7915_dev *dev)
void mt7915_unregister_device(struct mt7915_dev *dev)
{
cancel_work_sync(&dev->dump_work);
+ cancel_work_sync(&dev->reset_work);
+ cancel_work_sync(&dev->rc_work);
mt7915_unregister_ext_phy(dev);
mt7915_coredump_unregister(dev);
mt7915_unregister_thermal(&dev->phy);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/main.c b/drivers/net/wireless/mediatek/mt76/mt7915/main.c
index 0da672a86ae1..4707e7727f17 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/main.c
@@ -236,7 +236,11 @@ static int mt7915_add_interface(struct ieee80211_hw *hw,
dev->mt76.vif_mask |= BIT_ULL(mvif->mt76.idx);
phy->omac_mask |= BIT_ULL(mvif->mt76.omac_idx);
- idx = MT7915_WTBL_RESERVED - mvif->mt76.idx;
+ idx = mt76_wcid_alloc(dev->mt76.wcid_mask, mt7915_wtbl_size(dev));
+ if (idx < 0) {
+ ret = -ENOSPC;
+ goto err;
+ }
INIT_LIST_HEAD(&mvif->sta.rc_list);
INIT_LIST_HEAD(&mvif->sta.poll_list);
@@ -266,7 +270,17 @@ static int mt7915_add_interface(struct ieee80211_hw *hw,
mt7915_mcu_add_sta(dev, vif, NULL, true);
rcu_assign_pointer(dev->mt76.wcid[idx], &mvif->sta.wcid);
+ mutex_unlock(&dev->mt76.mutex);
+
+ return 0;
+
+err:
+ dev->mt76.vif_mask &= ~BIT_ULL(mvif->mt76.idx);
+ phy->omac_mask &= ~BIT_ULL(mvif->mt76.omac_idx);
+ mt7915_mcu_add_dev_info(phy, vif, false);
out:
+ if (phy->monitor_vif == vif)
+ phy->monitor_vif = NULL;
mutex_unlock(&dev->mt76.mutex);
return ret;
@@ -283,6 +297,7 @@ static void mt7915_remove_interface(struct ieee80211_hw *hw,
mt7915_mcu_add_bss_info(phy, vif, false);
mt7915_mcu_add_sta(dev, vif, NULL, false);
+ mt76_wcid_mask_clear(dev->mt76.wcid_mask, mvif->sta.wcid.idx);
mutex_lock(&dev->mt76.mutex);
mt76_testmode_reset(phy->mt76, true);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c
index f1f6bcc49949..6cea8b2791bb 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c
@@ -2751,7 +2751,7 @@ int mt7915_mcu_set_eeprom(struct mt7915_dev *dev)
&req, sizeof(req), true);
}
-int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset)
+int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset, u8 *read_buf)
{
struct mt7915_mcu_eeprom_info req = {
.addr = cpu_to_le32(round_down(offset,
@@ -2759,8 +2759,8 @@ int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset)
};
struct mt7915_mcu_eeprom_info *res;
struct sk_buff *skb;
+ u8 *buf = read_buf;
int ret;
- u8 *buf;
ret = mt76_mcu_send_and_get_msg(&dev->mt76,
MCU_EXT_QUERY(EFUSE_ACCESS),
@@ -2769,8 +2769,17 @@ int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset)
return ret;
res = (struct mt7915_mcu_eeprom_info *)skb->data;
- buf = dev->mt76.eeprom.data + le32_to_cpu(res->addr);
+ if (!buf) {
+ u32 addr = le32_to_cpu(res->addr);
+
+ if (addr > dev->mt76.eeprom.size - MT7915_EEPROM_BLOCK_SIZE) {
+ dev_kfree_skb(skb);
+ return -EINVAL;
+ }
+ buf = dev->mt76.eeprom.data + addr;
+ }
memcpy(buf, res->data, MT7915_EEPROM_BLOCK_SIZE);
+
dev_kfree_skb(skb);
return 0;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h
index ebe1cb0429d0..c2b3a28d7169 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h
+++ b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h
@@ -465,6 +465,7 @@ int mt7915_eeprom_get_target_power(struct mt7915_dev *dev,
struct ieee80211_channel *chan,
u8 chain_idx);
s8 mt7915_eeprom_get_power_delta(struct mt7915_dev *dev, int band);
+bool mt7915_eeprom_has_background_radar(struct mt7915_dev *dev);
int mt7915_dma_init(struct mt7915_dev *dev, struct mt7915_phy *phy2);
void mt7915_dma_prefetch(struct mt7915_dev *dev);
void mt7915_dma_cleanup(struct mt7915_dev *dev);
@@ -511,7 +512,7 @@ int mt7915_mcu_set_fixed_rate_ctrl(struct mt7915_dev *dev,
struct ieee80211_sta *sta,
void *data, u32 field);
int mt7915_mcu_set_eeprom(struct mt7915_dev *dev);
-int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset);
+int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset, u8 *read_buf);
int mt7915_mcu_get_eeprom_free_block(struct mt7915_dev *dev, u8 *block_num);
int mt7915_mcu_set_mac(struct mt7915_dev *dev, int band, bool enable,
bool hdr_trans);
diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/main.c b/drivers/net/wireless/mediatek/mt76/mt7921/main.c
index fd91d2c53760..2283682aec65 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7921/main.c
+++ b/drivers/net/wireless/mediatek/mt76/mt7921/main.c
@@ -716,6 +716,9 @@ int mt7921_mac_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif,
struct mt7921_vif *mvif = (struct mt7921_vif *)vif->drv_priv;
int ret, idx;
+ if (sta->aid > MT7921_MAX_AID)
+ return -ENOENT;
+
idx = mt76_wcid_alloc(dev->mt76.wcid_mask, MT7921_WTBL_STA - 1);
if (idx < 0)
return -ENOSPC;
@@ -756,6 +759,9 @@ void mt7921_mac_sta_assoc(struct mt76_dev *mdev, struct ieee80211_vif *vif,
struct mt7921_sta *msta = (struct mt7921_sta *)sta->drv_priv;
struct mt7921_vif *mvif = (struct mt7921_vif *)vif->drv_priv;
+ if (sta->aid > MT7921_MAX_AID)
+ return;
+
mt7921_mutex_acquire(dev);
if (vif->type == NL80211_IFTYPE_STATION && !sta->tdls)
@@ -1436,7 +1442,7 @@ static void mt7921_ipv6_addr_change(struct ieee80211_hw *hw,
struct inet6_dev *idev)
{
struct mt7921_vif *mvif = (struct mt7921_vif *)vif->drv_priv;
- struct mt7921_dev *dev = mvif->phy->dev;
+ struct mt7921_dev *dev = mt7921_hw_dev(hw);
struct inet6_ifaddr *ifa;
struct in6_addr ns_addrs[IEEE80211_BSS_ARP_ADDR_LIST_LEN];
struct sk_buff *skb;
diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h b/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h
index d36b940c0a07..1e2f914558a2 100644
--- a/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h
+++ b/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h
@@ -21,6 +21,8 @@
#define MT7921_WATCHDOG_TIME (HZ / 4)
#define MT7921_RESET_TIMEOUT (30 * HZ)
+#define MT7921_MAX_AID 20
+
#define MT7921_TX_RING_SIZE 2048
#define MT7921_TX_MCU_RING_SIZE 256
#define MT7921_TX_FWDL_RING_SIZE 128
diff --git a/drivers/net/wireless/mediatek/mt76/tx.c b/drivers/net/wireless/mediatek/mt76/tx.c
index 4482e4ff7804..3551a013208d 100644
--- a/drivers/net/wireless/mediatek/mt76/tx.c
+++ b/drivers/net/wireless/mediatek/mt76/tx.c
@@ -298,7 +298,6 @@ __mt76_tx_queue_skb(struct mt76_phy *phy, int qid, struct sk_buff *skb,
if (idx < 0 || !sta)
return idx;
- wcid = (struct mt76_wcid *)sta->drv_priv;
q->entry[idx].wcid = wcid->idx;
if (!non_aql)
diff --git a/drivers/net/wireless/microchip/wilc1000/wlan_cfg.c b/drivers/net/wireless/microchip/wilc1000/wlan_cfg.c
index 131388886acb..cfabd5aebb54 100644
--- a/drivers/net/wireless/microchip/wilc1000/wlan_cfg.c
+++ b/drivers/net/wireless/microchip/wilc1000/wlan_cfg.c
@@ -41,10 +41,10 @@ static const struct wilc_cfg_word g_cfg_word[] = {
};
static const struct wilc_cfg_str g_cfg_str[] = {
- {WID_FIRMWARE_VERSION, NULL},
- {WID_MAC_ADDR, NULL},
- {WID_ASSOC_RES_INFO, NULL},
- {WID_NIL, NULL}
+ {WID_FIRMWARE_VERSION, 0, NULL},
+ {WID_MAC_ADDR, 0, NULL},
+ {WID_ASSOC_RES_INFO, 0, NULL},
+ {WID_NIL, 0, NULL}
};
#define WILC_RESP_MSG_TYPE_CONFIG_REPLY 'R'
@@ -147,44 +147,58 @@ static void wilc_wlan_parse_response_frame(struct wilc *wl, u8 *info, int size)
switch (FIELD_GET(WILC_WID_TYPE, wid)) {
case WID_CHAR:
+ len = 3;
+ if (len + 2 > size)
+ return;
+
while (cfg->b[i].id != WID_NIL && cfg->b[i].id != wid)
i++;
if (cfg->b[i].id == wid)
cfg->b[i].val = info[4];
- len = 3;
break;
case WID_SHORT:
+ len = 4;
+ if (len + 2 > size)
+ return;
+
while (cfg->hw[i].id != WID_NIL && cfg->hw[i].id != wid)
i++;
if (cfg->hw[i].id == wid)
cfg->hw[i].val = get_unaligned_le16(&info[4]);
- len = 4;
break;
case WID_INT:
+ len = 6;
+ if (len + 2 > size)
+ return;
+
while (cfg->w[i].id != WID_NIL && cfg->w[i].id != wid)
i++;
if (cfg->w[i].id == wid)
cfg->w[i].val = get_unaligned_le32(&info[4]);
- len = 6;
break;
case WID_STR:
+ len = 2 + get_unaligned_le16(&info[2]);
+
while (cfg->s[i].id != WID_NIL && cfg->s[i].id != wid)
i++;
- if (cfg->s[i].id == wid)
+ if (cfg->s[i].id == wid) {
+ if (len > cfg->s[i].len || (len + 2 > size))
+ return;
+
memcpy(cfg->s[i].str, &info[2],
- get_unaligned_le16(&info[2]) + 2);
+ len);
+ }
- len = 2 + get_unaligned_le16(&info[2]);
break;
default:
@@ -384,12 +398,15 @@ int wilc_wlan_cfg_init(struct wilc *wl)
/* store the string cfg parameters */
wl->cfg.s[i].id = WID_FIRMWARE_VERSION;
wl->cfg.s[i].str = str_vals->firmware_version;
+ wl->cfg.s[i].len = sizeof(str_vals->firmware_version);
i++;
wl->cfg.s[i].id = WID_MAC_ADDR;
wl->cfg.s[i].str = str_vals->mac_address;
+ wl->cfg.s[i].len = sizeof(str_vals->mac_address);
i++;
wl->cfg.s[i].id = WID_ASSOC_RES_INFO;
wl->cfg.s[i].str = str_vals->assoc_rsp;
+ wl->cfg.s[i].len = sizeof(str_vals->assoc_rsp);
i++;
wl->cfg.s[i].id = WID_NIL;
wl->cfg.s[i].str = NULL;
diff --git a/drivers/net/wireless/microchip/wilc1000/wlan_cfg.h b/drivers/net/wireless/microchip/wilc1000/wlan_cfg.h
index 614c5673f232..b0987682892a 100644
--- a/drivers/net/wireless/microchip/wilc1000/wlan_cfg.h
+++ b/drivers/net/wireless/microchip/wilc1000/wlan_cfg.h
@@ -24,12 +24,13 @@ struct wilc_cfg_word {
struct wilc_cfg_str {
u16 id;
+ u16 len;
u8 *str;
};
struct wilc_cfg_str_vals {
- u8 mac_address[7];
- u8 firmware_version[129];
+ u8 mac_address[8];
+ u8 firmware_version[130];
u8 assoc_rsp[256];
};
diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c
index c1bc55f0e4c0..20a7537d2cae 100644
--- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c
+++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c
@@ -5263,14 +5263,19 @@ static void rtl8xxxu_queue_rx_urb(struct rtl8xxxu_priv *priv,
{
struct sk_buff *skb;
unsigned long flags;
- int pending = 0;
spin_lock_irqsave(&priv->rx_urb_lock, flags);
if (!priv->shutdown) {
list_add_tail(&rx_urb->list, &priv->rx_urb_pending_list);
priv->rx_urb_pending_count++;
- pending = priv->rx_urb_pending_count;
+ /*
+ * Arm the worker under rx_urb_lock so this is atomic with the
+ * shutdown check: moving it out of the lock would let a
+ * completion arm the work after rtl8xxxu_stop() canceled it.
+ */
+ if (priv->rx_urb_pending_count > RTL8XXXU_RX_URB_PENDING_WATER)
+ schedule_work(&priv->rx_urb_wq);
} else {
skb = (struct sk_buff *)rx_urb->urb.context;
dev_kfree_skb_irq(skb);
@@ -5278,9 +5283,6 @@ static void rtl8xxxu_queue_rx_urb(struct rtl8xxxu_priv *priv,
}
spin_unlock_irqrestore(&priv->rx_urb_lock, flags);
-
- if (pending > RTL8XXXU_RX_URB_PENDING_WATER)
- schedule_work(&priv->rx_urb_wq);
}
static void rtl8xxxu_rx_urb_work(struct work_struct *work)
@@ -6529,6 +6531,13 @@ static void rtl8xxxu_stop(struct ieee80211_hw *hw)
priv->shutdown = true;
spin_unlock_irqrestore(&priv->rx_urb_lock, flags);
+ /*
+ * Cancel before killing rx_anchor: the worker re-anchors every URB
+ * it drained via rtl8xxxu_submit_rx_urb(), so a worker still running
+ * after the kill could submit a URB that escapes it.
+ */
+ cancel_work_sync(&priv->rx_urb_wq);
+
usb_kill_anchored_urbs(&priv->rx_anchor);
usb_kill_anchored_urbs(&priv->tx_anchor);
if (priv->usb_interrupts)
diff --git a/drivers/net/wireless/realtek/rtw88/debug.c b/drivers/net/wireless/realtek/rtw88/debug.c
index abd750c3c28e..017783a1c2d9 100644
--- a/drivers/net/wireless/realtek/rtw88/debug.c
+++ b/drivers/net/wireless/realtek/rtw88/debug.c
@@ -43,6 +43,62 @@ struct rtw_debugfs_priv {
};
};
+struct rtw_debugfs {
+ struct rtw_debugfs_priv mac_0;
+ struct rtw_debugfs_priv mac_1;
+ struct rtw_debugfs_priv mac_2;
+ struct rtw_debugfs_priv mac_3;
+ struct rtw_debugfs_priv mac_4;
+ struct rtw_debugfs_priv mac_5;
+ struct rtw_debugfs_priv mac_6;
+ struct rtw_debugfs_priv mac_7;
+ struct rtw_debugfs_priv mac_10;
+ struct rtw_debugfs_priv mac_11;
+ struct rtw_debugfs_priv mac_12;
+ struct rtw_debugfs_priv mac_13;
+ struct rtw_debugfs_priv mac_14;
+ struct rtw_debugfs_priv mac_15;
+ struct rtw_debugfs_priv mac_16;
+ struct rtw_debugfs_priv mac_17;
+ struct rtw_debugfs_priv bb_8;
+ struct rtw_debugfs_priv bb_9;
+ struct rtw_debugfs_priv bb_a;
+ struct rtw_debugfs_priv bb_b;
+ struct rtw_debugfs_priv bb_c;
+ struct rtw_debugfs_priv bb_d;
+ struct rtw_debugfs_priv bb_e;
+ struct rtw_debugfs_priv bb_f;
+ struct rtw_debugfs_priv bb_18;
+ struct rtw_debugfs_priv bb_19;
+ struct rtw_debugfs_priv bb_1a;
+ struct rtw_debugfs_priv bb_1b;
+ struct rtw_debugfs_priv bb_1c;
+ struct rtw_debugfs_priv bb_1d;
+ struct rtw_debugfs_priv bb_1e;
+ struct rtw_debugfs_priv bb_1f;
+ struct rtw_debugfs_priv bb_2c;
+ struct rtw_debugfs_priv bb_2d;
+ struct rtw_debugfs_priv bb_40;
+ struct rtw_debugfs_priv bb_41;
+ struct rtw_debugfs_priv rf_dump;
+ struct rtw_debugfs_priv tx_pwr_tbl;
+ struct rtw_debugfs_priv write_reg;
+ struct rtw_debugfs_priv h2c;
+ struct rtw_debugfs_priv rf_write;
+ struct rtw_debugfs_priv rf_read;
+ struct rtw_debugfs_priv read_reg;
+ struct rtw_debugfs_priv fix_rate;
+ struct rtw_debugfs_priv dump_cam;
+ struct rtw_debugfs_priv rsvd_page;
+ struct rtw_debugfs_priv phy_info;
+ struct rtw_debugfs_priv coex_enable;
+ struct rtw_debugfs_priv coex_info;
+ struct rtw_debugfs_priv edcca_enable;
+ struct rtw_debugfs_priv fw_crash;
+ struct rtw_debugfs_priv force_lowest_basic_rate;
+ struct rtw_debugfs_priv dm_cap;
+};
+
static const char * const rtw_dm_cap_strs[] = {
[RTW_DM_CAP_NA] = "NA",
[RTW_DM_CAP_TXGAPK] = "TXGAPK",
@@ -512,7 +568,7 @@ static int rtw_debug_get_bb_page(struct seq_file *m, void *v)
return 0;
}
-static int rtw_debug_get_rf_dump(struct seq_file *m, void *v)
+static int rtw_debugfs_get_rf_dump(struct seq_file *m, void *v)
{
struct rtw_debugfs_priv *debugfs_priv = m->private;
struct rtw_dev *rtwdev = debugfs_priv->rtwdev;
@@ -1060,139 +1116,102 @@ static int rtw_debugfs_get_dm_cap(struct seq_file *m, void *v)
return 0;
}
-#define rtw_debug_impl_mac(page, addr) \
-static struct rtw_debugfs_priv rtw_debug_priv_mac_ ##page = { \
+#define rtw_debug_priv_mac(addr) \
+{ \
.cb_read = rtw_debug_get_mac_page, \
.cb_data = addr, \
}
-rtw_debug_impl_mac(0, 0x0000);
-rtw_debug_impl_mac(1, 0x0100);
-rtw_debug_impl_mac(2, 0x0200);
-rtw_debug_impl_mac(3, 0x0300);
-rtw_debug_impl_mac(4, 0x0400);
-rtw_debug_impl_mac(5, 0x0500);
-rtw_debug_impl_mac(6, 0x0600);
-rtw_debug_impl_mac(7, 0x0700);
-rtw_debug_impl_mac(10, 0x1000);
-rtw_debug_impl_mac(11, 0x1100);
-rtw_debug_impl_mac(12, 0x1200);
-rtw_debug_impl_mac(13, 0x1300);
-rtw_debug_impl_mac(14, 0x1400);
-rtw_debug_impl_mac(15, 0x1500);
-rtw_debug_impl_mac(16, 0x1600);
-rtw_debug_impl_mac(17, 0x1700);
-
-#define rtw_debug_impl_bb(page, addr) \
-static struct rtw_debugfs_priv rtw_debug_priv_bb_ ##page = { \
+#define rtw_debug_priv_bb(addr) \
+{ \
.cb_read = rtw_debug_get_bb_page, \
.cb_data = addr, \
}
-rtw_debug_impl_bb(8, 0x0800);
-rtw_debug_impl_bb(9, 0x0900);
-rtw_debug_impl_bb(a, 0x0a00);
-rtw_debug_impl_bb(b, 0x0b00);
-rtw_debug_impl_bb(c, 0x0c00);
-rtw_debug_impl_bb(d, 0x0d00);
-rtw_debug_impl_bb(e, 0x0e00);
-rtw_debug_impl_bb(f, 0x0f00);
-rtw_debug_impl_bb(18, 0x1800);
-rtw_debug_impl_bb(19, 0x1900);
-rtw_debug_impl_bb(1a, 0x1a00);
-rtw_debug_impl_bb(1b, 0x1b00);
-rtw_debug_impl_bb(1c, 0x1c00);
-rtw_debug_impl_bb(1d, 0x1d00);
-rtw_debug_impl_bb(1e, 0x1e00);
-rtw_debug_impl_bb(1f, 0x1f00);
-rtw_debug_impl_bb(2c, 0x2c00);
-rtw_debug_impl_bb(2d, 0x2d00);
-rtw_debug_impl_bb(40, 0x4000);
-rtw_debug_impl_bb(41, 0x4100);
-
-static struct rtw_debugfs_priv rtw_debug_priv_rf_dump = {
- .cb_read = rtw_debug_get_rf_dump,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_tx_pwr_tbl = {
- .cb_read = rtw_debugfs_get_tx_pwr_tbl,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_write_reg = {
- .cb_write = rtw_debugfs_set_write_reg,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_h2c = {
- .cb_write = rtw_debugfs_set_h2c,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_rf_write = {
- .cb_write = rtw_debugfs_set_rf_write,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_rf_read = {
- .cb_write = rtw_debugfs_set_rf_read,
- .cb_read = rtw_debugfs_get_rf_read,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_read_reg = {
- .cb_write = rtw_debugfs_set_read_reg,
- .cb_read = rtw_debugfs_get_read_reg,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_fix_rate = {
- .cb_write = rtw_debugfs_set_fix_rate,
- .cb_read = rtw_debugfs_get_fix_rate,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_dump_cam = {
- .cb_write = rtw_debugfs_set_single_input,
- .cb_read = rtw_debugfs_get_dump_cam,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_rsvd_page = {
- .cb_write = rtw_debugfs_set_rsvd_page,
- .cb_read = rtw_debugfs_get_rsvd_page,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_phy_info = {
- .cb_read = rtw_debugfs_get_phy_info,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_coex_enable = {
- .cb_write = rtw_debugfs_set_coex_enable,
- .cb_read = rtw_debugfs_get_coex_enable,
-};
-
-static struct rtw_debugfs_priv rtw_debug_priv_coex_info = {
- .cb_read = rtw_debugfs_get_coex_info,
-};
+#define rtw_debug_priv_get(name) \
+{ \
+ .cb_read = rtw_debugfs_get_ ##name, \
+}
-static struct rtw_debugfs_priv rtw_debug_priv_edcca_enable = {
- .cb_write = rtw_debugfs_set_edcca_enable,
- .cb_read = rtw_debugfs_get_edcca_enable,
-};
+#define rtw_debug_priv_set(name) \
+{ \
+ .cb_write = rtw_debugfs_set_ ##name, \
+}
-static struct rtw_debugfs_priv rtw_debug_priv_fw_crash = {
- .cb_write = rtw_debugfs_set_fw_crash,
- .cb_read = rtw_debugfs_get_fw_crash,
-};
+#define rtw_debug_priv_set_and_get(name) \
+{ \
+ .cb_write = rtw_debugfs_set_ ##name, \
+ .cb_read = rtw_debugfs_get_ ##name, \
+}
-static struct rtw_debugfs_priv rtw_debug_priv_force_lowest_basic_rate = {
- .cb_write = rtw_debugfs_set_force_lowest_basic_rate,
- .cb_read = rtw_debugfs_get_force_lowest_basic_rate,
-};
+#define rtw_debug_priv_set_single_and_get(name) \
+{ \
+ .cb_write = rtw_debugfs_set_single_input, \
+ .cb_read = rtw_debugfs_get_ ##name, \
+}
-static struct rtw_debugfs_priv rtw_debug_priv_dm_cap = {
- .cb_write = rtw_debugfs_set_dm_cap,
- .cb_read = rtw_debugfs_get_dm_cap,
+static const struct rtw_debugfs rtw_debugfs_templ = {
+ .mac_0 = rtw_debug_priv_mac(0x0000),
+ .mac_1 = rtw_debug_priv_mac(0x0100),
+ .mac_2 = rtw_debug_priv_mac(0x0200),
+ .mac_3 = rtw_debug_priv_mac(0x0300),
+ .mac_4 = rtw_debug_priv_mac(0x0400),
+ .mac_5 = rtw_debug_priv_mac(0x0500),
+ .mac_6 = rtw_debug_priv_mac(0x0600),
+ .mac_7 = rtw_debug_priv_mac(0x0700),
+ .mac_10 = rtw_debug_priv_mac(0x1000),
+ .mac_11 = rtw_debug_priv_mac(0x1100),
+ .mac_12 = rtw_debug_priv_mac(0x1200),
+ .mac_13 = rtw_debug_priv_mac(0x1300),
+ .mac_14 = rtw_debug_priv_mac(0x1400),
+ .mac_15 = rtw_debug_priv_mac(0x1500),
+ .mac_16 = rtw_debug_priv_mac(0x1600),
+ .mac_17 = rtw_debug_priv_mac(0x1700),
+ .bb_8 = rtw_debug_priv_bb(0x0800),
+ .bb_9 = rtw_debug_priv_bb(0x0900),
+ .bb_a = rtw_debug_priv_bb(0x0a00),
+ .bb_b = rtw_debug_priv_bb(0x0b00),
+ .bb_c = rtw_debug_priv_bb(0x0c00),
+ .bb_d = rtw_debug_priv_bb(0x0d00),
+ .bb_e = rtw_debug_priv_bb(0x0e00),
+ .bb_f = rtw_debug_priv_bb(0x0f00),
+ .bb_18 = rtw_debug_priv_bb(0x1800),
+ .bb_19 = rtw_debug_priv_bb(0x1900),
+ .bb_1a = rtw_debug_priv_bb(0x1a00),
+ .bb_1b = rtw_debug_priv_bb(0x1b00),
+ .bb_1c = rtw_debug_priv_bb(0x1c00),
+ .bb_1d = rtw_debug_priv_bb(0x1d00),
+ .bb_1e = rtw_debug_priv_bb(0x1e00),
+ .bb_1f = rtw_debug_priv_bb(0x1f00),
+ .bb_2c = rtw_debug_priv_bb(0x2c00),
+ .bb_2d = rtw_debug_priv_bb(0x2d00),
+ .bb_40 = rtw_debug_priv_bb(0x4000),
+ .bb_41 = rtw_debug_priv_bb(0x4100),
+ .rf_dump = rtw_debug_priv_get(rf_dump),
+ .tx_pwr_tbl = rtw_debug_priv_get(tx_pwr_tbl),
+ .write_reg = rtw_debug_priv_set(write_reg),
+ .h2c = rtw_debug_priv_set(h2c),
+ .rf_write = rtw_debug_priv_set(rf_write),
+ .rf_read = rtw_debug_priv_set_and_get(rf_read),
+ .read_reg = rtw_debug_priv_set_and_get(read_reg),
+ .fix_rate = rtw_debug_priv_set_and_get(fix_rate),
+ .dump_cam = rtw_debug_priv_set_single_and_get(dump_cam),
+ .rsvd_page = rtw_debug_priv_set_and_get(rsvd_page),
+ .phy_info = rtw_debug_priv_get(phy_info),
+ .coex_enable = rtw_debug_priv_set_and_get(coex_enable),
+ .coex_info = rtw_debug_priv_get(coex_info),
+ .edcca_enable = rtw_debug_priv_set_and_get(edcca_enable),
+ .fw_crash = rtw_debug_priv_set_and_get(fw_crash),
+ .force_lowest_basic_rate = rtw_debug_priv_set_and_get(force_lowest_basic_rate),
+ .dm_cap = rtw_debug_priv_set_and_get(dm_cap),
};
#define rtw_debugfs_add_core(name, mode, fopname, parent) \
do { \
- rtw_debug_priv_ ##name.rtwdev = rtwdev; \
+ struct rtw_debugfs_priv *priv = &rtwdev->debugfs->name; \
+ priv->rtwdev = rtwdev; \
if (IS_ERR(debugfs_create_file(#name, mode, \
- parent, &rtw_debug_priv_ ##name,\
+ parent, priv, \
&file_ops_ ##fopname))) \
pr_debug("Unable to initialize debugfs:%s\n", \
#name); \
@@ -1205,12 +1224,9 @@ static struct rtw_debugfs_priv rtw_debug_priv_dm_cap = {
#define rtw_debugfs_add_r(name) \
rtw_debugfs_add_core(name, S_IFREG | 0444, single_r, debugfs_topdir)
-void rtw_debugfs_init(struct rtw_dev *rtwdev)
+static
+void rtw_debugfs_add_basic(struct rtw_dev *rtwdev, struct dentry *debugfs_topdir)
{
- struct dentry *debugfs_topdir;
-
- debugfs_topdir = debugfs_create_dir("rtw88",
- rtwdev->hw->wiphy->debugfsdir);
rtw_debugfs_add_w(write_reg);
rtw_debugfs_add_rw(read_reg);
rtw_debugfs_add_w(rf_write);
@@ -1222,6 +1238,17 @@ void rtw_debugfs_init(struct rtw_dev *rtwdev)
rtw_debugfs_add_r(coex_info);
rtw_debugfs_add_rw(coex_enable);
rtw_debugfs_add_w(h2c);
+ rtw_debugfs_add_r(rf_dump);
+ rtw_debugfs_add_r(tx_pwr_tbl);
+ rtw_debugfs_add_rw(edcca_enable);
+ rtw_debugfs_add_rw(fw_crash);
+ rtw_debugfs_add_rw(force_lowest_basic_rate);
+ rtw_debugfs_add_rw(dm_cap);
+}
+
+static
+void rtw_debugfs_add_sec0(struct rtw_dev *rtwdev, struct dentry *debugfs_topdir)
+{
rtw_debugfs_add_r(mac_0);
rtw_debugfs_add_r(mac_1);
rtw_debugfs_add_r(mac_2);
@@ -1238,6 +1265,11 @@ void rtw_debugfs_init(struct rtw_dev *rtwdev)
rtw_debugfs_add_r(bb_d);
rtw_debugfs_add_r(bb_e);
rtw_debugfs_add_r(bb_f);
+}
+
+static
+void rtw_debugfs_add_sec1(struct rtw_dev *rtwdev, struct dentry *debugfs_topdir)
+{
rtw_debugfs_add_r(mac_10);
rtw_debugfs_add_r(mac_11);
rtw_debugfs_add_r(mac_12);
@@ -1260,14 +1292,29 @@ void rtw_debugfs_init(struct rtw_dev *rtwdev)
rtw_debugfs_add_r(bb_40);
rtw_debugfs_add_r(bb_41);
}
- rtw_debugfs_add_r(rf_dump);
- rtw_debugfs_add_r(tx_pwr_tbl);
- rtw_debugfs_add_rw(edcca_enable);
- rtw_debugfs_add_rw(fw_crash);
- rtw_debugfs_add_rw(force_lowest_basic_rate);
- rtw_debugfs_add_rw(dm_cap);
}
+void rtw_debugfs_init(struct rtw_dev *rtwdev)
+{
+ struct dentry *debugfs_topdir;
+
+ rtwdev->debugfs = kmemdup(&rtw_debugfs_templ, sizeof(rtw_debugfs_templ),
+ GFP_KERNEL);
+ if (!rtwdev->debugfs)
+ return;
+
+ debugfs_topdir = debugfs_create_dir("rtw88",
+ rtwdev->hw->wiphy->debugfsdir);
+
+ rtw_debugfs_add_basic(rtwdev, debugfs_topdir);
+ rtw_debugfs_add_sec0(rtwdev, debugfs_topdir);
+ rtw_debugfs_add_sec1(rtwdev, debugfs_topdir);
+}
+
+void rtw_debugfs_deinit(struct rtw_dev *rtwdev)
+{
+ kfree(rtwdev->debugfs);
+}
#endif /* CONFIG_RTW88_DEBUGFS */
#ifdef CONFIG_RTW88_DEBUG
diff --git a/drivers/net/wireless/realtek/rtw88/debug.h b/drivers/net/wireless/realtek/rtw88/debug.h
index 066792dd96af..3db8d0bd0ed9 100644
--- a/drivers/net/wireless/realtek/rtw88/debug.h
+++ b/drivers/net/wireless/realtek/rtw88/debug.h
@@ -31,11 +31,13 @@ enum rtw_debug_mask {
#ifdef CONFIG_RTW88_DEBUGFS
void rtw_debugfs_init(struct rtw_dev *rtwdev);
+void rtw_debugfs_deinit(struct rtw_dev *rtwdev);
void rtw_debugfs_get_simple_phy_info(struct seq_file *m);
#else
static inline void rtw_debugfs_init(struct rtw_dev *rtwdev) {}
+static inline void rtw_debugfs_deinit(struct rtw_dev *rtwdev) {}
#endif /* CONFIG_RTW88_DEBUGFS */
diff --git a/drivers/net/wireless/realtek/rtw88/main.c b/drivers/net/wireless/realtek/rtw88/main.c
index 53c61758d57a..32459c6ebecd 100644
--- a/drivers/net/wireless/realtek/rtw88/main.c
+++ b/drivers/net/wireless/realtek/rtw88/main.c
@@ -1605,11 +1605,13 @@ static u16 rtw_get_max_scan_ie_len(struct rtw_dev *rtwdev)
static void rtw_set_supported_band(struct ieee80211_hw *hw,
const struct rtw_chip_info *chip)
{
- struct rtw_dev *rtwdev = hw->priv;
struct ieee80211_supported_band *sband;
+ struct rtw_dev *rtwdev = hw->priv;
+ struct device *dev = rtwdev->dev;
if (chip->band & RTW_BAND_2G) {
- sband = kmemdup(&rtw_band_2ghz, sizeof(*sband), GFP_KERNEL);
+ sband = devm_kmemdup(dev, &rtw_band_2ghz, sizeof(*sband),
+ GFP_KERNEL);
if (!sband)
goto err_out;
if (chip->ht_supported)
@@ -1618,7 +1620,8 @@ static void rtw_set_supported_band(struct ieee80211_hw *hw,
}
if (chip->band & RTW_BAND_5G) {
- sband = kmemdup(&rtw_band_5ghz, sizeof(*sband), GFP_KERNEL);
+ sband = devm_kmemdup(dev, &rtw_band_5ghz, sizeof(*sband),
+ GFP_KERNEL);
if (!sband)
goto err_out;
if (chip->ht_supported)
@@ -1634,13 +1637,6 @@ static void rtw_set_supported_band(struct ieee80211_hw *hw,
rtw_err(rtwdev, "failed to set supported band\n");
}
-static void rtw_unset_supported_band(struct ieee80211_hw *hw,
- const struct rtw_chip_info *chip)
-{
- kfree(hw->wiphy->bands[NL80211_BAND_2GHZ]);
- kfree(hw->wiphy->bands[NL80211_BAND_5GHZ]);
-}
-
static void rtw_vif_smps_iter(void *data, u8 *mac,
struct ieee80211_vif *vif)
{
@@ -2238,10 +2234,8 @@ EXPORT_SYMBOL(rtw_register_hw);
void rtw_unregister_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw)
{
- const struct rtw_chip_info *chip = rtwdev->chip;
-
ieee80211_unregister_hw(hw);
- rtw_unset_supported_band(hw, chip);
+ rtw_debugfs_deinit(rtwdev);
}
EXPORT_SYMBOL(rtw_unregister_hw);
diff --git a/drivers/net/wireless/realtek/rtw88/main.h b/drivers/net/wireless/realtek/rtw88/main.h
index 2976e3fd7288..8d84ffcc29e2 100644
--- a/drivers/net/wireless/realtek/rtw88/main.h
+++ b/drivers/net/wireless/realtek/rtw88/main.h
@@ -50,6 +50,7 @@ extern const struct ieee80211_ops rtw_ops;
#define RTW_MAX_CHANNEL_NUM_5G 49
struct rtw_dev;
+struct rtw_debugfs;
enum rtw_hci_type {
RTW_HCI_TYPE_PCIE,
@@ -2036,7 +2037,7 @@ struct rtw_dev {
bool beacon_loss;
struct completion lps_leave_check;
- struct dentry *debugfs;
+ struct rtw_debugfs *debugfs;
u8 sta_cnt;
u32 rts_threshold;
diff --git a/drivers/net/wireless/realtek/rtw89/core.c b/drivers/net/wireless/realtek/rtw89/core.c
index ff3a31bf3a87..ec9712b74876 100644
--- a/drivers/net/wireless/realtek/rtw89/core.c
+++ b/drivers/net/wireless/realtek/rtw89/core.c
@@ -1258,14 +1258,18 @@ static int rtw89_core_rx_parse_phy_sts(struct rtw89_dev *rtwdev,
pos = (u8 *)phy_ppdu->buf + PHY_STS_HDR_LEN;
end = (u8 *)phy_ppdu->buf + phy_ppdu->len;
while (pos < end) {
- ie_len = rtw89_core_get_phy_status_ie_len(rtwdev, pos);
- rtw89_core_process_phy_status_ie(rtwdev, pos, phy_ppdu);
+ u8 *iehdr = pos;
+
+ ie_len = rtw89_core_get_phy_status_ie_len(rtwdev, iehdr);
pos += ie_len;
if (pos > end || ie_len == 0) {
rtw89_debug(rtwdev, RTW89_DBG_TXRX,
"phy status parse failed\n");
+
return -EINVAL;
}
+
+ rtw89_core_process_phy_status_ie(rtwdev, iehdr, phy_ppdu);
}
return 0;
diff --git a/drivers/net/wireless/realtek/rtw89/mac80211.c b/drivers/net/wireless/realtek/rtw89/mac80211.c
index f7880499aeb0..7827f90d5bdc 100644
--- a/drivers/net/wireless/realtek/rtw89/mac80211.c
+++ b/drivers/net/wireless/realtek/rtw89/mac80211.c
@@ -383,7 +383,7 @@ static void rtw89_ops_bss_info_changed(struct ieee80211_hw *hw,
* when disconnected by peer
*/
if (rtwdev->scanning)
- rtw89_hw_scan_abort(rtwdev, vif);
+ rtw89_hw_scan_abort(rtwdev, rtwdev->scan_info.scanning_vif);
}
}
@@ -808,11 +808,14 @@ static void rtw89_ops_cancel_hw_scan(struct ieee80211_hw *hw,
if (!RTW89_CHK_FW_FEATURE(SCAN_OFFLOAD, &rtwdev->fw))
return;
+ mutex_lock(&rtwdev->mutex);
+
if (!rtwdev->scanning)
- return;
+ goto out;
- mutex_lock(&rtwdev->mutex);
rtw89_hw_scan_abort(rtwdev, vif);
+
+out:
mutex_unlock(&rtwdev->mutex);
}
diff --git a/drivers/net/wireless/realtek/rtw89/pci.c b/drivers/net/wireless/realtek/rtw89/pci.c
index ac76697dc462..11634b6d4014 100644
--- a/drivers/net/wireless/realtek/rtw89/pci.c
+++ b/drivers/net/wireless/realtek/rtw89/pci.c
@@ -472,11 +472,16 @@ static void rtw89_pci_release_rpp(struct rtw89_dev *rtwdev,
tx_status = le32_get_bits(rpp->dword, RTW89_PCI_RPP_TX_STATUS);
txch = rtw89_core_get_ch_dma(rtwdev, qsel);
- if (txch == RTW89_TXCH_CH12) {
+ if (unlikely(txch == RTW89_TXCH_CH12)) {
rtw89_warn(rtwdev, "should no fwcmd release report\n");
return;
}
+ if (unlikely(seq >= RTW89_PCI_TXWD_NUM_MAX)) {
+ rtw89_warn(rtwdev, "invalid seq %d\n", seq);
+ return;
+ }
+
tx_ring = &rtwpci->tx_rings[txch];
wd_ring = &tx_ring->wd_ring;
txwd = &wd_ring->pages[seq];
diff --git a/drivers/net/wireless/silabs/wfx/hif_tx.c b/drivers/net/wireless/silabs/wfx/hif_tx.c
index 9402503fbde3..852406fc7f3c 100644
--- a/drivers/net/wireless/silabs/wfx/hif_tx.c
+++ b/drivers/net/wireless/silabs/wfx/hif_tx.c
@@ -189,9 +189,11 @@ int wfx_hif_read_mib(struct wfx_dev *wdev, int vif_id, u16 mib_id, void *val, si
dev_warn(wdev->dev, "%s: confirmation mismatch request\n", __func__);
ret = -EIO;
}
- if (ret == -ENOMEM)
+ if (!ret && le16_to_cpu(reply->length) > val_len) {
dev_err(wdev->dev, "buffer is too small to receive %s (%zu < %d)\n",
wfx_get_mib_name(mib_id), val_len, le16_to_cpu(reply->length));
+ ret = -EIO;
+ }
if (!ret)
memcpy(val, &reply->mib_data, le16_to_cpu(reply->length));
else
diff --git a/drivers/net/wireless/st/cw1200/bh.c b/drivers/net/wireless/st/cw1200/bh.c
index 37232ee22037..91d4287f5e4a 100644
--- a/drivers/net/wireless/st/cw1200/bh.c
+++ b/drivers/net/wireless/st/cw1200/bh.c
@@ -343,6 +343,7 @@ static int cw1200_bh_tx_helper(struct cw1200_common *priv,
int *tx_burst)
{
size_t tx_len;
+ size_t aligned_len;
u8 *data;
int ret;
struct wsm_hdr *wsm;
@@ -375,9 +376,14 @@ static int cw1200_bh_tx_helper(struct cw1200_common *priv,
atomic_inc(&priv->bh_tx);
- tx_len = priv->hwbus_ops->align_size(
+ aligned_len = priv->hwbus_ops->align_size(
priv->hwbus_priv, tx_len);
+ /* Zero the bus alignment padding so no stale data is sent. */
+ if (aligned_len > tx_len)
+ memset(data + tx_len, 0, aligned_len - tx_len);
+ tx_len = aligned_len;
+
/* Check if not exceeding CW1200 capabilities */
if (WARN_ON_ONCE(tx_len > EFFECTIVE_BUF_SIZE))
pr_debug("Write aligned len: %zu\n", tx_len);
diff --git a/drivers/net/wireless/st/cw1200/txrx.c b/drivers/net/wireless/st/cw1200/txrx.c
index e16e9ae90d20..413193aac2e0 100644
--- a/drivers/net/wireless/st/cw1200/txrx.c
+++ b/drivers/net/wireless/st/cw1200/txrx.c
@@ -14,6 +14,7 @@
#include "wsm.h"
#include "bh.h"
#include "sta.h"
+#include "hwbus.h"
#include "debug.h"
#define CW1200_INVALID_RATE_ID (0xFF)
@@ -716,6 +717,7 @@ void cw1200_tx(struct ieee80211_hw *dev,
struct ieee80211_sta *sta;
struct wsm_tx *wsm;
bool tid_update = false;
+ size_t pad_len;
u8 flags = 0;
int ret;
@@ -764,6 +766,18 @@ void cw1200_tx(struct ieee80211_hw *dev,
sta = t.sta;
+ /*
+ * The bus transfer is padded up to the bus alignment and the
+ * padding is transferred from the frame buffer. Make sure the
+ * buffer has room for the padding so the device never receives
+ * data past the end of the frame.
+ */
+ pad_len = priv->hwbus_ops->align_size(priv->hwbus_priv, skb->len) -
+ skb->len;
+ if (skb_tailroom(skb) < pad_len &&
+ pskb_expand_head(skb, 0, pad_len, GFP_ATOMIC))
+ goto drop;
+
spin_lock_bh(&priv->ps_state_lock);
{
tid_update = cw1200_tx_h_pm_state(priv, &t);
diff --git a/drivers/net/wireless/ti/wl12xx/main.c b/drivers/net/wireless/ti/wl12xx/main.c
index d06a2c419447..de045fe4ca1e 100644
--- a/drivers/net/wireless/ti/wl12xx/main.c
+++ b/drivers/net/wireless/ti/wl12xx/main.c
@@ -1919,7 +1919,7 @@ static int wl12xx_probe(struct platform_device *pdev)
return ret;
}
-static int wl12xx_remove(struct platform_device *pdev)
+static void wl12xx_remove(struct platform_device *pdev)
{
struct wl1271 *wl = platform_get_drvdata(pdev);
struct wl12xx_priv *priv;
@@ -1928,7 +1928,7 @@ static int wl12xx_remove(struct platform_device *pdev)
kfree(priv->rx_mem_addr);
- return wlcore_remove(pdev);
+ wlcore_remove(pdev);
}
static const struct platform_device_id wl12xx_id_table[] = {
@@ -1939,7 +1939,7 @@ MODULE_DEVICE_TABLE(platform, wl12xx_id_table);
static struct platform_driver wl12xx_driver = {
.probe = wl12xx_probe,
- .remove = wl12xx_remove,
+ .remove_new = wl12xx_remove,
.id_table = wl12xx_id_table,
.driver = {
.name = "wl12xx_driver",
diff --git a/drivers/net/wireless/ti/wl18xx/main.c b/drivers/net/wireless/ti/wl18xx/main.c
index 0b3cf8477c6c..d4a89401f2c4 100644
--- a/drivers/net/wireless/ti/wl18xx/main.c
+++ b/drivers/net/wireless/ti/wl18xx/main.c
@@ -2033,7 +2033,7 @@ MODULE_DEVICE_TABLE(platform, wl18xx_id_table);
static struct platform_driver wl18xx_driver = {
.probe = wl18xx_probe,
- .remove = wlcore_remove,
+ .remove_new = wlcore_remove,
.id_table = wl18xx_id_table,
.driver = {
.name = "wl18xx_driver",
diff --git a/drivers/net/wireless/ti/wlcore/main.c b/drivers/net/wireless/ti/wlcore/main.c
index 2c3f7fbd0433..ec3eaf788abf 100644
--- a/drivers/net/wireless/ti/wlcore/main.c
+++ b/drivers/net/wireless/ti/wlcore/main.c
@@ -6734,7 +6734,7 @@ int wlcore_probe(struct wl1271 *wl, struct platform_device *pdev)
}
EXPORT_SYMBOL_GPL(wlcore_probe);
-int wlcore_remove(struct platform_device *pdev)
+void wlcore_remove(struct platform_device *pdev)
{
struct wlcore_platdev_data *pdev_data = dev_get_platdata(&pdev->dev);
struct wl1271 *wl = platform_get_drvdata(pdev);
@@ -6748,8 +6748,10 @@ int wlcore_remove(struct platform_device *pdev)
if (pdev_data->family && pdev_data->family->nvs_name)
wait_for_completion(&wl->nvs_loading_complete);
- if (!wl->initialized)
- return 0;
+ if (!wl->initialized) {
+ pm_runtime_put_noidle(wl->dev);
+ return;
+ }
if (wl->wakeirq >= 0) {
dev_pm_clear_wake_irq(wl->dev);
@@ -6769,8 +6771,6 @@ int wlcore_remove(struct platform_device *pdev)
free_irq(wl->irq, wl);
wlcore_free_hw(wl);
-
- return 0;
}
EXPORT_SYMBOL_GPL(wlcore_remove);
diff --git a/drivers/net/wireless/ti/wlcore/wlcore.h b/drivers/net/wireless/ti/wlcore/wlcore.h
index 81c94d390623..1f8511bf9bb3 100644
--- a/drivers/net/wireless/ti/wlcore/wlcore.h
+++ b/drivers/net/wireless/ti/wlcore/wlcore.h
@@ -497,7 +497,7 @@ struct wl1271 {
};
int wlcore_probe(struct wl1271 *wl, struct platform_device *pdev);
-int wlcore_remove(struct platform_device *pdev);
+void wlcore_remove(struct platform_device *pdev);
struct ieee80211_hw *wlcore_alloc_hw(size_t priv_size, u32 aggr_buf_size,
u32 mbox_size);
int wlcore_free_hw(struct wl1271 *wl);
diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c
index dd40e0d98d7e..779a594b8dc9 100644
--- a/drivers/net/xen-netfront.c
+++ b/drivers/net/xen-netfront.c
@@ -1233,8 +1233,16 @@ static int handle_incoming_queue(struct netfront_queue *queue,
while ((skb = __skb_dequeue(rxq)) != NULL) {
int pull_to = NETFRONT_SKB_CB(skb)->pull_to;
- if (pull_to > skb_headlen(skb))
- __pskb_pull_tail(skb, pull_to - skb_headlen(skb));
+ /* pull_to comes from the first slot's length, which the
+ * backend controls. Make sure the head holds at least an
+ * Ethernet header for eth_type_trans().
+ */
+ if (!pskb_may_pull(skb, max(pull_to, ETH_HLEN))) {
+ kfree_skb(skb);
+ packets_dropped++;
+ queue->info->netdev->stats.rx_errors++;
+ continue;
+ }
/* Ethernet work: Delayed to here as it peeks the header. */
skb->protocol = eth_type_trans(skb, queue->info->netdev);
@@ -1337,8 +1345,10 @@ static int xennet_poll(struct napi_struct *napi, int budget)
skb->data_len = rx->status;
skb->len += rx->status;
- if (unlikely(xennet_fill_frags(queue, skb, &tmpq)))
+ if (unlikely(xennet_fill_frags(queue, skb, &tmpq))) {
+ __skb_queue_head(&tmpq, skb);
goto err;
+ }
if (rx->flags & XEN_NETRXF_csum_blank)
skb->ip_summed = CHECKSUM_PARTIAL;
diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c
index 6cd884d1d7eb..a0143aafa73b 100644
--- a/drivers/ntb/ntb_transport.c
+++ b/drivers/ntb/ntb_transport.c
@@ -211,6 +211,7 @@ struct ntb_transport_mw {
void *alloc_addr;
void *virt_addr;
dma_addr_t dma_addr;
+ dma_addr_t original_dma_addr;
};
struct ntb_transport_client_dev {
@@ -791,14 +792,14 @@ static void ntb_transport_msi_desc_changed(void *data)
static void ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw)
{
struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
- struct pci_dev *pdev = nt->ndev->pdev;
+ struct device *dma_dev = ntb_get_dma_dev(nt->ndev);
if (!mw->virt_addr)
return;
ntb_mw_clear_trans(nt->ndev, PIDX, num_mw);
- dma_free_coherent(&pdev->dev, mw->alloc_size,
- mw->alloc_addr, mw->dma_addr);
+ dma_free_coherent(dma_dev, mw->alloc_size,
+ mw->alloc_addr, mw->original_dma_addr);
mw->xlat_size = 0;
mw->buff_size = 0;
mw->alloc_size = 0;
@@ -834,6 +835,7 @@ static int ntb_alloc_mw_buffer(struct ntb_transport_mw *mw,
return -ENOMEM;
}
virt_addr = alloc_addr;
+ mw->original_dma_addr = dma_addr;
/*
* we must ensure that the memory address allocated is BAR size
@@ -867,7 +869,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
resource_size_t size)
{
struct ntb_transport_mw *mw = &nt->mw_vec[num_mw];
- struct pci_dev *pdev = nt->ndev->pdev;
+ struct device *dma_dev = ntb_get_dma_dev(nt->ndev);
size_t xlat_size, buff_size;
resource_size_t xlat_align;
resource_size_t xlat_align_size;
@@ -896,12 +898,12 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
mw->buff_size = buff_size;
mw->alloc_size = buff_size;
- rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align);
+ rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align);
if (rc) {
mw->alloc_size *= 2;
- rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align);
+ rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align);
if (rc) {
- dev_err(&pdev->dev,
+ dev_err(dma_dev,
"Unable to alloc aligned MW buff\n");
mw->xlat_size = 0;
mw->buff_size = 0;
@@ -914,7 +916,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw,
rc = ntb_mw_set_trans(nt->ndev, PIDX, num_mw, mw->dma_addr,
mw->xlat_size);
if (rc) {
- dev_err(&pdev->dev, "Unable to set mw%d translation", num_mw);
+ dev_err(dma_dev, "Unable to set mw%d translation", num_mw);
ntb_free_mw(nt, num_mw);
return -EIO;
}
diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c
index 90b0fdb72024..5bde34e7a143 100644
--- a/drivers/nvdimm/nd_virtio.c
+++ b/drivers/nvdimm/nd_virtio.c
@@ -9,6 +9,14 @@
#include "virtio_pmem.h"
#include "nd.h"
+static void virtio_pmem_req_release(struct kref *kref)
+{
+ struct virtio_pmem_request *req;
+
+ req = container_of(kref, struct virtio_pmem_request, kref);
+ kfree(req);
+}
+
/* The interrupt handler */
void virtio_pmem_host_ack(struct virtqueue *vq)
{
@@ -29,6 +37,7 @@ void virtio_pmem_host_ack(struct virtqueue *vq)
wake_up(&req_buf->wq_buf);
list_del(&req_buf->list);
}
+ kref_put(&req_data->kref, virtio_pmem_req_release);
}
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
}
@@ -59,6 +68,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
if (!req_data)
return -ENOMEM;
+ kref_init(&req_data->kref);
req_data->done = false;
init_waitqueue_head(&req_data->host_acked);
init_waitqueue_head(&req_data->wq_buf);
@@ -76,10 +86,23 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
* to req_list and wait for host_ack to wake us up when free
* slots are available.
*/
- while ((err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
- GFP_ATOMIC)) == -ENOSPC) {
+ for (;;) {
+ err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data,
+ GFP_ATOMIC);
+ if (!err) {
+ /*
+ * Take the virtqueue reference while @pmem_lock is
+ * held so completion cannot run concurrently.
+ */
+ kref_get(&req_data->kref);
+ break;
+ }
+
+ if (err != -ENOSPC)
+ break;
- dev_info(&vdev->dev, "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
+ dev_info_ratelimited(&vdev->dev,
+ "failed to send command to virtio pmem device, no free slots in the virtqueue\n");
req_data->wq_buf_avail = false;
list_add_tail(&req_data->list, &vpmem->req_list);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
@@ -88,6 +111,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
wait_event(req_data->wq_buf, req_data->wq_buf_avail);
spin_lock_irqsave(&vpmem->pmem_lock, flags);
}
+
err1 = virtqueue_kick(vpmem->req_vq);
spin_unlock_irqrestore(&vpmem->pmem_lock, flags);
/*
@@ -103,7 +127,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region)
err = le32_to_cpu(req_data->resp.ret);
}
- kfree(req_data);
+ kref_put(&req_data->kref, virtio_pmem_req_release);
return err;
};
diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h
index f72cf17f9518..1017e498c9b4 100644
--- a/drivers/nvdimm/virtio_pmem.h
+++ b/drivers/nvdimm/virtio_pmem.h
@@ -12,11 +12,13 @@
#include <linux/module.h>
#include <uapi/linux/virtio_pmem.h>
+#include <linux/kref.h>
#include <linux/libnvdimm.h>
#include <linux/mutex.h>
#include <linux/spinlock.h>
struct virtio_pmem_request {
+ struct kref kref;
struct virtio_pmem_req req;
struct virtio_pmem_resp resp;
diff --git a/drivers/nvme/host/apple.c b/drivers/nvme/host/apple.c
index 994cf755de09..2d8904505b5a 100644
--- a/drivers/nvme/host/apple.c
+++ b/drivers/nvme/host/apple.c
@@ -1387,7 +1387,7 @@ static void devm_apple_nvme_mempool_destroy(void *data)
mempool_destroy(data);
}
-static int apple_nvme_probe(struct platform_device *pdev)
+static struct apple_nvme *apple_nvme_alloc(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct apple_nvme *anv;
@@ -1395,7 +1395,7 @@ static int apple_nvme_probe(struct platform_device *pdev)
anv = devm_kzalloc(dev, sizeof(*anv), GFP_KERNEL);
if (!anv)
- return -ENOMEM;
+ return ERR_PTR(-ENOMEM);
anv->dev = get_device(dev);
anv->adminq.is_adminq = true;
@@ -1515,10 +1515,26 @@ static int apple_nvme_probe(struct platform_device *pdev)
goto put_dev;
}
+ return anv;
+put_dev:
+ put_device(anv->dev);
+ return ERR_PTR(ret);
+}
+
+static int apple_nvme_probe(struct platform_device *pdev)
+{
+ struct apple_nvme *anv;
+ int ret;
+
+ anv = apple_nvme_alloc(pdev);
+ if (IS_ERR(anv))
+ return PTR_ERR(anv);
+
anv->ctrl.admin_q = blk_mq_init_queue(&anv->admin_tagset);
if (IS_ERR(anv->ctrl.admin_q)) {
ret = -ENOMEM;
- goto put_dev;
+ anv->ctrl.admin_q = NULL;
+ goto out_uninit_ctrl;
}
if (!blk_get_queue(anv->ctrl.admin_q)) {
@@ -1527,7 +1543,7 @@ static int apple_nvme_probe(struct platform_device *pdev)
blk_put_queue(anv->ctrl.admin_q);
anv->ctrl.admin_q = NULL;
ret = -ENODEV;
- goto put_dev;
+ goto out_uninit_ctrl;
}
nvme_reset_ctrl(&anv->ctrl);
@@ -1535,8 +1551,9 @@ static int apple_nvme_probe(struct platform_device *pdev)
return 0;
-put_dev:
- put_device(anv->dev);
+out_uninit_ctrl:
+ nvme_uninit_ctrl(&anv->ctrl);
+ nvme_put_ctrl(&anv->ctrl);
return ret;
}
@@ -1549,6 +1566,15 @@ static int apple_nvme_remove(struct platform_device *pdev)
nvme_stop_ctrl(&anv->ctrl);
nvme_remove_namespaces(&anv->ctrl);
apple_nvme_disable(anv, true);
+ if (anv->ctrl.admin_q && !blk_queue_dying(anv->ctrl.admin_q)) {
+ /*
+ * If the controller was reset during removal, it's possible
+ * user requests may be waiting on a stopped queue. Start the
+ * queue to flush these to completion.
+ */
+ nvme_start_admin_queue(&anv->ctrl);
+ blk_mq_destroy_queue(anv->ctrl.admin_q);
+ }
nvme_uninit_ctrl(&anv->ctrl);
if (apple_rtkit_is_running(anv->rtk))
diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c
index 66600341bd6a..1faef93ce4ba 100644
--- a/drivers/nvme/host/core.c
+++ b/drivers/nvme/host/core.c
@@ -4437,6 +4437,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info)
if (list_empty(&ns->head->list))
list_del_init(&ns->head->entry);
mutex_unlock(&ctrl->subsys->lock);
+
+ /* guarantee not available in head->list */
+ synchronize_srcu(&ns->head->srcu);
nvme_put_ns_head(ns->head);
out_cleanup_disk:
put_disk(disk);
diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c
index ffe433c47089..c4d58e5f3ea7 100644
--- a/drivers/nvme/host/fabrics.c
+++ b/drivers/nvme/host/fabrics.c
@@ -153,7 +153,7 @@ int nvmf_reg_read32(struct nvme_ctrl *ctrl, u32 off, u32 *val)
cmd.prop_get.offset = cpu_to_le32(off);
ret = __nvme_submit_sync_cmd(ctrl->fabrics_q, &cmd, &res, NULL, 0,
- NVME_QID_ANY, 0, 0);
+ NVME_QID_ANY, 0, BLK_MQ_REQ_RESERVED);
if (ret >= 0)
*val = le64_to_cpu(res.u64);
@@ -199,7 +199,7 @@ int nvmf_reg_read64(struct nvme_ctrl *ctrl, u32 off, u64 *val)
cmd.prop_get.offset = cpu_to_le32(off);
ret = __nvme_submit_sync_cmd(ctrl->fabrics_q, &cmd, &res, NULL, 0,
- NVME_QID_ANY, 0, 0);
+ NVME_QID_ANY, 0, BLK_MQ_REQ_RESERVED);
if (ret >= 0)
*val = le64_to_cpu(res.u64);
@@ -244,7 +244,7 @@ int nvmf_reg_write32(struct nvme_ctrl *ctrl, u32 off, u32 val)
cmd.prop_set.value = cpu_to_le64(val);
ret = __nvme_submit_sync_cmd(ctrl->fabrics_q, &cmd, NULL, NULL, 0,
- NVME_QID_ANY, 0, 0);
+ NVME_QID_ANY, 0, BLK_MQ_REQ_RESERVED);
if (unlikely(ret))
dev_err(ctrl->device,
"Property Set error: %d, offset %#x\n",
@@ -898,6 +898,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
}
if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
pr_err("Invalid DH-CHAP secret %s\n", p);
+ kfree_sensitive(p);
ret = -EINVAL;
goto out;
}
@@ -912,6 +913,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts,
}
if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) {
pr_err("Invalid DH-CHAP secret %s\n", p);
+ kfree_sensitive(p);
ret = -EINVAL;
goto out;
}
diff --git a/drivers/nvme/host/fc.c b/drivers/nvme/host/fc.c
index 7fc35b16f629..914e48040855 100644
--- a/drivers/nvme/host/fc.c
+++ b/drivers/nvme/host/fc.c
@@ -2492,7 +2492,7 @@ __nvme_fc_abort_outstanding_ios(struct nvme_fc_ctrl *ctrl, bool start_queues)
* io requests back to the block layer as part of normal completions
* (but with error status).
*/
- if (ctrl->ctrl.queue_count > 1) {
+ if (ctrl->ctrl.queue_count > 1 && ctrl->ctrl.tagset) {
nvme_stop_queues(&ctrl->ctrl);
nvme_sync_io_queues(&ctrl->ctrl);
blk_mq_tagset_busy_iter(&ctrl->tag_set,
@@ -2925,6 +2925,11 @@ nvme_fc_create_io_queues(struct nvme_fc_ctrl *ctrl)
out_delete_hw_queues:
nvme_fc_delete_hw_io_queues(ctrl);
out_cleanup_tagset:
+ /*
+ * In CONNECTING state ctrl->ioerr_work will abort both admin
+ * and io tagsets. Cancel it first before removing io tagset.
+ */
+ cancel_work_sync(&ctrl->ioerr_work);
nvme_remove_io_tag_set(&ctrl->ctrl);
nvme_fc_free_io_queues(ctrl);
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 0a9de2d65a61..3ed604c8e9ca 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -616,7 +616,8 @@ static int nvme_parse_ana_log(struct nvme_ctrl *ctrl, void *data,
u32 nr_nsids;
size_t nsid_buf_size;
- if (WARN_ON_ONCE(offset > ctrl->ana_log_size - sizeof(*desc)))
+ if (WARN_ON_ONCE(offset > ctrl->ana_log_size ||
+ sizeof(*desc) > ctrl->ana_log_size - offset))
return -EINVAL;
nr_nsids = le32_to_cpu(desc->nnsids);
@@ -632,7 +633,7 @@ static int nvme_parse_ana_log(struct nvme_ctrl *ctrl, void *data,
return -EINVAL;
offset += sizeof(*desc);
- if (WARN_ON_ONCE(offset > ctrl->ana_log_size - nsid_buf_size))
+ if (WARN_ON_ONCE(nsid_buf_size > ctrl->ana_log_size - offset))
return -EINVAL;
error = cb(ctrl, desc, data);
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index a12d3ad54a1b..0ed96b0ecb9e 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -2620,11 +2620,23 @@ static void nvme_pci_alloc_tag_set(struct nvme_dev *dev)
dev->ctrl.tagset = set;
}
-static void nvme_pci_update_nr_queues(struct nvme_dev *dev)
+static bool nvme_pci_update_nr_queues(struct nvme_dev *dev)
{
+ /* Give up if we are racing with nvme_dev_disable() */
+ if (!mutex_trylock(&dev->shutdown_lock))
+ return false;
+
+ /* Check if nvme_dev_disable() has been executed already */
+ if (!dev->online_queues) {
+ mutex_unlock(&dev->shutdown_lock);
+ return false;
+ }
+
blk_mq_update_nr_hw_queues(&dev->tagset, dev->online_queues - 1);
/* free previously allocated queues that are no longer usable */
nvme_free_queues(dev, dev->online_queues);
+ mutex_unlock(&dev->shutdown_lock);
+ return true;
}
static int nvme_pci_enable(struct nvme_dev *dev)
@@ -2986,8 +2998,8 @@ static void nvme_reset_work(struct work_struct *work)
nvme_wait_freeze(&dev->ctrl);
if (!dev->ctrl.tagset)
nvme_pci_alloc_tag_set(dev);
- else
- nvme_pci_update_nr_queues(dev);
+ else if (!nvme_pci_update_nr_queues(dev))
+ goto out;
nvme_dbbuf_set(dev);
nvme_unfreeze(&dev->ctrl);
}
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index ac76f1ac1709..77a4727d503d 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -1120,13 +1120,7 @@ static int nvme_rdma_setup_ctrl(struct nvme_rdma_ctrl *ctrl, bool new)
nvme_rdma_free_io_queues(ctrl);
}
destroy_admin:
- nvme_stop_admin_queue(&ctrl->ctrl);
- blk_sync_queue(ctrl->ctrl.admin_q);
- nvme_rdma_stop_queue(&ctrl->queues[0]);
- nvme_cancel_admin_tagset(&ctrl->ctrl);
- if (new)
- nvme_remove_admin_tag_set(&ctrl->ctrl);
- nvme_rdma_destroy_admin_queue(ctrl);
+ nvme_rdma_teardown_admin_queue(ctrl, new);
return ret;
}
diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c
index e139f567a853..996940c972cd 100644
--- a/drivers/nvme/host/tcp.c
+++ b/drivers/nvme/host/tcp.c
@@ -189,6 +189,19 @@ static inline int nvme_tcp_queue_id(struct nvme_tcp_queue *queue)
return queue - queue->ctrl->queues;
}
+static inline bool nvme_tcp_recv_pdu_supported(enum nvme_tcp_pdu_type type)
+{
+ switch (type) {
+ case nvme_tcp_c2h_term:
+ case nvme_tcp_c2h_data:
+ case nvme_tcp_r2t:
+ case nvme_tcp_rsp:
+ return true;
+ default:
+ return false;
+ }
+}
+
static inline struct blk_mq_tags *nvme_tcp_tagset(struct nvme_tcp_queue *queue)
{
u32 queue_idx = nvme_tcp_queue_id(queue);
@@ -792,6 +805,16 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
return 0;
hdr = queue->pdu;
+ if (unlikely(hdr->hlen != sizeof(struct nvme_tcp_rsp_pdu))) {
+ if (!nvme_tcp_recv_pdu_supported(hdr->type))
+ goto unsupported_pdu;
+
+ dev_err(queue->ctrl->ctrl.device,
+ "pdu type %d has unexpected header length (%d)\n",
+ hdr->type, hdr->hlen);
+ return -EPROTO;
+ }
+
if (unlikely(hdr->type == nvme_tcp_c2h_term)) {
/*
* C2HTermReq never includes Header or Data digests.
@@ -824,10 +847,13 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb,
nvme_tcp_init_recv_ctx(queue);
return nvme_tcp_handle_r2t(queue, (void *)queue->pdu);
default:
- dev_err(queue->ctrl->ctrl.device,
- "unsupported pdu type (%d)\n", hdr->type);
- return -EINVAL;
+ goto unsupported_pdu;
}
+
+unsupported_pdu:
+ dev_err(queue->ctrl->ctrl.device,
+ "unsupported pdu type (%d)\n", hdr->type);
+ return -EINVAL;
}
static inline void nvme_tcp_end_request(struct request *rq, u16 status)
@@ -1629,6 +1655,7 @@ static int nvme_tcp_alloc_queue(struct nvme_ctrl *nctrl, int qid)
queue->sock->sk->sk_rcvtimeo = 10 * HZ;
queue->sock->sk->sk_allocation = GFP_ATOMIC;
+ queue->sock->sk->sk_use_task_frag = false;
nvme_tcp_set_queue_io_cpu(queue);
queue->request = NULL;
queue->data_remaining = 0;
diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c
index 74445bead85f..9d4180ceceb2 100644
--- a/drivers/nvme/target/admin-cmd.c
+++ b/drivers/nvme/target/admin-cmd.c
@@ -831,7 +831,7 @@ u16 nvmet_set_feat_async_event(struct nvmet_req *req, u32 mask)
void nvmet_execute_set_features(struct nvmet_req *req)
{
- struct nvmet_subsys *subsys = nvmet_req_subsys(req);
+ struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11);
u16 status = 0;
@@ -850,7 +850,7 @@ void nvmet_execute_set_features(struct nvmet_req *req)
break;
}
nvmet_set_result(req,
- (subsys->max_qid - 1) | ((subsys->max_qid - 1) << 16));
+ (ctrl->max_qid - 1) | ((ctrl->max_qid - 1) << 16));
break;
case NVME_FEAT_KATO:
status = nvmet_set_feat_kato(req);
@@ -905,7 +905,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req)
void nvmet_execute_get_features(struct nvmet_req *req)
{
- struct nvmet_subsys *subsys = nvmet_req_subsys(req);
+ struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req);
u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10);
u16 status = 0;
@@ -942,7 +942,7 @@ void nvmet_execute_get_features(struct nvmet_req *req)
break;
case NVME_FEAT_NUM_QUEUES:
nvmet_set_result(req,
- (subsys->max_qid-1) | ((subsys->max_qid-1) << 16));
+ (ctrl->max_qid-1) | ((ctrl->max_qid-1) << 16));
break;
case NVME_FEAT_KATO:
nvmet_get_feat_kato(req);
@@ -988,7 +988,7 @@ void nvmet_execute_async_event(struct nvmet_req *req)
ctrl->async_event_cmds[ctrl->nr_async_event_cmds++] = req;
mutex_unlock(&ctrl->lock);
- queue_work(nvmet_wq, &ctrl->async_event_work);
+ queue_work(nvmet_aen_wq, &ctrl->async_event_work);
}
void nvmet_execute_keep_alive(struct nvmet_req *req)
@@ -1011,6 +1011,27 @@ void nvmet_execute_keep_alive(struct nvmet_req *req)
nvmet_req_complete(req, status);
}
+u32 nvmet_admin_cmd_data_len(struct nvmet_req *req)
+{
+ struct nvme_command *cmd = req->cmd;
+
+ if (nvme_is_fabrics(cmd))
+ return nvmet_fabrics_admin_cmd_data_len(req);
+ if (nvmet_is_disc_subsys(nvmet_req_subsys(req)))
+ return nvmet_discovery_cmd_data_len(req);
+
+ switch (cmd->common.opcode) {
+ case nvme_admin_get_log_page:
+ return nvmet_get_log_page_len(cmd);
+ case nvme_admin_identify:
+ return NVME_IDENTIFY_DATA_SIZE;
+ case nvme_admin_get_features:
+ return nvmet_feat_data_len(req, le32_to_cpu(cmd->common.cdw10));
+ default:
+ return 0;
+ }
+}
+
u16 nvmet_parse_admin_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c
index 636a37130b22..b3a0e1946453 100644
--- a/drivers/nvme/target/auth.c
+++ b/drivers/nvme/target/auth.c
@@ -125,12 +125,11 @@ int nvmet_setup_dhgroup(struct nvmet_ctrl *ctrl, u8 dhgroup_id)
return ret;
}
-int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
+u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl)
{
int ret = 0;
struct nvmet_host_link *p;
struct nvmet_host *host = NULL;
- const char *hash_name;
down_read(&nvmet_config_sem);
if (nvmet_is_disc_subsys(ctrl->subsys))
@@ -148,13 +147,16 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
}
if (!host) {
pr_debug("host %s not found\n", ctrl->hostnqn);
- ret = -EPERM;
+ ret = NVME_AUTH_DHCHAP_FAILURE_FAILED;
goto out_unlock;
}
ret = nvmet_setup_dhgroup(ctrl, host->dhchap_dhgroup_id);
- if (ret < 0)
+ if (ret < 0) {
pr_warn("Failed to setup DH group");
+ ret = NVME_AUTH_DHCHAP_FAILURE_DHGROUP_UNUSABLE;
+ goto out_unlock;
+ }
if (!host->dhchap_secret) {
pr_debug("No authentication provided\n");
@@ -165,12 +167,6 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
pr_debug("Re-use existing hash ID %d\n",
ctrl->shash_id);
} else {
- hash_name = nvme_auth_hmac_name(host->dhchap_hash_id);
- if (!hash_name) {
- pr_warn("Hash ID %d invalid\n", host->dhchap_hash_id);
- ret = -EINVAL;
- goto out_unlock;
- }
ctrl->shash_id = host->dhchap_hash_id;
}
@@ -179,7 +175,7 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
ctrl->host_key = nvme_auth_extract_key(host->dhchap_secret + 10,
host->dhchap_key_hash);
if (IS_ERR(ctrl->host_key)) {
- ret = PTR_ERR(ctrl->host_key);
+ ret = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
ctrl->host_key = NULL;
goto out_free_hash;
}
@@ -197,7 +193,7 @@ int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
ctrl->ctrl_key = nvme_auth_extract_key(host->dhchap_ctrl_secret + 10,
host->dhchap_ctrl_key_hash);
if (IS_ERR(ctrl->ctrl_key)) {
- ret = PTR_ERR(ctrl->ctrl_key);
+ ret = NVME_AUTH_DHCHAP_FAILURE_NOT_USABLE;
ctrl->ctrl_key = NULL;
goto out_free_hash;
}
diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c
index 2e87718aa194..88e6adc54dcb 100644
--- a/drivers/nvme/target/configfs.c
+++ b/drivers/nvme/target/configfs.c
@@ -345,6 +345,7 @@ static ssize_t nvmet_ns_device_path_store(struct config_item *item,
struct nvmet_subsys *subsys = ns->subsys;
size_t len;
int ret;
+ char *new_path = NULL;
mutex_lock(&subsys->lock);
ret = -EBUSY;
@@ -356,12 +357,14 @@ static ssize_t nvmet_ns_device_path_store(struct config_item *item,
if (!len)
goto out_unlock;
- kfree(ns->device_path);
ret = -ENOMEM;
- ns->device_path = kmemdup_nul(page, len, GFP_KERNEL);
- if (!ns->device_path)
+ new_path = kmemdup_nul(page, len, GFP_KERNEL);
+ if (!new_path)
goto out_unlock;
+ kfree(ns->device_path);
+ ns->device_path = new_path;
+
mutex_unlock(&subsys->lock);
return count;
diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c
index 72703cdbff2a..ed92999609c7 100644
--- a/drivers/nvme/target/core.c
+++ b/drivers/nvme/target/core.c
@@ -23,6 +23,8 @@ static DEFINE_IDA(cntlid_ida);
struct workqueue_struct *nvmet_wq;
EXPORT_SYMBOL_GPL(nvmet_wq);
+struct workqueue_struct *nvmet_aen_wq;
+EXPORT_SYMBOL_GPL(nvmet_aen_wq);
/*
* This read/write semaphore is used to synchronize access to configuration
@@ -209,7 +211,7 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type,
list_add_tail(&aen->entry, &ctrl->async_events);
mutex_unlock(&ctrl->lock);
- queue_work(nvmet_wq, &ctrl->async_event_work);
+ queue_work(nvmet_aen_wq, &ctrl->async_event_work);
}
static void nvmet_add_to_changed_ns_log(struct nvmet_ctrl *ctrl, __le32 nsid)
@@ -882,6 +884,33 @@ static inline u16 nvmet_io_cmd_check_access(struct nvmet_req *req)
return 0;
}
+static u32 nvmet_io_cmd_transfer_len(struct nvmet_req *req)
+{
+ struct nvme_command *cmd = req->cmd;
+ u32 metadata_len = 0;
+
+ if (nvme_is_fabrics(cmd))
+ return nvmet_fabrics_io_cmd_data_len(req);
+
+ if (!req->ns)
+ return 0;
+
+ switch (req->cmd->common.opcode) {
+ case nvme_cmd_read:
+ case nvme_cmd_write:
+ case nvme_cmd_zone_append:
+ if (req->sq->ctrl->pi_support && nvmet_ns_has_pi(req->ns))
+ metadata_len = nvmet_rw_metadata_len(req);
+ return nvmet_rw_data_len(req) + metadata_len;
+ case nvme_cmd_dsm:
+ return nvmet_dsm_len(req);
+ case nvme_cmd_zone_mgmt_recv:
+ return (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2;
+ default:
+ return 0;
+ }
+}
+
static u16 nvmet_parse_io_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
@@ -1006,6 +1035,16 @@ void nvmet_req_uninit(struct nvmet_req *req)
}
EXPORT_SYMBOL_GPL(nvmet_req_uninit);
+size_t nvmet_req_transfer_len(struct nvmet_req *req)
+{
+ if (likely(req->sq->qid != 0))
+ return nvmet_io_cmd_transfer_len(req);
+ if (unlikely(!req->sq->ctrl))
+ return nvmet_connect_cmd_data_len(req);
+ return nvmet_admin_cmd_data_len(req);
+}
+EXPORT_SYMBOL_GPL(nvmet_req_transfer_len);
+
bool nvmet_check_transfer_len(struct nvmet_req *req, size_t len)
{
if (unlikely(len != req->transfer_len)) {
@@ -1053,7 +1092,7 @@ static int nvmet_req_alloc_p2pmem_sgls(struct pci_dev *p2p_dev,
return 0;
out_free_sg:
- pci_p2pmem_free_sgl(req->p2p_dev, req->sg);
+ pci_p2pmem_free_sgl(p2p_dev, req->sg);
out_err:
return -ENOMEM;
}
@@ -1431,7 +1470,25 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
if (!ctrl->changed_ns_list)
goto out_free_ctrl;
- ctrl->sqs = kcalloc(subsys->max_qid + 1,
+ /*
+ * Discovery controllers may use some arbitrary high value
+ * in order to cleanup stale discovery sessions
+ */
+ if (nvmet_is_disc_subsys(ctrl->subsys) && !kato)
+ kato = NVMET_DISC_KATO_MS;
+
+ /* keep-alive timeout in seconds */
+ ctrl->kato = DIV_ROUND_UP(kato, 1000);
+
+ ctrl->err_counter = 0;
+ spin_lock_init(&ctrl->error_lock);
+
+ down_read(&nvmet_config_sem);
+ mutex_lock(&subsys->lock);
+
+ ctrl->max_qid = subsys->max_qid;
+
+ ctrl->sqs = kcalloc(ctrl->max_qid + 1,
sizeof(struct nvmet_sq *),
GFP_KERNEL);
if (!ctrl->sqs)
@@ -1449,25 +1506,12 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
}
ctrl->cntlid = ret;
- /*
- * Discovery controllers may use some arbitrary high value
- * in order to cleanup stale discovery sessions
- */
- if (nvmet_is_disc_subsys(ctrl->subsys) && !kato)
- kato = NVMET_DISC_KATO_MS;
-
- /* keep-alive timeout in seconds */
- ctrl->kato = DIV_ROUND_UP(kato, 1000);
-
- ctrl->err_counter = 0;
- spin_lock_init(&ctrl->error_lock);
-
- nvmet_start_keep_alive_timer(ctrl);
-
- mutex_lock(&subsys->lock);
list_add_tail(&ctrl->subsys_entry, &subsys->ctrls);
nvmet_setup_p2p_ns_map(ctrl, req);
mutex_unlock(&subsys->lock);
+ up_read(&nvmet_config_sem);
+
+ nvmet_start_keep_alive_timer(ctrl);
*ctrlp = ctrl;
return 0;
@@ -1475,6 +1519,8 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn,
out_free_sqs:
kfree(ctrl->sqs);
out_free_changed_ns_list:
+ mutex_unlock(&subsys->lock);
+ up_read(&nvmet_config_sem);
kfree(ctrl->changed_ns_list);
out_free_ctrl:
kfree(ctrl);
@@ -1670,9 +1716,14 @@ static int __init nvmet_init(void)
if (!nvmet_wq)
goto out_free_buffered_work_queue;
+ nvmet_aen_wq = alloc_workqueue("nvmet-aen-wq",
+ WQ_MEM_RECLAIM | WQ_UNBOUND, 0);
+ if (!nvmet_aen_wq)
+ goto out_free_nvmet_work_queue;
+
error = nvmet_init_discovery();
if (error)
- goto out_free_nvmet_work_queue;
+ goto out_free_nvmet_aen_work_queue;
error = nvmet_init_configfs();
if (error)
@@ -1681,6 +1732,8 @@ static int __init nvmet_init(void)
out_exit_discovery:
nvmet_exit_discovery();
+out_free_nvmet_aen_work_queue:
+ destroy_workqueue(nvmet_aen_wq);
out_free_nvmet_work_queue:
destroy_workqueue(nvmet_wq);
out_free_buffered_work_queue:
@@ -1697,6 +1750,7 @@ static void __exit nvmet_exit(void)
nvmet_exit_configfs();
nvmet_exit_discovery();
ida_destroy(&cntlid_ida);
+ destroy_workqueue(nvmet_aen_wq);
destroy_workqueue(nvmet_wq);
destroy_workqueue(buffered_io_wq);
destroy_workqueue(zbd_wq);
diff --git a/drivers/nvme/target/discovery.c b/drivers/nvme/target/discovery.c
index af3c6793aff0..4ea79ee87f79 100644
--- a/drivers/nvme/target/discovery.c
+++ b/drivers/nvme/target/discovery.c
@@ -166,6 +166,7 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)
u64 offset = nvmet_get_log_page_offset(req->cmd);
size_t data_len = nvmet_get_log_page_len(req->cmd);
size_t alloc_len;
+ size_t copy_len;
struct nvmet_subsys_link *p;
struct nvmet_port *r;
u32 numrec = 0;
@@ -239,7 +240,27 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req)
up_read(&nvmet_config_sem);
- status = nvmet_copy_to_sgl(req, 0, buffer + offset, data_len);
+ /*
+ * Validate the host-supplied log page offset before copying out.
+ * Without this check, the host controls a 64-bit byte offset into
+ * a small kzalloc'd buffer: a value past the log page lets the
+ * subsequent memcpy read adjacent kernel heap, and a value aimed
+ * at unmapped kernel memory faults the in-kernel copy and crashes
+ * the target host. The Discovery controller is unauthenticated,
+ * so the bug is reachable from any reachable fabric peer.
+ */
+ if (offset > alloc_len) {
+ req->error_loc =
+ offsetof(struct nvme_get_log_page_command, lpo);
+ status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
+ goto out_free_buffer;
+ }
+
+ copy_len = min_t(size_t, data_len, alloc_len - offset);
+ status = nvmet_copy_to_sgl(req, 0, buffer + offset, copy_len);
+ if (!status && copy_len < data_len)
+ status = nvmet_zero_sgl(req, copy_len, data_len - copy_len);
+out_free_buffer:
kfree(buffer);
out:
nvmet_req_complete(req, status);
@@ -352,6 +373,20 @@ static void nvmet_execute_disc_get_features(struct nvmet_req *req)
nvmet_req_complete(req, stat);
}
+u32 nvmet_discovery_cmd_data_len(struct nvmet_req *req)
+{
+ struct nvme_command *cmd = req->cmd;
+
+ switch (cmd->common.opcode) {
+ case nvme_admin_get_log_page:
+ return nvmet_get_log_page_len(req->cmd);
+ case nvme_admin_identify:
+ return NVME_IDENTIFY_DATA_SIZE;
+ default:
+ return 0;
+ }
+}
+
u16 nvmet_parse_discovery_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c
index 04d070a5f8d8..990bb7af4dc8 100644
--- a/drivers/nvme/target/fabrics-cmd-auth.c
+++ b/drivers/nvme/target/fabrics-cmd-auth.c
@@ -31,7 +31,7 @@ void nvmet_auth_sq_init(struct nvmet_sq *sq)
sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
}
-static u16 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
+static u8 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
{
struct nvmet_ctrl *ctrl = req->sq->ctrl;
struct nvmf_auth_dhchap_negotiate_data *data = d;
@@ -109,13 +109,22 @@ static u16 nvmet_auth_negotiate(struct nvmet_req *req, void *d)
return 0;
}
-static u16 nvmet_auth_reply(struct nvmet_req *req, void *d)
+static u8 nvmet_auth_reply(struct nvmet_req *req, void *d, u32 tl)
{
struct nvmet_ctrl *ctrl = req->sq->ctrl;
struct nvmf_auth_dhchap_reply_data *data = d;
- u16 dhvlen = le16_to_cpu(data->dhvlen);
+ u16 dhvlen;
u8 *response;
+ if (tl < sizeof(*data))
+ return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+
+ dhvlen = le16_to_cpu(data->dhvlen);
+
+ /* Validate that hl and dhvlen fit within the transfer length */
+ if (sizeof(*data) + 2 * (size_t)data->hl + dhvlen > tl)
+ return NVME_AUTH_DHCHAP_FAILURE_INCORRECT_PAYLOAD;
+
pr_debug("%s: ctrl %d qid %d: data hl %d cvalid %d dhvlen %u\n",
__func__, ctrl->cntlid, req->sq->qid,
data->hl, data->cvalid, dhvlen);
@@ -172,13 +181,18 @@ static u16 nvmet_auth_reply(struct nvmet_req *req, void *d)
return 0;
}
-static u16 nvmet_auth_failure2(void *d)
+static u8 nvmet_auth_failure2(void *d)
{
struct nvmf_auth_dhchap_failure_data *data = d;
return data->rescode_exp;
}
+u32 nvmet_auth_send_data_len(struct nvmet_req *req)
+{
+ return le32_to_cpu(req->cmd->auth_send.tl);
+}
+
void nvmet_execute_auth_send(struct nvmet_req *req)
{
struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -186,6 +200,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
void *d;
u32 tl;
u16 status = 0;
+ u8 dhchap_status;
if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) {
status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
@@ -205,7 +220,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
offsetof(struct nvmf_auth_send_command, spsp1);
goto done;
}
- tl = le32_to_cpu(req->cmd->auth_send.tl);
+ tl = nvmet_auth_send_data_len(req);
if (!tl) {
status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
req->error_loc =
@@ -237,30 +252,32 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
if (data->auth_type == NVME_AUTH_COMMON_MESSAGES) {
if (data->auth_id == NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE) {
/* Restart negotiation */
- pr_debug("%s: ctrl %d qid %d reset negotiation\n", __func__,
- ctrl->cntlid, req->sq->qid);
+ pr_debug("%s: ctrl %d qid %d reset negotiation\n",
+ __func__, ctrl->cntlid, req->sq->qid);
if (!req->sq->qid) {
- if (nvmet_setup_auth(ctrl) < 0) {
- status = NVME_SC_INTERNAL;
- pr_err("ctrl %d qid 0 failed to setup"
- "re-authentication",
+ dhchap_status = nvmet_setup_auth(ctrl);
+ if (dhchap_status) {
+ pr_err("ctrl %d qid 0 failed to setup re-authentication\n",
ctrl->cntlid);
- goto done_failure1;
+ req->sq->dhchap_status = dhchap_status;
+ req->sq->dhchap_step =
+ NVME_AUTH_DHCHAP_MESSAGE_FAILURE1;
+ goto done_kfree;
}
}
- req->sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
+ req->sq->dhchap_step =
+ NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE;
} else if (data->auth_id != req->sq->dhchap_step)
goto done_failure1;
/* Validate negotiation parameters */
- status = nvmet_auth_negotiate(req, d);
- if (status == 0)
+ dhchap_status = nvmet_auth_negotiate(req, d);
+ if (dhchap_status == 0)
req->sq->dhchap_step =
NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE;
else {
req->sq->dhchap_step =
NVME_AUTH_DHCHAP_MESSAGE_FAILURE1;
- req->sq->dhchap_status = status;
- status = 0;
+ req->sq->dhchap_status = dhchap_status;
}
goto done_kfree;
}
@@ -284,15 +301,14 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
switch (data->auth_id) {
case NVME_AUTH_DHCHAP_MESSAGE_REPLY:
- status = nvmet_auth_reply(req, d);
- if (status == 0)
+ dhchap_status = nvmet_auth_reply(req, d, tl);
+ if (dhchap_status == 0)
req->sq->dhchap_step =
NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1;
else {
req->sq->dhchap_step =
NVME_AUTH_DHCHAP_MESSAGE_FAILURE1;
- req->sq->dhchap_status = status;
- status = 0;
+ req->sq->dhchap_status = dhchap_status;
}
goto done_kfree;
break;
@@ -303,13 +319,12 @@ void nvmet_execute_auth_send(struct nvmet_req *req)
goto done_kfree;
break;
case NVME_AUTH_DHCHAP_MESSAGE_FAILURE2:
- status = nvmet_auth_failure2(d);
- if (status) {
+ dhchap_status = nvmet_auth_failure2(d);
+ if (dhchap_status) {
pr_warn("ctrl %d qid %d: authentication failed (%d)\n",
- ctrl->cntlid, req->sq->qid, status);
- req->sq->dhchap_status = status;
+ ctrl->cntlid, req->sq->qid, dhchap_status);
+ req->sq->dhchap_status = dhchap_status;
req->sq->authenticated = false;
- status = 0;
}
goto done_kfree;
break;
@@ -359,7 +374,7 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al)
struct nvmet_ctrl *ctrl = req->sq->ctrl;
int ret = 0;
int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id);
- int data_size = sizeof(*d) + hash_len;
+ int data_size = sizeof(*data) + hash_len;
if (ctrl->dh_tfm)
data_size += ctrl->dh_keysize;
@@ -432,6 +447,35 @@ static void nvmet_auth_failure1(struct nvmet_req *req, void *d, int al)
data->rescode_exp = req->sq->dhchap_status;
}
+u32 nvmet_auth_receive_data_len(struct nvmet_req *req)
+{
+ struct nvmet_ctrl *ctrl = req->sq->ctrl;
+ u32 al = le32_to_cpu(req->cmd->auth_receive.al);
+ u32 min_len;
+
+ /*
+ * Reject too-short al before kmalloc(al), since the SUCCESS1 and
+ * FAILURE1/default builders write fixed response headers into it.
+ */
+ switch (req->sq->dhchap_step) {
+ case NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE:
+ return al;
+ case NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1:
+ min_len = sizeof(struct nvmf_auth_dhchap_success1_data);
+ if (req->sq->dhchap_c2)
+ min_len += nvme_auth_hmac_hash_len(ctrl->shash_id);
+ break;
+ default:
+ min_len = sizeof(struct nvmf_auth_dhchap_failure_data);
+ break;
+ }
+
+ if (al < min_len)
+ return 0;
+
+ return al;
+}
+
void nvmet_execute_auth_receive(struct nvmet_req *req)
{
struct nvmet_ctrl *ctrl = req->sq->ctrl;
@@ -457,7 +501,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req)
offsetof(struct nvmf_auth_receive_command, spsp1);
goto done;
}
- al = le32_to_cpu(req->cmd->auth_receive.al);
+ al = nvmet_auth_receive_data_len(req);
if (!al) {
status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR;
req->error_loc =
diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c
index e6b659badc52..7807cee08790 100644
--- a/drivers/nvme/target/fabrics-cmd.c
+++ b/drivers/nvme/target/fabrics-cmd.c
@@ -82,6 +82,22 @@ static void nvmet_execute_prop_get(struct nvmet_req *req)
nvmet_req_complete(req, status);
}
+u32 nvmet_fabrics_admin_cmd_data_len(struct nvmet_req *req)
+{
+ struct nvme_command *cmd = req->cmd;
+
+ switch (cmd->fabrics.fctype) {
+#ifdef CONFIG_NVME_TARGET_AUTH
+ case nvme_fabrics_type_auth_send:
+ return nvmet_auth_send_data_len(req);
+ case nvme_fabrics_type_auth_receive:
+ return nvmet_auth_receive_data_len(req);
+#endif
+ default:
+ return 0;
+ }
+}
+
u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
@@ -111,6 +127,22 @@ u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req)
return 0;
}
+u32 nvmet_fabrics_io_cmd_data_len(struct nvmet_req *req)
+{
+ struct nvme_command *cmd = req->cmd;
+
+ switch (cmd->fabrics.fctype) {
+#ifdef CONFIG_NVME_TARGET_AUTH
+ case nvme_fabrics_type_auth_send:
+ return nvmet_auth_send_data_len(req);
+ case nvme_fabrics_type_auth_receive:
+ return nvmet_auth_receive_data_len(req);
+#endif
+ default:
+ return 0;
+ }
+}
+
u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
@@ -210,7 +242,7 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
struct nvmf_connect_data *d;
struct nvmet_ctrl *ctrl = NULL;
u16 status;
- int ret;
+ u8 dhchap_status;
if (!nvmet_check_transfer_len(req, sizeof(struct nvmf_connect_data)))
return;
@@ -252,11 +284,12 @@ static void nvmet_execute_admin_connect(struct nvmet_req *req)
uuid_copy(&ctrl->hostid, &d->hostid);
- ret = nvmet_setup_auth(ctrl);
- if (ret < 0) {
- pr_err("Failed to setup authentication, error %d\n", ret);
+ dhchap_status = nvmet_setup_auth(ctrl);
+ if (dhchap_status) {
+ pr_err("Failed to setup authentication, dhchap status %u\n",
+ dhchap_status);
nvmet_ctrl_put(ctrl);
- if (ret == -EPERM)
+ if (dhchap_status == NVME_AUTH_DHCHAP_FAILURE_FAILED)
status = (NVME_SC_CONNECT_INVALID_HOST | NVME_STATUS_DNR);
else
status = NVME_SC_INTERNAL;
@@ -318,7 +351,7 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
goto out;
}
- if (unlikely(qid > ctrl->subsys->max_qid)) {
+ if (unlikely(qid > ctrl->max_qid)) {
pr_warn("invalid queue id (%d)\n", qid);
status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR;
req->cqe->result.u32 = IPO_IATTR_CONNECT_SQE(qid);
@@ -342,6 +375,17 @@ static void nvmet_execute_io_connect(struct nvmet_req *req)
goto out;
}
+u32 nvmet_connect_cmd_data_len(struct nvmet_req *req)
+{
+ struct nvme_command *cmd = req->cmd;
+
+ if (!nvme_is_fabrics(cmd) ||
+ cmd->fabrics.fctype != nvme_fabrics_type_connect)
+ return 0;
+
+ return sizeof(struct nvmf_connect_data);
+}
+
u16 nvmet_parse_connect_cmd(struct nvmet_req *req)
{
struct nvme_command *cmd = req->cmd;
diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h
index 90c63971402a..09778c5d2679 100644
--- a/drivers/nvme/target/nvmet.h
+++ b/drivers/nvme/target/nvmet.h
@@ -111,8 +111,8 @@ struct nvmet_sq {
struct delayed_work auth_expired_work;
bool authenticated;
u16 dhchap_tid;
- u16 dhchap_status;
- int dhchap_step;
+ u8 dhchap_status;
+ u8 dhchap_step;
u8 *dhchap_c1;
u8 *dhchap_c2;
u32 dhchap_s1;
@@ -190,6 +190,7 @@ struct nvmet_ctrl {
uuid_t hostid;
u16 cntlid;
+ u16 max_qid;
u32 kato;
struct nvmet_port *port;
@@ -397,6 +398,7 @@ extern struct kmem_cache *nvmet_bvec_cache;
extern struct workqueue_struct *buffered_io_wq;
extern struct workqueue_struct *zbd_wq;
extern struct workqueue_struct *nvmet_wq;
+extern struct workqueue_struct *nvmet_aen_wq;
static inline void nvmet_set_result(struct nvmet_req *req, u32 result)
{
@@ -443,18 +445,24 @@ void nvmet_start_keep_alive_timer(struct nvmet_ctrl *ctrl);
void nvmet_stop_keep_alive_timer(struct nvmet_ctrl *ctrl);
u16 nvmet_parse_connect_cmd(struct nvmet_req *req);
+u32 nvmet_connect_cmd_data_len(struct nvmet_req *req);
void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id);
u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req);
u16 nvmet_file_parse_io_cmd(struct nvmet_req *req);
u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req);
+u32 nvmet_admin_cmd_data_len(struct nvmet_req *req);
u16 nvmet_parse_admin_cmd(struct nvmet_req *req);
+u32 nvmet_discovery_cmd_data_len(struct nvmet_req *req);
u16 nvmet_parse_discovery_cmd(struct nvmet_req *req);
u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req);
+u32 nvmet_fabrics_admin_cmd_data_len(struct nvmet_req *req);
u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req);
+u32 nvmet_fabrics_io_cmd_data_len(struct nvmet_req *req);
bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq,
struct nvmet_sq *sq, const struct nvmet_fabrics_ops *ops);
void nvmet_req_uninit(struct nvmet_req *req);
+size_t nvmet_req_transfer_len(struct nvmet_req *req);
bool nvmet_check_transfer_len(struct nvmet_req *req, size_t len);
bool nvmet_check_data_len_lte(struct nvmet_req *req, size_t data_len);
void nvmet_req_complete(struct nvmet_req *req, u16 status);
@@ -610,6 +618,11 @@ static inline struct nvmet_subsys *nvmet_req_subsys(struct nvmet_req *req)
return req->sq->ctrl->subsys;
}
+static inline struct nvmet_ctrl *nvmet_req_ctrl(struct nvmet_req *req)
+{
+ return req->sq->ctrl;
+}
+
static inline bool nvmet_is_disc_subsys(struct nvmet_subsys *subsys)
{
return subsys->type != NVME_NQN_NVME;
@@ -700,12 +713,14 @@ static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio)
}
#ifdef CONFIG_NVME_TARGET_AUTH
+u32 nvmet_auth_send_data_len(struct nvmet_req *req);
void nvmet_execute_auth_send(struct nvmet_req *req);
+u32 nvmet_auth_receive_data_len(struct nvmet_req *req);
void nvmet_execute_auth_receive(struct nvmet_req *req);
int nvmet_auth_set_key(struct nvmet_host *host, const char *secret,
bool set_ctrl);
int nvmet_auth_set_host_hash(struct nvmet_host *host, const char *hash);
-int nvmet_setup_auth(struct nvmet_ctrl *ctrl);
+u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl);
void nvmet_auth_sq_init(struct nvmet_sq *sq);
void nvmet_destroy_auth(struct nvmet_ctrl *ctrl);
void nvmet_auth_sq_free(struct nvmet_sq *sq);
@@ -725,7 +740,7 @@ int nvmet_auth_ctrl_exponential(struct nvmet_req *req,
int nvmet_auth_ctrl_sesskey(struct nvmet_req *req,
u8 *buf, int buf_size);
#else
-static inline int nvmet_setup_auth(struct nvmet_ctrl *ctrl)
+static inline u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl)
{
return 0;
}
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index 0dfe264968ff..380c0aea840d 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -2098,6 +2098,7 @@ static void nvmet_rdma_remove_one(struct ib_device *ib_device, void *client_data
mutex_unlock(&nvmet_rdma_queue_mutex);
flush_workqueue(nvmet_wq);
+ flush_workqueue(nvmet_aen_wq);
}
static struct ib_client nvmet_rdma_ib_client = {
diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c
index 8af62bd24486..6ae240232760 100644
--- a/drivers/nvme/target/tcp.c
+++ b/drivers/nvme/target/tcp.c
@@ -1744,6 +1744,14 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port,
nvmet_tcp_free_cmd(&queue->connect);
out_ida_remove:
ida_free(&nvmet_tcp_queue_ida, queue->idx);
+ /*
+ * Drain the page fragment cache if any allocations were done.
+ * The first allocation using pf_cache is nvmet_tcp_alloc_cmd()
+ * for queue->connect after ida_alloc().
+ */
+ if (queue->pf_cache.va)
+ __page_frag_cache_drain(virt_to_head_page(queue->pf_cache.va),
+ queue->pf_cache.pagecnt_bias);
out_free_queue:
kfree(queue);
return ret;
diff --git a/drivers/of/irq.c b/drivers/of/irq.c
index 24e2273b9df2..1e093400bd1a 100644
--- a/drivers/of/irq.c
+++ b/drivers/of/irq.c
@@ -467,10 +467,16 @@ int of_irq_get(struct device_node *dev, int index)
return rc;
domain = irq_find_host(oirq.np);
- if (!domain)
- return -EPROBE_DEFER;
+ if (!domain) {
+ rc = -EPROBE_DEFER;
+ goto out;
+ }
+
+ rc = irq_create_of_mapping(&oirq);
+out:
+ of_node_put(oirq.np);
- return irq_create_of_mapping(&oirq);
+ return rc;
}
EXPORT_SYMBOL_GPL(of_irq_get);
@@ -654,6 +660,8 @@ void __init of_irq_init(const struct of_device_id *matches)
list_for_each_entry_safe(desc, temp_desc, &intc_parent_list, list) {
list_del(&desc->list);
+ of_node_put(desc->interrupt_parent);
+ of_node_put(desc->dev);
kfree(desc);
}
err:
diff --git a/drivers/of/overlay.c b/drivers/of/overlay.c
index e663d5585a05..7d8205b7e563 100644
--- a/drivers/of/overlay.c
+++ b/drivers/of/overlay.c
@@ -241,6 +241,9 @@ static struct property *dup_and_fixup_symbol_prop(
if (!target_path)
return NULL;
target_path_len = strlen(target_path);
+ /* a root target renders as "/"; drop it to avoid "//" results */
+ if (target_path_len == 1 && target_path[0] == '/' && path_tail_len)
+ target_path_len = 0;
new_prop = kzalloc(sizeof(*new_prop), GFP_KERNEL);
if (!new_prop)
@@ -346,12 +349,13 @@ static int add_changeset_property(struct overlay_changeset *ovcs,
return -ENOMEM;
if (!prop) {
- if (!target->in_livetree) {
+ ret = of_changeset_add_property(&ovcs->cset, target->np,
+ new_prop);
+ /* the detached node owns the property until the apply */
+ if (!ret && !target->in_livetree) {
new_prop->next = target->np->deadprops;
target->np->deadprops = new_prop;
}
- ret = of_changeset_add_property(&ovcs->cset, target->np,
- new_prop);
} else {
ret = of_changeset_update_property(&ovcs->cset, target->np,
new_prop);
@@ -829,6 +833,10 @@ static int init_overlay_changeset(struct overlay_changeset *ovcs)
err_out:
pr_err("%s() failed, ret = %d\n", __func__, ret);
+ /* let free_overlay_changeset() put the fragments set up so far */
+ if (ovcs->fragments)
+ ovcs->count = cnt;
+
return ret;
}
@@ -839,7 +847,8 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs)
if (ovcs->cset.entries.next)
of_changeset_destroy(&ovcs->cset);
- if (ovcs->id) {
+ /* a failed idr_alloc() leaves its negative error in ovcs->id */
+ if (ovcs->id > 0) {
idr_remove(&ovcs_idr, ovcs->id);
list_del(&ovcs->ovcs_list);
ovcs->id = 0;
diff --git a/drivers/of/unittest.c b/drivers/of/unittest.c
index 598e0891533f..2dc62174e167 100644
--- a/drivers/of/unittest.c
+++ b/drivers/of/unittest.c
@@ -1546,7 +1546,6 @@ static void attach_node_and_children(struct device_node *np)
*/
static int __init unittest_data_add(void)
{
- void *unittest_data;
void *unittest_data_align;
struct device_node *unittest_data_node = NULL, *np;
/*
@@ -1565,7 +1564,7 @@ static int __init unittest_data_add(void)
}
/* creating copy */
- unittest_data = kmalloc(size + FDT_ALIGN_SIZE, GFP_KERNEL);
+ void *unittest_data __free(kfree) = kmalloc(size + FDT_ALIGN_SIZE, GFP_KERNEL);
if (!unittest_data)
return -ENOMEM;
@@ -1575,12 +1574,10 @@ static int __init unittest_data_add(void)
ret = of_fdt_unflatten_tree(unittest_data_align, NULL, &unittest_data_node);
if (!ret) {
pr_warn("%s: unflatten testcases tree failed\n", __func__);
- kfree(unittest_data);
return -ENODATA;
}
if (!unittest_data_node) {
pr_warn("%s: testcases tree is empty\n", __func__);
- kfree(unittest_data);
return -ENODATA;
}
@@ -1602,6 +1599,7 @@ static int __init unittest_data_add(void)
__of_attach_node_sysfs(np);
of_aliases = of_find_node_by_path("/aliases");
of_chosen = of_find_node_by_path("/chosen");
+ no_free_ptr(unittest_data);
of_overlay_mutex_unlock();
return 0;
}
@@ -1622,6 +1620,7 @@ static int __init unittest_data_add(void)
EXPECT_END(KERN_INFO,
"Duplicate name in testcase-data, renamed to \"duplicate-name#1\"");
+ no_free_ptr(unittest_data);
of_overlay_mutex_unlock();
return 0;
diff --git a/drivers/opp/core.c b/drivers/opp/core.c
index 4211cfb27350..0ba25aba48ab 100644
--- a/drivers/opp/core.c
+++ b/drivers/opp/core.c
@@ -1993,12 +1993,12 @@ int _opp_add(struct device *dev, struct dev_pm_opp *new_opp,
return ret;
}
- list_add(&new_opp->node, head);
- mutex_unlock(&opp_table->lock);
-
new_opp->opp_table = opp_table;
kref_init(&new_opp->kref);
+ list_add(&new_opp->node, head);
+ mutex_unlock(&opp_table->lock);
+
opp_debug_create_one(new_opp, opp_table);
if (!_opp_supported_by_regulators(new_opp, opp_table)) {
diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c
index 3b3f079d0d2d..7a68574b90d3 100644
--- a/drivers/pci/controller/dwc/pcie-designware-ep.c
+++ b/drivers/pci/controller/dwc/pcie-designware-ep.c
@@ -247,6 +247,23 @@ static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no,
func_offset = dw_pcie_ep_func_select(ep, func_no);
+ /*
+ * Certain EPF drivers dynamically change the physical address of a BAR
+ * (i.e. they call set_bar() twice, without ever calling clear_bar(), as
+ * calling clear_bar() would clear the BAR's PCI address assigned by the
+ * host).
+ */
+ if (ep->epf_bar[bar]) {
+ /*
+ * We can only dynamically change a BAR if the new BAR size and
+ * BAR flags do not differ from the existing configuration.
+ */
+ if (ep->epf_bar[bar]->barno != bar ||
+ ep->epf_bar[bar]->size != size ||
+ ep->epf_bar[bar]->flags != flags)
+ return -EINVAL;
+ }
+
reg = PCI_BASE_ADDRESS_0 + (4 * bar) + func_offset;
if (!(flags & PCI_BASE_ADDRESS_SPACE))
@@ -616,6 +633,9 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no,
writel(msg_data, ep->msi_mem + aligned_offset);
+ /* flush posted write before unmap */
+ readl(ep->msi_mem + aligned_offset);
+
dw_pcie_ep_unmap_addr(epc, func_no, 0, ep->msi_mem_phys);
return 0;
diff --git a/drivers/pci/endpoint/functions/pci-epf-ntb.c b/drivers/pci/endpoint/functions/pci-epf-ntb.c
index e17dbcc9c8db..6abe474fd940 100644
--- a/drivers/pci/endpoint/functions/pci-epf-ntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-ntb.c
@@ -2073,6 +2073,11 @@ static int __init epf_ntb_init(void)
kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM |
WQ_HIGHPRI, 0);
+ if (!kpcintb_workqueue) {
+ pr_err("Failed to allocate kpcintb workqueue\n");
+ return -ENOMEM;
+ }
+
ret = pci_epf_register_driver(&epf_ntb_driver);
if (ret) {
destroy_workqueue(kpcintb_workqueue);
diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c
index 3fe2187cdd0b..53f1236cd91d 100644
--- a/drivers/pci/endpoint/functions/pci-epf-test.c
+++ b/drivers/pci/endpoint/functions/pci-epf-test.c
@@ -735,6 +735,7 @@ static void pci_epf_test_unbind(struct pci_epf *epf)
epf_bar);
pci_epf_free_space(epf, epf_test->reg[bar], bar,
PRIMARY_INTERFACE);
+ epf_test->reg[bar] = NULL;
}
}
}
@@ -769,6 +770,7 @@ static int pci_epf_test_set_bar(struct pci_epf *epf)
if (ret) {
pci_epf_free_space(epf, epf_test->reg[bar], bar,
PRIMARY_INTERFACE);
+ epf_test->reg[bar] = NULL;
dev_err(dev, "Failed to set BAR%d\n", bar);
if (bar == test_reg_bar)
return ret;
@@ -782,19 +784,11 @@ static int pci_epf_test_core_init(struct pci_epf *epf)
{
struct pci_epf_test *epf_test = epf_get_drvdata(epf);
struct pci_epf_header *header = epf->header;
- const struct pci_epc_features *epc_features;
+ const struct pci_epc_features *epc_features = epf_test->epc_features;
struct pci_epc *epc = epf->epc;
struct device *dev = &epf->dev;
- bool msix_capable = false;
- bool msi_capable = true;
int ret;
- epc_features = pci_epc_get_features(epc, epf->func_no, epf->vfunc_no);
- if (epc_features) {
- msix_capable = epc_features->msix_capable;
- msi_capable = epc_features->msi_capable;
- }
-
if (epf->vfunc_no <= 1) {
ret = pci_epc_write_header(epc, epf->func_no, epf->vfunc_no, header);
if (ret) {
@@ -807,7 +801,7 @@ static int pci_epf_test_core_init(struct pci_epf *epf)
if (ret)
return ret;
- if (msi_capable) {
+ if (epc_features->msi_capable) {
ret = pci_epc_set_msi(epc, epf->func_no, epf->vfunc_no,
epf->msi_interrupts);
if (ret) {
@@ -816,7 +810,7 @@ static int pci_epf_test_core_init(struct pci_epf *epf)
}
}
- if (msix_capable) {
+ if (epc_features->msix_capable) {
ret = pci_epc_set_msix(epc, epf->func_no, epf->vfunc_no,
epf->msix_interrupts,
epf_test->test_reg_bar,
diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c
index 88ff30659909..15c707840cc2 100644
--- a/drivers/pci/endpoint/functions/pci-epf-vntb.c
+++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c
@@ -1433,6 +1433,11 @@ static int __init epf_ntb_init(void)
kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM |
WQ_HIGHPRI, 0);
+ if (!kpcintb_workqueue) {
+ pr_err("Failed to allocate kpcintb workqueue\n");
+ return -ENOMEM;
+ }
+
ret = pci_epf_register_driver(&epf_ntb_driver);
if (ret) {
destroy_workqueue(kpcintb_workqueue);
diff --git a/drivers/pci/hotplug/pnv_php.c b/drivers/pci/hotplug/pnv_php.c
index ec7828ad6661..2bd2b7599d1c 100644
--- a/drivers/pci/hotplug/pnv_php.c
+++ b/drivers/pci/hotplug/pnv_php.c
@@ -799,7 +799,7 @@ static struct pnv_php_slot *pnv_php_alloc_slot(struct device_node *dn)
if (dn->child && PCI_DN(dn->child))
php_slot->slot_no = PCI_SLOT(PCI_DN(dn->child)->devfn);
else
- php_slot->slot_no = -1; /* Placeholder slot */
+ php_slot->slot_no = PCI_SLOT_PLACEHOLDER; /* Placeholder slot */
kref_init(&php_slot->kref);
php_slot->state = PNV_PHP_STATE_INITIALIZED;
diff --git a/drivers/pci/hotplug/rpaphp_slot.c b/drivers/pci/hotplug/rpaphp_slot.c
index 779eab12e981..ac8c512190df 100644
--- a/drivers/pci/hotplug/rpaphp_slot.c
+++ b/drivers/pci/hotplug/rpaphp_slot.c
@@ -85,7 +85,7 @@ int rpaphp_register_slot(struct slot *slot)
struct device_node *child;
u32 my_index;
int retval;
- int slotno = -1;
+ int slotno = PCI_SLOT_PLACEHOLDER;
dbg("%s registering slot:path[%pOF] index[%x], name[%s] pdomain[%x] type[%d]\n",
__func__, slot->dn, slot->index, slot->name,
diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c
index fa3165e5b921..1fbf42f2c961 100644
--- a/drivers/pci/pci-driver.c
+++ b/drivers/pci/pci-driver.c
@@ -29,6 +29,47 @@ struct pci_dynid {
struct pci_device_id id;
};
+/**
+ * do_pci_add_dynid - Add a new PCI device ID to this driver and re-probe
+ * @drv: target PCI driver
+ * @id: ID to be added
+ * @check_dup: whether to check if matching ID is already present
+ *
+ * Add a new dynamic PCI device ID to this driver and causes the driver to
+ * probe for all devices again. @drv must have been registered prior to calling
+ * this function.
+ *
+ * Context: Does GFP_KERNEL allocation.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+static int do_pci_add_dynid(struct pci_driver *drv,
+ const struct pci_device_id *id,
+ bool check_dup)
+{
+ struct pci_dynid *dynid, *existing_dynid;
+
+ dynid = kzalloc(sizeof(*dynid), GFP_KERNEL);
+ if (!dynid)
+ return -ENOMEM;
+
+ dynid->id = *id;
+
+ scoped_guard(spinlock, &drv->dynids.lock) {
+ if (check_dup) {
+ list_for_each_entry(existing_dynid, &drv->dynids.list, node) {
+ if (pci_match_one_id(&existing_dynid->id, id)) {
+ kfree(dynid);
+ return -EEXIST;
+ }
+ }
+ }
+ list_add_tail(&dynid->node, &drv->dynids.list);
+ }
+
+ return driver_attach(&drv->driver);
+}
+
/**
* pci_add_dynid - add a new PCI device ID to this driver and re-probe devices
* @drv: target pci driver
@@ -56,25 +97,17 @@ int pci_add_dynid(struct pci_driver *drv,
unsigned int class, unsigned int class_mask,
unsigned long driver_data)
{
- struct pci_dynid *dynid;
-
- dynid = kzalloc(sizeof(*dynid), GFP_KERNEL);
- if (!dynid)
- return -ENOMEM;
-
- dynid->id.vendor = vendor;
- dynid->id.device = device;
- dynid->id.subvendor = subvendor;
- dynid->id.subdevice = subdevice;
- dynid->id.class = class;
- dynid->id.class_mask = class_mask;
- dynid->id.driver_data = driver_data;
-
- spin_lock(&drv->dynids.lock);
- list_add_tail(&dynid->node, &drv->dynids.list);
- spin_unlock(&drv->dynids.lock);
-
- return driver_attach(&drv->driver);
+ struct pci_device_id id = {
+ .vendor = vendor,
+ .device = device,
+ .subvendor = subvendor,
+ .subdevice = subdevice,
+ .class = class,
+ .class_mask = class_mask,
+ .driver_data = driver_data,
+ };
+
+ return do_pci_add_dynid(drv, &id, false);
}
EXPORT_SYMBOL_GPL(pci_add_dynid);
@@ -90,6 +123,31 @@ static void pci_free_dynids(struct pci_driver *drv)
spin_unlock(&drv->dynids.lock);
}
+/**
+ * do_pci_match_id - See if a PCI ID matches a given pci_id table
+ * @ids: array of PCI device ID structures to search in
+ * @dev_id: the PCI device ID structure to match against.
+ * @include_override_only: also match against device ID entries marked as
+ * override only.
+ *
+ * Returns the matching pci_device_id structure or %NULL if there is no match.
+ */
+static const struct pci_device_id *
+do_pci_match_id(const struct pci_device_id *ids,
+ const struct pci_device_id *dev_id,
+ bool include_override_only)
+{
+ if (ids) {
+ while (ids->vendor || ids->subvendor || ids->class_mask) {
+ if ((!ids->override_only || include_override_only) &&
+ pci_match_one_id(ids, dev_id))
+ return ids;
+ ids++;
+ }
+ }
+ return NULL;
+}
+
/**
* pci_match_id - See if a PCI device matches a given pci_id table
* @ids: array of PCI device ID structures to search in
@@ -105,14 +163,9 @@ static void pci_free_dynids(struct pci_driver *drv)
const struct pci_device_id *pci_match_id(const struct pci_device_id *ids,
struct pci_dev *dev)
{
- if (ids) {
- while (ids->vendor || ids->subvendor || ids->class_mask) {
- if (pci_match_one_device(ids, dev))
- return ids;
- ids++;
- }
- }
- return NULL;
+ struct pci_device_id dev_id = pci_id_from_device(dev);
+
+ return do_pci_match_id(ids, &dev_id, true);
}
EXPORT_SYMBOL(pci_match_id);
@@ -127,6 +180,7 @@ static const struct pci_device_id pci_device_id_any = {
* pci_match_device - See if a device matches a driver's list of IDs
* @drv: the PCI driver to match against
* @dev: the PCI device structure to match against
+ * @id_copy: place to store copy of pci_device_id for dynamic ID
*
* Used by a driver to check whether a PCI device is in its list of
* supported devices or in the dynids list, which may have been augmented
@@ -134,43 +188,33 @@ static const struct pci_device_id pci_device_id_any = {
* structure or %NULL if there is no match.
*/
static const struct pci_device_id *pci_match_device(struct pci_driver *drv,
- struct pci_dev *dev)
+ struct pci_dev *dev,
+ struct pci_device_id *id_copy)
{
- struct pci_dynid *dynid;
- const struct pci_device_id *found_id = NULL, *ids;
+ const struct pci_device_id *found_id = NULL;
+ struct pci_device_id dev_id;
/* When driver_override is set, only bind to the matching driver */
if (dev->driver_override && strcmp(dev->driver_override, drv->name))
return NULL;
+ dev_id = pci_id_from_device(dev);
/* Look at the dynamic ids first, before the static ones */
- spin_lock(&drv->dynids.lock);
- list_for_each_entry(dynid, &drv->dynids.list, node) {
- if (pci_match_one_device(&dynid->id, dev)) {
- found_id = &dynid->id;
- break;
+ scoped_guard(spinlock, &drv->dynids.lock) {
+ struct pci_dynid *dynid;
+
+ list_for_each_entry(dynid, &drv->dynids.list, node) {
+ if (pci_match_one_id(&dynid->id, &dev_id)) {
+ *id_copy = dynid->id;
+ return id_copy;
+ }
}
}
- spin_unlock(&drv->dynids.lock);
+ found_id = do_pci_match_id(drv->id_table, &dev_id, !!dev->driver_override);
if (found_id)
return found_id;
- for (ids = drv->id_table; (found_id = pci_match_id(ids, dev));
- ids = found_id + 1) {
- /*
- * The match table is split based on driver_override.
- * In case override_only was set, enforce driver_override
- * matching.
- */
- if (found_id->override_only) {
- if (dev->driver_override)
- return found_id;
- } else {
- return found_id;
- }
- }
-
/* driver_override will always match, send a dummy id */
if (dev->driver_override)
return &pci_device_id_any;
@@ -190,36 +234,22 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
{
struct pci_driver *pdrv = to_pci_driver(driver);
const struct pci_device_id *ids = pdrv->id_table;
- u32 vendor, device, subvendor = PCI_ANY_ID,
- subdevice = PCI_ANY_ID, class = 0, class_mask = 0;
- unsigned long driver_data = 0;
- int fields = 0;
+ struct pci_device_id id = {
+ .subvendor = PCI_ANY_ID,
+ .subdevice = PCI_ANY_ID
+ };
+ int fields;
int retval = 0;
fields = sscanf(buf, "%x %x %x %x %x %x %lx",
- &vendor, &device, &subvendor, &subdevice,
- &class, &class_mask, &driver_data);
+ &id.vendor, &id.device, &id.subvendor, &id.subdevice,
+ &id.class, &id.class_mask, &id.driver_data);
if (fields < 2)
return -EINVAL;
if (fields != 7) {
- struct pci_dev *pdev = kzalloc(sizeof(*pdev), GFP_KERNEL);
- if (!pdev)
- return -ENOMEM;
-
- pdev->vendor = vendor;
- pdev->device = device;
- pdev->subsystem_vendor = subvendor;
- pdev->subsystem_device = subdevice;
- pdev->class = class;
-
- if (pci_match_device(pdrv, pdev))
- retval = -EEXIST;
-
- kfree(pdev);
-
- if (retval)
- return retval;
+ if (do_pci_match_id(pdrv->id_table, &id, false))
+ return -EEXIST;
}
/* Only accept driver_data values that match an existing id_table
@@ -227,7 +257,7 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
if (ids) {
retval = -EINVAL;
while (ids->vendor || ids->subvendor || ids->class_mask) {
- if (driver_data == ids->driver_data) {
+ if (id.driver_data == ids->driver_data) {
retval = 0;
break;
}
@@ -237,8 +267,7 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf,
return retval;
}
- retval = pci_add_dynid(pdrv, vendor, device, subvendor, subdevice,
- class, class_mask, driver_data);
+ retval = do_pci_add_dynid(pdrv, &id, fields != 7);
if (retval)
return retval;
return count;
@@ -407,12 +436,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev,
static int __pci_device_probe(struct pci_driver *drv, struct pci_dev *pci_dev)
{
const struct pci_device_id *id;
+ struct pci_device_id id_copy;
int error = 0;
if (drv->probe) {
error = -ENODEV;
- id = pci_match_device(drv, pci_dev);
+ id = pci_match_device(drv, pci_dev, &id_copy);
if (id)
error = pci_call_probe(drv, pci_dev, id);
}
@@ -1517,12 +1547,13 @@ static int pci_bus_match(struct device *dev, struct device_driver *drv)
struct pci_dev *pci_dev = to_pci_dev(dev);
struct pci_driver *pci_drv;
const struct pci_device_id *found_id;
+ struct pci_device_id id_copy;
if (!pci_dev->match_driver)
return 0;
pci_drv = to_pci_driver(drv);
- found_id = pci_match_device(pci_drv, pci_dev);
+ found_id = pci_match_device(pci_drv, pci_dev, &id_copy);
if (found_id)
return 1;
diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c
index d0ec90890830..528b4efc95d7 100644
--- a/drivers/pci/pci-sysfs.c
+++ b/drivers/pci/pci-sysfs.c
@@ -1433,71 +1433,84 @@ static const struct attribute_group pci_dev_reset_attr_group = {
.is_visible = pci_dev_reset_attr_is_visible,
};
+static ssize_t __resource_resize_show(struct device *dev, int n, char *buf)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ ssize_t ret;
+
+ pci_config_pm_runtime_get(pdev);
+
+ ret = sysfs_emit(buf, "%016llx\n",
+ (u64)pci_rebar_get_possible_sizes(pdev, n));
+
+ pci_config_pm_runtime_put(pdev);
+
+ return ret;
+}
+
+static ssize_t __resource_resize_store(struct device *dev, int n,
+ const char *buf, size_t count)
+{
+ struct pci_dev *pdev = to_pci_dev(dev);
+ unsigned long size;
+ int ret;
+ u16 cmd;
+
+ if (!capable(CAP_SYS_ADMIN))
+ return -EPERM;
+
+ if (kstrtoul(buf, 0, &size) < 0)
+ return -EINVAL;
+
+ device_lock(dev);
+ if (dev->driver) {
+ ret = -EBUSY;
+ goto unlock;
+ }
+
+ pci_config_pm_runtime_get(pdev);
+
+ if ((pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA) {
+ ret = aperture_remove_conflicting_pci_devices(pdev,
+ "resourceN_resize");
+ if (ret)
+ goto pm_put;
+ }
+
+ pci_read_config_word(pdev, PCI_COMMAND, &cmd);
+ pci_write_config_word(pdev, PCI_COMMAND,
+ cmd & ~PCI_COMMAND_MEMORY);
+
+ pci_remove_resource_files(pdev);
+
+ ret = pci_resize_resource(pdev, n, size, 0);
+
+ pci_assign_unassigned_bus_resources(pdev->bus);
+
+ if (pci_create_resource_files(pdev))
+ pci_warn(pdev, "Failed to recreate resource files after BAR resizing\n");
+
+ pci_write_config_word(pdev, PCI_COMMAND, cmd);
+pm_put:
+ pci_config_pm_runtime_put(pdev);
+unlock:
+ device_unlock(dev);
+
+ return ret ? ret : count;
+}
+
#define pci_dev_resource_resize_attr(n) \
static ssize_t resource##n##_resize_show(struct device *dev, \
struct device_attribute *attr, \
- char * buf) \
+ char *buf) \
{ \
- struct pci_dev *pdev = to_pci_dev(dev); \
- ssize_t ret; \
- \
- pci_config_pm_runtime_get(pdev); \
- \
- ret = sysfs_emit(buf, "%016llx\n", \
- (u64)pci_rebar_get_possible_sizes(pdev, n)); \
- \
- pci_config_pm_runtime_put(pdev); \
- \
- return ret; \
+ return __resource_resize_show(dev, n, buf); \
} \
- \
static ssize_t resource##n##_resize_store(struct device *dev, \
struct device_attribute *attr,\
const char *buf, size_t count)\
{ \
- struct pci_dev *pdev = to_pci_dev(dev); \
- unsigned long size; \
- int ret; \
- u16 cmd; \
- \
- if (kstrtoul(buf, 0, &size) < 0) \
- return -EINVAL; \
- \
- device_lock(dev); \
- if (dev->driver) { \
- ret = -EBUSY; \
- goto unlock; \
- } \
- \
- pci_config_pm_runtime_get(pdev); \
- \
- if ((pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA) { \
- ret = aperture_remove_conflicting_pci_devices(pdev, \
- "resourceN_resize"); \
- if (ret) \
- goto pm_put; \
- } \
- \
- pci_read_config_word(pdev, PCI_COMMAND, &cmd); \
- pci_write_config_word(pdev, PCI_COMMAND, \
- cmd & ~PCI_COMMAND_MEMORY); \
- \
- pci_remove_resource_files(pdev); \
- \
- ret = pci_resize_resource(pdev, n, size, 0); \
- \
- pci_assign_unassigned_bus_resources(pdev->bus); \
- \
- if (pci_create_resource_files(pdev)) \
- pci_warn(pdev, "Failed to recreate resource files after BAR resizing\n");\
- \
- pci_write_config_word(pdev, PCI_COMMAND, cmd); \
-pm_put: \
- pci_config_pm_runtime_put(pdev); \
-unlock: \
- device_unlock(dev); \
- \
- return ret ? ret : count; \
+ return __resource_resize_store(dev, n, buf, count); \
} \
static DEVICE_ATTR_RW(resource##n##_resize)
diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c
index b0d4c3bc205b..bc0588b5da73 100644
--- a/drivers/pci/pci.c
+++ b/drivers/pci/pci.c
@@ -3822,8 +3822,21 @@ int pci_enable_atomic_ops_to_root(struct pci_dev *dev, u32 cap_mask)
}
root = pcie_find_root_port(dev);
- if (!root)
+ if (!root) {
+
+ /*
+ * A hypervisor may expose a topology with the root port
+ * not visible to the guest. If the hypervisor has already
+ * set AtomicOp Requester Enable in the endpoint, we assume
+ * it has verified support in the root port, so it is safe
+ * for the driver to use AtomicOps.
+ */
+ pcie_capability_read_dword(dev, PCI_EXP_DEVCTL2, &ctl2);
+ if (ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ)
+ return 0;
+
return -EINVAL;
+ }
pcie_capability_read_dword(root, PCI_EXP_DEVCAP2, &cap);
if ((cap & cap_mask) != cap_mask)
@@ -5119,8 +5132,9 @@ static int pci_reset_hotplug_slot(struct hotplug_slot *hotplug, bool probe)
static int pci_dev_reset_slot_function(struct pci_dev *dev, bool probe)
{
- if (dev->multifunction || dev->subordinate || !dev->slot ||
- dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET)
+ if (dev->subordinate || !dev->slot ||
+ dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET ||
+ (dev->multifunction && !dev->slot->per_func_slot))
return -ENOTTY;
return pci_reset_hotplug_slot(dev->slot->hotplug, probe);
diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h
index 32cc16ce8a68..49a0f23ebe4a 100644
--- a/drivers/pci/pci.h
+++ b/drivers/pci/pci.h
@@ -204,21 +204,40 @@ extern unsigned long pci_hotplug_mmio_pref_size;
extern unsigned long pci_hotplug_bus_size;
/**
- * pci_match_one_device - Tell if a PCI device structure has a matching
- * PCI device id structure
- * @id: single PCI device id structure to match
- * @dev: the PCI device structure to match against
+ * pci_id_from_device - Obtain a pci_device_id from a PCI device
+ * @dev: the PCI device
*
- * Returns the matching pci_device_id structure or %NULL if there is no match.
+ * Return: a pci_device_id filled.
*/
-static inline const struct pci_device_id *
-pci_match_one_device(const struct pci_device_id *id, const struct pci_dev *dev)
+static inline struct pci_device_id pci_id_from_device(const struct pci_dev *dev)
{
- if ((id->vendor == PCI_ANY_ID || id->vendor == dev->vendor) &&
- (id->device == PCI_ANY_ID || id->device == dev->device) &&
- (id->subvendor == PCI_ANY_ID || id->subvendor == dev->subsystem_vendor) &&
- (id->subdevice == PCI_ANY_ID || id->subdevice == dev->subsystem_device) &&
- !((id->class ^ dev->class) & id->class_mask))
+ return (struct pci_device_id) {
+ .vendor = dev->vendor,
+ .device = dev->device,
+ .subvendor = dev->subsystem_vendor,
+ .subdevice = dev->subsystem_device,
+ .class = dev->class,
+ };
+}
+
+/**
+ * pci_match_one_id - Tell if a PCI device ID matches a needle PCI device ID
+ * @id: single PCI device id structure to match against (needle)
+ * @dev_id: the actual ID from the PCI device
+ *
+ * ID can be retrieved from device using pci_id_from_device().
+ *
+ * Return: the matching pci_device_id structure or %NULL if there is no match.
+ */
+static inline const struct pci_device_id *
+pci_match_one_id(const struct pci_device_id *id,
+ const struct pci_device_id *dev_id)
+{
+ if ((id->vendor == PCI_ANY_ID || id->vendor == dev_id->vendor) &&
+ (id->device == PCI_ANY_ID || id->device == dev_id->device) &&
+ (id->subvendor == PCI_ANY_ID || id->subvendor == dev_id->subvendor) &&
+ (id->subdevice == PCI_ANY_ID || id->subdevice == dev_id->subdevice) &&
+ !((id->class ^ dev_id->class) & id->class_mask))
return id;
return NULL;
}
diff --git a/drivers/pci/rebar.c b/drivers/pci/rebar.c
index 920ef3ab4bda..d5e432d084e5 100644
--- a/drivers/pci/rebar.c
+++ b/drivers/pci/rebar.c
@@ -177,7 +177,6 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size,
int exclude_bars)
{
struct pci_host_bridge *host;
- int old, ret;
u32 sizes;
u16 cmd;
@@ -197,22 +196,6 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size,
if (!(sizes & BIT(size)))
return -EINVAL;
- old = pci_rebar_get_current_size(dev, resno);
- if (old < 0)
- return old;
-
- ret = pci_rebar_set_size(dev, resno, size);
- if (ret)
- return ret;
-
- ret = pci_do_resource_release_and_resize(dev, resno, size, exclude_bars);
- if (ret)
- goto error_resize;
-
- return 0;
-
-error_resize:
- pci_rebar_set_size(dev, resno, old);
- return ret;
+ return pci_do_resource_release_and_resize(dev, resno, size, exclude_bars);
}
EXPORT_SYMBOL(pci_resize_resource);
diff --git a/drivers/pci/search.c b/drivers/pci/search.c
index b4c138a6ec02..376f5cbe08a8 100644
--- a/drivers/pci/search.c
+++ b/drivers/pci/search.c
@@ -243,8 +243,10 @@ static int match_pci_dev_by_id(struct device *dev, const void *data)
{
struct pci_dev *pdev = to_pci_dev(dev);
const struct pci_device_id *id = data;
+ struct pci_device_id dev_id;
- if (pci_match_one_device(id, pdev))
+ dev_id = pci_id_from_device(pdev);
+ if (pci_match_one_id(id, &dev_id))
return 1;
return 0;
}
diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c
index df345ebbb991..cd09f1e26b8a 100644
--- a/drivers/pci/setup-bus.c
+++ b/drivers/pci/setup-bus.c
@@ -2294,6 +2294,11 @@ static int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long t
return 0;
}
+/* Keep the resize follow-up compatible with the 6.12 resource helpers. */
+#define pbus_reassign_bridge_resources(bus, res, saved) \
+ pci_reassign_bridge_resources((bus)->self, (res)->flags, (saved))
+#define resource_assigned(res) ((res)->parent != NULL)
+
int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size,
int exclude_bars)
{
@@ -2301,10 +2306,21 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size
unsigned long flags = res->flags;
struct pci_dev_resource *dev_res;
struct pci_bus *bus = pdev->bus;
- struct resource *r;
+ struct pci_dev *bridge = pci_upstream_bridge(pdev);
+ struct resource *b_win, *r;
LIST_HEAD(saved);
unsigned int i;
- int ret = 0;
+ int old, ret;
+
+ b_win = res->parent;
+
+ old = pci_rebar_get_current_size(pdev, resno);
+ if (old < 0)
+ return old;
+
+ ret = pci_rebar_set_size(pdev, resno, size);
+ if (ret)
+ return ret;
down_read(&pci_bus_sem);
@@ -2315,7 +2331,10 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size
if (exclude_bars & BIT(i))
continue;
- if (!pci_resource_len(pdev, i) || r->flags != flags)
+ if (!pci_resource_len(pdev, i))
+ continue;
+
+ if (b_win ? r->parent != b_win : r->flags != flags)
continue;
ret = add_to_list(&saved, pdev, r, 0, 0);
@@ -2326,12 +2345,21 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size
res->end = res->start + pci_rebar_size_to_bytes(size) - 1;
- if (!bus->self)
- goto out;
+ if (bridge) {
+ ret = pbus_reassign_bridge_resources(bus, res, &saved);
+ if (ret)
+ goto restore;
+ } else {
+ /* No bridge window to adjust; let the core reassign the bus. */
+ pci_bus_assign_resources(bus);
- ret = pci_reassign_bridge_resources(bus->self, res->flags, &saved);
- if (ret)
- goto restore;
+ list_for_each_entry(dev_res, &saved, list) {
+ if (!resource_assigned(dev_res->res)) {
+ ret = -ENOSPC;
+ goto restore;
+ }
+ }
+ }
out:
up_read(&pci_bus_sem);
@@ -2339,14 +2367,22 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size
return ret;
restore:
- /* Revert to the old configuration */
+ /*
+ * Revert to the old configuration.
+ *
+ * BAR Size must be restored first because it affects the read-only
+ * bits in BAR (the old address might not be restorable otherwise
+ * due to low address bits).
+ */
+ pci_rebar_set_size(pdev, resno, old);
+
list_for_each_entry(dev_res, &saved, list) {
struct resource *res = dev_res->res;
struct pci_dev *dev = dev_res->dev;
i = res - dev->resource;
- if (res->parent) {
+ if (resource_assigned(res)) {
release_child_resources(res);
pci_release_resource(dev, i);
}
diff --git a/drivers/pci/slot.c b/drivers/pci/slot.c
index 7f4bcf359b91..94e7c61ee23e 100644
--- a/drivers/pci/slot.c
+++ b/drivers/pci/slot.c
@@ -38,7 +38,7 @@ static const struct sysfs_ops pci_slot_sysfs_ops = {
static ssize_t address_read_file(struct pci_slot *slot, char *buf)
{
- if (slot->number == 0xff)
+ if (slot->number == PCI_SLOT_PLACEHOLDER)
return sysfs_emit(buf, "%04x:%02x\n",
pci_domain_nr(slot->bus),
slot->bus->number);
@@ -64,6 +64,22 @@ static ssize_t cur_speed_read_file(struct pci_slot *slot, char *buf)
return bus_speed_read(slot->bus->cur_bus_speed, buf);
}
+static bool pci_dev_matches_slot(struct pci_dev *dev, struct pci_slot *slot)
+{
+ if (slot->per_func_slot)
+ return dev->devfn == slot->number;
+
+ return PCI_SLOT(dev->devfn) == slot->number;
+}
+
+static bool pci_slot_enabled_per_func(void)
+{
+ if (IS_ENABLED(CONFIG_S390))
+ return true;
+
+ return false;
+}
+
static void pci_slot_release(struct kobject *kobj)
{
struct pci_dev *dev;
@@ -74,7 +90,7 @@ static void pci_slot_release(struct kobject *kobj)
down_read(&pci_bus_sem);
list_for_each_entry(dev, &slot->bus->devices, bus_list)
- if (PCI_SLOT(dev->devfn) == slot->number)
+ if (pci_dev_matches_slot(dev, slot))
dev->slot = NULL;
up_read(&pci_bus_sem);
@@ -167,7 +183,7 @@ void pci_dev_assign_slot(struct pci_dev *dev)
mutex_lock(&pci_slot_mutex);
list_for_each_entry(slot, &dev->bus->slots, list)
- if (PCI_SLOT(dev->devfn) == slot->number)
+ if (pci_dev_matches_slot(dev, slot))
dev->slot = slot;
mutex_unlock(&pci_slot_mutex);
}
@@ -189,7 +205,7 @@ static struct pci_slot *get_slot(struct pci_bus *parent, int slot_nr)
/**
* pci_create_slot - create or increment refcount for physical PCI slot
* @parent: struct pci_bus of parent bridge
- * @slot_nr: PCI_SLOT(pci_dev->devfn) or -1 for placeholder
+ * @slot_nr: PCI_SLOT(pci_dev->devfn) or PCI_SLOT_PLACEHOLDER for placeholder
* @name: user visible string presented in /sys/bus/pci/slots/<name>
* @hotplug: set if caller is hotplug driver, NULL otherwise
*
@@ -214,15 +230,16 @@ static struct pci_slot *get_slot(struct pci_bus *parent, int slot_nr)
* In most cases, @pci_bus, @slot_nr will be sufficient to uniquely identify
* a slot. There is one notable exception - pSeries (rpaphp), where the
* @slot_nr cannot be determined until a device is actually inserted into
- * the slot. In this scenario, the caller may pass -1 for @slot_nr.
+ * the slot. In this scenario, the caller may pass PCI_SLOT_PLACEHOLDER for @slot_nr.
*
* The following semantics are imposed when the caller passes @slot_nr ==
- * -1. First, we no longer check for an existing %struct pci_slot, as there
- * may be many slots with @slot_nr of -1. The other change in semantics is
- * user-visible, which is the 'address' parameter presented in sysfs will
- * consist solely of a dddd:bb tuple, where dddd is the PCI domain of the
- * %struct pci_bus and bb is the bus number. In other words, the devfn of
- * the 'placeholder' slot will not be displayed.
+ * PCI_SLOT_PLACEHOLDER. First, we no longer check for an existing %struct
+ * pci_slot, as there may be many slots with @slot_nr of
+ * PCI_SLOT_PLACEHOLDER. The other change in semantics is user-visible,
+ * which is the 'address' parameter presented in sysfs will consist solely
+ * of a dddd:bb tuple, where dddd is the PCI domain of the %struct pci_bus
+ * and bb is the bus number. In other words, the devfn of the 'placeholder'
+ * slot will not be displayed.
*/
struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr,
const char *name,
@@ -235,7 +252,7 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr,
mutex_lock(&pci_slot_mutex);
- if (slot_nr == -1)
+ if (slot_nr == PCI_SLOT_PLACEHOLDER)
goto placeholder;
/*
@@ -265,6 +282,9 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr,
slot->bus = pci_bus_get(parent);
slot->number = slot_nr;
+ if (pci_slot_enabled_per_func())
+ slot->per_func_slot = 1;
+
slot->kobj.kset = pci_slots_kset;
slot_name = make_slot_name(name);
@@ -287,7 +307,7 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr,
down_read(&pci_bus_sem);
list_for_each_entry(dev, &parent->devices, bus_list)
- if (PCI_SLOT(dev->devfn) == slot_nr)
+ if (pci_dev_matches_slot(dev, slot))
dev->slot = slot;
up_read(&pci_bus_sem);
diff --git a/drivers/perf/riscv_pmu_legacy.c b/drivers/perf/riscv_pmu_legacy.c
index ca9e20bfc7ac..c70ada8178de 100644
--- a/drivers/perf/riscv_pmu_legacy.c
+++ b/drivers/perf/riscv_pmu_legacy.c
@@ -82,8 +82,9 @@ static void pmu_legacy_init(struct riscv_pmu *pmu)
{
pr_info("Legacy PMU implementation is available\n");
- pmu->cmask = BIT(RISCV_PMU_LEGACY_CYCLE) |
- BIT(RISCV_PMU_LEGACY_INSTRET);
+ bitmap_zero(pmu->cmask, RISCV_MAX_COUNTERS);
+ set_bit(RISCV_PMU_LEGACY_CYCLE, pmu->cmask);
+ set_bit(RISCV_PMU_LEGACY_INSTRET, pmu->cmask);
pmu->ctr_start = pmu_legacy_ctr_start;
pmu->ctr_stop = NULL;
pmu->event_map = pmu_legacy_event_map;
diff --git a/drivers/perf/riscv_pmu_sbi.c b/drivers/perf/riscv_pmu_sbi.c
index 5aab43a3ffb9..ed7b95854928 100644
--- a/drivers/perf/riscv_pmu_sbi.c
+++ b/drivers/perf/riscv_pmu_sbi.c
@@ -43,12 +43,15 @@ static const struct attribute_group *riscv_pmu_attr_groups[] = {
};
/*
- * RISC-V doesn't have hetergenous harts yet. This need to be part of
+ * RISC-V doesn't have heterogeneous harts yet. This need to be part of
* per_cpu in case of harts with different pmu counters
*/
static union sbi_pmu_ctr_info *pmu_ctr_list;
static unsigned int riscv_pmu_irq;
+/* Cache the available counters in a bitmask */
+static DECLARE_BITMAP(cmask, RISCV_MAX_COUNTERS);
+
struct sbi_pmu_event_data {
union {
union {
@@ -264,14 +267,60 @@ static bool pmu_sbi_ctr_is_fw(int cidx)
return (info->type == SBI_PMU_CTR_TYPE_FW) ? true : false;
}
+/*
+ * Returns the counter width of a programmable counter and number of hardware
+ * counters. As we don't support heterogeneous CPUs yet, it is okay to just
+ * return the counter width of the first programmable counter.
+ */
+int riscv_pmu_get_hpm_info(u32 *hw_ctr_width, u32 *num_hw_ctr)
+{
+ int i;
+ union sbi_pmu_ctr_info *info;
+ u32 hpm_width = 0, hpm_count = 0;
+
+ if (bitmap_empty(cmask, RISCV_MAX_COUNTERS))
+ return -EINVAL;
+
+ for_each_set_bit(i, cmask, RISCV_MAX_COUNTERS) {
+ info = &pmu_ctr_list[i];
+ if (!info)
+ continue;
+ if (!hpm_width && info->csr != CSR_CYCLE && info->csr != CSR_INSTRET)
+ hpm_width = info->width;
+ if (info->type == SBI_PMU_CTR_TYPE_HW)
+ hpm_count++;
+ }
+
+ *hw_ctr_width = hpm_width;
+ *num_hw_ctr = hpm_count;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(riscv_pmu_get_hpm_info);
+
+static struct sbiret pmu_sbi_ctr_cfg_match(unsigned long cbase,
+ unsigned long ctr_mask,
+ unsigned long cflags,
+ unsigned long event_idx,
+ u64 config)
+{
+#if defined(CONFIG_32BIT)
+ return sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase,
+ ctr_mask, cflags, event_idx, config, config >> 32);
+#else
+ return sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase,
+ ctr_mask, cflags, event_idx, config, 0);
+#endif
+}
+
static int pmu_sbi_ctr_get_idx(struct perf_event *event)
{
struct hw_perf_event *hwc = &event->hw;
struct riscv_pmu *rvpmu = to_riscv_pmu(event->pmu);
struct cpu_hw_events *cpuc = this_cpu_ptr(rvpmu->hw_events);
struct sbiret ret;
- int idx;
- uint64_t cbase = 0;
+ int idx, i;
+ u64 cbase = 0;
unsigned long cflags = 0;
if (event->attr.exclude_kernel)
@@ -280,14 +329,16 @@ static int pmu_sbi_ctr_get_idx(struct perf_event *event)
cflags |= SBI_PMU_CFG_FLAG_SET_UINH;
/* retrieve the available counter index */
-#if defined(CONFIG_32BIT)
- ret = sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase,
- rvpmu->cmask, cflags, hwc->event_base, hwc->config,
- hwc->config >> 32);
-#else
- ret = sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase,
- rvpmu->cmask, cflags, hwc->event_base, hwc->config, 0);
-#endif
+ ret.error = SBI_ERR_NOT_SUPPORTED;
+ for (i = 0; i < BITS_TO_LONGS(RISCV_MAX_COUNTERS); i++) {
+ if (!rvpmu->cmask[i])
+ continue;
+ cbase = i * BITS_PER_LONG;
+ ret = pmu_sbi_ctr_cfg_match(cbase, rvpmu->cmask[i], cflags,
+ hwc->event_base, hwc->config);
+ if (!ret.error)
+ break;
+ }
if (ret.error) {
pr_debug("Not able to find a counter for event %lx config %llx\n",
hwc->event_base, hwc->config);
@@ -295,7 +346,7 @@ static int pmu_sbi_ctr_get_idx(struct perf_event *event)
}
idx = ret.value;
- if (!test_bit(idx, &rvpmu->cmask) || !pmu_ctr_list[idx].value)
+ if (!test_bit(idx, rvpmu->cmask) || !pmu_ctr_list[idx].value)
return -ENOENT;
/* Additional sanity check for the counter id */
@@ -480,7 +531,7 @@ static int pmu_sbi_get_ctrinfo(int nctr, unsigned long *mask)
/* The logical counter ids are not expected to be contiguous */
continue;
- *mask |= BIT(i);
+ set_bit(i, mask);
cinfo.value = ret.value;
if (cinfo.type == SBI_PMU_CTR_TYPE_FW)
@@ -497,12 +548,19 @@ static int pmu_sbi_get_ctrinfo(int nctr, unsigned long *mask)
static inline void pmu_sbi_stop_all(struct riscv_pmu *pmu)
{
+ int i;
+
/*
* No need to check the error because we are disabling all the counters
* which may include counters that are not enabled yet.
*/
- sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_STOP,
- 0, pmu->cmask, SBI_PMU_STOP_FLAG_RESET, 0, 0, 0);
+ for (i = 0; i < BITS_TO_LONGS(RISCV_MAX_COUNTERS); i++) {
+ if (!pmu->cmask[i])
+ continue;
+ sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_STOP,
+ i * BITS_PER_LONG, pmu->cmask[i],
+ SBI_PMU_STOP_FLAG_RESET, 0, 0, 0);
+ }
}
static inline void pmu_sbi_stop_hw_ctrs(struct riscv_pmu *pmu)
@@ -617,14 +675,14 @@ static irqreturn_t pmu_sbi_ovf_handler(int irq, void *dev)
/* compute hardware counter index */
hidx = info->csr - CSR_CYCLE;
/* check if the corresponding bit is set in sscountovf */
- if (!(overflow & (1 << hidx)))
+ if (!(overflow & BIT(hidx)))
continue;
/*
* Keep a track of overflowed counters so that they can be started
* with updated initial value.
*/
- overflowed_ctrs |= 1 << lidx;
+ overflowed_ctrs |= BIT(lidx);
hw_evt = &event->hw;
riscv_pmu_event_update(event);
perf_sample_data_init(&data, 0, hw_evt->last_period);
@@ -798,7 +856,6 @@ static void riscv_pmu_destroy(struct riscv_pmu *pmu)
static int pmu_sbi_device_probe(struct platform_device *pdev)
{
struct riscv_pmu *pmu = NULL;
- unsigned long cmask = 0;
int ret = -ENODEV;
int num_counters;
@@ -813,8 +870,14 @@ static int pmu_sbi_device_probe(struct platform_device *pdev)
goto out_free;
}
+ /* It is possible to get from SBI more than max number of counters */
+ if (num_counters > RISCV_MAX_COUNTERS) {
+ num_counters = RISCV_MAX_COUNTERS;
+ pr_info("SBI returned more than maximum number of counters. Limiting the number of counters to %d\n", num_counters);
+ }
+
/* cache all the information about counters now */
- if (pmu_sbi_get_ctrinfo(num_counters, &cmask))
+ if (pmu_sbi_get_ctrinfo(num_counters, cmask))
goto out_free;
ret = pmu_sbi_setup_irqs(pmu, pdev);
@@ -825,7 +888,7 @@ static int pmu_sbi_device_probe(struct platform_device *pdev)
}
pmu->pmu.attr_groups = riscv_pmu_attr_groups;
- pmu->cmask = cmask;
+ bitmap_copy(pmu->cmask, cmask, RISCV_MAX_COUNTERS);
pmu->ctr_start = pmu_sbi_ctr_start;
pmu->ctr_stop = pmu_sbi_ctr_stop;
pmu->event_map = pmu_sbi_event_map;
diff --git a/drivers/phy/qualcomm/phy-qcom-qusb2.c b/drivers/phy/qualcomm/phy-qcom-qusb2.c
index 2ef638b32e8f..9ee689e3ae33 100644
--- a/drivers/phy/qualcomm/phy-qcom-qusb2.c
+++ b/drivers/phy/qualcomm/phy-qcom-qusb2.c
@@ -1061,31 +1061,31 @@ static int qusb2_phy_probe(struct platform_device *pdev)
or->hsdisc_trim.override = true;
}
- pm_runtime_set_active(dev);
- pm_runtime_enable(dev);
+ dev_set_drvdata(dev, qphy);
+
/*
- * Prevent runtime pm from being ON by default. Users can enable
- * it using power/control in sysfs.
+ * Enable runtime PM support, but forbid it by default.
+ * Users can allow it again via the power/control attribute in sysfs.
*/
+ pm_runtime_set_active(dev);
pm_runtime_forbid(dev);
+ ret = devm_pm_runtime_enable(dev);
+ if (ret)
+ return ret;
generic_phy = devm_phy_create(dev, NULL, &qusb2_phy_gen_ops);
if (IS_ERR(generic_phy)) {
ret = PTR_ERR(generic_phy);
dev_err(dev, "failed to create phy, %d\n", ret);
- pm_runtime_disable(dev);
return ret;
}
qphy->phy = generic_phy;
- dev_set_drvdata(dev, qphy);
phy_set_drvdata(generic_phy, qphy);
phy_provider = devm_of_phy_provider_register(dev, of_phy_simple_xlate);
if (!IS_ERR(phy_provider))
dev_info(dev, "Registered Qcom-QUSB2 phy\n");
- else
- pm_runtime_disable(dev);
return PTR_ERR_OR_ZERO(phy_provider);
}
diff --git a/drivers/pinctrl/core.c b/drivers/pinctrl/core.c
index 03bfdcbc3325..1c3dcb270ea0 100644
--- a/drivers/pinctrl/core.c
+++ b/drivers/pinctrl/core.c
@@ -221,6 +221,9 @@ static int pinctrl_register_one_pin(struct pinctrl_dev *pctldev,
/* Set owner */
pindesc->pctldev = pctldev;
+#ifdef CONFIG_PINMUX
+ mutex_init(&pindesc->mux_lock);
+#endif
/* Copy basic pin info */
if (pin->name) {
diff --git a/drivers/pinctrl/core.h b/drivers/pinctrl/core.h
index 840103c40c14..f5e4ca19c17b 100644
--- a/drivers/pinctrl/core.h
+++ b/drivers/pinctrl/core.h
@@ -167,6 +167,7 @@ struct pin_desc {
const char *mux_owner;
const struct pinctrl_setting_mux *mux_setting;
const char *gpio_owner;
+ struct mutex mux_lock;
#endif
};
diff --git a/drivers/pinctrl/freescale/pinctrl-scu.c b/drivers/pinctrl/freescale/pinctrl-scu.c
index 59b5f8a35111..c38ec7ee439d 100644
--- a/drivers/pinctrl/freescale/pinctrl-scu.c
+++ b/drivers/pinctrl/freescale/pinctrl-scu.c
@@ -77,7 +77,7 @@ int imx_pinconf_set_scu(struct pinctrl_dev *pctldev, unsigned pin_id,
struct imx_sc_msg_req_pad_set msg;
struct imx_sc_rpc_msg *hdr = &msg.hdr;
unsigned int mux = configs[0];
- unsigned int conf = configs[1];
+ unsigned int conf;
unsigned int val;
int ret;
@@ -85,6 +85,7 @@ int imx_pinconf_set_scu(struct pinctrl_dev *pctldev, unsigned pin_id,
* Set mux and conf together in one IPC call
*/
WARN_ON(num_configs != 2);
+ conf = configs[1];
val = conf | BM_PAD_CTL_IFMUX_ENABLE | BM_PAD_CTL_GP_ENABLE;
val |= mux << BP_PAD_CTL_IFMUX;
diff --git a/drivers/pinctrl/mediatek/pinctrl-moore.c b/drivers/pinctrl/mediatek/pinctrl-moore.c
index 319da66d4f32..e91987d33fc6 100644
--- a/drivers/pinctrl/mediatek/pinctrl-moore.c
+++ b/drivers/pinctrl/mediatek/pinctrl-moore.c
@@ -538,7 +538,7 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
chip->ngpio = hw->soc->npins;
chip->of_gpio_n_cells = 2;
- ret = gpiochip_add_data(chip, hw);
+ ret = devm_gpiochip_add_data(hw->dev, chip, hw);
if (ret < 0)
return ret;
@@ -552,10 +552,8 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
if (!of_find_property(hw->dev->of_node, "gpio-ranges", NULL)) {
ret = gpiochip_add_pin_range(chip, dev_name(hw->dev), 0, 0,
chip->ngpio);
- if (ret < 0) {
- gpiochip_remove(chip);
+ if (ret < 0)
return ret;
- }
}
return 0;
diff --git a/drivers/pinctrl/mediatek/pinctrl-mtk-common.c b/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
index fe44867b4c28..1dada2b64b9e 100644
--- a/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
+++ b/drivers/pinctrl/mediatek/pinctrl-mtk-common.c
@@ -1121,30 +1121,24 @@ int mtk_pctrl_init(struct platform_device *pdev,
pctl->chip->parent = &pdev->dev;
pctl->chip->base = -1;
- ret = gpiochip_add_data(pctl->chip, pctl);
+ ret = devm_gpiochip_add_data(&pdev->dev, pctl->chip, pctl);
if (ret)
return -EINVAL;
/* Register the GPIO to pin mappings. */
ret = gpiochip_add_pin_range(pctl->chip, dev_name(&pdev->dev),
0, 0, pctl->devdata->npins);
- if (ret) {
- ret = -EINVAL;
- goto chip_error;
- }
+ if (ret)
+ return -EINVAL;
/* Only initialize EINT if we have EINT pins */
if (data->eint_hw.ap_num > 0) {
ret = mtk_eint_init(pctl, pdev);
if (ret)
- goto chip_error;
+ return ret;
}
return 0;
-
-chip_error:
- gpiochip_remove(pctl->chip);
- return ret;
}
int mtk_pctrl_common_probe(struct platform_device *pdev)
diff --git a/drivers/pinctrl/mediatek/pinctrl-paris.c b/drivers/pinctrl/mediatek/pinctrl-paris.c
index ab1c34e5a582..bb9c70a1333b 100644
--- a/drivers/pinctrl/mediatek/pinctrl-paris.c
+++ b/drivers/pinctrl/mediatek/pinctrl-paris.c
@@ -770,7 +770,6 @@ static const struct pinmux_ops mtk_pmxops = {
.get_function_name = mtk_pmx_get_func_name,
.get_function_groups = mtk_pmx_get_func_groups,
.set_mux = mtk_pmx_set_mux,
- .gpio_set_direction = mtk_pinmux_gpio_set_direction,
.gpio_request_enable = mtk_pinmux_gpio_request_enable,
};
@@ -899,7 +898,7 @@ static int mtk_gpio_direction_input(struct gpio_chip *chip, unsigned int gpio)
if (gpio >= hw->soc->npins)
return -EINVAL;
- return pinctrl_gpio_direction_input(chip, gpio);
+ return mtk_pinmux_gpio_set_direction(hw->pctrl, NULL, gpio, true);
}
static int mtk_gpio_direction_output(struct gpio_chip *chip, unsigned int gpio,
@@ -912,7 +911,7 @@ static int mtk_gpio_direction_output(struct gpio_chip *chip, unsigned int gpio,
mtk_gpio_set(chip, gpio, value);
- return pinctrl_gpio_direction_output(chip, gpio);
+ return mtk_pinmux_gpio_set_direction(hw->pctrl, NULL, gpio, false);
}
static int mtk_gpio_to_irq(struct gpio_chip *chip, unsigned int offset)
@@ -970,7 +969,7 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw)
chip->ngpio = hw->soc->npins;
chip->of_gpio_n_cells = 2;
- ret = gpiochip_add_data(chip, hw);
+ ret = devm_gpiochip_add_data(hw->dev, chip, hw);
if (ret < 0)
return ret;
diff --git a/drivers/pinctrl/pinctrl-amd.c b/drivers/pinctrl/pinctrl-amd.c
index 51506492302e..95de96cdd807 100644
--- a/drivers/pinctrl/pinctrl-amd.c
+++ b/drivers/pinctrl/pinctrl-amd.c
@@ -871,6 +871,15 @@ static void amd_gpio_irq_init(struct amd_gpio *gpio_dev)
mask = BIT(WAKE_CNTRL_OFF_S0I3) | BIT(WAKE_CNTRL_OFF_S3);
+#ifdef CONFIG_ACPI
+ /*
+ * Platforms without _AEI leave GPIO wake sources to firmware, so
+ * preserve their S4 wake bits.
+ */
+ if (acpi_has_method(ACPI_HANDLE(&gpio_dev->pdev->dev), "_AEI"))
+ mask |= BIT(WAKE_CNTRL_OFF_S4);
+#endif
+
for (i = 0; i < desc->npins; i++) {
int pin = desc->pins[i].number;
const struct pin_desc *pd = pin_desc_get(gpio_dev->pctrl, pin);
diff --git a/drivers/pinctrl/pinmux.c b/drivers/pinctrl/pinmux.c
index 0cbf698d16db..7290a127f13c 100644
--- a/drivers/pinctrl/pinmux.c
+++ b/drivers/pinctrl/pinmux.c
@@ -13,6 +13,7 @@
#define pr_fmt(fmt) "pinmux core: " fmt
#include <linux/ctype.h>
+#include <linux/cleanup.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/init.h>
@@ -90,6 +91,7 @@ bool pinmux_can_be_used_for_gpio(struct pinctrl_dev *pctldev, unsigned pin)
if (!desc || !ops)
return true;
+ guard(mutex)(&desc->mux_lock);
if (ops->strict && desc->mux_usecount)
return false;
@@ -124,29 +126,31 @@ static int pin_request(struct pinctrl_dev *pctldev,
dev_dbg(pctldev->dev, "request pin %d (%s) for %s\n",
pin, desc->name, owner);
- if ((!gpio_range || ops->strict) &&
- desc->mux_usecount && strcmp(desc->mux_owner, owner)) {
- dev_err(pctldev->dev,
- "pin %s already requested by %s; cannot claim for %s\n",
- desc->name, desc->mux_owner, owner);
- goto out;
- }
+ scoped_guard(mutex, &desc->mux_lock) {
+ if ((!gpio_range || ops->strict) &&
+ desc->mux_usecount && strcmp(desc->mux_owner, owner)) {
+ dev_err(pctldev->dev,
+ "pin %s already requested by %s; cannot claim for %s\n",
+ desc->name, desc->mux_owner, owner);
+ goto out;
+ }
- if ((gpio_range || ops->strict) && desc->gpio_owner) {
- dev_err(pctldev->dev,
- "pin %s already requested by %s; cannot claim for %s\n",
- desc->name, desc->gpio_owner, owner);
- goto out;
- }
+ if ((gpio_range || ops->strict) && desc->gpio_owner) {
+ dev_err(pctldev->dev,
+ "pin %s already requested by %s; cannot claim for %s\n",
+ desc->name, desc->gpio_owner, owner);
+ goto out;
+ }
- if (gpio_range) {
- desc->gpio_owner = owner;
- } else {
- desc->mux_usecount++;
- if (desc->mux_usecount > 1)
- return 0;
+ if (gpio_range) {
+ desc->gpio_owner = owner;
+ } else {
+ desc->mux_usecount++;
+ if (desc->mux_usecount > 1)
+ return 0;
- desc->mux_owner = owner;
+ desc->mux_owner = owner;
+ }
}
/* Let each pin increase references to this module */
@@ -177,12 +181,14 @@ static int pin_request(struct pinctrl_dev *pctldev,
out_free_pin:
if (status) {
- if (gpio_range) {
- desc->gpio_owner = NULL;
- } else {
- desc->mux_usecount--;
- if (!desc->mux_usecount)
- desc->mux_owner = NULL;
+ scoped_guard(mutex, &desc->mux_lock) {
+ if (gpio_range) {
+ desc->gpio_owner = NULL;
+ } else {
+ desc->mux_usecount--;
+ if (!desc->mux_usecount)
+ desc->mux_owner = NULL;
+ }
}
}
out:
@@ -218,15 +224,26 @@ static const char *pin_free(struct pinctrl_dev *pctldev, int pin,
return NULL;
}
- if (!gpio_range) {
- /*
- * A pin should not be freed more times than allocated.
- */
- if (WARN_ON(!desc->mux_usecount))
- return NULL;
- desc->mux_usecount--;
- if (desc->mux_usecount)
- return NULL;
+ scoped_guard(mutex, &desc->mux_lock) {
+ if (!gpio_range) {
+ /*
+ * A pin should not be freed more times than allocated.
+ */
+ if (WARN_ON(!desc->mux_usecount))
+ return NULL;
+ desc->mux_usecount--;
+ if (desc->mux_usecount)
+ return NULL;
+ }
+
+ if (gpio_range) {
+ owner = desc->gpio_owner;
+ desc->gpio_owner = NULL;
+ } else {
+ owner = desc->mux_owner;
+ desc->mux_owner = NULL;
+ desc->mux_setting = NULL;
+ }
}
/*
@@ -238,15 +255,6 @@ static const char *pin_free(struct pinctrl_dev *pctldev, int pin,
else if (ops->free)
ops->free(pctldev, pin);
- if (gpio_range) {
- owner = desc->gpio_owner;
- desc->gpio_owner = NULL;
- } else {
- owner = desc->mux_owner;
- desc->mux_owner = NULL;
- desc->mux_setting = NULL;
- }
-
module_put(pctldev->owner);
return owner;
@@ -458,7 +466,8 @@ int pinmux_enable_setting(const struct pinctrl_setting *setting)
pins[i]);
continue;
}
- desc->mux_setting = &(setting->data.mux);
+ scoped_guard(mutex, &desc->mux_lock)
+ desc->mux_setting = &(setting->data.mux);
}
ret = ops->set_mux(pctldev, setting->data.mux.func,
@@ -472,8 +481,10 @@ int pinmux_enable_setting(const struct pinctrl_setting *setting)
err_set_mux:
for (i = 0; i < num_pins; i++) {
desc = pin_desc_get(pctldev, pins[i]);
- if (desc)
- desc->mux_setting = NULL;
+ if (desc) {
+ scoped_guard(mutex, &desc->mux_lock)
+ desc->mux_setting = NULL;
+ }
}
err_pin_request:
/* On error release all taken pins */
@@ -492,6 +503,7 @@ void pinmux_disable_setting(const struct pinctrl_setting *setting)
unsigned num_pins = 0;
int i;
struct pin_desc *desc;
+ bool is_equal;
if (pctlops->get_group_pins)
ret = pctlops->get_group_pins(pctldev, setting->data.mux.group,
@@ -517,7 +529,10 @@ void pinmux_disable_setting(const struct pinctrl_setting *setting)
pins[i]);
continue;
}
- if (desc->mux_setting == &(setting->data.mux)) {
+ scoped_guard(mutex, &desc->mux_lock)
+ is_equal = (desc->mux_setting == &(setting->data.mux));
+
+ if (is_equal) {
pin_free(pctldev, pins[i], NULL);
} else {
const char *gname;
@@ -609,40 +624,42 @@ static int pinmux_pins_show(struct seq_file *s, void *what)
if (desc == NULL)
continue;
- if (desc->mux_owner &&
- !strcmp(desc->mux_owner, pinctrl_dev_get_name(pctldev)))
- is_hog = true;
-
- if (pmxops->strict) {
- if (desc->mux_owner)
- seq_printf(s, "pin %d (%s): device %s%s",
- pin, desc->name, desc->mux_owner,
+ scoped_guard(mutex, &desc->mux_lock) {
+ if (desc->mux_owner &&
+ !strcmp(desc->mux_owner, pinctrl_dev_get_name(pctldev)))
+ is_hog = true;
+
+ if (pmxops->strict) {
+ if (desc->mux_owner)
+ seq_printf(s, "pin %d (%s): device %s%s",
+ pin, desc->name, desc->mux_owner,
+ is_hog ? " (HOG)" : "");
+ else if (desc->gpio_owner)
+ seq_printf(s, "pin %d (%s): GPIO %s",
+ pin, desc->name, desc->gpio_owner);
+ else
+ seq_printf(s, "pin %d (%s): UNCLAIMED",
+ pin, desc->name);
+ } else {
+ /* For non-strict controllers */
+ seq_printf(s, "pin %d (%s): %s %s%s", pin, desc->name,
+ desc->mux_owner ? desc->mux_owner
+ : "(MUX UNCLAIMED)",
+ desc->gpio_owner ? desc->gpio_owner
+ : "(GPIO UNCLAIMED)",
is_hog ? " (HOG)" : "");
- else if (desc->gpio_owner)
- seq_printf(s, "pin %d (%s): GPIO %s",
- pin, desc->name, desc->gpio_owner);
+ }
+
+ /* If mux: print function+group claiming the pin */
+ if (desc->mux_setting)
+ seq_printf(s, " function %s group %s\n",
+ pmxops->get_function_name(pctldev,
+ desc->mux_setting->func),
+ pctlops->get_group_name(pctldev,
+ desc->mux_setting->group));
else
- seq_printf(s, "pin %d (%s): UNCLAIMED",
- pin, desc->name);
- } else {
- /* For non-strict controllers */
- seq_printf(s, "pin %d (%s): %s %s%s", pin, desc->name,
- desc->mux_owner ? desc->mux_owner
- : "(MUX UNCLAIMED)",
- desc->gpio_owner ? desc->gpio_owner
- : "(GPIO UNCLAIMED)",
- is_hog ? " (HOG)" : "");
+ seq_putc(s, '\n');
}
-
- /* If mux: print function+group claiming the pin */
- if (desc->mux_setting)
- seq_printf(s, " function %s group %s\n",
- pmxops->get_function_name(pctldev,
- desc->mux_setting->func),
- pctlops->get_group_name(pctldev,
- desc->mux_setting->group));
- else
- seq_putc(s, '\n');
}
mutex_unlock(&pctldev->mutex);
diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.c b/drivers/pinctrl/sunxi/pinctrl-sunxi.c
index 508aa911c8d9..c7001f0eb676 100644
--- a/drivers/pinctrl/sunxi/pinctrl-sunxi.c
+++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.c
@@ -793,6 +793,21 @@ static void sunxi_pmx_set(struct pinctrl_dev *pctldev,
writel((readl(pctl->membase + reg) & ~mask) | config << shift,
pctl->membase + reg);
+ /*
+ * A pin muxed to gpio_out directly through a pinmux node bypasses
+ * sunxi_pinctrl_gpio_set() and drives whatever its output latch
+ * holds. Now that the pin is in output mode the data register
+ * reads back the latch, so refresh the shadow to keep such pins
+ * driving their pre-existing level.
+ */
+ if (config == SUN4I_FUNC_OUTPUT) {
+ u32 *shadow = &pctl->dat_shadow[pin / PINS_PER_BANK];
+
+ sunxi_data_reg(pctl, pin, ®, &shift, &mask);
+ *shadow = (*shadow & ~mask) |
+ (readl(pctl->membase + reg) & mask);
+ }
+
raw_spin_unlock_irqrestore(&pctl->lock, flags);
}
@@ -946,21 +961,29 @@ static void sunxi_pinctrl_gpio_set(struct gpio_chip *chip,
unsigned offset, int value)
{
struct sunxi_pinctrl *pctl = gpiochip_get_data(chip);
- u32 reg, shift, mask, val;
+ u32 *shadow = &pctl->dat_shadow[offset / PINS_PER_BANK];
+ u32 reg, shift, mask;
unsigned long flags;
sunxi_data_reg(pctl, offset, ®, &shift, &mask);
raw_spin_lock_irqsave(&pctl->lock, flags);
- val = readl(pctl->membase + reg);
-
+ /*
+ * Reading the data register returns the pin level, not the output
+ * latch, for pins muxed as inputs. A read-modify-write based on
+ * the register would therefore corrupt the latches of input-muxed
+ * pins in the same bank (e.g. an emulated open-drain I2C line
+ * released high), making them drive the wrong level once switched
+ * to output. Base the read-modify-write on a shadow copy of the
+ * latches instead.
+ */
if (value)
- val |= mask;
+ *shadow |= mask;
else
- val &= ~mask;
+ *shadow &= ~mask;
- writel(val, pctl->membase + reg);
+ writel(*shadow, pctl->membase + reg);
raw_spin_unlock_irqrestore(&pctl->lock, flags);
}
@@ -1484,7 +1507,7 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
struct pinctrl_pin_desc *pins;
struct sunxi_pinctrl *pctl;
struct pinmux_ops *pmxops;
- int i, ret, last_pin, pin_idx;
+ int i, ret, last_pin, pin_idx, nbanks;
struct clk *clk;
pctl = devm_kzalloc(&pdev->dev, sizeof(*pctl), GFP_KERNEL);
@@ -1518,6 +1541,37 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
if (!pctl->irq_array)
return -ENOMEM;
+ /*
+ * The bus clock has to be enabled before the pinctrl device
+ * registers, as the pin hogs claimed from there access registers.
+ */
+ ret = of_clk_get_parent_count(node);
+ clk = devm_clk_get_enabled(&pdev->dev, ret == 1 ? NULL : "apb");
+ if (IS_ERR(clk))
+ return PTR_ERR(clk);
+
+ /*
+ * Seed the output latch shadow from the hardware so pins the
+ * bootloader left in output mode keep their state; see
+ * sunxi_pinctrl_gpio_set() for why a shadow is needed. This must
+ * happen before the pinctrl device registers, as pin hogs can mux
+ * pins to gpio_out and thereby update the shadow.
+ */
+ last_pin = pctl->desc->pins[pctl->desc->npins - 1].pin.number;
+ nbanks = DIV_ROUND_UP(last_pin + 1 - pctl->desc->pin_base,
+ PINS_PER_BANK);
+ pctl->dat_shadow = devm_kcalloc(&pdev->dev, nbanks,
+ sizeof(*pctl->dat_shadow), GFP_KERNEL);
+ if (!pctl->dat_shadow)
+ return -ENOMEM;
+
+ for (i = 0; i < nbanks; i++) {
+ u32 reg, shift, mask;
+
+ sunxi_data_reg(pctl, i * PINS_PER_BANK, ®, &shift, &mask);
+ pctl->dat_shadow[i] = readl(pctl->membase + reg);
+ }
+
ret = sunxi_pinctrl_build_state(pdev);
if (ret) {
dev_err(&pdev->dev, "dt probe failed: %d\n", ret);
@@ -1572,7 +1626,6 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
if (!pctl->chip)
return -ENOMEM;
- last_pin = pctl->desc->pins[pctl->desc->npins - 1].pin.number;
pctl->chip->owner = THIS_MODULE;
pctl->chip->request = gpiochip_generic_request;
pctl->chip->free = gpiochip_generic_free;
@@ -1605,31 +1658,20 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
goto gpiochip_error;
}
- ret = of_clk_get_parent_count(node);
- clk = devm_clk_get(&pdev->dev, ret == 1 ? NULL : "apb");
- if (IS_ERR(clk)) {
- ret = PTR_ERR(clk);
- goto gpiochip_error;
- }
-
- ret = clk_prepare_enable(clk);
- if (ret)
- goto gpiochip_error;
-
pctl->irq = devm_kcalloc(&pdev->dev,
pctl->desc->irq_banks,
sizeof(*pctl->irq),
GFP_KERNEL);
if (!pctl->irq) {
ret = -ENOMEM;
- goto clk_error;
+ goto gpiochip_error;
}
for (i = 0; i < pctl->desc->irq_banks; i++) {
pctl->irq[i] = platform_get_irq(pdev, i);
if (pctl->irq[i] < 0) {
ret = pctl->irq[i];
- goto clk_error;
+ goto gpiochip_error;
}
}
@@ -1640,7 +1682,7 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
if (!pctl->domain) {
dev_err(&pdev->dev, "Couldn't register IRQ domain\n");
ret = -ENOMEM;
- goto clk_error;
+ goto gpiochip_error;
}
for (i = 0; i < (pctl->desc->irq_banks * IRQ_PER_BANK); i++) {
@@ -1672,8 +1714,6 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev,
return 0;
-clk_error:
- clk_disable_unprepare(clk);
gpiochip_error:
gpiochip_remove(pctl->chip);
return ret;
diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.h b/drivers/pinctrl/sunxi/pinctrl-sunxi.h
index a87a2f944d60..244b0039480a 100644
--- a/drivers/pinctrl/sunxi/pinctrl-sunxi.h
+++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.h
@@ -80,6 +80,7 @@
#define IO_BIAS_MASK GENMASK(3, 0)
#define SUN4I_FUNC_INPUT 0
+#define SUN4I_FUNC_OUTPUT 1
#define SUN4I_FUNC_IRQ 6
#define PINCTRL_SUN5I_A10S BIT(1)
@@ -173,6 +174,12 @@ struct sunxi_pinctrl {
int *irq;
unsigned *irq_array;
raw_spinlock_t lock;
+ /*
+ * Output latch shadow, one word per bank. Seeded lockless at
+ * probe before the pinctrl device registers, protected by @lock
+ * afterwards.
+ */
+ u32 *dat_shadow;
struct pinctrl_dev *pctl_dev;
unsigned long variant;
u32 bank_mem_size;
diff --git a/drivers/platform/x86/amd/pmc.c b/drivers/platform/x86/amd/pmc.c
index 8eaeb1e8f975..8eb5ddeca0d4 100644
--- a/drivers/platform/x86/amd/pmc.c
+++ b/drivers/platform/x86/amd/pmc.c
@@ -872,21 +872,31 @@ static int amd_pmc_s2d_init(struct amd_pmc_dev *dev)
u32 phys_addr_low, phys_addr_hi;
u64 stb_phys_addr;
u32 size = 0;
+ int ret;
/* Spill to DRAM feature uses separate SMU message port */
dev->msg_port = 1;
- amd_pmc_send_cmd(dev, S2D_TELEMETRY_SIZE, &size, STB_SPILL_TO_DRAM, 1);
- if (size != S2D_TELEMETRY_BYTES_MAX)
- return -EIO;
+ ret = amd_pmc_send_cmd(dev, S2D_TELEMETRY_SIZE, &size, STB_SPILL_TO_DRAM, 1);
+ if (ret)
+ goto out;
+ if (size != S2D_TELEMETRY_BYTES_MAX) {
+ ret = -EIO;
+ goto out;
+ }
/* Get STB DRAM address */
- amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_LOW, &phys_addr_low, STB_SPILL_TO_DRAM, 1);
- amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_HIGH, &phys_addr_hi, STB_SPILL_TO_DRAM, 1);
+ ret = amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_LOW, &phys_addr_low, STB_SPILL_TO_DRAM, 1);
+ if (ret)
+ goto out;
+ ret = amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_HIGH, &phys_addr_hi, STB_SPILL_TO_DRAM, 1);
+ if (ret)
+ goto out;
if (!phys_addr_hi && !phys_addr_low) {
dev_err(dev->dev, "STB is not enabled on the system; disable enable_stb or contact system vendor\n");
- return -EINVAL;
+ ret = -EINVAL;
+ goto out;
}
stb_phys_addr = ((u64)phys_addr_hi << 32 | phys_addr_low);
@@ -899,6 +909,10 @@ static int amd_pmc_s2d_init(struct amd_pmc_dev *dev)
return -ENOMEM;
return 0;
+
+out:
+ dev->msg_port = 0;
+ return ret;
}
#ifdef CONFIG_SUSPEND
diff --git a/drivers/platform/x86/asus-wmi.c b/drivers/platform/x86/asus-wmi.c
index ed6ea8782558..8814b258caa5 100644
--- a/drivers/platform/x86/asus-wmi.c
+++ b/drivers/platform/x86/asus-wmi.c
@@ -900,7 +900,7 @@ static ssize_t charge_control_end_threshold_show(struct device *device,
static DEVICE_ATTR_RW(charge_control_end_threshold);
-static int asus_wmi_battery_add(struct power_supply *battery)
+static int asus_wmi_battery_add(struct power_supply *battery, struct acpi_battery_hook *hook)
{
/* The WMI method does not provide a way to specific a battery, so we
* just assume it is the first battery.
@@ -927,7 +927,7 @@ static int asus_wmi_battery_add(struct power_supply *battery)
return 0;
}
-static int asus_wmi_battery_remove(struct power_supply *battery)
+static int asus_wmi_battery_remove(struct power_supply *battery, struct acpi_battery_hook *hook)
{
device_remove_file(&battery->dev,
&dev_attr_charge_control_end_threshold);
@@ -3859,13 +3859,13 @@ static int asus_wmi_add(struct platform_device *pdev)
fail_rfkill:
asus_wmi_led_exit(asus);
fail_leds:
+fail_custom_fan_curve:
fail_hwmon:
asus_wmi_input_exit(asus);
fail_input:
asus_wmi_sysfs_exit(asus->platform_device);
fail_sysfs:
fail_throttle_thermal_policy:
-fail_custom_fan_curve:
fail_platform_profile_setup:
if (asus->platform_profile_support)
platform_profile_remove();
diff --git a/drivers/platform/x86/dell/dell-laptop.c b/drivers/platform/x86/dell/dell-laptop.c
index 215861c11dc4..5288f080dd14 100644
--- a/drivers/platform/x86/dell/dell-laptop.c
+++ b/drivers/platform/x86/dell/dell-laptop.c
@@ -361,29 +361,6 @@ static const struct dmi_system_id dell_quirks[] __initconst = {
{ }
};
-static void dell_fill_request(struct calling_interface_buffer *buffer,
- u32 arg0, u32 arg1, u32 arg2, u32 arg3)
-{
- memset(buffer, 0, sizeof(struct calling_interface_buffer));
- buffer->input[0] = arg0;
- buffer->input[1] = arg1;
- buffer->input[2] = arg2;
- buffer->input[3] = arg3;
-}
-
-static int dell_send_request(struct calling_interface_buffer *buffer,
- u16 class, u16 select)
-{
- int ret;
-
- buffer->cmd_class = class;
- buffer->cmd_select = select;
- ret = dell_smbios_call(buffer);
- if (ret != 0)
- return ret;
- return dell_smbios_error(buffer->output[0]);
-}
-
/*
* Derived from information in smbios-wireless-ctl:
*
@@ -2287,6 +2264,11 @@ static int __init dell_init(void)
if (micmute_led_registered)
led_classdev_unregister(&micmute_led_cdev);
fail_led:
+ dell_laptop_unregister_notifier(&dell_laptop_notifier);
+ debugfs_remove_recursive(dell_laptop_dir);
+ kbd_led_exit();
+ if (quirks && quirks->touchpad_led)
+ touchpad_led_exit();
dell_cleanup_rfkill();
fail_rfkill:
platform_device_del(platform_device);
diff --git a/drivers/platform/x86/dell/dell-smbios-base.c b/drivers/platform/x86/dell/dell-smbios-base.c
index 9a9b9feac416..636d881c9477 100644
--- a/drivers/platform/x86/dell/dell-smbios-base.c
+++ b/drivers/platform/x86/dell/dell-smbios-base.c
@@ -320,6 +320,31 @@ int dell_smbios_call(struct calling_interface_buffer *buffer)
}
EXPORT_SYMBOL_GPL(dell_smbios_call);
+void dell_fill_request(struct calling_interface_buffer *buffer,
+ u32 arg0, u32 arg1, u32 arg2, u32 arg3)
+{
+ memset(buffer, 0, sizeof(struct calling_interface_buffer));
+ buffer->input[0] = arg0;
+ buffer->input[1] = arg1;
+ buffer->input[2] = arg2;
+ buffer->input[3] = arg3;
+}
+EXPORT_SYMBOL_GPL(dell_fill_request);
+
+int dell_send_request(struct calling_interface_buffer *buffer,
+ u16 class, u16 select)
+{
+ int ret;
+
+ buffer->cmd_class = class;
+ buffer->cmd_select = select;
+ ret = dell_smbios_call(buffer);
+ if (ret != 0)
+ return ret;
+ return dell_smbios_error(buffer->output[0]);
+}
+EXPORT_SYMBOL_GPL(dell_send_request);
+
struct calling_interface_token *dell_smbios_find_token(int tokenid)
{
int i;
diff --git a/drivers/platform/x86/dell/dell-smbios.h b/drivers/platform/x86/dell/dell-smbios.h
index 75fa8ea0476d..5706fdd1318c 100644
--- a/drivers/platform/x86/dell/dell-smbios.h
+++ b/drivers/platform/x86/dell/dell-smbios.h
@@ -62,6 +62,11 @@ int dell_smbios_call_filter(struct device *d,
struct calling_interface_buffer *buffer);
int dell_smbios_call(struct calling_interface_buffer *buffer);
+void dell_fill_request(struct calling_interface_buffer *buffer,
+ u32 arg0, u32 arg1, u32 arg2, u32 arg3);
+int dell_send_request(struct calling_interface_buffer *buffer,
+ u16 class, u16 select);
+
struct calling_interface_token *dell_smbios_find_token(int tokenid);
enum dell_laptop_notifier_actions {
diff --git a/drivers/platform/x86/dell/dell-wmi-base.c b/drivers/platform/x86/dell/dell-wmi-base.c
index 713ec2c12a04..4838992ee5f9 100644
--- a/drivers/platform/x86/dell/dell-wmi-base.c
+++ b/drivers/platform/x86/dell/dell-wmi-base.c
@@ -442,7 +442,7 @@ static int dell_wmi_process_key(struct wmi_device *wdev, int type, int code, u16
SW_TABLET_MODE, !buffer[0]);
return 1;
} else if (type == 0x0012 && code == 0x000d && remaining > 0) {
- value = (buffer[2] == 2);
+ value = (buffer[0] == 2);
used = 1;
}
diff --git a/drivers/platform/x86/huawei-wmi.c b/drivers/platform/x86/huawei-wmi.c
index 201e1f2e2c2f..3721879bb847 100644
--- a/drivers/platform/x86/huawei-wmi.c
+++ b/drivers/platform/x86/huawei-wmi.c
@@ -474,7 +474,7 @@ static DEVICE_ATTR_RW(charge_control_start_threshold);
static DEVICE_ATTR_RW(charge_control_end_threshold);
static DEVICE_ATTR_RW(charge_control_thresholds);
-static int huawei_wmi_battery_add(struct power_supply *battery)
+static int huawei_wmi_battery_add(struct power_supply *battery, struct acpi_battery_hook *hook)
{
int err = 0;
@@ -489,7 +489,7 @@ static int huawei_wmi_battery_add(struct power_supply *battery)
return err;
}
-static int huawei_wmi_battery_remove(struct power_supply *battery)
+static int huawei_wmi_battery_remove(struct power_supply *battery, struct acpi_battery_hook *hook)
{
device_remove_file(&battery->dev, &dev_attr_charge_control_start_threshold);
device_remove_file(&battery->dev, &dev_attr_charge_control_end_threshold);
diff --git a/drivers/platform/x86/intel/int1092/intel_sar.c b/drivers/platform/x86/intel/int1092/intel_sar.c
index e03943e6380a..0a67b26d53a1 100644
--- a/drivers/platform/x86/intel/int1092/intel_sar.c
+++ b/drivers/platform/x86/intel/int1092/intel_sar.c
@@ -91,8 +91,10 @@ static acpi_status parse_package(struct wwan_sar_context *context, union acpi_ob
item->package.count <= data->total_dev_mode)
return AE_ERROR;
- data->device_mode_info = kmalloc_array(data->total_dev_mode,
- sizeof(struct wwan_device_mode_info), GFP_KERNEL);
+ data->device_mode_info = devm_kmalloc_array(&context->sar_device->dev,
+ data->total_dev_mode,
+ sizeof(*data->device_mode_info),
+ GFP_KERNEL);
if (!data->device_mode_info)
return AE_ERROR;
@@ -246,21 +248,26 @@ static void sar_get_data(int reg, struct wwan_sar_context *context)
static int sar_probe(struct platform_device *device)
{
struct wwan_sar_context *context;
+ acpi_handle handle;
int reg;
int result;
- context = kzalloc(sizeof(*context), GFP_KERNEL);
+ handle = ACPI_HANDLE(&device->dev);
+ if (!handle)
+ return -ENODEV;
+
+ context = devm_kzalloc(&device->dev, sizeof(*context), GFP_KERNEL);
if (!context)
return -ENOMEM;
context->sar_device = device;
- context->handle = ACPI_HANDLE(&device->dev);
+ context->handle = handle;
dev_set_drvdata(&device->dev, context);
result = guid_parse(SAR_DSM_UUID, &context->guid);
if (result) {
dev_err(&device->dev, "SAR UUID parse error: %d\n", result);
- goto r_free;
+ return result;
}
for (reg = 0; reg < MAX_REGULATORY; reg++)
@@ -268,43 +275,29 @@ static int sar_probe(struct platform_device *device)
if (sar_get_device_mode(device) != AE_OK) {
dev_err(&device->dev, "Failed to get device mode\n");
- result = -EIO;
- goto r_free;
+ return -EIO;
}
result = sysfs_create_group(&device->dev.kobj, &intcsar_group);
if (result) {
dev_err(&device->dev, "sysfs creation failed\n");
- goto r_free;
+ return result;
}
if (acpi_install_notify_handler(ACPI_HANDLE(&device->dev), ACPI_DEVICE_NOTIFY,
sar_notify, (void *)device) != AE_OK) {
dev_err(&device->dev, "Failed acpi_install_notify_handler\n");
- result = -EIO;
- goto r_sys;
+ sysfs_remove_group(&device->dev.kobj, &intcsar_group);
+ return -EIO;
}
return 0;
-
-r_sys:
- sysfs_remove_group(&device->dev.kobj, &intcsar_group);
-r_free:
- kfree(context);
- return result;
}
static int sar_remove(struct platform_device *device)
{
- struct wwan_sar_context *context = dev_get_drvdata(&device->dev);
- int reg;
-
acpi_remove_notify_handler(ACPI_HANDLE(&device->dev),
ACPI_DEVICE_NOTIFY, sar_notify);
sysfs_remove_group(&device->dev.kobj, &intcsar_group);
- for (reg = 0; reg < MAX_REGULATORY; reg++)
- kfree(context->config_data[reg].device_mode_info);
-
- kfree(context);
return 0;
}
diff --git a/drivers/platform/x86/lg-laptop.c b/drivers/platform/x86/lg-laptop.c
index 5704981d1848..cd66db74093e 100644
--- a/drivers/platform/x86/lg-laptop.c
+++ b/drivers/platform/x86/lg-laptop.c
@@ -71,6 +71,7 @@ static u32 inited;
#define INIT_SPARSE_KEYMAP 0x80
static int battery_limit_use_wmbb;
+static bool kbd_backlight_available;
static struct led_classdev kbd_backlight;
static enum led_brightness get_kbd_backlight_level(struct device *dev);
@@ -188,6 +189,7 @@ static void wmi_notify(u32 value, void *context)
union acpi_object *obj;
acpi_status status;
long data = (long)context;
+ unsigned int brightness;
pr_debug("event guid %li\n", data);
status = wmi_get_event_data(value, &response);
@@ -205,8 +207,11 @@ static void wmi_notify(u32 value, void *context)
struct key_entry *key;
if (eventcode == 0x10000000) {
- led_classdev_notify_brightness_hw_changed(
- &kbd_backlight, get_kbd_backlight_level(kbd_backlight.dev->parent));
+ if (kbd_backlight_available) {
+ brightness = get_kbd_backlight_level(kbd_backlight.dev->parent);
+ led_classdev_notify_brightness_hw_changed(&kbd_backlight,
+ brightness);
+ }
} else {
key = sparse_keymap_entry_from_scancode(
wmi_input_dev, eventcode);
@@ -522,7 +527,7 @@ static DEVICE_ATTR_RW(fn_lock);
static DEVICE_ATTR_RW(charge_control_end_threshold);
static DEVICE_ATTR_RW(battery_care_limit);
-static int lg_battery_add(struct power_supply *battery)
+static int lg_battery_add(struct power_supply *battery, struct acpi_battery_hook *hook)
{
if (device_create_file(&battery->dev,
&dev_attr_charge_control_end_threshold))
@@ -531,7 +536,7 @@ static int lg_battery_add(struct power_supply *battery)
return 0;
}
-static int lg_battery_remove(struct power_supply *battery)
+static int lg_battery_remove(struct power_supply *battery, struct acpi_battery_hook *hook)
{
device_remove_file(&battery->dev,
&dev_attr_charge_control_end_threshold);
@@ -725,7 +730,12 @@ static int acpi_add(struct acpi_device *device)
goto out_platform_device;
/* LEDs are optional */
- led_classdev_register(&pf_device->dev, &kbd_backlight);
+ ret = led_classdev_register(&pf_device->dev, &kbd_backlight);
+ if (ret < 0)
+ kbd_backlight_available = false;
+ else
+ kbd_backlight_available = true;
+
led_classdev_register(&pf_device->dev, &tpad_led);
wmi_input_setup();
@@ -744,11 +754,13 @@ static int acpi_remove(struct acpi_device *device)
{
sysfs_remove_group(&pf_device->dev.kobj, &dev_attribute_group);
+ battery_hook_unregister(&battery_hook);
+ wmi_input_destroy();
+
+ kbd_backlight_available = false;
led_classdev_unregister(&tpad_led);
led_classdev_unregister(&kbd_backlight);
- battery_hook_unregister(&battery_hook);
- wmi_input_destroy();
platform_device_unregister(pf_device);
pf_device = NULL;
platform_driver_unregister(&pf_driver);
diff --git a/drivers/platform/x86/system76_acpi.c b/drivers/platform/x86/system76_acpi.c
index 958df41ad509..9031bd53253f 100644
--- a/drivers/platform/x86/system76_acpi.c
+++ b/drivers/platform/x86/system76_acpi.c
@@ -254,7 +254,7 @@ static struct attribute *system76_battery_attrs[] = {
ATTRIBUTE_GROUPS(system76_battery);
-static int system76_battery_add(struct power_supply *battery)
+static int system76_battery_add(struct power_supply *battery, struct acpi_battery_hook *hook)
{
// System76 EC only supports 1 battery
if (strcmp(battery->desc->name, "BAT0") != 0)
@@ -266,7 +266,7 @@ static int system76_battery_add(struct power_supply *battery)
return 0;
}
-static int system76_battery_remove(struct power_supply *battery)
+static int system76_battery_remove(struct power_supply *battery, struct acpi_battery_hook *hook)
{
device_remove_groups(&battery->dev, system76_battery_groups);
return 0;
diff --git a/drivers/platform/x86/thinkpad_acpi.c b/drivers/platform/x86/thinkpad_acpi.c
index 3f2098e686f7..d2124f69948e 100644
--- a/drivers/platform/x86/thinkpad_acpi.c
+++ b/drivers/platform/x86/thinkpad_acpi.c
@@ -10102,7 +10102,7 @@ ATTRIBUTE_GROUPS(tpacpi_battery);
/* ACPI battery hooking */
-static int tpacpi_battery_add(struct power_supply *battery)
+static int tpacpi_battery_add(struct power_supply *battery, struct acpi_battery_hook *hook)
{
int batteryid = tpacpi_battery_get_id(battery->desc->name);
@@ -10113,7 +10113,7 @@ static int tpacpi_battery_add(struct power_supply *battery)
return 0;
}
-static int tpacpi_battery_remove(struct power_supply *battery)
+static int tpacpi_battery_remove(struct power_supply *battery, struct acpi_battery_hook *hook)
{
device_remove_groups(&battery->dev, tpacpi_battery_groups);
return 0;
diff --git a/drivers/platform/x86/toshiba_acpi.c b/drivers/platform/x86/toshiba_acpi.c
index 033e28aaeea6..3541c2048905 100644
--- a/drivers/platform/x86/toshiba_acpi.c
+++ b/drivers/platform/x86/toshiba_acpi.c
@@ -3124,7 +3124,7 @@ static struct attribute *toshiba_acpi_battery_attrs[] = {
ATTRIBUTE_GROUPS(toshiba_acpi_battery);
-static int toshiba_acpi_battery_add(struct power_supply *battery)
+static int toshiba_acpi_battery_add(struct power_supply *battery, struct acpi_battery_hook *hook)
{
if (toshiba_acpi == NULL) {
pr_err("Init order issue\n");
@@ -3137,7 +3137,7 @@ static int toshiba_acpi_battery_add(struct power_supply *battery)
return 0;
}
-static int toshiba_acpi_battery_remove(struct power_supply *battery)
+static int toshiba_acpi_battery_remove(struct power_supply *battery, struct acpi_battery_hook *hook)
{
device_remove_groups(&battery->dev, toshiba_acpi_battery_groups);
return 0;
diff --git a/drivers/platform/x86/x86-android-tablets.c b/drivers/platform/x86/x86-android-tablets.c
index 94710471d7dd..2063e1ac9d0b 100644
--- a/drivers/platform/x86/x86-android-tablets.c
+++ b/drivers/platform/x86/x86-android-tablets.c
@@ -1748,19 +1748,19 @@ static void x86_android_tablet_cleanup(void)
{
int i;
- for (i = 0; i < serdev_count; i++) {
+ for (i = serdev_count - 1; i >= 0; i--) {
if (serdevs[i])
serdev_device_remove(serdevs[i]);
}
kfree(serdevs);
- for (i = 0; i < pdev_count; i++)
+ for (i = pdev_count - 1; i >= 0; i--)
platform_device_unregister(pdevs[i]);
kfree(pdevs);
- for (i = 0; i < i2c_client_count; i++)
+ for (i = i2c_client_count - 1; i >= 0; i--)
i2c_unregister_device(i2c_clients[i]);
kfree(i2c_clients);
diff --git a/drivers/power/supply/charger-manager.c b/drivers/power/supply/charger-manager.c
index 75df8c15df05..7f7ddecf2250 100644
--- a/drivers/power/supply/charger-manager.c
+++ b/drivers/power/supply/charger-manager.c
@@ -1017,6 +1017,29 @@ static int charger_extcon_init(struct charger_manager *cm,
return 0;
}
+static int charger_manager_get_regulators(struct charger_manager *cm)
+{
+ struct charger_desc *desc = cm->desc;
+ struct charger_regulator *charger;
+ int i, ret;
+
+ for (i = 0; i < desc->num_charger_regulators; i++) {
+ charger = &desc->charger_regulators[i];
+ charger->consumer = regulator_get(cm->dev,
+ charger->regulator_name);
+ if (IS_ERR(charger->consumer)) {
+ dev_err(cm->dev, "Cannot find charger(%s)\n",
+ charger->regulator_name);
+ ret = PTR_ERR(charger->consumer);
+ while (i-- > 0)
+ regulator_put(desc->charger_regulators[i].consumer);
+ return ret;
+ }
+ charger->cm = cm;
+ }
+ return 0;
+}
+
/**
* charger_manager_register_extcon - Register extcon device to receive state
* of charger cable.
@@ -1039,15 +1062,6 @@ static int charger_manager_register_extcon(struct charger_manager *cm)
for (i = 0; i < desc->num_charger_regulators; i++) {
charger = &desc->charger_regulators[i];
- charger->consumer = regulator_get(cm->dev,
- charger->regulator_name);
- if (IS_ERR(charger->consumer)) {
- dev_err(cm->dev, "Cannot find charger(%s)\n",
- charger->regulator_name);
- return PTR_ERR(charger->consumer);
- }
- charger->cm = cm;
-
for (j = 0; j < charger->num_cables; j++) {
struct charger_cable *cable = &charger->cables[j];
@@ -1582,13 +1596,23 @@ static int charger_manager_probe(struct platform_device *pdev)
}
psy_cfg.attr_grp = desc->sysfs_groups;
+ /*
+ * Acquire charger regulators before exposing the sysfs entries, so
+ * userspace cannot reach externally_control before the regulators
+ * (and charger->cm) are available. Mirrors the order in remove().
+ */
+ ret = charger_manager_get_regulators(cm);
+ if (ret < 0)
+ return ret;
+
cm->charger_psy = power_supply_register(&pdev->dev,
&cm->charger_psy_desc,
&psy_cfg);
if (IS_ERR(cm->charger_psy)) {
dev_err(&pdev->dev, "Cannot register charger-manager with name \"%s\"\n",
cm->charger_psy_desc.name);
- return PTR_ERR(cm->charger_psy);
+ ret = PTR_ERR(cm->charger_psy);
+ goto err_regulator;
}
/* Register extcon device for charger cable */
@@ -1622,15 +1646,15 @@ static int charger_manager_probe(struct platform_device *pdev)
return 0;
err_reg_extcon:
+ power_supply_unregister(cm->charger_psy);
+err_regulator:
for (i = 0; i < desc->num_charger_regulators; i++)
regulator_put(desc->charger_regulators[i].consumer);
- power_supply_unregister(cm->charger_psy);
-
return ret;
}
-static int charger_manager_remove(struct platform_device *pdev)
+static void charger_manager_remove(struct platform_device *pdev)
{
struct charger_manager *cm = platform_get_drvdata(pdev);
struct charger_desc *desc = cm->desc;
@@ -1644,14 +1668,12 @@ static int charger_manager_remove(struct platform_device *pdev)
cancel_work_sync(&setup_polling);
cancel_delayed_work_sync(&cm_monitor_work);
- for (i = 0 ; i < desc->num_charger_regulators ; i++)
- regulator_put(desc->charger_regulators[i].consumer);
+ try_charger_enable(cm, false);
power_supply_unregister(cm->charger_psy);
- try_charger_enable(cm, false);
-
- return 0;
+ for (i = 0 ; i < desc->num_charger_regulators ; i++)
+ regulator_put(desc->charger_regulators[i].consumer);
}
static const struct platform_device_id charger_manager_id[] = {
@@ -1742,7 +1764,7 @@ static struct platform_driver charger_manager_driver = {
.of_match_table = charger_manager_match,
},
.probe = charger_manager_probe,
- .remove = charger_manager_remove,
+ .remove_new = charger_manager_remove,
.id_table = charger_manager_id,
};
diff --git a/drivers/power/supply/ds2780_battery.c b/drivers/power/supply/ds2780_battery.c
index 2b8c90d84325..0a1f79600c60 100644
--- a/drivers/power/supply/ds2780_battery.c
+++ b/drivers/power/supply/ds2780_battery.c
@@ -304,7 +304,7 @@ static int ds2780_get_capacity(struct ds2780_device_info *dev_info,
return ret;
*capacity = raw;
- return raw;
+ return 0;
}
static int ds2780_get_status(struct ds2780_device_info *dev_info, int *status)
diff --git a/drivers/power/supply/lp8788-charger.c b/drivers/power/supply/lp8788-charger.c
index 56c57529c228..cf797bf7f648 100644
--- a/drivers/power/supply/lp8788-charger.c
+++ b/drivers/power/supply/lp8788-charger.c
@@ -406,12 +406,6 @@ static const struct power_supply_desc lp8788_psy_battery_desc = {
.get_property = lp8788_battery_get_property,
};
-static void lp8788_psy_unregister(struct lp8788_charger *pchg)
-{
- power_supply_unregister(pchg->battery);
- power_supply_unregister(pchg->charger);
-}
-
static void lp8788_charger_event(struct work_struct *work)
{
struct lp8788_charger *pchg =
@@ -667,18 +661,16 @@ static int lp8788_psy_register(struct platform_device *pdev,
charger_cfg.supplied_to = battery_supplied_to;
charger_cfg.num_supplicants = ARRAY_SIZE(battery_supplied_to);
- pchg->charger = power_supply_register(&pdev->dev,
- &lp8788_psy_charger_desc,
- &charger_cfg);
+ pchg->charger = devm_power_supply_register(&pdev->dev,
+ &lp8788_psy_charger_desc,
+ &charger_cfg);
if (IS_ERR(pchg->charger))
return -EPERM;
- pchg->battery = power_supply_register(&pdev->dev,
- &lp8788_psy_battery_desc, NULL);
- if (IS_ERR(pchg->battery)) {
- power_supply_unregister(pchg->charger);
+ pchg->battery = devm_power_supply_register(&pdev->dev,
+ &lp8788_psy_battery_desc, NULL);
+ if (IS_ERR(pchg->battery))
return -EPERM;
- }
return 0;
}
@@ -715,20 +707,17 @@ static int lp8788_charger_probe(struct platform_device *pdev)
return 0;
}
-static int lp8788_charger_remove(struct platform_device *pdev)
+static void lp8788_charger_remove(struct platform_device *pdev)
{
struct lp8788_charger *pchg = platform_get_drvdata(pdev);
- flush_work(&pchg->charger_work);
lp8788_irq_unregister(pdev, pchg);
- lp8788_psy_unregister(pchg);
-
- return 0;
+ cancel_work_sync(&pchg->charger_work);
}
static struct platform_driver lp8788_charger_driver = {
.probe = lp8788_charger_probe,
- .remove = lp8788_charger_remove,
+ .remove_new = lp8788_charger_remove,
.driver = {
.name = LP8788_DEV_CHARGER,
},
diff --git a/drivers/power/supply/max17040_battery.c b/drivers/power/supply/max17040_battery.c
index 04b61692da30..509b18d35934 100644
--- a/drivers/power/supply/max17040_battery.c
+++ b/drivers/power/supply/max17040_battery.c
@@ -190,8 +190,11 @@ static int max17040_raw_vcell_to_uvolts(struct max17040_chip *chip, u16 vcell)
static int max17040_get_vcell(struct max17040_chip *chip)
{
u32 vcell;
+ int ret;
- regmap_read(chip->regmap, MAX17040_VCELL, &vcell);
+ ret = regmap_read(chip->regmap, MAX17040_VCELL, &vcell);
+ if (ret)
+ return ret;
return max17040_raw_vcell_to_uvolts(chip, vcell);
}
@@ -199,8 +202,11 @@ static int max17040_get_vcell(struct max17040_chip *chip)
static int max17040_get_soc(struct max17040_chip *chip)
{
u32 soc;
+ int ret;
- regmap_read(chip->regmap, MAX17040_SOC, &soc);
+ ret = regmap_read(chip->regmap, MAX17040_SOC, &soc);
+ if (ret)
+ return ret;
return soc >> (chip->quirk_double_soc ? 9 : 8);
}
@@ -259,7 +265,11 @@ static int max17040_get_of_data(struct max17040_chip *chip)
static void max17040_check_changes(struct max17040_chip *chip)
{
- chip->soc = max17040_get_soc(chip);
+ int soc;
+
+ soc = max17040_get_soc(chip);
+ if (soc >= 0)
+ chip->soc = soc;
}
static void max17040_queue_work(struct max17040_chip *chip)
@@ -386,16 +396,23 @@ static int max17040_get_property(struct power_supply *psy,
union power_supply_propval *val)
{
struct max17040_chip *chip = power_supply_get_drvdata(psy);
+ int ret;
switch (psp) {
case POWER_SUPPLY_PROP_ONLINE:
val->intval = max17040_get_online(chip);
break;
case POWER_SUPPLY_PROP_VOLTAGE_NOW:
- val->intval = max17040_get_vcell(chip);
+ ret = max17040_get_vcell(chip);
+ if (ret < 0)
+ return ret;
+ val->intval = ret;
break;
case POWER_SUPPLY_PROP_CAPACITY:
- val->intval = max17040_get_soc(chip);
+ ret = max17040_get_soc(chip);
+ if (ret < 0)
+ return ret;
+ val->intval = ret;
break;
case POWER_SUPPLY_PROP_CAPACITY_ALERT_MIN:
val->intval = chip->low_soc_alert;
@@ -430,19 +447,15 @@ static const struct power_supply_desc max17040_battery_desc = {
.num_properties = ARRAY_SIZE(max17040_battery_props),
};
-static int max17040_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static int max17040_probe(struct i2c_client *client)
{
- struct i2c_adapter *adapter = client->adapter;
+ const struct i2c_device_id *id = i2c_client_get_device_id(client);
struct power_supply_config psy_cfg = {};
struct max17040_chip *chip;
enum chip_id chip_id;
bool enable_irq = false;
int ret;
- if (!i2c_check_functionality(adapter, I2C_FUNC_SMBUS_BYTE))
- return -EIO;
-
chip = devm_kzalloc(&client->dev, sizeof(*chip), GFP_KERNEL);
if (!chip)
return -ENOMEM;
@@ -599,7 +612,7 @@ static struct i2c_driver max17040_i2c_driver = {
.of_match_table = max17040_of_match,
.pm = MAX17040_PM_OPS,
},
- .probe = max17040_probe,
+ .probe_new = max17040_probe,
.id_table = max17040_id,
};
module_i2c_driver(max17040_i2c_driver);
diff --git a/drivers/power/supply/rt9455_charger.c b/drivers/power/supply/rt9455_charger.c
index 566243a423c8..62ffc3acc2ae 100644
--- a/drivers/power/supply/rt9455_charger.c
+++ b/drivers/power/supply/rt9455_charger.c
@@ -1583,8 +1583,20 @@ static const struct regmap_config rt9455_regmap_config = {
.cache_type = REGCACHE_RBTREE,
};
-static int rt9455_probe(struct i2c_client *client,
- const struct i2c_device_id *id)
+static void rt9455_cancel_all_delayed_works(void *data)
+{
+ struct rt9455_info *info = data;
+
+ /*
+ * Both pwr_rdy_work and batt_presence_work can queue
+ * max_charging_time_work, so cancel them first.
+ */
+ cancel_delayed_work_sync(&info->pwr_rdy_work);
+ cancel_delayed_work_sync(&info->batt_presence_work);
+ cancel_delayed_work_sync(&info->max_charging_time_work);
+}
+
+static int rt9455_probe(struct i2c_client *client)
{
struct i2c_adapter *adapter = client->adapter;
struct device *dev = &client->dev;
@@ -1674,6 +1686,10 @@ static int rt9455_probe(struct i2c_client *client,
goto put_usb_notifier;
}
+ ret = devm_add_action_or_reset(dev, rt9455_cancel_all_delayed_works, info);
+ if (ret)
+ goto put_usb_notifier;
+
ret = devm_request_threaded_irq(dev, client->irq, NULL,
rt9455_irq_handler_thread,
IRQF_TRIGGER_LOW | IRQF_ONESHOT,
@@ -1714,10 +1730,6 @@ static void rt9455_remove(struct i2c_client *client)
if (info->nb.notifier_call)
usb_unregister_notifier(info->usb_phy, &info->nb);
#endif
-
- cancel_delayed_work_sync(&info->pwr_rdy_work);
- cancel_delayed_work_sync(&info->max_charging_time_work);
- cancel_delayed_work_sync(&info->batt_presence_work);
}
static const struct i2c_device_id rt9455_i2c_id_table[] = {
@@ -1741,7 +1753,7 @@ MODULE_DEVICE_TABLE(acpi, rt9455_i2c_acpi_match);
#endif
static struct i2c_driver rt9455_driver = {
- .probe = rt9455_probe,
+ .probe_new = rt9455_probe,
.remove = rt9455_remove,
.id_table = rt9455_i2c_id_table,
.driver = {
diff --git a/drivers/ptp/ptp_ocp.c b/drivers/ptp/ptp_ocp.c
index f2332b954bcf..0661e1e2dca1 100644
--- a/drivers/ptp/ptp_ocp.c
+++ b/drivers/ptp/ptp_ocp.c
@@ -3673,8 +3673,7 @@ ptp_ocp_detach(struct ptp_ocp *bp)
ptp_ocp_debugfs_remove_device(bp);
ptp_ocp_detach_sysfs(bp);
ptp_ocp_attr_group_del(bp);
- if (timer_pending(&bp->watchdog))
- del_timer_sync(&bp->watchdog);
+ del_timer_sync(&bp->watchdog);
if (bp->ts0)
ptp_ocp_unregister_ext(bp->ts0);
if (bp->ts1)
diff --git a/drivers/ptp/ptp_private.h b/drivers/ptp/ptp_private.h
index a431eb79fe77..ef6b9ef145ef 100644
--- a/drivers/ptp/ptp_private.h
+++ b/drivers/ptp/ptp_private.h
@@ -19,6 +19,7 @@
#define PTP_MAX_TIMESTAMPS 128
#define PTP_BUF_TIMESTAMPS 30
#define PTP_DEFAULT_MAX_VCLOCKS 20
+#define PTP_MAX_VCLOCKS_LIMIT (KMALLOC_MAX_SIZE/(sizeof(int)))
struct timestamp_event_queue {
struct ptp_extts_event buf[PTP_MAX_TIMESTAMPS];
diff --git a/drivers/ptp/ptp_sysfs.c b/drivers/ptp/ptp_sysfs.c
index 1263612ef275..4ff6afb15d91 100644
--- a/drivers/ptp/ptp_sysfs.c
+++ b/drivers/ptp/ptp_sysfs.c
@@ -271,7 +271,7 @@ static ssize_t max_vclocks_store(struct device *dev,
size_t size;
u32 max;
- if (kstrtou32(buf, 0, &max) || max == 0)
+ if (kstrtou32(buf, 0, &max) || max == 0 || max > PTP_MAX_VCLOCKS_LIMIT)
return -EINVAL;
if (max == ptp->max_vclocks)
diff --git a/drivers/pwm/pwm-samsung.c b/drivers/pwm/pwm-samsung.c
index 9c5b4f515641..96f3ff4e6be1 100644
--- a/drivers/pwm/pwm-samsung.c
+++ b/drivers/pwm/pwm-samsung.c
@@ -102,7 +102,7 @@ struct samsung_pwm_chip {
* IP. Should this change, both drivers will need to be modified to
* properly synchronize accesses to particular instances.
*/
-static DEFINE_SPINLOCK(samsung_pwm_lock);
+static DEFINE_RAW_SPINLOCK(samsung_pwm_lock);
#endif
static inline
@@ -141,14 +141,14 @@ static void pwm_samsung_set_divisor(struct samsung_pwm_chip *pwm,
bits = (fls(divisor) - 1) - pwm->variant.div_base;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
reg = readl(pwm->base + REG_TCFG1);
reg &= ~(TCFG1_MUX_MASK << shift);
reg |= bits << shift;
writel(reg, pwm->base + REG_TCFG1);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int pwm_samsung_is_tdiv(struct samsung_pwm_chip *chip, unsigned int chan)
@@ -258,7 +258,7 @@ static int pwm_samsung_enable(struct pwm_chip *chip, struct pwm_device *pwm)
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(our_chip->base + REG_TCON);
@@ -272,7 +272,7 @@ static int pwm_samsung_enable(struct pwm_chip *chip, struct pwm_device *pwm)
our_chip->disabled_mask &= ~BIT(pwm->hwpwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
return 0;
}
@@ -284,7 +284,7 @@ static void pwm_samsung_disable(struct pwm_chip *chip, struct pwm_device *pwm)
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(our_chip->base + REG_TCON);
tcon &= ~TCON_AUTORELOAD(tcon_chan);
@@ -299,7 +299,7 @@ static void pwm_samsung_disable(struct pwm_chip *chip, struct pwm_device *pwm)
our_chip->disabled_mask |= BIT(pwm->hwpwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static void pwm_samsung_manual_update(struct samsung_pwm_chip *chip,
@@ -307,11 +307,11 @@ static void pwm_samsung_manual_update(struct samsung_pwm_chip *chip,
{
unsigned long flags;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
__pwm_samsung_manual_update(chip, pwm);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int __pwm_samsung_config(struct pwm_chip *chip, struct pwm_device *pwm,
@@ -400,7 +400,7 @@ static void pwm_samsung_set_invert(struct samsung_pwm_chip *chip,
unsigned long flags;
u32 tcon;
- spin_lock_irqsave(&samsung_pwm_lock, flags);
+ raw_spin_lock_irqsave(&samsung_pwm_lock, flags);
tcon = readl(chip->base + REG_TCON);
@@ -414,7 +414,7 @@ static void pwm_samsung_set_invert(struct samsung_pwm_chip *chip,
writel(tcon, chip->base + REG_TCON);
- spin_unlock_irqrestore(&samsung_pwm_lock, flags);
+ raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags);
}
static int pwm_samsung_set_polarity(struct pwm_chip *chip,
diff --git a/drivers/regulator/ltc3676.c b/drivers/regulator/ltc3676.c
index f4d8ea21f173..405baea41357 100644
--- a/drivers/regulator/ltc3676.c
+++ b/drivers/regulator/ltc3676.c
@@ -232,7 +232,7 @@ static const struct regulator_desc ltc3676_regulators[LTC3676_NUM_REGULATORS] =
LTC3676_FIXED_REG(LDO4, ldo4, LDOB, 2),
};
-static bool ltc3676_readable_writeable_reg(struct device *dev, unsigned int reg)
+static bool ltc3676_writeable_reg(struct device *dev, unsigned int reg)
{
switch (reg) {
case LTC3676_BUCK1 ... LTC3676_IRQSTAT:
@@ -243,6 +243,15 @@ static bool ltc3676_readable_writeable_reg(struct device *dev, unsigned int reg)
return false;
}
+static bool ltc3676_readable_reg(struct device *dev, unsigned int reg)
+{
+ switch (reg) {
+ case LTC3676_BUCK1 ... LTC3676_IRQSTAT:
+ return true;
+ }
+ return false;
+}
+
static bool ltc3676_volatile_reg(struct device *dev, unsigned int reg)
{
switch (reg) {
@@ -255,8 +264,8 @@ static bool ltc3676_volatile_reg(struct device *dev, unsigned int reg)
static const struct regmap_config ltc3676_regmap_config = {
.reg_bits = 8,
.val_bits = 8,
- .writeable_reg = ltc3676_readable_writeable_reg,
- .readable_reg = ltc3676_readable_writeable_reg,
+ .writeable_reg = ltc3676_writeable_reg,
+ .readable_reg = ltc3676_readable_reg,
.volatile_reg = ltc3676_volatile_reg,
.max_register = LTC3676_CLIRQ,
.use_single_read = true,
diff --git a/drivers/remoteproc/qcom_wcnss.c b/drivers/remoteproc/qcom_wcnss.c
index af96541c9b69..44301a77db93 100644
--- a/drivers/remoteproc/qcom_wcnss.c
+++ b/drivers/remoteproc/qcom_wcnss.c
@@ -430,25 +430,31 @@ static void wcnss_release_pds(struct qcom_wcnss *wcnss)
}
static int wcnss_init_regulators(struct qcom_wcnss *wcnss,
- const struct wcnss_vreg_info *info,
- int num_vregs, int num_pd_vregs)
+ const struct wcnss_data *data)
{
+ const struct wcnss_vreg_info *info = data->vregs;
struct regulator_bulk_data *bulk;
+ size_t i, possible_pds = 0, num_vregs = data->num_vregs;
int ret;
- int i;
+
+ for (i = 0; i < WCNSS_MAX_PDS; i++)
+ if (data->pd_names[i])
+ possible_pds++;
/*
* If attaching the power domains suceeded we can skip requesting
* the regulators for the power domains. For old device trees we need to
* reserve extra space to manage them through the regulator interface.
*/
- if (wcnss->num_pds) {
+ if (possible_pds >= num_vregs) {
+ /* Do nothing if vregs do not include PD regulators (pronto-v3) */
+ } else if (wcnss->num_pds) {
info += wcnss->num_pds;
/* Handle single power domain case */
- if (wcnss->num_pds < num_pd_vregs)
- num_vregs += num_pd_vregs - wcnss->num_pds;
+ if (wcnss->num_pds < data->num_pd_vregs)
+ num_vregs += data->num_pd_vregs - wcnss->num_pds;
} else {
- num_vregs += num_pd_vregs;
+ num_vregs += data->num_pd_vregs;
}
bulk = devm_kcalloc(wcnss->dev,
@@ -607,8 +613,7 @@ static int wcnss_probe(struct platform_device *pdev)
if (ret && (ret != -ENODATA || !data->num_pd_vregs))
goto free_rproc;
- ret = wcnss_init_regulators(wcnss, data->vregs, data->num_vregs,
- data->num_pd_vregs);
+ ret = wcnss_init_regulators(wcnss, data);
if (ret)
goto detach_pds;
diff --git a/drivers/rtc/dev.c b/drivers/rtc/dev.c
index 4aad9bb99868..8ff910017b10 100644
--- a/drivers/rtc/dev.c
+++ b/drivers/rtc/dev.c
@@ -207,7 +207,7 @@ static long rtc_dev_ioctl(struct file *file,
struct rtc_device *rtc = file->private_data;
const struct rtc_class_ops *ops = rtc->ops;
struct rtc_time tm;
- struct rtc_wkalrm alarm;
+ struct rtc_wkalrm alarm = { 0 };
struct rtc_param param;
void __user *uarg = (void __user *)arg;
diff --git a/drivers/rtc/rtc-mpfs.c b/drivers/rtc/rtc-mpfs.c
index d329e0eb66c4..e530b7868f99 100644
--- a/drivers/rtc/rtc-mpfs.c
+++ b/drivers/rtc/rtc-mpfs.c
@@ -256,8 +256,12 @@ static int mpfs_rtc_probe(struct platform_device *pdev)
return ret;
}
+ clk = devm_clk_get(&pdev->dev, "rtcref");
+ if (IS_ERR(clk))
+ return PTR_ERR(clk);
+
/* prescaler hardware adds 1 to reg value */
- prescaler = clk_get_rate(devm_clk_get(&pdev->dev, "rtcref")) - 1;
+ prescaler = clk_get_rate(clk) - 1;
if (prescaler > MAX_PRESCALER_COUNT) {
dev_dbg(&pdev->dev, "invalid prescaler %lu\n", prescaler);
return -EINVAL;
diff --git a/drivers/rtc/rtc-rzn1.c b/drivers/rtc/rtc-rzn1.c
index 36381385d262..cf9154a62d46 100644
--- a/drivers/rtc/rtc-rzn1.c
+++ b/drivers/rtc/rtc-rzn1.c
@@ -193,13 +193,24 @@ static int rzn1_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
if (ret)
return ret;
+ ctl1 = readl(rtc->base + RZN1_RTC_CTL1);
+ alrm->enabled = !!(ctl1 & RZN1_RTC_CTL1_ALME);
+
min = readl(rtc->base + RZN1_RTC_ALM);
hour = readl(rtc->base + RZN1_RTC_ALH);
- wday = readl(rtc->base + RZN1_RTC_ALW);
tm->tm_sec = 0;
tm->tm_min = bcd2bin(min);
tm->tm_hour = bcd2bin(hour);
+
+ /*
+ * If wday is zero, no bit is set in RZN1_RTC_ALW. This is the
+ * register's power-on reset value.
+ */
+ wday = readl(rtc->base + RZN1_RTC_ALW);
+ if (!wday)
+ return 0;
+
delta_days = ((fls(wday) - 1) - tm->tm_wday + 7) % 7;
tm->tm_wday = fls(wday) - 1;
@@ -208,9 +219,6 @@ static int rzn1_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm)
rtc_time64_to_tm(alarm, tm);
}
- ctl1 = readl(rtc->base + RZN1_RTC_CTL1);
- alrm->enabled = !!(ctl1 & RZN1_RTC_CTL1_ALME);
-
return 0;
}
diff --git a/drivers/rtc/rtc-spear.c b/drivers/rtc/rtc-spear.c
index 736fe535cd45..7303f5bf1192 100644
--- a/drivers/rtc/rtc-spear.c
+++ b/drivers/rtc/rtc-spear.c
@@ -363,6 +363,12 @@ static int spear_rtc_probe(struct platform_device *pdev)
if (irq < 0)
return irq;
+ config->ioaddr = devm_platform_ioremap_resource(pdev, 0);
+ if (IS_ERR(config->ioaddr))
+ return PTR_ERR(config->ioaddr);
+
+ spin_lock_init(&config->lock);
+
status = devm_request_irq(&pdev->dev, irq, spear_rtc_irq, 0, pdev->name,
config);
if (status) {
@@ -371,10 +377,6 @@ static int spear_rtc_probe(struct platform_device *pdev)
return status;
}
- config->ioaddr = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(config->ioaddr))
- return PTR_ERR(config->ioaddr);
-
config->clk = devm_clk_get(&pdev->dev, NULL);
if (IS_ERR(config->clk))
return PTR_ERR(config->clk);
@@ -383,7 +385,6 @@ static int spear_rtc_probe(struct platform_device *pdev)
if (status < 0)
return status;
- spin_lock_init(&config->lock);
platform_set_drvdata(pdev, config);
config->rtc->ops = &spear_rtc_ops;
diff --git a/drivers/s390/block/dasd_diag.c b/drivers/s390/block/dasd_diag.c
index 5802aead09f3..12db1046aad0 100644
--- a/drivers/s390/block/dasd_diag.c
+++ b/drivers/s390/block/dasd_diag.c
@@ -24,7 +24,7 @@
#include <asm/debug.h>
#include <asm/diag.h>
#include <asm/ebcdic.h>
-#include <asm/io.h>
+#include <linux/io.h>
#include <asm/irq.h>
#include <asm/vtoc.h>
diff --git a/drivers/s390/block/dasd_eckd.c b/drivers/s390/block/dasd_eckd.c
index b2e238467e70..e83a43c09d74 100644
--- a/drivers/s390/block/dasd_eckd.c
+++ b/drivers/s390/block/dasd_eckd.c
@@ -21,13 +21,14 @@
#include <linux/compat.h>
#include <linux/init.h>
#include <linux/seq_file.h>
+#include <linux/uaccess.h>
+#include <linux/io.h>
+#include <linux/overflow.h>
#include <asm/css_chars.h>
#include <asm/debug.h>
#include <asm/idals.h>
#include <asm/ebcdic.h>
-#include <asm/io.h>
-#include <linux/uaccess.h>
#include <asm/cio.h>
#include <asm/ccwdev.h>
#include <asm/itcw.h>
@@ -3518,11 +3519,11 @@ static int dasd_eckd_check_device_format(struct dasd_device *base,
{
struct dasd_eckd_private *private = base->private;
struct eckd_count *fmt_buffer;
- struct irb irb;
+ size_t fmt_buffer_size;
+ unsigned int trkcount;
int rpt_max, rpt_exp;
- int fmt_buffer_size;
+ struct irb irb;
int trk_per_cyl;
- int trkcount;
int tpm = 0;
int rc;
@@ -3533,7 +3534,9 @@ static int dasd_eckd_check_device_format(struct dasd_device *base,
rpt_exp = recs_per_track(&private->rdc_data, 0, cdata->expect.blksize);
trkcount = cdata->expect.stop_unit - cdata->expect.start_unit + 1;
- fmt_buffer_size = trkcount * rpt_max * sizeof(struct eckd_count);
+ if (check_mul_overflow(trkcount, rpt_max, &fmt_buffer_size) ||
+ check_mul_overflow(fmt_buffer_size, sizeof(struct eckd_count), &fmt_buffer_size))
+ return -EINVAL;
fmt_buffer = kzalloc(fmt_buffer_size, GFP_KERNEL | GFP_DMA);
if (!fmt_buffer)
diff --git a/drivers/s390/block/dasd_fba.c b/drivers/s390/block/dasd_fba.c
index cddfb01a3dca..0a8241b3bfdf 100644
--- a/drivers/s390/block/dasd_fba.c
+++ b/drivers/s390/block/dasd_fba.c
@@ -16,10 +16,10 @@
#include <linux/bio.h>
#include <linux/module.h>
#include <linux/init.h>
+#include <linux/io.h>
#include <asm/idals.h>
#include <asm/ebcdic.h>
-#include <asm/io.h>
#include <asm/ccwdev.h>
#include "dasd_int.h"
diff --git a/drivers/s390/block/dcssblk.c b/drivers/s390/block/dcssblk.c
index 735ee0ca4a13..038aa93a3ec8 100644
--- a/drivers/s390/block/dcssblk.c
+++ b/drivers/s390/block/dcssblk.c
@@ -20,8 +20,8 @@
#include <linux/pfn_t.h>
#include <linux/uio.h>
#include <linux/dax.h>
+#include <linux/io.h>
#include <asm/extmem.h>
-#include <asm/io.h>
#define DCSSBLK_NAME "dcssblk"
#define DCSSBLK_MINORS_PER_DISK 1
diff --git a/drivers/s390/char/con3215.c b/drivers/s390/char/con3215.c
index 033eb2ba21f0..98e28a5a90ec 100644
--- a/drivers/s390/char/con3215.c
+++ b/drivers/s390/char/con3215.c
@@ -25,7 +25,7 @@
#include <linux/slab.h>
#include <asm/ccwdev.h>
#include <asm/cio.h>
-#include <asm/io.h>
+#include <linux/io.h>
#include <asm/ebcdic.h>
#include <linux/uaccess.h>
#include <asm/delay.h>
diff --git a/drivers/s390/char/monwriter.c b/drivers/s390/char/monwriter.c
index 7d9180847ccf..54debecd3cb7 100644
--- a/drivers/s390/char/monwriter.c
+++ b/drivers/s390/char/monwriter.c
@@ -22,8 +22,8 @@
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/uaccess.h>
+#include <linux/io.h>
#include <asm/ebcdic.h>
-#include <asm/io.h>
#include <asm/appldata.h>
#include <asm/monwriter.h>
diff --git a/drivers/s390/cio/vfio_ccw_cp.c b/drivers/s390/cio/vfio_ccw_cp.c
index e64ac6e9b291..96e8d5eb2a9f 100644
--- a/drivers/s390/cio/vfio_ccw_cp.c
+++ b/drivers/s390/cio/vfio_ccw_cp.c
@@ -376,6 +376,7 @@ static struct ccwchain *ccwchain_alloc(struct channel_program *cp, int len)
chain->ch_len = len;
list_add_tail(&chain->next, &cp->ccwchain_list);
+ cp->ccwchain_count++;
return chain;
}
@@ -478,6 +479,10 @@ static int ccwchain_handle_ccw(u32 cda, struct channel_program *cp)
if (len < 0)
return len;
+ /* Limit number of chains in a single channel program */
+ if (cp->ccwchain_count >= CCWCHAIN_COUNT_MAX)
+ return -EINVAL;
+
/* Need alloc a new chain for this one. */
chain = ccwchain_alloc(cp, len);
if (!chain)
@@ -612,6 +617,13 @@ static int ccwchain_fetch_direct(struct ccwchain *chain,
if (ret)
goto out_unpin;
+ /* Unexpected mismatch from earlier read */
+ if (idaws[0] != iova) {
+ pa->pa_nr = 0;
+ ret = -EINVAL;
+ goto out_unpin;
+ }
+
/*
* Copy guest IDAWs into page_array, in case the memory they
* occupy is not contiguous.
@@ -709,6 +721,7 @@ int cp_init(struct channel_program *cp, union orb *orb)
vdev->dev,
"Prefetching channel program even though prefetch not specified in ORB");
+ cp->ccwchain_count = 0;
INIT_LIST_HEAD(&cp->ccwchain_list);
memcpy(&cp->orb, orb, sizeof(*orb));
diff --git a/drivers/s390/cio/vfio_ccw_cp.h b/drivers/s390/cio/vfio_ccw_cp.h
index 54d26e242533..5c51d1197ba9 100644
--- a/drivers/s390/cio/vfio_ccw_cp.h
+++ b/drivers/s390/cio/vfio_ccw_cp.h
@@ -23,12 +23,19 @@
*/
#define CCWCHAIN_LEN_MAX 256
+/*
+ * Maximum number of chains
+ */
+#define CCWCHAIN_COUNT_MAX 16
+
/**
* struct channel_program - manage information for channel program
* @ccwchain_list: list head of ccwchains
* @orb: orb for the currently processed ssch request
* @mdev: the mediated device to perform page pinning/unpinning
* @initialized: whether this instance is actually initialized
+ * @guest_cp: copy of guest channel program
+ * @ccwchain_count: number of channel program segments (linked by TIC)
*
* @ccwchain_list is the head of a ccwchain list, that contents the
* translated result of the guest channel program that pointed out by
@@ -39,6 +46,7 @@ struct channel_program {
union orb orb;
bool initialized;
struct ccw1 *guest_cp;
+ unsigned int ccwchain_count;
};
int cp_init(struct channel_program *cp, union orb *orb);
diff --git a/drivers/s390/crypto/pkey_api.c b/drivers/s390/crypto/pkey_api.c
index 6a94f4965a2d..78c7ab04c7cb 100644
--- a/drivers/s390/crypto/pkey_api.c
+++ b/drivers/s390/crypto/pkey_api.c
@@ -273,7 +273,8 @@ static int pkey_verifykey(const struct pkey_seckey *seckey,
int rc;
/* check the secure key for valid AES secure key */
- rc = cca_check_secaeskeytoken(debug_info, 3, (u8 *)seckey, 0);
+ rc = cca_check_secaeskeytoken(debug_info, 3, (u8 *)seckey,
+ sizeof(*seckey), 0);
if (rc)
goto out;
if (pattributes)
@@ -688,7 +689,7 @@ static int pkey_verifykey2(const u8 *key, size_t keylen,
hdr->version == TOKVER_CCA_AES) {
struct secaeskeytoken *t = (struct secaeskeytoken *)key;
- rc = cca_check_secaeskeytoken(debug_info, 3, key, 0);
+ rc = cca_check_secaeskeytoken(debug_info, 3, key, keylen, 0);
if (rc)
goto out;
if (ktype)
@@ -718,7 +719,7 @@ static int pkey_verifykey2(const u8 *key, size_t keylen,
hdr->version == TOKVER_CCA_VLSC) {
struct cipherkeytoken *t = (struct cipherkeytoken *)key;
- rc = cca_check_secaescipherkey(debug_info, 3, key, 0, 1);
+ rc = cca_check_secaescipherkey(debug_info, 3, key, keylen, 0, 1);
if (rc)
goto out;
if (ktype)
@@ -801,12 +802,12 @@ static int pkey_keyblob2pkey2(const struct pkey_apqn *apqns, size_t nr_apqns,
if (hdr->version == TOKVER_CCA_AES) {
if (keylen != sizeof(struct secaeskeytoken))
return -EINVAL;
- if (cca_check_secaeskeytoken(debug_info, 3, key, 0))
+ if (cca_check_secaeskeytoken(debug_info, 3, key, keylen, 0))
return -EINVAL;
} else if (hdr->version == TOKVER_CCA_VLSC) {
if (keylen < hdr->len || keylen > MAXCCAVLSCTOKENSIZE)
return -EINVAL;
- if (cca_check_secaescipherkey(debug_info, 3, key, 0, 1))
+ if (cca_check_secaescipherkey(debug_info, 3, key, keylen, 0, 1))
return -EINVAL;
} else {
DEBUG_ERR("%s unknown CCA internal token version %d\n",
@@ -1070,13 +1071,13 @@ static int pkey_keyblob2pkey3(const struct pkey_apqn *apqns, size_t nr_apqns,
/* CCA AES data key */
if (keylen != sizeof(struct secaeskeytoken))
return -EINVAL;
- if (cca_check_secaeskeytoken(debug_info, 3, key, 0))
+ if (cca_check_secaeskeytoken(debug_info, 3, key, keylen, 0))
return -EINVAL;
} else if (hdr->version == TOKVER_CCA_VLSC) {
/* CCA AES cipher key */
if (keylen < hdr->len || keylen > MAXCCAVLSCTOKENSIZE)
return -EINVAL;
- if (cca_check_secaescipherkey(debug_info, 3, key, 0, 1))
+ if (cca_check_secaescipherkey(debug_info, 3, key, keylen, 0, 1))
return -EINVAL;
} else {
DEBUG_ERR("%s unknown CCA internal token version %d\n",
@@ -1173,10 +1174,9 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
rc = cca_genseckey(kgs.cardnr, kgs.domain,
kgs.keytype, kgs.seckey.seckey);
DEBUG_DBG("%s cca_genseckey()=%d\n", __func__, rc);
- if (rc)
- break;
- if (copy_to_user(ugs, &kgs, sizeof(kgs)))
- return -EFAULT;
+ if (!rc && copy_to_user(ugs, &kgs, sizeof(kgs)))
+ rc = -EFAULT;
+ memzero_explicit(&kgs, sizeof(kgs));
break;
}
case PKEY_CLR2SECK: {
@@ -1188,9 +1188,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
rc = cca_clr2seckey(kcs.cardnr, kcs.domain, kcs.keytype,
kcs.clrkey.clrkey, kcs.seckey.seckey);
DEBUG_DBG("%s cca_clr2seckey()=%d\n", __func__, rc);
- if (rc)
- break;
- if (copy_to_user(ucs, &kcs, sizeof(kcs)))
+ if (!rc && copy_to_user(ucs, &kcs, sizeof(kcs)))
rc = -EFAULT;
memzero_explicit(&kcs, sizeof(kcs));
break;
@@ -1205,10 +1203,9 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
ksp.seckey.seckey, ksp.protkey.protkey,
&ksp.protkey.len, &ksp.protkey.type);
DEBUG_DBG("%s cca_sec2protkey()=%d\n", __func__, rc);
- if (rc)
- break;
- if (copy_to_user(usp, &ksp, sizeof(ksp)))
- return -EFAULT;
+ if (!rc && copy_to_user(usp, &ksp, sizeof(ksp)))
+ rc = -EFAULT;
+ memzero_explicit(&ksp, sizeof(ksp));
break;
}
case PKEY_CLR2PROTK: {
@@ -1220,9 +1217,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
rc = pkey_clr2protkey(kcp.keytype,
&kcp.clrkey, &kcp.protkey);
DEBUG_DBG("%s pkey_clr2protkey()=%d\n", __func__, rc);
- if (rc)
- break;
- if (copy_to_user(ucp, &kcp, sizeof(kcp)))
+ if (!rc && copy_to_user(ucp, &kcp, sizeof(kcp)))
rc = -EFAULT;
memzero_explicit(&kcp, sizeof(kcp));
break;
@@ -1250,10 +1245,9 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
return -EFAULT;
rc = pkey_skey2pkey(ksp.seckey.seckey, &ksp.protkey);
DEBUG_DBG("%s pkey_skey2pkey()=%d\n", __func__, rc);
- if (rc)
- break;
- if (copy_to_user(usp, &ksp, sizeof(ksp)))
- return -EFAULT;
+ if (!rc && copy_to_user(usp, &ksp, sizeof(ksp)))
+ rc = -EFAULT;
+ memzero_explicit(&ksp, sizeof(ksp));
break;
}
case PKEY_VERIFYKEY: {
@@ -1265,10 +1259,9 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
rc = pkey_verifykey(&kvk.seckey, &kvk.cardnr, &kvk.domain,
&kvk.keysize, &kvk.attributes);
DEBUG_DBG("%s pkey_verifykey()=%d\n", __func__, rc);
- if (rc)
- break;
- if (copy_to_user(uvk, &kvk, sizeof(kvk)))
- return -EFAULT;
+ if (!rc && copy_to_user(uvk, &kvk, sizeof(kvk)))
+ rc = -EFAULT;
+ memzero_explicit(&kvk, sizeof(kvk));
break;
}
case PKEY_GENPROTK: {
@@ -1279,10 +1272,9 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
return -EFAULT;
rc = pkey_genprotkey(kgp.keytype, &kgp.protkey);
DEBUG_DBG("%s pkey_genprotkey()=%d\n", __func__, rc);
- if (rc)
- break;
- if (copy_to_user(ugp, &kgp, sizeof(kgp)))
- return -EFAULT;
+ if (!rc && copy_to_user(ugp, &kgp, sizeof(kgp)))
+ rc = -EFAULT;
+ memzero_explicit(&kgp, sizeof(kgp));
break;
}
case PKEY_VERIFYPROTK: {
@@ -1293,6 +1285,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
return -EFAULT;
rc = pkey_verifyprotkey(&kvp.protkey);
DEBUG_DBG("%s pkey_verifyprotkey()=%d\n", __func__, rc);
+ memzero_explicit(&kvp, sizeof(kvp));
break;
}
case PKEY_KBLOB2PROTK: {
@@ -1307,12 +1300,10 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
return PTR_ERR(kkey);
rc = pkey_keyblob2pkey(kkey, ktp.keylen, &ktp.protkey);
DEBUG_DBG("%s pkey_keyblob2pkey()=%d\n", __func__, rc);
- memzero_explicit(kkey, ktp.keylen);
- kfree(kkey);
- if (rc)
- break;
- if (copy_to_user(utp, &ktp, sizeof(ktp)))
- return -EFAULT;
+ kfree_sensitive(kkey);
+ if (!rc && copy_to_user(utp, &ktp, sizeof(ktp)))
+ rc = -EFAULT;
+ memzero_explicit(&ktp, sizeof(ktp));
break;
}
case PKEY_GENSECK2: {
@@ -1338,23 +1329,23 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
DEBUG_DBG("%s pkey_genseckey2()=%d\n", __func__, rc);
kfree(apqns);
if (rc) {
- kfree(kkey);
+ kfree_sensitive(kkey);
break;
}
if (kgs.key) {
if (kgs.keylen < klen) {
- kfree(kkey);
+ kfree_sensitive(kkey);
return -EINVAL;
}
if (copy_to_user(kgs.key, kkey, klen)) {
- kfree(kkey);
+ kfree_sensitive(kkey);
return -EFAULT;
}
}
kgs.keylen = klen;
if (copy_to_user(ugs, &kgs, sizeof(kgs)))
rc = -EFAULT;
- kfree(kkey);
+ kfree_sensitive(kkey);
break;
}
case PKEY_CLR2SECK2: {
@@ -1367,11 +1358,14 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
if (copy_from_user(&kcs, ucs, sizeof(kcs)))
return -EFAULT;
apqns = _copy_apqns_from_user(kcs.apqns, kcs.apqn_entries);
- if (IS_ERR(apqns))
+ if (IS_ERR(apqns)) {
+ memzero_explicit(&kcs, sizeof(kcs));
return PTR_ERR(apqns);
+ }
kkey = kzalloc(klen, GFP_KERNEL);
if (!kkey) {
kfree(apqns);
+ memzero_explicit(&kcs, sizeof(kcs));
return -ENOMEM;
}
rc = pkey_clr2seckey2(apqns, kcs.apqn_entries,
@@ -1380,16 +1374,19 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
DEBUG_DBG("%s pkey_clr2seckey2()=%d\n", __func__, rc);
kfree(apqns);
if (rc) {
- kfree(kkey);
+ kfree_sensitive(kkey);
+ memzero_explicit(&kcs, sizeof(kcs));
break;
}
if (kcs.key) {
if (kcs.keylen < klen) {
- kfree(kkey);
+ kfree_sensitive(kkey);
+ memzero_explicit(&kcs, sizeof(kcs));
return -EINVAL;
}
if (copy_to_user(kcs.key, kkey, klen)) {
- kfree(kkey);
+ kfree_sensitive(kkey);
+ memzero_explicit(&kcs, sizeof(kcs));
return -EFAULT;
}
}
@@ -1397,7 +1394,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
if (copy_to_user(ucs, &kcs, sizeof(kcs)))
rc = -EFAULT;
memzero_explicit(&kcs, sizeof(kcs));
- kfree(kkey);
+ kfree_sensitive(kkey);
break;
}
case PKEY_VERIFYKEY2: {
@@ -1414,7 +1411,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
&kvk.cardnr, &kvk.domain,
&kvk.type, &kvk.size, &kvk.flags);
DEBUG_DBG("%s pkey_verifykey2()=%d\n", __func__, rc);
- kfree(kkey);
+ kfree_sensitive(kkey);
if (rc)
break;
if (copy_to_user(uvk, &kvk, sizeof(kvk)))
@@ -1441,12 +1438,10 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
kkey, ktp.keylen, &ktp.protkey);
DEBUG_DBG("%s pkey_keyblob2pkey2()=%d\n", __func__, rc);
kfree(apqns);
- memzero_explicit(kkey, ktp.keylen);
- kfree(kkey);
- if (rc)
- break;
- if (copy_to_user(utp, &ktp, sizeof(ktp)))
- return -EFAULT;
+ kfree_sensitive(kkey);
+ if (!rc && copy_to_user(utp, &ktp, sizeof(ktp)))
+ rc = -EFAULT;
+ memzero_explicit(&ktp, sizeof(ktp));
break;
}
case PKEY_APQNS4K: {
@@ -1474,7 +1469,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
rc = pkey_apqns4key(kkey, kak.keylen, kak.flags,
apqns, &nr_apqns);
DEBUG_DBG("%s pkey_apqns4key()=%d\n", __func__, rc);
- kfree(kkey);
+ kfree_sensitive(kkey);
if (rc && rc != -ENOSPC) {
kfree(apqns);
break;
@@ -1560,7 +1555,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
protkey = kmalloc(protkeylen, GFP_KERNEL);
if (!protkey) {
kfree(apqns);
- kfree(kkey);
+ kfree_sensitive(kkey);
return -ENOMEM;
}
rc = pkey_keyblob2pkey3(apqns, ktp.apqn_entries, kkey,
@@ -1568,23 +1563,22 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd,
protkey, &protkeylen);
DEBUG_DBG("%s pkey_keyblob2pkey3()=%d\n", __func__, rc);
kfree(apqns);
- memzero_explicit(kkey, ktp.keylen);
- kfree(kkey);
+ kfree_sensitive(kkey);
if (rc) {
- kfree(protkey);
+ kfree_sensitive(protkey);
break;
}
if (ktp.pkey && ktp.pkeylen) {
if (protkeylen > ktp.pkeylen) {
- kfree(protkey);
+ kfree_sensitive(protkey);
return -EINVAL;
}
if (copy_to_user(ktp.pkey, protkey, protkeylen)) {
- kfree(protkey);
+ kfree_sensitive(protkey);
return -EFAULT;
}
}
- kfree(protkey);
+ kfree_sensitive(protkey);
ktp.pkeylen = protkeylen;
if (copy_to_user(utp, &ktp, sizeof(ktp)))
return -EFAULT;
diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c
index 34dfe7493173..f6a543599ba8 100644
--- a/drivers/s390/crypto/vfio_ap_ops.c
+++ b/drivers/s390/crypto/vfio_ap_ops.c
@@ -45,15 +45,19 @@ static int vfio_ap_mdev_reset_queue(struct vfio_ap_queue *q, unsigned int retry)
* 1. matrix_dev->guests_lock: required to use the KVM pointer to update a KVM
* guest's APCB.
* 2. kvm->lock: required to update a guest's APCB
- * 3. matrix_dev->mdevs_lock: required to access data stored in a matrix_mdev
+ * 3. kvm->arch.crypto.pqap_hook_rwsem: required to update pqap_hook and
+ * serialize against PQAP intercepts
+ * 4. matrix_dev->mdevs_lock: required to access data stored in a matrix_mdev
*
- * Note: If @kvm is NULL, the KVM lock will not be taken.
+ * Note: If @kvm is NULL, the KVM lock and pqap_hook_rwsem will not be taken.
*/
static inline void get_update_locks_for_kvm(struct kvm *kvm)
{
mutex_lock(&matrix_dev->guests_lock);
- if (kvm)
+ if (kvm) {
mutex_lock(&kvm->lock);
+ down_write(&kvm->arch.crypto.pqap_hook_rwsem);
+ }
mutex_lock(&matrix_dev->mdevs_lock);
}
@@ -65,16 +69,19 @@ static inline void get_update_locks_for_kvm(struct kvm *kvm)
*
* The proper unlocking order is:
* 1. matrix_dev->mdevs_lock
- * 2. kvm->lock
- * 3. matrix_dev->guests_lock
+ * 2. kvm->arch.crypto.pqap_hook_rwsem
+ * 3. kvm->lock
+ * 4. matrix_dev->guests_lock
*
- * Note: If @kvm is NULL, the KVM lock will not be released.
+ * Note: If @kvm is NULL, the KVM lock and pqap_hook_rwsem will not be released.
*/
static inline void release_update_locks_for_kvm(struct kvm *kvm)
{
mutex_unlock(&matrix_dev->mdevs_lock);
- if (kvm)
+ if (kvm) {
+ up_write(&kvm->arch.crypto.pqap_hook_rwsem);
mutex_unlock(&kvm->lock);
+ }
mutex_unlock(&matrix_dev->guests_lock);
}
@@ -714,10 +721,17 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev)
ret = vfio_register_emulated_iommu_dev(&matrix_mdev->vdev);
if (ret)
goto err_put_vdev;
+
+ /*
+ * Take the matrix_dev->guests_lock mutex before adding the matrix_mdev
+ * to the mdev_list. All functions that traverse the list must also hold
+ * this lock to guard against additions to or removals from the list
+ * while it is being traversed.
+ */
+ mutex_lock(&matrix_dev->guests_lock);
dev_set_drvdata(&mdev->dev, matrix_mdev);
- mutex_lock(&matrix_dev->mdevs_lock);
list_add(&matrix_mdev->node, &matrix_dev->mdev_list);
- mutex_unlock(&matrix_dev->mdevs_lock);
+ mutex_unlock(&matrix_dev->guests_lock);
return 0;
err_put_vdev:
@@ -1524,26 +1538,17 @@ static const struct attribute_group *vfio_ap_mdev_attr_groups[] = {
static int vfio_ap_mdev_set_kvm(struct ap_matrix_mdev *matrix_mdev,
struct kvm *kvm)
{
- struct ap_matrix_mdev *m;
-
if (kvm->arch.crypto.crycbd) {
- down_write(&kvm->arch.crypto.pqap_hook_rwsem);
- kvm->arch.crypto.pqap_hook = &matrix_mdev->pqap_hook;
- up_write(&kvm->arch.crypto.pqap_hook_rwsem);
-
get_update_locks_for_kvm(kvm);
-
- list_for_each_entry(m, &matrix_dev->mdev_list, node) {
- if (m != matrix_mdev && m->kvm == kvm) {
- release_update_locks_for_kvm(kvm);
- return -EPERM;
- }
+ if (kvm->arch.crypto.pqap_hook) {
+ release_update_locks_for_kvm(kvm);
+ return -EPERM;
}
+ kvm->arch.crypto.pqap_hook = &matrix_mdev->pqap_hook;
kvm_get_kvm(kvm);
matrix_mdev->kvm = kvm;
vfio_ap_mdev_update_guest_apcb(matrix_mdev);
-
release_update_locks_for_kvm(kvm);
}
@@ -1570,18 +1575,15 @@ static void vfio_ap_mdev_unset_kvm(struct ap_matrix_mdev *matrix_mdev)
struct kvm *kvm = matrix_mdev->kvm;
if (kvm && kvm->arch.crypto.crycbd) {
- down_write(&kvm->arch.crypto.pqap_hook_rwsem);
- kvm->arch.crypto.pqap_hook = NULL;
- up_write(&kvm->arch.crypto.pqap_hook_rwsem);
-
get_update_locks_for_kvm(kvm);
+ kvm->arch.crypto.pqap_hook = NULL;
kvm_arch_crypto_clear_masks(kvm);
vfio_ap_mdev_reset_queues(&matrix_mdev->qtable);
- kvm_put_kvm(kvm);
matrix_mdev->kvm = NULL;
release_update_locks_for_kvm(kvm);
+ kvm_put_kvm(kvm);
}
}
@@ -1747,6 +1749,8 @@ static struct ap_matrix_mdev *vfio_ap_mdev_for_queue(struct vfio_ap_queue *q)
unsigned long apid = AP_QID_CARD(q->apqn);
unsigned long apqi = AP_QID_QUEUE(q->apqn);
+ lockdep_assert_held(&matrix_dev->guests_lock);
+
list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
if (test_bit_inv(apid, matrix_mdev->matrix.apm) &&
test_bit_inv(apqi, matrix_mdev->matrix.aqm))
@@ -1766,8 +1770,26 @@ static ssize_t status_show(struct device *dev,
struct ap_matrix_mdev *matrix_mdev;
struct ap_device *apdev = to_ap_dev(dev);
+ mutex_lock(&matrix_dev->guests_lock);
mutex_lock(&matrix_dev->mdevs_lock);
q = dev_get_drvdata(&apdev->device);
+
+ /*
+ * Make sure the drvdata has been set before proceeding. There is a
+ * possibility that the drvdata was not set if the vfio_ap_queue object
+ * could not be allocated when the queue device was probed. In that case,
+ * the locks used in vfio_ap_mdev_probe_queue() are released prior to
+ * removing the sysfs status attribute to avoid a lockdep
+ * splat. That opens a very small window where the status attribute is
+ * still available without the vfio_ap_queue object having been
+ * stored in the device drvdata. In that case, indicate the queue is not
+ * assigned.
+ */
+ if (!q) {
+ nchars = sysfs_emit(buf, "%s\n", AP_QUEUE_UNASSIGNED);
+ goto done;
+ }
+
matrix_mdev = vfio_ap_mdev_for_queue(q);
/* If the queue is assigned to the matrix mediated device, then
@@ -1795,7 +1817,9 @@ static ssize_t status_show(struct device *dev,
AP_QUEUE_UNASSIGNED);
}
+done:
mutex_unlock(&matrix_dev->mdevs_lock);
+ mutex_unlock(&matrix_dev->guests_lock);
return nchars;
}
@@ -1864,13 +1888,16 @@ void vfio_ap_mdev_unregister(void)
int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
{
- int ret;
+ int ret, apqn;
struct vfio_ap_queue *q;
struct ap_matrix_mdev *matrix_mdev;
+ apqn = to_ap_queue(&apdev->device)->qid;
+ matrix_mdev = get_update_locks_by_apqn(apqn);
+
ret = sysfs_create_group(&apdev->device.kobj, &vfio_queue_attr_group);
if (ret)
- return ret;
+ goto err_release_locks;
q = kzalloc(sizeof(*q), GFP_KERNEL);
if (!q) {
@@ -1878,9 +1905,8 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
goto err_remove_group;
}
- q->apqn = to_ap_queue(&apdev->device)->qid;
+ q->apqn = apqn;
q->saved_isc = VFIO_AP_ISC_INVALID;
- matrix_mdev = get_update_locks_by_apqn(q->apqn);
if (matrix_mdev) {
vfio_ap_mdev_link_queue(matrix_mdev, q);
@@ -1907,8 +1933,13 @@ int vfio_ap_mdev_probe_queue(struct ap_device *apdev)
return 0;
err_remove_group:
+ release_update_locks_for_mdev(matrix_mdev);
sysfs_remove_group(&apdev->device.kobj, &vfio_queue_attr_group);
return ret;
+
+err_release_locks:
+ release_update_locks_for_mdev(matrix_mdev);
+ return ret;
}
void vfio_ap_mdev_remove_queue(struct ap_device *apdev)
@@ -2038,8 +2069,20 @@ static void vfio_ap_mdev_cfg_remove(unsigned long *ap_remove,
DECLARE_BITMAP(cdrem, AP_DOMAINS);
int do_remove;
+ /*
+ * It is safe to traverse this list here because the
+ * required guard - matrix_dev->guests_lock - is taken in the
+ * vfio_ap_on_cfg_changed function prior to this function getting
+ * called.
+ */
list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
- mutex_lock(&matrix_mdev->kvm->lock);
+ /*
+ * The mdevs_lock must be held to access fields within matrix_mdev,
+ * and kvm->lock must be taken before mdevs_lock to satisfy the lock
+ * ordering requirement and prevent a lockdep splat.
+ */
+ if (matrix_mdev->kvm)
+ mutex_lock(&matrix_mdev->kvm->lock);
mutex_lock(&matrix_dev->mdevs_lock);
do_remove = bitmap_and(aprem, ap_remove,
@@ -2057,7 +2100,8 @@ static void vfio_ap_mdev_cfg_remove(unsigned long *ap_remove,
cdrem);
mutex_unlock(&matrix_dev->mdevs_lock);
- mutex_unlock(&matrix_mdev->kvm->lock);
+ if (matrix_mdev->kvm)
+ mutex_unlock(&matrix_mdev->kvm->lock);
}
}
@@ -2183,6 +2227,12 @@ static void vfio_ap_mdev_cfg_add(unsigned long *apm_add, unsigned long *aqm_add,
vfio_ap_filter_apid_by_qtype(apm_add, aqm_add);
+ /*
+ * It is safe to traverse this list here because the
+ * required guard - matrix_dev->guests_lock - is taken in the
+ * vfio_ap_on_cfg_changed function prior to this function getting
+ * called.
+ */
list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
/*
* The mdevs_lock must be held in order to access fields
@@ -2250,6 +2300,10 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *cur_cfg_info,
if (!cur_cfg_info || !prev_cfg_info)
return;
+ /*
+ * Take the guests_lock mutex here to guard access to the
+ * matrix_dev->mdev_list in the two functions called below.
+ */
mutex_lock(&matrix_dev->guests_lock);
vfio_ap_mdev_on_cfg_remove(cur_cfg_info, prev_cfg_info);
@@ -2263,9 +2317,6 @@ static void vfio_ap_mdev_hot_plug_cfg(struct ap_matrix_mdev *matrix_mdev)
{
bool filter_domains, filter_adapters, filter_cdoms, do_hotplug = false;
- mutex_lock(&matrix_mdev->kvm->lock);
- mutex_lock(&matrix_dev->mdevs_lock);
-
filter_adapters = bitmap_intersects(matrix_mdev->matrix.apm,
matrix_mdev->apm_add, AP_DEVICES);
filter_domains = bitmap_intersects(matrix_mdev->matrix.aqm,
@@ -2281,9 +2332,6 @@ static void vfio_ap_mdev_hot_plug_cfg(struct ap_matrix_mdev *matrix_mdev)
if (do_hotplug)
vfio_ap_mdev_update_guest_apcb(matrix_mdev);
-
- mutex_unlock(&matrix_dev->mdevs_lock);
- mutex_unlock(&matrix_mdev->kvm->lock);
}
void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,
@@ -2294,15 +2342,29 @@ void vfio_ap_on_scan_complete(struct ap_config_info *new_config_info,
mutex_lock(&matrix_dev->guests_lock);
list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) {
+ /*
+ * The mdevs_lock must be held to access fields within matrix_mdev,
+ * and kvm->lock must be taken before mdevs_lock to satisfy the lock
+ * ordering requirement and prevent a lockdep splat.
+ */
+ if (matrix_mdev->kvm)
+ mutex_lock(&matrix_mdev->kvm->lock);
+ mutex_lock(&matrix_dev->mdevs_lock);
+
if (bitmap_empty(matrix_mdev->apm_add, AP_DEVICES) &&
bitmap_empty(matrix_mdev->aqm_add, AP_DOMAINS) &&
bitmap_empty(matrix_mdev->adm_add, AP_DOMAINS))
- continue;
+ goto do_unlock;
vfio_ap_mdev_hot_plug_cfg(matrix_mdev);
bitmap_clear(matrix_mdev->apm_add, 0, AP_DEVICES);
bitmap_clear(matrix_mdev->aqm_add, 0, AP_DOMAINS);
bitmap_clear(matrix_mdev->adm_add, 0, AP_DOMAINS);
+
+do_unlock:
+ mutex_unlock(&matrix_dev->mdevs_lock);
+ if (matrix_mdev->kvm)
+ mutex_unlock(&matrix_mdev->kvm->lock);
}
mutex_unlock(&matrix_dev->guests_lock);
diff --git a/drivers/s390/crypto/zcrypt_api.c b/drivers/s390/crypto/zcrypt_api.c
index d75affe6f4b6..d0e18a080624 100644
--- a/drivers/s390/crypto/zcrypt_api.c
+++ b/drivers/s390/crypto/zcrypt_api.c
@@ -26,6 +26,7 @@
#include <linux/cdev.h>
#include <linux/ctype.h>
#include <linux/capability.h>
+#include <linux/nospec.h>
#include <asm/debug.h>
#define CREATE_TRACE_POINTS
@@ -929,6 +930,7 @@ static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms,
tdom = *domain;
if (perms != &ap_perms && tdom < AP_DOMAINS) {
if (ap_msg.flags & AP_MSG_FLAG_ADMIN) {
+ tdom = array_index_nospec(tdom, AP_DOMAINS);
if (!test_bit_inv(tdom, perms->adm)) {
rc = -ENODEV;
goto out;
@@ -1121,6 +1123,7 @@ static long _zcrypt_send_ep11_cprb(bool userspace, struct ap_perms *perms,
if (perms != &ap_perms && domain < AP_DOMAINS) {
if (ap_msg.flags & AP_MSG_FLAG_ADMIN) {
+ domain = array_index_nospec(domain, AP_DOMAINS);
if (!test_bit_inv(domain, perms->adm)) {
rc = -ENODEV;
goto out_free;
diff --git a/drivers/s390/crypto/zcrypt_ccamisc.c b/drivers/s390/crypto/zcrypt_ccamisc.c
index 622d960098ad..bb2b51169d92 100644
--- a/drivers/s390/crypto/zcrypt_ccamisc.c
+++ b/drivers/s390/crypto/zcrypt_ccamisc.c
@@ -51,12 +51,18 @@ static DEFINE_SPINLOCK(cca_info_list_lock);
* also checked. Returns 0 on success or errno value on failure.
*/
int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl,
- const u8 *token, int keybitsize)
+ const u8 *token, u32 keysize, int keybitsize)
{
struct secaeskeytoken *t = (struct secaeskeytoken *)token;
#define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__)
+ if (keysize < sizeof(*t)) {
+ if (dbg)
+ DBF("%s keysize %u < min token size %zu\n",
+ __func__, keysize, sizeof(*t));
+ return -EINVAL;
+ }
if (t->type != TOKTYPE_CCA_INTERNAL) {
if (dbg)
DBF("%s token check failed, type 0x%02x != 0x%02x\n",
@@ -90,14 +96,20 @@ EXPORT_SYMBOL(cca_check_secaeskeytoken);
* Returns 0 on success or errno value on failure.
*/
int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl,
- const u8 *token, int keybitsize,
- int checkcpacfexport)
+ const u8 *token, u32 keysize,
+ int keybitsize, int checkcpacfexport)
{
struct cipherkeytoken *t = (struct cipherkeytoken *)token;
bool keybitsizeok = true;
#define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__)
+ if (keysize < sizeof(*t)) {
+ if (dbg)
+ DBF("%s keysize %u < min token size %zu\n",
+ __func__, keysize, sizeof(*t));
+ return -EINVAL;
+ }
if (t->type != TOKTYPE_CCA_INTERNAL) {
if (dbg)
DBF("%s token check failed, type 0x%02x != 0x%02x\n",
@@ -110,6 +122,18 @@ int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl,
__func__, (int)t->version, TOKVER_CCA_VLSC);
return -EINVAL;
}
+ if (t->len > keysize) {
+ if (dbg)
+ DBF("%s token check failed, len %d > keysize %u\n",
+ __func__, (int)t->len, keysize);
+ return -EINVAL;
+ }
+ if (t->len < sizeof(*t)) {
+ if (dbg)
+ DBF("%s token check failed, len %d < min token size %zu\n",
+ __func__, (int)t->len, sizeof(*t));
+ return -EINVAL;
+ }
if (t->algtype != 0x02) {
if (dbg)
DBF("%s token check failed, algtype 0x%02x != 0x02\n",
@@ -184,6 +208,12 @@ int cca_check_sececckeytoken(debug_info_t *dbg, int dbflvl,
#define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__)
+ if (keysize < sizeof(*t)) {
+ if (dbg)
+ DBF("%s keysize %zu < min token size %zu\n",
+ __func__, keysize, sizeof(*t));
+ return -EINVAL;
+ }
if (t->type != TOKTYPE_CCA_INTERNAL_PKA) {
if (dbg)
DBF("%s token check failed, type 0x%02x != 0x%02x\n",
@@ -196,6 +226,12 @@ int cca_check_sececckeytoken(debug_info_t *dbg, int dbflvl,
__func__, (int)t->len, keysize);
return -EINVAL;
}
+ if (t->len < sizeof(*t)) {
+ if (dbg)
+ DBF("%s token check failed, len %d < min token size %zu\n",
+ __func__, (int)t->len, sizeof(*t));
+ return -EINVAL;
+ }
if (t->secid != 0x20) {
if (dbg)
DBF("%s token check failed, secid 0x%02x != 0x20\n",
@@ -419,7 +455,8 @@ int cca_genseckey(u16 cardnr, u16 domain,
/* check secure key token */
rc = cca_check_secaeskeytoken(zcrypt_dbf_info, DBF_ERR,
- prepparm->lv3.keyblock.tok, 8 * keysize);
+ prepparm->lv3.keyblock.tok,
+ seckeysize, 8 * keysize);
if (rc) {
rc = -EIO;
goto out;
@@ -557,7 +594,8 @@ int cca_clr2seckey(u16 cardnr, u16 domain, u32 keybitsize,
/* check secure key token */
rc = cca_check_secaeskeytoken(zcrypt_dbf_info, DBF_ERR,
- prepparm->lv3.keyblock.tok, 8 * keysize);
+ prepparm->lv3.keyblock.tok,
+ seckeysize, 8 * keysize);
if (rc) {
rc = -EIO;
goto out;
@@ -816,6 +854,7 @@ int cca_gencipherkey(u16 cardnr, u16 domain, u32 keybitsize, u32 keygenflags,
} kb;
} __packed * prepparm;
struct cipherkeytoken *t;
+ u32 keybuflen;
/* get already prepared memory for 2 cprbs with param block each */
rc = alloc_and_prep_cprbmem(PARMBSIZE, &mem, &preqcblk, &prepcblk);
@@ -912,23 +951,28 @@ int cca_gencipherkey(u16 cardnr, u16 domain, u32 keybitsize, u32 keygenflags,
}
/* and some checks on the generated key */
+ t = (struct cipherkeytoken *)prepparm->kb.tlv1.gen_key;
+ if (prepparm->kb.tlv1.len < 2 * sizeof(uint16_t) + sizeof(*t)) {
+ rc = -EIO;
+ goto out;
+ }
+ keybuflen = prepparm->kb.tlv1.len - 2 * sizeof(uint16_t);
rc = cca_check_secaescipherkey(zcrypt_dbf_info, DBF_ERR,
prepparm->kb.tlv1.gen_key,
- keybitsize, 1);
+ keybuflen, keybitsize, 1);
if (rc) {
rc = -EIO;
goto out;
}
/* copy the generated vlsc key token */
- t = (struct cipherkeytoken *)prepparm->kb.tlv1.gen_key;
if (keybuf) {
- if (*keybufsize >= t->len)
- memcpy(keybuf, t, t->len);
+ if (*keybufsize >= keybuflen)
+ memcpy(keybuf, t, keybuflen);
else
rc = -EINVAL;
}
- *keybufsize = t->len;
+ *keybufsize = keybuflen;
out:
free_cprbmem(mem, PARMBSIZE, 0);
diff --git a/drivers/s390/crypto/zcrypt_ccamisc.h b/drivers/s390/crypto/zcrypt_ccamisc.h
index 78bf5631848e..0b151bcd4b55 100644
--- a/drivers/s390/crypto/zcrypt_ccamisc.h
+++ b/drivers/s390/crypto/zcrypt_ccamisc.h
@@ -135,7 +135,7 @@ struct eccprivkeytoken {
* also checked. Returns 0 on success or errno value on failure.
*/
int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl,
- const u8 *token, int keybitsize);
+ const u8 *token, u32 keysize, int keybitsize);
/*
* Simple check if the token is a valid CCA secure AES cipher key
@@ -145,8 +145,8 @@ int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl,
* Returns 0 on success or errno value on failure.
*/
int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl,
- const u8 *token, int keybitsize,
- int checkcpacfexport);
+ const u8 *token, u32 keysize,
+ int keybitsize, int checkcpacfexport);
/*
* Simple check if the token is a valid CCA secure ECC private
diff --git a/drivers/s390/net/ctcm_mpc.c b/drivers/s390/net/ctcm_mpc.c
index 55216d2fbea0..10a4e73da49f 100644
--- a/drivers/s390/net/ctcm_mpc.c
+++ b/drivers/s390/net/ctcm_mpc.c
@@ -43,13 +43,13 @@
#include <linux/netdevice.h>
#include <net/dst.h>
-#include <linux/io.h> /* instead of <asm/io.h> ok ? */
-#include <asm/ccwdev.h>
-#include <asm/ccwgroup.h>
-#include <linux/bitops.h> /* instead of <asm/bitops.h> ok ? */
-#include <linux/uaccess.h> /* instead of <asm/uaccess.h> ok ? */
+#include <linux/io.h>
+#include <linux/bitops.h>
+#include <linux/uaccess.h>
#include <linux/wait.h>
#include <linux/moduleparam.h>
+#include <asm/ccwdev.h>
+#include <asm/ccwgroup.h>
#include <asm/idals.h>
#include "ctcm_main.h"
@@ -824,7 +824,7 @@ static void mpc_action_go_ready(fsm_instance *fsm, int event, void *arg)
fsm_deltimer(&grp->timer);
- if (grp->saved_xid2->xid2_flag2 == 0x40) {
+ if (priv->xid->xid2_flag2 == 0x40) {
priv->xid->xid2_flag2 = 0x00;
if (grp->estconnfunc) {
grp->estconnfunc(grp->port_num, 1,
@@ -1633,7 +1633,6 @@ static int mpc_validate_xid(struct mpcg_info *mpcginfo)
"The XID used in the MPC protocol is not valid, "
"rc = %d\n", rc);
priv->xid->xid2_flag2 = 0x40;
- grp->saved_xid2->xid2_flag2 = 0x40;
}
return rc;
diff --git a/drivers/s390/net/ism.h b/drivers/s390/net/ism.h
index 38fe90c2597d..6f21e05b9c6c 100644
--- a/drivers/s390/net/ism.h
+++ b/drivers/s390/net/ism.h
@@ -5,6 +5,7 @@
#include <linux/spinlock.h>
#include <linux/types.h>
#include <linux/pci.h>
+#include <linux/ism.h>
#include <net/smc.h>
#include <asm/pci_insn.h>
@@ -191,6 +192,7 @@ struct ism_sba {
struct ism_dev {
spinlock_t lock;
+ spinlock_t cmd_lock; /* serializes cmds */
struct pci_dev *pdev;
struct smcd_dev *smcd;
diff --git a/drivers/s390/net/ism_drv.c b/drivers/s390/net/ism_drv.c
index d34bb6ec1490..c4150dba6cd4 100644
--- a/drivers/s390/net/ism_drv.c
+++ b/drivers/s390/net/ism_drv.c
@@ -40,6 +40,7 @@ static int ism_cmd(struct ism_dev *ism, void *cmd)
struct ism_req_hdr *req = cmd;
struct ism_resp_hdr *resp = cmd;
+ spin_lock(&ism->cmd_lock);
__ism_write_cmd(ism, req + 1, sizeof(*req), req->len - sizeof(*req));
__ism_write_cmd(ism, req, 0, sizeof(*req));
@@ -53,6 +54,7 @@ static int ism_cmd(struct ism_dev *ism, void *cmd)
}
__ism_read_cmd(ism, resp + 1, sizeof(*resp), resp->len - sizeof(*resp));
out:
+ spin_unlock(&ism->cmd_lock);
return resp->ret;
}
@@ -151,13 +153,16 @@ static int unregister_sba(struct ism_dev *ism)
if (ret && ret != ISM_ERROR)
return -EIO;
+ return 0;
+}
+
+static void ism_free_sba(struct ism_dev *ism)
+{
dma_free_coherent(&ism->pdev->dev, PAGE_SIZE,
ism->sba, ism->sba_dma_addr);
ism->sba = NULL;
ism->sba_dma_addr = 0;
-
- return 0;
}
static int unregister_ieq(struct ism_dev *ism)
@@ -171,13 +176,16 @@ static int unregister_ieq(struct ism_dev *ism)
if (ret && ret != ISM_ERROR)
return -EIO;
+ return 0;
+}
+
+static void ism_free_ieq(struct ism_dev *ism)
+{
dma_free_coherent(&ism->pdev->dev, PAGE_SIZE,
ism->ieq, ism->ieq_dma_addr);
ism->ieq = NULL;
ism->ieq_dma_addr = 0;
-
- return 0;
}
static int ism_read_local_gid(struct ism_dev *ism)
@@ -215,14 +223,14 @@ static int ism_query_rgid(struct smcd_dev *smcd, u64 rgid, u32 vid_valid,
return ism_cmd(ism, &cmd);
}
-static void ism_free_dmb(struct ism_dev *ism, struct smcd_dmb *dmb)
+static void ism_free_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
{
clear_bit(dmb->sba_idx, ism->sba_bitmap);
dma_free_coherent(&ism->pdev->dev, dmb->dmb_len,
dmb->cpu_addr, dmb->dma_addr);
}
-static int ism_alloc_dmb(struct ism_dev *ism, struct smcd_dmb *dmb)
+static int ism_alloc_dmb(struct ism_dev *ism, struct ism_dmb *dmb)
{
unsigned long bit;
@@ -250,7 +258,7 @@ static int ism_alloc_dmb(struct ism_dev *ism, struct smcd_dmb *dmb)
return dmb->cpu_addr ? 0 : -ENOMEM;
}
-static int ism_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
+static int ism_register_dmb(struct smcd_dev *smcd, struct ism_dmb *dmb)
{
struct ism_dev *ism = smcd->priv;
union ism_reg_dmb cmd;
@@ -281,7 +289,12 @@ static int ism_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
return ret;
}
-static int ism_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
+static int smcd_register_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
+{
+ return ism_register_dmb(smcd, (struct ism_dmb *)dmb);
+}
+
+static int ism_unregister_dmb(struct smcd_dev *smcd, struct ism_dmb *dmb)
{
struct ism_dev *ism = smcd->priv;
union ism_unreg_dmb cmd;
@@ -302,6 +315,11 @@ static int ism_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
return ret;
}
+static int smcd_unregister_dmb(struct smcd_dev *smcd, struct smcd_dmb *dmb)
+{
+ return ism_unregister_dmb(smcd, (struct ism_dmb *)dmb);
+}
+
static int ism_add_vlan_id(struct smcd_dev *smcd, u64 vlan_id)
{
struct ism_dev *ism = smcd->priv;
@@ -474,8 +492,8 @@ static irqreturn_t ism_handle_irq(int irq, void *data)
static const struct smcd_ops ism_ops = {
.query_remote_gid = ism_query_rgid,
- .register_dmb = ism_register_dmb,
- .unregister_dmb = ism_unregister_dmb,
+ .register_dmb = smcd_register_dmb,
+ .unregister_dmb = smcd_unregister_dmb,
.add_vlan_id = ism_add_vlan_id,
.del_vlan_id = ism_del_vlan_id,
.set_vlan_required = ism_set_vlan_required,
@@ -525,8 +543,10 @@ static int ism_dev_init(struct ism_dev *ism)
unreg_ieq:
unregister_ieq(ism);
+ ism_free_ieq(ism);
unreg_sba:
unregister_sba(ism);
+ ism_free_sba(ism);
free_irq:
free_irq(pci_irq_vector(pdev, 0), ism);
free_vectors:
@@ -545,6 +565,7 @@ static int ism_probe(struct pci_dev *pdev, const struct pci_device_id *id)
return -ENOMEM;
spin_lock_init(&ism->lock);
+ spin_lock_init(&ism->cmd_lock);
dev_set_drvdata(&pdev->dev, ism);
ism->pdev = pdev;
@@ -598,9 +619,13 @@ static void ism_dev_exit(struct ism_dev *ism)
if (SYSTEM_EID.serial_number[0] != '0' ||
SYSTEM_EID.type[0] != '0')
ism_del_vlan_id(ism->smcd, ISM_RESERVED_VLANID);
+ /* ism will only generate new IRQs while ieq & sba are registered */
unregister_ieq(ism);
unregister_sba(ism);
+ /* drain ongoing irpt handlers */
free_irq(pci_irq_vector(pdev, 0), ism);
+ ism_free_ieq(ism);
+ ism_free_sba(ism);
pci_free_irq_vectors(pdev);
}
diff --git a/drivers/s390/net/netiucv.c b/drivers/s390/net/netiucv.c
index 66076cada8ae..8852b03f943b 100644
--- a/drivers/s390/net/netiucv.c
+++ b/drivers/s390/net/netiucv.c
@@ -47,7 +47,7 @@
#include <linux/ctype.h>
#include <net/dst.h>
-#include <asm/io.h>
+#include <linux/io.h>
#include <linux/uaccess.h>
#include <asm/ebcdic.h>
diff --git a/drivers/s390/virtio/virtio_ccw.c b/drivers/s390/virtio/virtio_ccw.c
index a10dbe632ef9..630db9f4d9f5 100644
--- a/drivers/s390/virtio/virtio_ccw.c
+++ b/drivers/s390/virtio/virtio_ccw.c
@@ -1141,13 +1141,11 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
vcdev->err = -EIO;
}
virtio_ccw_check_activity(vcdev, activity);
-#ifdef CONFIG_VIRTIO_HARDEN_NOTIFICATION
/*
* Paired with virtio_ccw_synchronize_cbs() and interrupts are
* disabled here.
*/
read_lock(&vcdev->irq_lock);
-#endif
for_each_set_bit(i, indicators(vcdev),
sizeof(*indicators(vcdev)) * BITS_PER_BYTE) {
/* The bit clear must happen before the vring kick. */
@@ -1156,9 +1154,7 @@ static void virtio_ccw_int_handler(struct ccw_device *cdev,
vq = virtio_ccw_vq_by_ind(vcdev, i);
vring_interrupt(0, vq);
}
-#ifdef CONFIG_VIRTIO_HARDEN_NOTIFICATION
read_unlock(&vcdev->irq_lock);
-#endif
if (test_bit(0, indicators2(vcdev))) {
virtio_config_changed(&vcdev->vdev);
clear_bit(0, indicators2(vcdev));
diff --git a/drivers/scsi/ch.c b/drivers/scsi/ch.c
index 7ab29eaec6f3..fa0317402ba4 100644
--- a/drivers/scsi/ch.c
+++ b/drivers/scsi/ch.c
@@ -110,7 +110,6 @@ typedef struct {
int minor;
char name[8];
struct scsi_device *device;
- struct scsi_device **dt; /* ptrs to data transfer elements */
u_int firsts[CH_TYPES];
u_int counts[CH_TYPES];
u_int unit_attention;
@@ -357,16 +356,10 @@ ch_readconfig(scsi_changer *ch)
vendor_labels[i]);
}
- /* look up the devices of the data transfer elements */
- ch->dt = kcalloc(ch->counts[CHET_DT], sizeof(*ch->dt),
- GFP_KERNEL);
-
- if (!ch->dt) {
- kfree(buffer);
- return -ENOMEM;
- }
-
+ /* report the devices of the data transfer elements */
for (elem = 0; elem < ch->counts[CHET_DT]; elem++) {
+ struct scsi_device *sdev;
+
id = -1;
lun = 0;
if (elem < CH_DT_MAX && -1 != dt_id[elem]) {
@@ -382,10 +375,8 @@ ch_readconfig(scsi_changer *ch)
VPRINTK(KERN_INFO, "dt 0x%x: ",elem+ch->firsts[CHET_DT]);
if (data[6] & 0x80) {
VPRINTK(KERN_CONT, "not this SCSI bus\n");
- ch->dt[elem] = NULL;
} else if (0 == (data[6] & 0x30)) {
VPRINTK(KERN_CONT, "ID/LUN unknown\n");
- ch->dt[elem] = NULL;
} else {
id = ch->device->id;
lun = 0;
@@ -395,18 +386,16 @@ ch_readconfig(scsi_changer *ch)
}
if (-1 != id) {
VPRINTK(KERN_CONT, "ID %i, LUN %i, ",id,lun);
- ch->dt[elem] =
- scsi_device_lookup(ch->device->host,
- ch->device->channel,
- id,lun);
- if (!ch->dt[elem]) {
+ sdev = scsi_device_lookup(ch->device->host,
+ ch->device->channel,
+ id, lun);
+ if (!sdev) {
/* should not happen */
VPRINTK(KERN_CONT, "Huh? device not found!\n");
} else {
VPRINTK(KERN_CONT, "name: %8.8s %16.16s %4.4s\n",
- ch->dt[elem]->vendor,
- ch->dt[elem]->model,
- ch->dt[elem]->rev);
+ sdev->vendor, sdev->model, sdev->rev);
+ scsi_device_put(sdev);
}
}
}
@@ -570,7 +559,6 @@ static void ch_destroy(struct kref *ref)
scsi_changer *ch = container_of(ref, scsi_changer, ref);
ch->device = NULL;
- kfree(ch->dt);
kfree(ch);
}
diff --git a/drivers/scsi/hisi_sas/hisi_sas.h b/drivers/scsi/hisi_sas/hisi_sas.h
index 2ecf97f7ae52..b0f3110c6055 100644
--- a/drivers/scsi/hisi_sas/hisi_sas.h
+++ b/drivers/scsi/hisi_sas/hisi_sas.h
@@ -104,6 +104,7 @@ enum {
enum dev_status {
HISI_SAS_DEV_INIT,
HISI_SAS_DEV_NORMAL,
+ HISI_SAS_DEV_NCQ_ERR,
};
enum {
@@ -341,7 +342,7 @@ struct hisi_sas_hw {
u8 reg_index, u8 reg_count, u8 *write_data);
void (*wait_cmds_complete_timeout)(struct hisi_hba *hisi_hba,
int delay_ms, int timeout_ms);
- void (*debugfs_snapshot_regs)(struct hisi_hba *hisi_hba);
+ int (*debugfs_snapshot_regs)(struct hisi_hba *hisi_hba);
int complete_hdr_size;
struct scsi_host_template *sht;
};
@@ -449,7 +450,6 @@ struct hisi_hba {
const struct hisi_sas_hw *hw; /* Low level hw interface */
unsigned long sata_dev_bitmap[BITS_TO_LONGS(HISI_SAS_MAX_DEVICES)];
struct work_struct rst_work;
- struct work_struct debugfs_work;
u32 phy_state;
u32 intr_coal_ticks; /* Time of interrupt coalesce in us */
u32 intr_coal_count; /* Interrupt count to coalesce */
diff --git a/drivers/scsi/hisi_sas/hisi_sas_main.c b/drivers/scsi/hisi_sas/hisi_sas_main.c
index 290c9915fb9b..16ddf63ee9b8 100644
--- a/drivers/scsi/hisi_sas/hisi_sas_main.c
+++ b/drivers/scsi/hisi_sas/hisi_sas_main.c
@@ -1537,7 +1537,7 @@ static int hisi_sas_controller_prereset(struct hisi_hba *hisi_hba)
return -EPERM;
}
- if (hisi_sas_debugfs_enable && hisi_hba->debugfs_itct[0].itct)
+ if (hisi_sas_debugfs_enable)
hisi_hba->hw->debugfs_snapshot_regs(hisi_hba);
return 0;
@@ -1574,6 +1574,7 @@ static int hisi_sas_abort_task(struct sas_task *task)
struct hisi_sas_internal_abort_data internal_abort_data = { false };
struct domain_device *device = task->dev;
struct hisi_sas_device *sas_dev = device->lldd_dev;
+ struct hisi_sas_slot *slot = task->lldd_task;
struct hisi_hba *hisi_hba;
struct device *dev;
int rc = TMF_RESP_FUNC_FAILED;
@@ -1587,7 +1588,6 @@ static int hisi_sas_abort_task(struct sas_task *task)
spin_lock_irqsave(&task->task_state_lock, flags);
if (task->task_state_flags & SAS_TASK_STATE_DONE) {
- struct hisi_sas_slot *slot = task->lldd_task;
struct hisi_sas_cq *cq;
if (slot) {
@@ -1605,8 +1605,10 @@ static int hisi_sas_abort_task(struct sas_task *task)
task->task_state_flags |= SAS_TASK_STATE_ABORTED;
spin_unlock_irqrestore(&task->task_state_lock, flags);
- if (task->lldd_task && task->task_proto & SAS_PROTOCOL_SSP) {
- struct hisi_sas_slot *slot = task->lldd_task;
+ if (!slot)
+ goto out;
+
+ if (task->task_proto & SAS_PROTOCOL_SSP) {
u16 tag = slot->idx;
int rc2;
@@ -1632,17 +1634,29 @@ static int hisi_sas_abort_task(struct sas_task *task)
} else if (task->task_proto & SAS_PROTOCOL_SATA ||
task->task_proto & SAS_PROTOCOL_STP) {
if (task->dev->dev_type == SAS_SATA_DEV) {
+ struct ata_queued_cmd *qc = task->uldd_task;
+
rc = hisi_sas_internal_task_abort_dev(sas_dev, false);
if (rc < 0) {
dev_err(dev, "abort task: internal abort failed\n");
goto out;
}
hisi_sas_dereg_device(hisi_hba, device);
- rc = hisi_sas_softreset_ata_disk(device);
+
+ /*
+ * If an ATA internal command times out in ATA EH, it
+ * need to execute soft reset, so check the scsicmd
+ */
+ if ((sas_dev->dev_status == HISI_SAS_DEV_NCQ_ERR) &&
+ qc && qc->scsicmd) {
+ hisi_sas_do_release_task(hisi_hba, task, slot, true);
+ rc = TMF_RESP_FUNC_COMPLETE;
+ } else {
+ rc = hisi_sas_softreset_ata_disk(device);
+ }
}
- } else if (task->lldd_task && task->task_proto & SAS_PROTOCOL_SMP) {
+ } else if (task->task_proto & SAS_PROTOCOL_SMP) {
/* SMP */
- struct hisi_sas_slot *slot = task->lldd_task;
u32 tag = slot->idx;
struct hisi_sas_cq *cq = &hisi_hba->cq[slot->dlvry_queue];
@@ -1758,6 +1772,9 @@ static int hisi_sas_I_T_nexus_reset(struct domain_device *device)
struct device *dev = hisi_hba->dev;
int rc;
+ if (sas_dev->dev_status == HISI_SAS_DEV_NCQ_ERR)
+ sas_dev->dev_status = HISI_SAS_DEV_NORMAL;
+
rc = hisi_sas_internal_task_abort_dev(sas_dev, false);
if (rc < 0) {
dev_err(dev, "I_T nexus reset: internal abort (%d)\n", rc);
@@ -1893,8 +1910,19 @@ static bool hisi_sas_internal_abort_timeout(struct sas_task *task,
struct hisi_hba *hisi_hba = dev_to_hisi_hba(device);
struct hisi_sas_internal_abort_data *timeout = data;
- if (hisi_sas_debugfs_enable && hisi_hba->debugfs_itct[0].itct)
- queue_work(hisi_hba->wq, &hisi_hba->debugfs_work);
+ if (hisi_sas_debugfs_enable) {
+ /*
+ * If timeout occurs in device gone scenario, to avoid
+ * circular dependency like:
+ * hisi_sas_dev_gone() -> down() -> ... ->
+ * hisi_sas_internal_abort_timeout() -> down().
+ */
+ if (!timeout->rst_ha_timeout)
+ down(&hisi_hba->sem);
+ hisi_hba->hw->debugfs_snapshot_regs(hisi_hba);
+ if (!timeout->rst_ha_timeout)
+ up(&hisi_hba->sem);
+ }
if (task->task_state_flags & SAS_TASK_STATE_DONE) {
pr_err("Internal abort: timeout %016llx\n",
@@ -2450,7 +2478,7 @@ int hisi_sas_probe(struct platform_device *pdev,
shost->transportt = hisi_sas_stt;
shost->max_id = HISI_SAS_MAX_DEVICES;
shost->max_lun = ~0;
- shost->max_channel = 1;
+ shost->max_channel = 0;
shost->max_cmd_len = 16;
if (hisi_hba->hw->slot_index_alloc) {
shost->can_queue = HISI_SAS_MAX_COMMANDS;
diff --git a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
index 3db206c74383..27203628e646 100644
--- a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
+++ b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c
@@ -408,6 +408,11 @@
#define CMPLT_HDR_CMPLT_MSK (0x3 << CMPLT_HDR_CMPLT_OFF)
#define CMPLT_HDR_ERROR_PHASE_OFF 2
#define CMPLT_HDR_ERROR_PHASE_MSK (0xff << CMPLT_HDR_ERROR_PHASE_OFF)
+/* bit[9:2] Error Phase */
+#define ERR_PHASE_RESPONSE_FRAME_REV_STAGE_OFF \
+ 8
+#define ERR_PHASE_RESPONSE_FRAME_REV_STAGE_MSK \
+ (0x1 << ERR_PHASE_RESPONSE_FRAME_REV_STAGE_OFF)
#define CMPLT_HDR_RSPNS_XFRD_OFF 10
#define CMPLT_HDR_RSPNS_XFRD_MSK (0x1 << CMPLT_HDR_RSPNS_XFRD_OFF)
#define CMPLT_HDR_RSPNS_GOOD_OFF 11
@@ -427,8 +432,15 @@
#define CMPLT_HDR_DEV_ID_OFF 16
#define CMPLT_HDR_DEV_ID_MSK (0xffff << CMPLT_HDR_DEV_ID_OFF)
/* dw3 */
+#define CMPLT_HDR_SATA_DISK_ERR_OFF 16
+#define CMPLT_HDR_SATA_DISK_ERR_MSK (0x1 << CMPLT_HDR_SATA_DISK_ERR_OFF)
#define CMPLT_HDR_IO_IN_TARGET_OFF 17
#define CMPLT_HDR_IO_IN_TARGET_MSK (0x1 << CMPLT_HDR_IO_IN_TARGET_OFF)
+/* bit[23:18] ERR_FIS_ATA_STATUS */
+#define FIS_ATA_STATUS_ERR_OFF 18
+#define FIS_ATA_STATUS_ERR_MSK (0x1 << FIS_ATA_STATUS_ERR_OFF)
+#define FIS_TYPE_SDB_OFF 31
+#define FIS_TYPE_SDB_MSK (0x1 << FIS_TYPE_SDB_OFF)
/* ITCT header */
/* qw0 */
@@ -542,8 +554,7 @@ static int prot_mask;
module_param(prot_mask, int, 0444);
MODULE_PARM_DESC(prot_mask, " host protection capabilities mask, def=0x0 ");
-static void debugfs_work_handler_v3_hw(struct work_struct *work);
-static void debugfs_snapshot_regs_v3_hw(struct hisi_hba *hisi_hba);
+static int debugfs_snapshot_regs_v3_hw(struct hisi_hba *hisi_hba);
static u32 hisi_sas_read32(struct hisi_hba *hisi_hba, u32 off)
{
@@ -2160,6 +2171,18 @@ static irqreturn_t fatal_axi_int_v3_hw(int irq_no, void *p)
return IRQ_HANDLED;
}
+static bool is_ncq_err_v3_hw(struct hisi_sas_complete_v3_hdr *complete_hdr)
+{
+ u32 dw0, dw3;
+
+ dw0 = le32_to_cpu(complete_hdr->dw0);
+ dw3 = le32_to_cpu(complete_hdr->dw3);
+
+ return (dw0 & ERR_PHASE_RESPONSE_FRAME_REV_STAGE_MSK) &&
+ (dw3 & FIS_TYPE_SDB_MSK) &&
+ (dw3 & FIS_ATA_STATUS_ERR_MSK);
+}
+
static bool
slot_err_v3_hw(struct hisi_hba *hisi_hba, struct sas_task *task,
struct hisi_sas_slot *slot)
@@ -2383,15 +2406,14 @@ static void slot_complete_v3_hw(struct hisi_hba *hisi_hba,
task->task_done(task);
}
-static irqreturn_t cq_thread_v3_hw(int irq_no, void *p)
+static void complete_v3_hw(struct hisi_sas_cq *cq)
{
- struct hisi_sas_cq *cq = p;
- struct hisi_hba *hisi_hba = cq->hisi_hba;
- struct hisi_sas_slot *slot;
struct hisi_sas_complete_v3_hdr *complete_queue;
- u32 rd_point = cq->rd_point, wr_point;
+ struct hisi_hba *hisi_hba = cq->hisi_hba;
+ u32 rd_point, wr_point;
int queue = cq->id;
+ rd_point = cq->rd_point;
complete_queue = hisi_hba->complete_hdr[queue];
wr_point = hisi_sas_read32(hisi_hba, COMPL_Q_0_WR_PTR +
@@ -2400,14 +2422,35 @@ static irqreturn_t cq_thread_v3_hw(int irq_no, void *p)
while (rd_point != wr_point) {
struct hisi_sas_complete_v3_hdr *complete_hdr;
struct device *dev = hisi_hba->dev;
- u32 dw1;
+ struct hisi_sas_slot *slot;
+ u32 dw0, dw1, dw3;
int iptt;
complete_hdr = &complete_queue[rd_point];
+ dw0 = le32_to_cpu(complete_hdr->dw0);
dw1 = le32_to_cpu(complete_hdr->dw1);
+ dw3 = le32_to_cpu(complete_hdr->dw3);
iptt = dw1 & CMPLT_HDR_IPTT_MSK;
- if (likely(iptt < HISI_SAS_COMMAND_ENTRIES_V3_HW)) {
+ if (unlikely((dw0 & CMPLT_HDR_CMPLT_MSK) == 0x3) &&
+ (dw3 & CMPLT_HDR_SATA_DISK_ERR_MSK)) {
+ int device_id = (dw1 & CMPLT_HDR_DEV_ID_MSK) >>
+ CMPLT_HDR_DEV_ID_OFF;
+ struct hisi_sas_itct *itct =
+ &hisi_hba->itct[device_id];
+ struct hisi_sas_device *sas_dev =
+ &hisi_hba->devices[device_id];
+ struct domain_device *device = sas_dev->sas_device;
+
+ dev_err(dev, "erroneous completion disk err dev id=%d sas_addr=0x%llx CQ hdr: 0x%x 0x%x 0x%x 0x%x\n",
+ device_id, itct->sas_addr, dw0, dw1,
+ complete_hdr->act, dw3);
+
+ if (is_ncq_err_v3_hw(complete_hdr))
+ sas_dev->dev_status = HISI_SAS_DEV_NCQ_ERR;
+
+ sas_ata_device_link_abort(device, true);
+ } else if (likely(iptt < HISI_SAS_COMMAND_ENTRIES_V3_HW)) {
slot = &hisi_hba->slot_info[iptt];
slot->cmplt_queue_slot = rd_point;
slot->cmplt_queue = queue;
@@ -2422,6 +2465,13 @@ static irqreturn_t cq_thread_v3_hw(int irq_no, void *p)
/* update rd_point */
cq->rd_point = rd_point;
hisi_sas_write32(hisi_hba, COMPL_Q_0_RD_PTR + (0x14 * queue), rd_point);
+}
+
+static irqreturn_t cq_thread_v3_hw(int irq_no, void *p)
+{
+ struct hisi_sas_cq *cq = p;
+
+ complete_v3_hw(cq);
cond_resched();
return IRQ_HANDLED;
@@ -2810,7 +2860,7 @@ static int slave_configure_v3_hw(struct scsi_device *sdev)
return 0;
if (!device_link_add(&sdev->sdev_gendev, dev,
- DL_FLAG_PM_RUNTIME | DL_FLAG_RPM_ACTIVE)) {
+ DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME | DL_FLAG_RPM_ACTIVE)) {
if (pm_runtime_enabled(dev)) {
dev_info(dev, "add device link failed, disable runtime PM for the host\n");
pm_runtime_disable(dev);
@@ -2820,6 +2870,15 @@ static int slave_configure_v3_hw(struct scsi_device *sdev)
return 0;
}
+static void hisi_sas_sdev_destroy(struct scsi_device *sdev)
+{
+ struct Scsi_Host *shost = dev_to_shost(&sdev->sdev_gendev);
+ struct hisi_hba *hisi_hba = shost_priv(shost);
+ struct device *dev = hisi_hba->dev;
+
+ device_link_remove(&sdev->sdev_gendev, dev);
+}
+
static struct attribute *host_v3_hw_attrs[] = {
&dev_attr_phy_event_threshold.attr,
&dev_attr_intr_conv_v3_hw.attr,
@@ -3208,6 +3267,7 @@ static struct scsi_host_template sht_v3_hw = {
.eh_device_reset_handler = sas_eh_device_reset_handler,
.eh_target_reset_handler = sas_eh_target_reset_handler,
.slave_alloc = hisi_sas_slave_alloc,
+ .slave_destroy = hisi_sas_sdev_destroy,
.target_destroy = sas_target_destroy,
.ioctl = sas_ioctl,
#ifdef CONFIG_COMPAT
@@ -3260,7 +3320,6 @@ hisi_sas_shost_alloc_pci(struct pci_dev *pdev)
hisi_hba = shost_priv(shost);
INIT_WORK(&hisi_hba->rst_work, hisi_sas_rst_work_handler);
- INIT_WORK(&hisi_hba->debugfs_work, debugfs_work_handler_v3_hw);
hisi_hba->hw = &hisi_sas_v3_hw;
hisi_hba->pci_dev = pdev;
hisi_hba->dev = dev;
@@ -3418,6 +3477,11 @@ debugfs_to_reg_name_v3_hw(int off, int base_off,
return NULL;
}
+static bool debugfs_dump_is_generated_v3_hw(void *p)
+{
+ return p ? true : false;
+}
+
static void debugfs_print_reg_v3_hw(u32 *regs_val, struct seq_file *s,
const struct hisi_sas_debugfs_reg *reg)
{
@@ -3443,6 +3507,9 @@ static int debugfs_global_v3_hw_show(struct seq_file *s, void *p)
{
struct hisi_sas_debugfs_regs *global = s->private;
+ if (!debugfs_dump_is_generated_v3_hw(global->data))
+ return -EPERM;
+
debugfs_print_reg_v3_hw(global->data, s,
&debugfs_global_reg);
@@ -3454,6 +3521,9 @@ static int debugfs_axi_v3_hw_show(struct seq_file *s, void *p)
{
struct hisi_sas_debugfs_regs *axi = s->private;
+ if (!debugfs_dump_is_generated_v3_hw(axi->data))
+ return -EPERM;
+
debugfs_print_reg_v3_hw(axi->data, s,
&debugfs_axi_reg);
@@ -3465,6 +3535,9 @@ static int debugfs_ras_v3_hw_show(struct seq_file *s, void *p)
{
struct hisi_sas_debugfs_regs *ras = s->private;
+ if (!debugfs_dump_is_generated_v3_hw(ras->data))
+ return -EPERM;
+
debugfs_print_reg_v3_hw(ras->data, s,
&debugfs_ras_reg);
@@ -3477,6 +3550,9 @@ static int debugfs_port_v3_hw_show(struct seq_file *s, void *p)
struct hisi_sas_debugfs_port *port = s->private;
const struct hisi_sas_debugfs_reg *reg_port = &debugfs_port_reg;
+ if (!debugfs_dump_is_generated_v3_hw(port->data))
+ return -EPERM;
+
debugfs_print_reg_v3_hw(port->data, s, reg_port);
return 0;
@@ -3532,6 +3608,9 @@ static int debugfs_cq_v3_hw_show(struct seq_file *s, void *p)
struct hisi_sas_debugfs_cq *debugfs_cq = s->private;
int slot;
+ if (!debugfs_dump_is_generated_v3_hw(debugfs_cq->complete_hdr))
+ return -EPERM;
+
for (slot = 0; slot < HISI_SAS_QUEUE_SLOTS; slot++)
debugfs_cq_show_slot_v3_hw(s, slot, debugfs_cq);
@@ -3553,8 +3632,12 @@ static void debugfs_dq_show_slot_v3_hw(struct seq_file *s, int slot,
static int debugfs_dq_v3_hw_show(struct seq_file *s, void *p)
{
+ struct hisi_sas_debugfs_dq *debugfs_dq = s->private;
int slot;
+ if (!debugfs_dump_is_generated_v3_hw(debugfs_dq->hdr))
+ return -EPERM;
+
for (slot = 0; slot < HISI_SAS_QUEUE_SLOTS; slot++)
debugfs_dq_show_slot_v3_hw(s, slot, s->private);
@@ -3568,6 +3651,9 @@ static int debugfs_iost_v3_hw_show(struct seq_file *s, void *p)
struct hisi_sas_iost *iost = debugfs_iost->iost;
int i, max_command_entries = HISI_SAS_MAX_COMMANDS;
+ if (!debugfs_dump_is_generated_v3_hw(iost))
+ return -EPERM;
+
for (i = 0; i < max_command_entries; i++, iost++) {
__le64 *data = &iost->qw0;
@@ -3587,6 +3673,9 @@ static int debugfs_iost_cache_v3_hw_show(struct seq_file *s, void *p)
int i, tab_idx;
__le64 *iost;
+ if (!debugfs_dump_is_generated_v3_hw(iost_cache))
+ return -EPERM;
+
for (i = 0; i < HISI_SAS_IOST_ITCT_CACHE_NUM; i++, iost_cache++) {
/*
* Data struct of IOST cache:
@@ -3610,6 +3699,9 @@ static int debugfs_itct_v3_hw_show(struct seq_file *s, void *p)
struct hisi_sas_debugfs_itct *debugfs_itct = s->private;
struct hisi_sas_itct *itct = debugfs_itct->itct;
+ if (!debugfs_dump_is_generated_v3_hw(itct))
+ return -EPERM;
+
for (i = 0; i < HISI_SAS_MAX_ITCT_ENTRIES; i++, itct++) {
__le64 *data = &itct->qw0;
@@ -3629,6 +3721,9 @@ static int debugfs_itct_cache_v3_hw_show(struct seq_file *s, void *p)
int i, tab_idx;
__le64 *itct;
+ if (!debugfs_dump_is_generated_v3_hw(itct_cache))
+ return -EPERM;
+
for (i = 0; i < HISI_SAS_IOST_ITCT_CACHE_NUM; i++, itct_cache++) {
/*
* Data struct of ITCT cache:
@@ -3646,10 +3741,9 @@ static int debugfs_itct_cache_v3_hw_show(struct seq_file *s, void *p)
}
DEFINE_SHOW_ATTRIBUTE(debugfs_itct_cache_v3_hw);
-static void debugfs_create_files_v3_hw(struct hisi_hba *hisi_hba)
+static void debugfs_create_files_v3_hw(struct hisi_hba *hisi_hba, int index)
{
u64 *debugfs_timestamp;
- int dump_index = hisi_hba->debugfs_dump_index;
struct dentry *dump_dentry;
struct dentry *dentry;
char name[256];
@@ -3657,17 +3751,17 @@ static void debugfs_create_files_v3_hw(struct hisi_hba *hisi_hba)
int c;
int d;
- snprintf(name, 256, "%d", dump_index);
+ snprintf(name, 256, "%d", index);
dump_dentry = debugfs_create_dir(name, hisi_hba->debugfs_dump_dentry);
- debugfs_timestamp = &hisi_hba->debugfs_timestamp[dump_index];
+ debugfs_timestamp = &hisi_hba->debugfs_timestamp[index];
debugfs_create_u64("timestamp", 0400, dump_dentry,
debugfs_timestamp);
debugfs_create_file("global", 0400, dump_dentry,
- &hisi_hba->debugfs_regs[dump_index][DEBUGFS_GLOBAL],
+ &hisi_hba->debugfs_regs[index][DEBUGFS_GLOBAL],
&debugfs_global_v3_hw_fops);
/* Create port dir and files */
@@ -3676,7 +3770,7 @@ static void debugfs_create_files_v3_hw(struct hisi_hba *hisi_hba)
snprintf(name, 256, "%d", p);
debugfs_create_file(name, 0400, dentry,
- &hisi_hba->debugfs_port_reg[dump_index][p],
+ &hisi_hba->debugfs_port_reg[index][p],
&debugfs_port_v3_hw_fops);
}
@@ -3686,7 +3780,7 @@ static void debugfs_create_files_v3_hw(struct hisi_hba *hisi_hba)
snprintf(name, 256, "%d", c);
debugfs_create_file(name, 0400, dentry,
- &hisi_hba->debugfs_cq[dump_index][c],
+ &hisi_hba->debugfs_cq[index][c],
&debugfs_cq_v3_hw_fops);
}
@@ -3696,66 +3790,35 @@ static void debugfs_create_files_v3_hw(struct hisi_hba *hisi_hba)
snprintf(name, 256, "%d", d);
debugfs_create_file(name, 0400, dentry,
- &hisi_hba->debugfs_dq[dump_index][d],
+ &hisi_hba->debugfs_dq[index][d],
&debugfs_dq_v3_hw_fops);
}
debugfs_create_file("iost", 0400, dump_dentry,
- &hisi_hba->debugfs_iost[dump_index],
+ &hisi_hba->debugfs_iost[index],
&debugfs_iost_v3_hw_fops);
debugfs_create_file("iost_cache", 0400, dump_dentry,
- &hisi_hba->debugfs_iost_cache[dump_index],
+ &hisi_hba->debugfs_iost_cache[index],
&debugfs_iost_cache_v3_hw_fops);
debugfs_create_file("itct", 0400, dump_dentry,
- &hisi_hba->debugfs_itct[dump_index],
+ &hisi_hba->debugfs_itct[index],
&debugfs_itct_v3_hw_fops);
debugfs_create_file("itct_cache", 0400, dump_dentry,
- &hisi_hba->debugfs_itct_cache[dump_index],
+ &hisi_hba->debugfs_itct_cache[index],
&debugfs_itct_cache_v3_hw_fops);
debugfs_create_file("axi", 0400, dump_dentry,
- &hisi_hba->debugfs_regs[dump_index][DEBUGFS_AXI],
+ &hisi_hba->debugfs_regs[index][DEBUGFS_AXI],
&debugfs_axi_v3_hw_fops);
debugfs_create_file("ras", 0400, dump_dentry,
- &hisi_hba->debugfs_regs[dump_index][DEBUGFS_RAS],
+ &hisi_hba->debugfs_regs[index][DEBUGFS_RAS],
&debugfs_ras_v3_hw_fops);
}
-static void debugfs_snapshot_regs_v3_hw(struct hisi_hba *hisi_hba)
-{
- int debugfs_dump_index = hisi_hba->debugfs_dump_index;
- struct device *dev = hisi_hba->dev;
- u64 timestamp = local_clock();
-
- if (debugfs_dump_index >= hisi_sas_debugfs_dump_count) {
- dev_warn(dev, "dump count exceeded!\n");
- return;
- }
-
- do_div(timestamp, NSEC_PER_MSEC);
- hisi_hba->debugfs_timestamp[debugfs_dump_index] = timestamp;
-
- debugfs_snapshot_prepare_v3_hw(hisi_hba);
-
- debugfs_snapshot_global_reg_v3_hw(hisi_hba);
- debugfs_snapshot_port_reg_v3_hw(hisi_hba);
- debugfs_snapshot_axi_reg_v3_hw(hisi_hba);
- debugfs_snapshot_ras_reg_v3_hw(hisi_hba);
- debugfs_snapshot_cq_reg_v3_hw(hisi_hba);
- debugfs_snapshot_dq_reg_v3_hw(hisi_hba);
- debugfs_snapshot_itct_reg_v3_hw(hisi_hba);
- debugfs_snapshot_iost_reg_v3_hw(hisi_hba);
-
- debugfs_create_files_v3_hw(hisi_hba);
-
- debugfs_snapshot_restore_v3_hw(hisi_hba);
- hisi_hba->debugfs_dump_index++;
-}
-
static ssize_t debugfs_trigger_dump_v3_hw_write(struct file *file,
const char __user *user_buf,
size_t count, loff_t *ppos)
@@ -3763,9 +3826,6 @@ static ssize_t debugfs_trigger_dump_v3_hw_write(struct file *file,
struct hisi_hba *hisi_hba = file->f_inode->i_private;
char buf[8];
- if (hisi_hba->debugfs_dump_index >= hisi_sas_debugfs_dump_count)
- return -EFAULT;
-
if (count > 8)
return -EFAULT;
@@ -3775,7 +3835,12 @@ static ssize_t debugfs_trigger_dump_v3_hw_write(struct file *file,
if (buf[0] != '1')
return -EFAULT;
- queue_work(hisi_hba->wq, &hisi_hba->debugfs_work);
+ down(&hisi_hba->sem);
+ if (debugfs_snapshot_regs_v3_hw(hisi_hba)) {
+ up(&hisi_hba->sem);
+ return -EFAULT;
+ }
+ up(&hisi_hba->sem);
return count;
}
@@ -4526,36 +4591,40 @@ static void debugfs_fifo_init_v3_hw(struct hisi_hba *hisi_hba)
}
}
-static void debugfs_work_handler_v3_hw(struct work_struct *work)
-{
- struct hisi_hba *hisi_hba =
- container_of(work, struct hisi_hba, debugfs_work);
-
- debugfs_snapshot_regs_v3_hw(hisi_hba);
-}
-
static void debugfs_release_v3_hw(struct hisi_hba *hisi_hba, int dump_index)
{
struct device *dev = hisi_hba->dev;
int i;
devm_kfree(dev, hisi_hba->debugfs_iost_cache[dump_index].cache);
+ hisi_hba->debugfs_iost_cache[dump_index].cache = NULL;
devm_kfree(dev, hisi_hba->debugfs_itct_cache[dump_index].cache);
+ hisi_hba->debugfs_itct_cache[dump_index].cache = NULL;
devm_kfree(dev, hisi_hba->debugfs_iost[dump_index].iost);
+ hisi_hba->debugfs_iost[dump_index].iost = NULL;
devm_kfree(dev, hisi_hba->debugfs_itct[dump_index].itct);
+ hisi_hba->debugfs_itct[dump_index].itct = NULL;
- for (i = 0; i < hisi_hba->queue_count; i++)
+ for (i = 0; i < hisi_hba->queue_count; i++) {
devm_kfree(dev, hisi_hba->debugfs_dq[dump_index][i].hdr);
+ hisi_hba->debugfs_dq[dump_index][i].hdr = NULL;
+ }
- for (i = 0; i < hisi_hba->queue_count; i++)
+ for (i = 0; i < hisi_hba->queue_count; i++) {
devm_kfree(dev,
hisi_hba->debugfs_cq[dump_index][i].complete_hdr);
+ hisi_hba->debugfs_cq[dump_index][i].complete_hdr = NULL;
+ }
- for (i = 0; i < DEBUGFS_REGS_NUM; i++)
+ for (i = 0; i < DEBUGFS_REGS_NUM; i++) {
devm_kfree(dev, hisi_hba->debugfs_regs[dump_index][i].data);
+ hisi_hba->debugfs_regs[dump_index][i].data = NULL;
+ }
- for (i = 0; i < hisi_hba->n_phy; i++)
+ for (i = 0; i < hisi_hba->n_phy; i++) {
devm_kfree(dev, hisi_hba->debugfs_port_reg[dump_index][i].data);
+ hisi_hba->debugfs_port_reg[dump_index][i].data = NULL;
+ }
}
static const struct hisi_sas_debugfs_reg *debugfs_reg_array_v3_hw[DEBUGFS_REGS_NUM] = {
@@ -4568,7 +4637,7 @@ static int debugfs_alloc_v3_hw(struct hisi_hba *hisi_hba, int dump_index)
{
const struct hisi_sas_hw *hw = hisi_hba->hw;
struct device *dev = hisi_hba->dev;
- int p, c, d, r, i;
+ int p, c, d, r;
size_t sz;
for (r = 0; r < DEBUGFS_REGS_NUM; r++) {
@@ -4648,11 +4717,46 @@ static int debugfs_alloc_v3_hw(struct hisi_hba *hisi_hba, int dump_index)
return 0;
fail:
- for (i = 0; i < hisi_sas_debugfs_dump_count; i++)
- debugfs_release_v3_hw(hisi_hba, i);
+ debugfs_release_v3_hw(hisi_hba, dump_index);
return -ENOMEM;
}
+static int debugfs_snapshot_regs_v3_hw(struct hisi_hba *hisi_hba)
+{
+ int debugfs_dump_index = hisi_hba->debugfs_dump_index;
+ struct device *dev = hisi_hba->dev;
+ u64 timestamp = local_clock();
+
+ if (debugfs_dump_index >= hisi_sas_debugfs_dump_count) {
+ dev_warn(dev, "dump count exceeded!\n");
+ return -EINVAL;
+ }
+
+ if (debugfs_alloc_v3_hw(hisi_hba, debugfs_dump_index)) {
+ dev_warn(dev, "failed to alloc memory\n");
+ return -ENOMEM;
+ }
+
+ do_div(timestamp, NSEC_PER_MSEC);
+ hisi_hba->debugfs_timestamp[debugfs_dump_index] = timestamp;
+
+ debugfs_snapshot_prepare_v3_hw(hisi_hba);
+
+ debugfs_snapshot_global_reg_v3_hw(hisi_hba);
+ debugfs_snapshot_port_reg_v3_hw(hisi_hba);
+ debugfs_snapshot_axi_reg_v3_hw(hisi_hba);
+ debugfs_snapshot_ras_reg_v3_hw(hisi_hba);
+ debugfs_snapshot_cq_reg_v3_hw(hisi_hba);
+ debugfs_snapshot_dq_reg_v3_hw(hisi_hba);
+ debugfs_snapshot_itct_reg_v3_hw(hisi_hba);
+ debugfs_snapshot_iost_reg_v3_hw(hisi_hba);
+
+ debugfs_snapshot_restore_v3_hw(hisi_hba);
+ hisi_hba->debugfs_dump_index++;
+
+ return 0;
+}
+
static void debugfs_phy_down_cnt_init_v3_hw(struct hisi_hba *hisi_hba)
{
struct dentry *dir = debugfs_create_dir("phy_down_cnt",
@@ -4730,6 +4834,17 @@ static void debugfs_bist_init_v3_hw(struct hisi_hba *hisi_hba)
hisi_hba->debugfs_bist_linkrate = SAS_LINK_RATE_1_5_GBPS;
}
+static void debugfs_dump_init_v3_hw(struct hisi_hba *hisi_hba)
+{
+ int i;
+
+ hisi_hba->debugfs_dump_dentry =
+ debugfs_create_dir("dump", hisi_hba->debugfs_dir);
+
+ for (i = 0; i < hisi_sas_debugfs_dump_count; i++)
+ debugfs_create_files_v3_hw(hisi_hba, i);
+}
+
static void debugfs_exit_v3_hw(struct hisi_hba *hisi_hba)
{
debugfs_remove_recursive(hisi_hba->debugfs_dir);
@@ -4739,7 +4854,6 @@ static void debugfs_exit_v3_hw(struct hisi_hba *hisi_hba)
static void debugfs_init_v3_hw(struct hisi_hba *hisi_hba)
{
struct device *dev = hisi_hba->dev;
- int i;
hisi_hba->debugfs_dir = debugfs_create_dir(dev_name(dev),
hisi_sas_debugfs_dir);
@@ -4751,19 +4865,10 @@ static void debugfs_init_v3_hw(struct hisi_hba *hisi_hba)
/* create bist structures */
debugfs_bist_init_v3_hw(hisi_hba);
- hisi_hba->debugfs_dump_dentry =
- debugfs_create_dir("dump", hisi_hba->debugfs_dir);
+ debugfs_dump_init_v3_hw(hisi_hba);
debugfs_phy_down_cnt_init_v3_hw(hisi_hba);
debugfs_fifo_init_v3_hw(hisi_hba);
-
- for (i = 0; i < hisi_sas_debugfs_dump_count; i++) {
- if (debugfs_alloc_v3_hw(hisi_hba, i)) {
- debugfs_exit_v3_hw(hisi_hba);
- dev_dbg(dev, "failed to init debugfs!\n");
- break;
- }
- }
}
static int
@@ -4828,7 +4933,7 @@ hisi_sas_v3_probe(struct pci_dev *pdev, const struct pci_device_id *id)
shost->transportt = hisi_sas_stt;
shost->max_id = HISI_SAS_MAX_DEVICES;
shost->max_lun = ~0;
- shost->max_channel = 1;
+ shost->max_channel = 0;
shost->max_cmd_len = 16;
shost->can_queue = HISI_SAS_UNRESERVED_IPTT;
shost->cmd_per_lun = HISI_SAS_UNRESERVED_IPTT;
diff --git a/drivers/scsi/hosts.c b/drivers/scsi/hosts.c
index 28cf473fcf96..a280a3fd9bea 100644
--- a/drivers/scsi/hosts.c
+++ b/drivers/scsi/hosts.c
@@ -344,6 +344,7 @@ static void scsi_host_dev_release(struct device *dev)
/* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */
rcu_barrier();
+ cancel_work_sync(&shost->eh_work);
if (shost->tmf_work_q)
destroy_workqueue(shost->tmf_work_q);
if (shost->ehandler)
@@ -409,6 +410,7 @@ struct Scsi_Host *scsi_host_alloc(struct scsi_host_template *sht, int privsize)
INIT_LIST_HEAD(&shost->starved_list);
init_waitqueue_head(&shost->host_wait);
mutex_init(&shost->scan_mutex);
+ INIT_WORK(&shost->eh_work, scsi_rcu_eh_wakeup);
index = ida_alloc(&host_index_ida, GFP_KERNEL);
if (index < 0) {
diff --git a/drivers/scsi/imm.c b/drivers/scsi/imm.c
index 7a499d621c25..9754c0e28c36 100644
--- a/drivers/scsi/imm.c
+++ b/drivers/scsi/imm.c
@@ -1269,6 +1269,7 @@ static void imm_detach(struct parport *pb)
if (dev->dev->port == pb) {
list_del_init(&dev->list);
scsi_remove_host(dev->host);
+ cancel_delayed_work_sync(&dev->imm_tq);
scsi_host_put(dev->host);
parport_unregister_device(dev->dev);
kfree(dev);
diff --git a/drivers/scsi/iscsi_tcp.c b/drivers/scsi/iscsi_tcp.c
index 56ade4630970..050a0ca92bc0 100644
--- a/drivers/scsi/iscsi_tcp.c
+++ b/drivers/scsi/iscsi_tcp.c
@@ -742,6 +742,7 @@ iscsi_sw_tcp_conn_bind(struct iscsi_cls_session *cls_session,
sk->sk_reuse = SK_CAN_REUSE;
sk->sk_sndtimeo = 15 * HZ; /* FIXME: make it configurable */
sk->sk_allocation = GFP_ATOMIC;
+ sk->sk_use_task_frag = false;
sk_set_memalloc(sk);
sock_no_linger(sk);
diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c
index b858efe3972e..99a0a77e2b2c 100644
--- a/drivers/scsi/libiscsi.c
+++ b/drivers/scsi/libiscsi.c
@@ -1142,13 +1142,15 @@ static int iscsi_handle_reject(struct iscsi_conn *conn, struct iscsi_hdr *hdr,
spin_unlock(&conn->session->frwd_lock);
spin_lock(&conn->session->back_lock);
} else {
- struct iscsi_task *task;
+ struct iscsi_task *task = NULL;
/*
* Our nop as ping got dropped. We know the target
* and transport are ok so just clean up
*/
- task = iscsi_itt_to_task(conn, rejected_pdu.itt);
- if (!task) {
+ if (!iscsi_verify_itt(conn, rejected_pdu.itt))
+ task = iscsi_itt_to_task(conn, rejected_pdu.itt);
+ if (!task || task->state == ISCSI_TASK_FREE ||
+ task->sc) {
iscsi_conn_printk(KERN_ERR, conn,
"Invalid pdu reject. Could "
"not lookup rejected task.\n");
@@ -1191,7 +1193,7 @@ struct iscsi_task *iscsi_itt_to_task(struct iscsi_conn *conn, itt_t itt)
session->tt->parse_pdu_itt(conn, itt, &i, NULL);
else
i = get_itt(itt);
- if (i >= session->cmds_max)
+ if (i < 0 || i >= session->cmds_max)
return NULL;
return session->cmds[i];
@@ -1284,7 +1286,7 @@ int __iscsi_complete_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr,
case ISCSI_OP_SCSI_TMFUNC_RSP:
case ISCSI_OP_NOOP_IN:
task = iscsi_itt_to_task(conn, hdr->itt);
- if (!task)
+ if (!task || task->state == ISCSI_TASK_FREE || task->sc)
return ISCSI_ERR_BAD_ITT;
break;
default:
@@ -1384,7 +1386,7 @@ int iscsi_verify_itt(struct iscsi_conn *conn, itt_t itt)
return ISCSI_ERR_BAD_ITT;
}
- if (i >= session->cmds_max) {
+ if (i < 0 || i >= session->cmds_max) {
iscsi_conn_printk(KERN_ERR, conn,
"received invalid itt index %u (max cmds "
"%u.\n", i, session->cmds_max);
diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
index 03d367c2f0a7..ade2685e7d38 100644
--- a/drivers/scsi/libsas/sas_expander.c
+++ b/drivers/scsi/libsas/sas_expander.c
@@ -40,7 +40,11 @@ static int smp_execute_task_sg(struct domain_device *dev,
to_sas_internal(dev->port->ha->shost->transportt);
struct sas_ha_struct *ha = dev->port->ha;
- pm_runtime_get_sync(ha->dev);
+ /*
+ * Non-blocking: a sync resume here would deadlock against
+ * sas_drain_work() during HA resume.
+ */
+ pm_runtime_get_noresume(ha->dev);
mutex_lock(&dev->ex_dev.cmd_mutex);
for (retry = 0; retry < 3; retry++) {
if (test_bit(SAS_DEV_GONE, &dev->state)) {
@@ -113,7 +117,7 @@ static int smp_execute_task_sg(struct domain_device *dev,
}
}
mutex_unlock(&dev->ex_dev.cmd_mutex);
- pm_runtime_put_sync(ha->dev);
+ pm_runtime_put(ha->dev);
BUG_ON(retry == 3 && task != NULL);
sas_free_task(task);
@@ -2159,6 +2163,11 @@ void sas_smp_handler(struct bsg_job *job, struct Scsi_Host *shost,
goto out;
}
+ /* The host may have autosuspended, resume it here. */
+ ret = pm_runtime_resume_and_get(dev->port->ha->dev);
+ if (ret)
+ goto out;
+
ret = smp_execute_task_sg(dev, job->request_payload.sg_list,
job->reply_payload.sg_list);
if (ret >= 0) {
@@ -2166,6 +2175,7 @@ void sas_smp_handler(struct bsg_job *job, struct Scsi_Host *shost,
rcvlen = job->reply_payload.payload_len - ret;
ret = 0;
}
+ pm_runtime_put(dev->port->ha->dev);
out:
bsg_job_done(job, ret, rcvlen);
diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c
index 13ebbe4a71c4..fc5a640b7f74 100644
--- a/drivers/scsi/lpfc/lpfc_init.c
+++ b/drivers/scsi/lpfc/lpfc_init.c
@@ -8175,7 +8175,6 @@ lpfc_sli4_driver_resource_setup(struct lpfc_hba *phba)
}
}
}
- mempool_free(mboxq, phba->mbox_mem_pool);
lpfc_nvme_mod_param_dep(phba);
@@ -8209,6 +8208,7 @@ lpfc_sli4_driver_resource_setup(struct lpfc_hba *phba)
goto out_free_bsmbx;
}
}
+ mempool_free(mboxq, phba->mbox_mem_pool);
/*
* 1 for cmd, 1 for rsp, NVME adds an extra one
diff --git a/drivers/scsi/mpi3mr/mpi3mr.h b/drivers/scsi/mpi3mr/mpi3mr.h
index ffeadd1e3543..422cb50b3577 100644
--- a/drivers/scsi/mpi3mr/mpi3mr.h
+++ b/drivers/scsi/mpi3mr/mpi3mr.h
@@ -130,8 +130,6 @@ extern atomic64_t event_counter;
#define MPI3MR_WATCHDOG_INTERVAL 1000 /* in milli seconds */
-#define MPI3MR_DEFAULT_CFG_PAGE_SZ 1024 /* in bytes */
-
#define MPI3MR_RESET_TOPOLOGY_SETTLE_TIME 10
#define MPI3MR_SCMD_TIMEOUT (60 * HZ)
@@ -971,6 +969,7 @@ struct scmd_priv {
* @ts_update_counter: Timestamp update counter
* @reset_in_progress: Reset in progress flag
* @unrecoverable: Controller unrecoverable flag
+ * @io_admin_reset_sync: Manage state of I/O ops during an admin reset process
* @prev_reset_result: Result of previous reset
* @reset_mutex: Controller reset mutex
* @reset_waitq: Controller reset wait queue
@@ -1013,9 +1012,6 @@ struct scmd_priv {
* @io_throttle_low: I/O size to stop throttle in 512b blocks
* @num_io_throttle_group: Maximum number of throttle groups
* @throttle_groups: Pointer to throttle group info structures
- * @cfg_page: Default memory for configuration pages
- * @cfg_page_dma: Configuration page DMA address
- * @cfg_page_sz: Default configuration page memory size
* @sas_transport_enabled: SAS transport enabled or not
* @scsi_device_channel: Channel ID for SCSI devices
* @transport_cmds: Command tracker for SAS transport commands
@@ -1146,6 +1142,7 @@ struct mpi3mr_ioc {
u32 ts_update_counter;
u8 reset_in_progress;
u8 unrecoverable;
+ u8 io_admin_reset_sync;
int prev_reset_result;
struct mutex reset_mutex;
wait_queue_head_t reset_waitq;
@@ -1198,10 +1195,6 @@ struct mpi3mr_ioc {
u16 num_io_throttle_group;
struct mpi3mr_throttle_group_info *throttle_groups;
- void *cfg_page;
- dma_addr_t cfg_page_dma;
- u16 cfg_page_sz;
-
u8 sas_transport_enabled;
u8 scsi_device_channel;
struct mpi3mr_drv_cmd transport_cmds;
diff --git a/drivers/scsi/mpi3mr/mpi3mr_app.c b/drivers/scsi/mpi3mr/mpi3mr_app.c
index c77803bd9b00..28f19209c58d 100644
--- a/drivers/scsi/mpi3mr/mpi3mr_app.c
+++ b/drivers/scsi/mpi3mr/mpi3mr_app.c
@@ -1531,6 +1531,7 @@ void mpi3mr_bsg_init(struct mpi3mr_ioc *mrioc)
{
struct device *bsg_dev = &mrioc->bsg_dev;
struct device *parent = &mrioc->shost->shost_gendev;
+ struct request_queue *q;
device_initialize(bsg_dev);
@@ -1546,9 +1547,9 @@ void mpi3mr_bsg_init(struct mpi3mr_ioc *mrioc)
return;
}
- mrioc->bsg_queue = bsg_setup_queue(bsg_dev, dev_name(bsg_dev),
+ q = bsg_setup_queue(bsg_dev, dev_name(bsg_dev),
mpi3mr_bsg_request, NULL, 0);
- if (IS_ERR(mrioc->bsg_queue)) {
+ if (IS_ERR(q)) {
ioc_err(mrioc, "%s: bsg registration failed\n",
dev_name(bsg_dev));
device_del(bsg_dev);
@@ -1556,8 +1557,10 @@ void mpi3mr_bsg_init(struct mpi3mr_ioc *mrioc)
return;
}
- blk_queue_max_segments(mrioc->bsg_queue, MPI3MR_MAX_APP_XFER_SEGMENTS);
- blk_queue_max_hw_sectors(mrioc->bsg_queue, MPI3MR_MAX_APP_XFER_SECTORS);
+ mrioc->bsg_queue = q;
+
+ blk_queue_max_segments(q, MPI3MR_MAX_APP_XFER_SEGMENTS);
+ blk_queue_max_hw_sectors(q, MPI3MR_MAX_APP_XFER_SECTORS);
return;
}
diff --git a/drivers/scsi/mpi3mr/mpi3mr_fw.c b/drivers/scsi/mpi3mr/mpi3mr_fw.c
index d4747ff4d800..8cdbb593c8e5 100644
--- a/drivers/scsi/mpi3mr/mpi3mr_fw.c
+++ b/drivers/scsi/mpi3mr/mpi3mr_fw.c
@@ -17,7 +17,7 @@ static void mpi3mr_process_factsdata(struct mpi3mr_ioc *mrioc,
struct mpi3_ioc_facts_data *facts_data);
static void mpi3mr_pel_wait_complete(struct mpi3mr_ioc *mrioc,
struct mpi3mr_drv_cmd *drv_cmd);
-
+static int mpi3mr_check_op_admin_proc(struct mpi3mr_ioc *mrioc);
static int poll_queues;
module_param(poll_queues, int, 0444);
MODULE_PARM_DESC(poll_queues, "Number of queues for io_uring poll mode. (Range 1 - 126)");
@@ -449,7 +449,7 @@ int mpi3mr_process_admin_reply_q(struct mpi3mr_ioc *mrioc)
}
do {
- if (mrioc->unrecoverable)
+ if (mrioc->unrecoverable || mrioc->io_admin_reset_sync)
break;
mrioc->admin_req_ci = le16_to_cpu(reply_desc->request_queue_ci);
@@ -538,7 +538,7 @@ int mpi3mr_process_op_reply_q(struct mpi3mr_ioc *mrioc,
}
do {
- if (mrioc->unrecoverable)
+ if (mrioc->unrecoverable || mrioc->io_admin_reset_sync)
break;
req_q_idx = le16_to_cpu(reply_desc->request_queue_id) - 1;
@@ -3864,17 +3864,6 @@ int mpi3mr_init_ioc(struct mpi3mr_ioc *mrioc)
mpi3mr_print_ioc_info(mrioc);
- if (!mrioc->cfg_page) {
- dprint_init(mrioc, "allocating config page buffers\n");
- mrioc->cfg_page_sz = MPI3MR_DEFAULT_CFG_PAGE_SZ;
- mrioc->cfg_page = dma_alloc_coherent(&mrioc->pdev->dev,
- mrioc->cfg_page_sz, &mrioc->cfg_page_dma, GFP_KERNEL);
- if (!mrioc->cfg_page) {
- retval = -1;
- goto out_failed_noretry;
- }
- }
-
if (!mrioc->init_cmds.reply) {
retval = mpi3mr_alloc_reply_sense_bufs(mrioc);
if (retval) {
@@ -4001,6 +3990,7 @@ int mpi3mr_reinit_ioc(struct mpi3mr_ioc *mrioc, u8 is_resume)
goto out_failed_noretry;
}
+ mrioc->io_admin_reset_sync = 0;
if (is_resume) {
dprint_reset(mrioc, "setting up single ISR\n");
retval = mpi3mr_setup_isr(mrioc, 1);
@@ -4418,11 +4408,7 @@ void mpi3mr_free_mem(struct mpi3mr_ioc *mrioc)
mrioc->admin_req_base, mrioc->admin_req_dma);
mrioc->admin_req_base = NULL;
}
- if (mrioc->cfg_page) {
- dma_free_coherent(&mrioc->pdev->dev, mrioc->cfg_page_sz,
- mrioc->cfg_page, mrioc->cfg_page_dma);
- mrioc->cfg_page = NULL;
- }
+
if (mrioc->pel_seqnum_virt) {
dma_free_coherent(&mrioc->pdev->dev, mrioc->pel_seqnum_sz,
mrioc->pel_seqnum_virt, mrioc->pel_seqnum_dma);
@@ -4839,6 +4825,55 @@ void mpi3mr_pel_get_seqnum_complete(struct mpi3mr_ioc *mrioc,
drv_cmd->retry_count = 0;
}
+/**
+ * mpi3mr_check_op_admin_proc -
+ * @mrioc: Adapter instance reference
+ *
+ * Check if any of the operation reply queues
+ * or the admin reply queue are currently in use.
+ * If any queue is in use, this function waits for
+ * a maximum of 10 seconds for them to become available.
+ *
+ * Return: 0 on success, non-zero on failure.
+ */
+static int mpi3mr_check_op_admin_proc(struct mpi3mr_ioc *mrioc)
+{
+
+ u16 timeout = 10 * 10;
+ u16 elapsed_time = 0;
+ bool op_admin_in_use = false;
+
+ do {
+ op_admin_in_use = false;
+
+ /* Check admin_reply queue first to exit early */
+ if (atomic_read(&mrioc->admin_reply_q_in_use) == 1)
+ op_admin_in_use = true;
+ else {
+ /* Check op_reply queues */
+ int i;
+
+ for (i = 0; i < mrioc->num_queues; i++) {
+ if (atomic_read(&mrioc->op_reply_qinfo[i].in_use) == 1) {
+ op_admin_in_use = true;
+ break;
+ }
+ }
+ }
+
+ if (!op_admin_in_use)
+ break;
+
+ msleep(100);
+
+ } while (++elapsed_time < timeout);
+
+ if (op_admin_in_use)
+ return 1;
+
+ return 0;
+}
+
/**
* mpi3mr_soft_reset_handler - Reset the controller
* @mrioc: Adapter instance reference
@@ -4912,6 +4947,7 @@ int mpi3mr_soft_reset_handler(struct mpi3mr_ioc *mrioc,
mpi3mr_wait_for_host_io(mrioc, MPI3MR_RESET_HOST_IOWAIT_TIMEOUT);
mpi3mr_ioc_disable_intr(mrioc);
+ mrioc->io_admin_reset_sync = 1;
if (snapdump) {
mpi3mr_set_diagsave(mrioc);
@@ -4935,6 +4971,16 @@ int mpi3mr_soft_reset_handler(struct mpi3mr_ioc *mrioc,
ioc_err(mrioc, "Failed to issue soft reset to the ioc\n");
goto out;
}
+
+ retval = mpi3mr_check_op_admin_proc(mrioc);
+ if (retval) {
+ ioc_err(mrioc, "Soft reset failed due to an Admin or I/O queue polling\n"
+ "thread still processing replies even after a 10 second\n"
+ "timeout. Marking the controller as unrecoverable!\n");
+
+ goto out;
+ }
+
if (mrioc->num_io_throttle_group !=
mrioc->facts.max_io_throttle_group) {
ioc_err(mrioc,
@@ -5007,55 +5053,6 @@ int mpi3mr_soft_reset_handler(struct mpi3mr_ioc *mrioc,
return retval;
}
-
-/**
- * mpi3mr_free_config_dma_memory - free memory for config page
- * @mrioc: Adapter instance reference
- * @mem_desc: memory descriptor structure
- *
- * Check whether the size of the buffer specified by the memory
- * descriptor is greater than the default page size if so then
- * free the memory pointed by the descriptor.
- *
- * Return: Nothing.
- */
-static void mpi3mr_free_config_dma_memory(struct mpi3mr_ioc *mrioc,
- struct dma_memory_desc *mem_desc)
-{
- if ((mem_desc->size > mrioc->cfg_page_sz) && mem_desc->addr) {
- dma_free_coherent(&mrioc->pdev->dev, mem_desc->size,
- mem_desc->addr, mem_desc->dma_addr);
- mem_desc->addr = NULL;
- }
-}
-
-/**
- * mpi3mr_alloc_config_dma_memory - Alloc memory for config page
- * @mrioc: Adapter instance reference
- * @mem_desc: Memory descriptor to hold dma memory info
- *
- * This function allocates new dmaable memory or provides the
- * default config page dmaable memory based on the memory size
- * described by the descriptor.
- *
- * Return: 0 on success, non-zero on failure.
- */
-static int mpi3mr_alloc_config_dma_memory(struct mpi3mr_ioc *mrioc,
- struct dma_memory_desc *mem_desc)
-{
- if (mem_desc->size > mrioc->cfg_page_sz) {
- mem_desc->addr = dma_alloc_coherent(&mrioc->pdev->dev,
- mem_desc->size, &mem_desc->dma_addr, GFP_KERNEL);
- if (!mem_desc->addr)
- return -ENOMEM;
- } else {
- mem_desc->addr = mrioc->cfg_page;
- mem_desc->dma_addr = mrioc->cfg_page_dma;
- memset(mem_desc->addr, 0, mrioc->cfg_page_sz);
- }
- return 0;
-}
-
/**
* mpi3mr_post_cfg_req - Issue config requests and wait
* @mrioc: Adapter instance reference
@@ -5211,8 +5208,12 @@ static int mpi3mr_process_cfg_req(struct mpi3mr_ioc *mrioc,
cfg_req->page_length = cfg_hdr->page_length;
cfg_req->page_version = cfg_hdr->page_version;
}
- if (mpi3mr_alloc_config_dma_memory(mrioc, &mem_desc))
- goto out;
+
+ mem_desc.addr = dma_alloc_coherent(&mrioc->pdev->dev,
+ mem_desc.size, &mem_desc.dma_addr, GFP_KERNEL);
+
+ if (!mem_desc.addr)
+ return retval;
mpi3mr_add_sg_single(&cfg_req->sgl, sgl_flags, mem_desc.size,
mem_desc.dma_addr);
@@ -5241,7 +5242,12 @@ static int mpi3mr_process_cfg_req(struct mpi3mr_ioc *mrioc,
}
out:
- mpi3mr_free_config_dma_memory(mrioc, &mem_desc);
+ if (mem_desc.addr) {
+ dma_free_coherent(&mrioc->pdev->dev, mem_desc.size,
+ mem_desc.addr, mem_desc.dma_addr);
+ mem_desc.addr = NULL;
+ }
+
return retval;
}
diff --git a/drivers/scsi/qedi/qedi_main.c b/drivers/scsi/qedi/qedi_main.c
index 825606c845e1..e180fa177f6d 100644
--- a/drivers/scsi/qedi/qedi_main.c
+++ b/drivers/scsi/qedi/qedi_main.c
@@ -2664,8 +2664,6 @@ static int __qedi_probe(struct pci_dev *pdev, int mode)
snprintf(host_buf, sizeof(host_buf), "host_%d", qedi->shost->host_no);
qedi_ops->common->set_name(qedi->cdev, host_buf);
- qedi_ops->register_ops(qedi->cdev, &qedi_cb_ops, qedi);
-
memset(¶ms, 0, sizeof(params));
params.mtu = DEF_PATH_MTU + IPV6_HDR_LEN + TCP_HDR_LEN;
qedi->ll2_mtu = DEF_PATH_MTU;
@@ -2682,6 +2680,7 @@ static int __qedi_probe(struct pci_dev *pdev, int mode)
spin_lock_init(&qedi->task_idx_lock);
mutex_init(&qedi->stats_lock);
}
+ qedi_ops->register_ops(qedi->cdev, &qedi_cb_ops, qedi);
qedi_ops->ll2->register_cb_ops(qedi->cdev, &qedi_ll2_cb_ops, qedi);
qedi_ops->ll2->start(qedi->cdev, ¶ms);
diff --git a/drivers/scsi/qla2xxx/qla_bsg.c b/drivers/scsi/qla2xxx/qla_bsg.c
index 3982f7d576a8..0fbf6dba5a36 100644
--- a/drivers/scsi/qla2xxx/qla_bsg.c
+++ b/drivers/scsi/qla2xxx/qla_bsg.c
@@ -158,6 +158,12 @@ qla24xx_proc_fcp_prio_cfg_cmd(struct bsg_job *bsg_job)
goto exit_fcp_prio_cfg;
}
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t)) {
+ ret = -EINVAL;
+ goto exit_fcp_prio_cfg;
+ }
+
/* Get the sub command */
oper = bsg_request->rqst_data.h_vendor.vendor_cmd[1];
@@ -756,6 +762,10 @@ qla2x00_process_loopback(struct bsg_job *bsg_job)
return -EIO;
}
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 3 * sizeof(uint32_t))
+ return -EINVAL;
+
memset(&elreq, 0, sizeof(elreq));
elreq.req_sg_cnt = dma_map_sg(&ha->pdev->dev,
@@ -988,6 +998,10 @@ qla84xx_reset(struct bsg_job *bsg_job)
return -EINVAL;
}
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t))
+ return -EINVAL;
+
flag = bsg_request->rqst_data.h_vendor.vendor_cmd[1];
rval = qla84xx_reset_chip(vha, flag == A84_ISSUE_RESET_DIAG_FW);
@@ -1032,6 +1046,10 @@ qla84xx_updatefw(struct bsg_job *bsg_job)
return -EINVAL;
}
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t))
+ return -EINVAL;
+
sg_cnt = dma_map_sg(&ha->pdev->dev, bsg_job->request_payload.sg_list,
bsg_job->request_payload.sg_cnt, DMA_TO_DEVICE);
if (!sg_cnt) {
@@ -1482,6 +1500,10 @@ qla2x00_read_optrom(struct bsg_job *bsg_job)
struct qla_hw_data *ha = vha->hw;
int rval = 0;
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t))
+ return -EINVAL;
+
if (ha->flags.nic_core_reset_hdlr_active)
return -EBUSY;
@@ -1519,6 +1541,10 @@ qla2x00_update_optrom(struct bsg_job *bsg_job)
struct qla_hw_data *ha = vha->hw;
int rval = 0;
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t))
+ return -EINVAL;
+
mutex_lock(&ha->optrom_mutex);
rval = qla2x00_optrom_setup(bsg_job, vha, 1);
if (rval) {
@@ -2010,6 +2036,11 @@ qlafx00_mgmt_cmd(struct bsg_job *bsg_job)
struct fc_port *fcport;
char *type = "FC_BSG_HST_FX_MGMT";
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + sizeof(uint32_t) +
+ sizeof(struct qla_mt_iocb_rqst_fx00))
+ return -EINVAL;
+
/* Copy the IOCB specific information */
piocb_rqst = (struct qla_mt_iocb_rqst_fx00 *)
&bsg_request->rqst_data.h_vendor.vendor_cmd[1];
@@ -2412,7 +2443,7 @@ qla2x00_do_dport_diagnostics(struct bsg_job *bsg_job)
!IS_QLA28XX(vha->hw))
return -EPERM;
- dd = kmalloc(sizeof(*dd), GFP_KERNEL);
+ dd = kzalloc(sizeof(*dd), GFP_KERNEL);
if (!dd) {
ql_log(ql_log_warn, vha, 0x70db,
"Failed to allocate memory for dport.\n");
@@ -2899,6 +2930,13 @@ qla2x00_process_vendor_specific(struct scsi_qla_host *vha, struct bsg_job *bsg_j
{
struct fc_bsg_request *bsg_request = bsg_job->request;
+ if (bsg_job->request_len <
+ sizeof(struct fc_bsg_request) + sizeof(uint32_t)) {
+ ql_log(ql_log_warn, vha, 0x7000,
+ "BSG request too small for vendor cmd.\n");
+ return -EINVAL;
+ }
+
ql_dbg(ql_dbg_edif, vha, 0x911b, "%s FC_BSG_HST_VENDOR cmd[0]=0x%x\n",
__func__, bsg_request->rqst_data.h_vendor.vendor_cmd[0]);
@@ -3024,8 +3062,11 @@ qla24xx_bsg_request(struct bsg_job *bsg_job)
}
/* Disable port will bring down the chip, allow enable command */
- if (bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_MANAGE_HOST_PORT ||
- bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_GET_HOST_STATS)
+ if (bsg_request->msgcode == FC_BSG_HST_VENDOR &&
+ bsg_job->request_len >=
+ sizeof(struct fc_bsg_request) + sizeof(uint32_t) &&
+ (bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_MANAGE_HOST_PORT ||
+ bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_GET_HOST_STATS))
goto skip_chip_chk;
if (vha->hw->flags.port_isolated) {
diff --git a/drivers/scsi/qla2xxx/qla_init.c b/drivers/scsi/qla2xxx/qla_init.c
index ace35581fd94..16470cfa333b 100644
--- a/drivers/scsi/qla2xxx/qla_init.c
+++ b/drivers/scsi/qla2xxx/qla_init.c
@@ -9908,9 +9908,26 @@ int qla2xxx_delete_qpair(struct scsi_qla_host *vha, struct qla_qpair *qpair)
{
int ret = QLA_FUNCTION_FAILED;
struct qla_hw_data *ha = qpair->hw;
+ struct rsp_que *rsp = qpair->rsp;
qpair->delete_in_progress = 1;
+ /*
+ * The response-queue interrupt schedules qla_do_work(), which
+ * dereferences qpair->rsp->req. Release the interrupt and flush
+ * any pending work before the request queue is freed below so a
+ * late completion cannot touch the freed request queue. The
+ * firmware queue-delete order (request then response) is kept.
+ */
+ if (rsp && rsp->msix && rsp->msix->have_irq) {
+ free_irq(rsp->msix->vector, rsp->msix->handle);
+ rsp->msix->have_irq = 0;
+ rsp->msix->in_use = 0;
+ rsp->msix->handle = NULL;
+ }
+ if (rsp && ha->wq)
+ cancel_work_sync(&qpair->q_work);
+
ret = qla25xx_delete_req_que(vha, qpair->req);
if (ret != QLA_SUCCESS)
goto fail;
diff --git a/drivers/scsi/qla2xxx/qla_iocb.c b/drivers/scsi/qla2xxx/qla_iocb.c
index 7bccd525ee19..4dec45f19269 100644
--- a/drivers/scsi/qla2xxx/qla_iocb.c
+++ b/drivers/scsi/qla2xxx/qla_iocb.c
@@ -2347,11 +2347,13 @@ __qla2x00_alloc_iocbs(struct qla_qpair *qpair, srb_t *sp)
/* Prep packet */
req->cnt -= req_cnt;
pkt = req->ring_ptr;
- memset(pkt, 0, REQUEST_ENTRY_SIZE);
if (IS_QLAFX00(ha)) {
+ memset_io((void __iomem __force *)pkt, 0,
+ REQUEST_ENTRY_SIZE);
wrt_reg_byte((u8 __force __iomem *)&pkt->entry_count, req_cnt);
wrt_reg_dword((__le32 __force __iomem *)&pkt->handle, handle);
} else {
+ memset(pkt, 0, REQUEST_ENTRY_SIZE);
pkt->entry_count = req_cnt;
pkt->handle = handle;
}
@@ -2752,7 +2754,6 @@ qla24xx_els_dcmd_iocb(scsi_qla_host_t *vha, int els_opcode,
if (!elsio->u.els_logo.els_logo_pyld) {
/* ref: INIT */
kref_put(&sp->cmd_kref, qla2x00_sp_release);
- qla2x00_free_fcport(fcport);
return QLA_FUNCTION_FAILED;
}
@@ -2777,7 +2778,6 @@ qla24xx_els_dcmd_iocb(scsi_qla_host_t *vha, int els_opcode,
if (rval != QLA_SUCCESS) {
/* ref: INIT */
kref_put(&sp->cmd_kref, qla2x00_sp_release);
- qla2x00_free_fcport(fcport);
return QLA_FUNCTION_FAILED;
}
@@ -3826,6 +3826,12 @@ qla25xx_ctrlvp_iocb(srb_t *sp, struct vp_ctrl_entry_24xx *vce)
*/
map = (sp->u.iocb_cmd.u.ctrlvp.vp_index - 1) / 8;
pos = (sp->u.iocb_cmd.u.ctrlvp.vp_index - 1) & 7;
+ if (map >= ARRAY_SIZE(vce->vp_idx_map)) {
+ ql_log(ql_log_warn, sp->vha, 0x307c,
+ "ctrlvp: vp_index %u exceeds vp_idx_map capacity\n",
+ sp->u.iocb_cmd.u.ctrlvp.vp_index);
+ return;
+ }
vce->vp_idx_map[map] |= 1 << pos;
}
diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c
index 72ecd39a7583..2b3885bd47ff 100644
--- a/drivers/scsi/qla2xxx/qla_isr.c
+++ b/drivers/scsi/qla2xxx/qla_isr.c
@@ -1053,14 +1053,6 @@ qla27xx_copy_fpin_pkt(struct scsi_qla_host *vha, void **pkt,
do {
while ((total_bytes > 0) && (entry_count_remaining > 0)) {
- if (rsp_q->ring_ptr->signature == RESPONSE_PROCESSED) {
- ql_dbg(ql_dbg_async, vha, 0x5084,
- "Ran out of IOCBs, partial data 0x%x\n",
- buffer_copy_offset);
- cpu_relax();
- continue;
- }
-
new_pkt = (sts_cont_entry_t *)rsp_q->ring_ptr;
*pkt = new_pkt;
@@ -3993,9 +3985,24 @@ void qla24xx_process_response_queue(struct scsi_qla_host *vha,
"SCM not active for this port\n");
break;
}
+ if (qla_chk_cont_iocb_avail(vha, rsp,
+ (response_t *)pkt, rsp_in)) {
+ /*
+ * ring_ptr and ring_index were
+ * pre-incremented above. Reset them
+ * back to current. Wait for next
+ * interrupt with all IOCBs to arrive
+ * and re-process.
+ */
+ rsp->ring_ptr = (response_t *)pkt;
+ rsp->ring_index = cur_ring_index;
+
+ ql_dbg(ql_dbg_init, vha, 0x5095,
+ "Defer processing FPIN...\n");
+ return;
+ }
pure_item = qla27xx_copy_fpin_pkt(vha,
(void **)&pkt, &rsp);
- __update_rsp_in(is_shadow_hba, rsp, rsp_in);
if (!pure_item)
break;
qla24xx_queue_purex_item(vha, pure_item,
diff --git a/drivers/scsi/qla2xxx/qla_mbx.c b/drivers/scsi/qla2xxx/qla_mbx.c
index ee22639fb54b..828c424abb3d 100644
--- a/drivers/scsi/qla2xxx/qla_mbx.c
+++ b/drivers/scsi/qla2xxx/qla_mbx.c
@@ -4147,6 +4147,8 @@ qla24xx_report_id_acquisition(scsi_qla_host_t *vha,
spin_lock_irqsave(&ha->vport_slock, flags);
list_for_each_entry(vp, &ha->vp_list, list) {
if (rptid_entry->vp_idx == vp->vp_idx) {
+ if (test_bit(VPORT_DELETE, &vp->dpc_flags))
+ break;
found = 1;
atomic_inc(&vp->vref_count);
break;
@@ -5656,7 +5658,7 @@ qla2x00_get_data_rate(scsi_qla_host_t *vha)
ql_dbg(ql_dbg_mbx, vha, 0x1107,
"Failed=%x mb[0]=%x.\n", rval, mcp->mb[0]);
} else {
- if (mcp->mb[1] != 0x7)
+ if (mcp->mb[1] != 0x7 || IS_QLA28XX(ha))
ha->link_data_rate = mcp->mb[1];
if (IS_QLA83XX(ha) || IS_QLA27XX(ha) || IS_QLA28XX(ha)) {
@@ -5667,8 +5669,6 @@ qla2x00_get_data_rate(scsi_qla_host_t *vha)
ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x1108,
"Done %s.\n", __func__);
- if (mcp->mb[1] != 0x7)
- ha->link_data_rate = mcp->mb[1];
}
return rval;
diff --git a/drivers/scsi/qla2xxx/qla_mid.c b/drivers/scsi/qla2xxx/qla_mid.c
index 79e4319bbbd2..d8e0f969012d 100644
--- a/drivers/scsi/qla2xxx/qla_mid.c
+++ b/drivers/scsi/qla2xxx/qla_mid.c
@@ -583,16 +583,19 @@ qla25xx_free_req_que(struct scsi_qla_host *vha, struct req_que *req)
{
struct qla_hw_data *ha = vha->hw;
uint16_t que_id = req->id;
+ size_t req_entry_size = sizeof(request_t);
- dma_free_coherent(&ha->pdev->dev, (req->length + 1) *
- sizeof(request_t), req->ring, req->dma);
+ if (req->ring)
+ dma_free_coherent(&ha->pdev->dev,
+ (req->length + 1) * req_entry_size,
+ req->ring, req->dma);
req->ring = NULL;
req->dma = 0;
if (que_id) {
+ mutex_lock(&ha->mq_lock);
ha->req_q_map[que_id] = NULL;
- mutex_lock(&ha->vport_lock);
clear_bit(que_id, ha->req_qid_map);
- mutex_unlock(&ha->vport_lock);
+ mutex_unlock(&ha->mq_lock);
}
kfree(req->outstanding_cmds);
kfree(req);
@@ -603,6 +606,7 @@ qla25xx_free_rsp_que(struct scsi_qla_host *vha, struct rsp_que *rsp)
{
struct qla_hw_data *ha = vha->hw;
uint16_t que_id = rsp->id;
+ size_t rsp_entry_size = sizeof(response_t);
if (rsp->msix && rsp->msix->have_irq) {
free_irq(rsp->msix->vector, rsp->msix->handle);
@@ -610,15 +614,22 @@ qla25xx_free_rsp_que(struct scsi_qla_host *vha, struct rsp_que *rsp)
rsp->msix->in_use = 0;
rsp->msix->handle = NULL;
}
- dma_free_coherent(&ha->pdev->dev, (rsp->length + 1) *
- sizeof(response_t), rsp->ring, rsp->dma);
+
+ /* Flush any queued response work before freeing the queue/qpair. */
+ if (rsp->qpair && ha->wq)
+ cancel_work_sync(&rsp->qpair->q_work);
+
+ if (rsp->ring)
+ dma_free_coherent(&ha->pdev->dev,
+ (rsp->length + 1) * rsp_entry_size,
+ rsp->ring, rsp->dma);
rsp->ring = NULL;
rsp->dma = 0;
if (que_id) {
+ mutex_lock(&ha->mq_lock);
ha->rsp_q_map[que_id] = NULL;
- mutex_lock(&ha->vport_lock);
clear_bit(que_id, ha->rsp_qid_map);
- mutex_unlock(&ha->vport_lock);
+ mutex_unlock(&ha->mq_lock);
}
kfree(rsp);
}
@@ -803,9 +814,6 @@ qla25xx_create_req_que(struct qla_hw_data *ha, uint16_t options,
if (ret != QLA_SUCCESS) {
ql_log(ql_log_fatal, base_vha, 0x00df,
"%s failed.\n", __func__);
- mutex_lock(&ha->mq_lock);
- clear_bit(que_id, ha->req_qid_map);
- mutex_unlock(&ha->mq_lock);
goto que_failed;
}
vha->flags.qpairs_req_created = 1;
@@ -917,9 +925,6 @@ qla25xx_create_rsp_que(struct qla_hw_data *ha, uint16_t options,
if (ret != QLA_SUCCESS) {
ql_log(ql_log_fatal, base_vha, 0x00e7,
"%s failed.\n", __func__);
- mutex_lock(&ha->mq_lock);
- clear_bit(que_id, ha->rsp_qid_map);
- mutex_unlock(&ha->mq_lock);
goto que_failed;
}
vha->flags.qpairs_rsp_created = 1;
@@ -966,6 +971,14 @@ int qla24xx_control_vp(scsi_qla_host_t *vha, int cmd)
if (vp_index == 0 || vp_index >= ha->max_npiv_vports)
return QLA_PARAMETER_ERROR;
+ /*
+ * The VP_CTRL IOCB selects the target VP through a fixed 128-bit
+ * (16-byte) vp_idx_map bitmap, so vp_index must fit within it even
+ * if firmware advertises more NPIV vports.
+ */
+ if (vp_index > sizeof_field(struct vp_ctrl_entry_24xx, vp_idx_map) * 8)
+ return QLA_PARAMETER_ERROR;
+
/* ref: INIT */
sp = qla2x00_get_sp(base_vha, NULL, GFP_KERNEL);
if (!sp)
diff --git a/drivers/scsi/qla2xxx/qla_nvme.c b/drivers/scsi/qla2xxx/qla_nvme.c
index 622b11660b67..20a904142810 100644
--- a/drivers/scsi/qla2xxx/qla_nvme.c
+++ b/drivers/scsi/qla2xxx/qla_nvme.c
@@ -309,8 +309,8 @@ static void qla_nvme_ls_abort(struct nvme_fc_local_port *lport,
}
spin_unlock_irqrestore(&priv->cmd_lock, flags);
- INIT_WORK(&priv->abort_work, qla_nvme_abort_work);
- schedule_work(&priv->abort_work);
+ if (!schedule_work(&priv->abort_work))
+ kref_put(&priv->sp->cmd_kref, priv->sp->put_fn);
}
static int qla_nvme_ls_req(struct nvme_fc_local_port *lport,
@@ -347,6 +347,7 @@ static int qla_nvme_ls_req(struct nvme_fc_local_port *lport,
priv->sp = sp;
kref_init(&sp->cmd_kref);
spin_lock_init(&priv->cmd_lock);
+ INIT_WORK(&priv->abort_work, qla_nvme_abort_work);
nvme = &sp->u.iocb_cmd;
priv->fd = fd;
nvme->u.nvme.desc = fd;
@@ -391,8 +392,8 @@ static void qla_nvme_fcp_abort(struct nvme_fc_local_port *lport,
}
spin_unlock_irqrestore(&priv->cmd_lock, flags);
- INIT_WORK(&priv->abort_work, qla_nvme_abort_work);
- schedule_work(&priv->abort_work);
+ if (!schedule_work(&priv->abort_work))
+ kref_put(&priv->sp->cmd_kref, priv->sp->put_fn);
}
static inline int qla2x00_start_nvme_mq(srb_t *sp)
@@ -657,6 +658,7 @@ static int qla_nvme_post_cmd(struct nvme_fc_local_port *lport,
kref_init(&sp->cmd_kref);
spin_lock_init(&priv->cmd_lock);
+ INIT_WORK(&priv->abort_work, qla_nvme_abort_work);
sp->priv = priv;
priv->sp = sp;
sp->type = SRB_NVME_CMD;
diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c
index e1578808e27e..38e2b3123702 100644
--- a/drivers/scsi/qla2xxx/qla_os.c
+++ b/drivers/scsi/qla2xxx/qla_os.c
@@ -1883,10 +1883,26 @@ __qla2x00_abort_all_cmds(struct qla_qpair *qp, int res)
continue;
}
cmd = (struct qla_tgt_cmd *)sp;
- cmd->aborted = 1;
+
+ if (cmd->sg_mapped)
+ qlt_unmap_sg(vha, cmd);
+
+ if (cmd->state == QLA_TGT_STATE_NEED_DATA) {
+ cmd->aborted = 1;
+ cmd->write_data_transferred = 0;
+ cmd->state = QLA_TGT_STATE_DATA_IN;
+ ha->tgt.tgt_ops->handle_data(cmd);
+ } else {
+ ha->tgt.tgt_ops->free_cmd(cmd);
+ }
break;
case TYPE_TGT_TMCMD:
- /* Skip task management functions. */
+ /*
+ * Currently, only ABTS response gets on the
+ * outstanding_cmds[]
+ */
+ ha->tgt.tgt_ops->free_mcmd(
+ (struct qla_tgt_mgmt_cmd *) sp);
break;
default:
break;
@@ -4464,23 +4480,36 @@ qla2x00_mem_alloc(struct qla_hw_data *ha, uint16_t req_len, uint16_t rsp_len,
fail_elsrej:
dma_pool_destroy(ha->purex_dma_pool);
+ ha->purex_dma_pool = NULL;
fail_flt:
dma_free_coherent(&ha->pdev->dev, sizeof(struct qla_flt_header) + FLT_REGIONS_SIZE,
ha->flt, ha->flt_dma);
+ ha->flt = NULL;
+ ha->flt_dma = 0;
fail_flt_buffer:
dma_free_coherent(&ha->pdev->dev, SFP_DEV_SIZE,
ha->sfp_data, ha->sfp_data_dma);
+ ha->sfp_data = NULL;
+ ha->sfp_data_dma = 0;
fail_sfp_data:
kfree(ha->loop_id_map);
+ ha->loop_id_map = NULL;
fail_loop_id_map:
dma_pool_free(ha->s_dma_pool, ha->async_pd, ha->async_pd_dma);
+ ha->async_pd = NULL;
+ ha->async_pd_dma = 0;
fail_async_pd:
dma_pool_free(ha->s_dma_pool, ha->sf_init_cb, ha->sf_init_cb_dma);
+ ha->sf_init_cb = NULL;
+ ha->sf_init_cb_dma = 0;
fail_sf_init_cb:
dma_pool_free(ha->s_dma_pool, ha->ex_init_cb, ha->ex_init_cb_dma);
+ ha->ex_init_cb = NULL;
+ ha->ex_init_cb_dma = 0;
fail_ex_init_cb:
kfree(ha->npiv_info);
+ ha->npiv_info = NULL;
fail_npiv_info:
dma_free_coherent(&ha->pdev->dev, ((*rsp)->length + 1) *
sizeof(response_t), (*rsp)->ring, (*rsp)->dma);
diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c
index 043cfa10c716..7136ec5ded35 100644
--- a/drivers/scsi/qla2xxx/qla_target.c
+++ b/drivers/scsi/qla2xxx/qla_target.c
@@ -2504,7 +2504,7 @@ static int qlt_pci_map_calc_cnt(struct qla_tgt_prm *prm)
return -1;
}
-static void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd)
+void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd)
{
struct qla_hw_data *ha;
struct qla_qpair *qpair;
@@ -3834,9 +3834,6 @@ int qlt_abort_cmd(struct qla_tgt_cmd *cmd)
spin_lock_irqsave(&cmd->cmd_lock, flags);
if (cmd->aborted) {
- if (cmd->sg_mapped)
- qlt_unmap_sg(vha, cmd);
-
spin_unlock_irqrestore(&cmd->cmd_lock, flags);
/*
* It's normal to see 2 calls in this path:
diff --git a/drivers/scsi/qla2xxx/qla_target.h b/drivers/scsi/qla2xxx/qla_target.h
index 7df86578214f..e674a81eb527 100644
--- a/drivers/scsi/qla2xxx/qla_target.h
+++ b/drivers/scsi/qla2xxx/qla_target.h
@@ -1060,6 +1060,7 @@ extern int qlt_abort_cmd(struct qla_tgt_cmd *);
extern void qlt_xmit_tm_rsp(struct qla_tgt_mgmt_cmd *);
extern void qlt_free_mcmd(struct qla_tgt_mgmt_cmd *);
extern void qlt_free_cmd(struct qla_tgt_cmd *cmd);
+extern void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd);
extern void qlt_async_event(uint16_t, struct scsi_qla_host *, uint16_t *);
extern void qlt_enable_vha(struct scsi_qla_host *);
extern void qlt_vport_create(struct scsi_qla_host *, struct qla_hw_data *);
diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c
index f6d3f3564763..b458a8cb619f 100644
--- a/drivers/scsi/scsi_error.c
+++ b/drivers/scsi/scsi_error.c
@@ -73,6 +73,26 @@ void scsi_eh_wakeup(struct Scsi_Host *shost, unsigned int busy)
}
}
+void scsi_rcu_eh_wakeup(struct work_struct *work)
+{
+ struct Scsi_Host *shost = container_of(work, struct Scsi_Host, eh_work);
+ unsigned long flags;
+ unsigned int busy;
+
+ /*
+ * Ensure any running scsi_dec_host_busy has completed its rcu section
+ * so changes to host state and host_eh_scheduled are visible to all
+ * future calls of scsi_dec_host_busy
+ */
+ synchronize_rcu();
+
+ busy = scsi_host_busy(shost);
+
+ spin_lock_irqsave(shost->host_lock, flags);
+ scsi_eh_wakeup(shost, busy);
+ spin_unlock_irqrestore(shost->host_lock, flags);
+}
+
/**
* scsi_schedule_eh - schedule EH for SCSI host
* @shost: SCSI host to invoke error handling on.
@@ -88,7 +108,7 @@ void scsi_schedule_eh(struct Scsi_Host *shost)
if (scsi_host_set_state(shost, SHOST_RECOVERY) == 0 ||
scsi_host_set_state(shost, SHOST_CANCEL_RECOVERY) == 0) {
shost->host_eh_scheduled++;
- scsi_eh_wakeup(shost, scsi_host_busy(shost));
+ queue_work(shost->tmf_work_q, &shost->eh_work);
}
spin_unlock_irqrestore(shost->host_lock, flags);
@@ -2257,6 +2277,7 @@ static void scsi_unjam_host(struct Scsi_Host *shost)
int scsi_error_handler(void *data)
{
struct Scsi_Host *shost = data;
+ bool eh_noresume;
/*
* We use TASK_INTERRUPTIBLE so that the thread is not
@@ -2298,7 +2319,8 @@ int scsi_error_handler(void *data)
* what we need to do to get it up and online again (if we can).
* If we fail, we end up taking the thing offline.
*/
- if (!shost->eh_noresume && scsi_autopm_get_host(shost) != 0) {
+ eh_noresume = READ_ONCE(shost->eh_noresume);
+ if (!eh_noresume && scsi_autopm_get_host(shost) != 0) {
SCSI_LOG_ERROR_RECOVERY(1,
shost_printk(KERN_ERR, shost,
"scsi_eh_%d: unable to autoresume\n",
@@ -2322,7 +2344,7 @@ int scsi_error_handler(void *data)
* which are still online.
*/
scsi_restart_operations(shost);
- if (!shost->eh_noresume)
+ if (!eh_noresume)
scsi_autopm_put_host(shost);
}
__set_current_state(TASK_RUNNING);
diff --git a/drivers/scsi/scsi_priv.h b/drivers/scsi/scsi_priv.h
index 9b5fb30e684b..459dfbdf64fd 100644
--- a/drivers/scsi/scsi_priv.h
+++ b/drivers/scsi/scsi_priv.h
@@ -87,6 +87,7 @@ extern enum blk_eh_timer_return scsi_timeout(struct request *req);
extern int scsi_error_handler(void *host);
extern enum scsi_disposition scsi_decide_disposition(struct scsi_cmnd *cmd);
extern void scsi_eh_wakeup(struct Scsi_Host *shost, unsigned int busy);
+extern void scsi_rcu_eh_wakeup(struct work_struct *work);
extern void scsi_eh_scmd_add(struct scsi_cmnd *);
void scsi_eh_ready_devs(struct Scsi_Host *shost,
struct list_head *work_q,
diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c
index 5e6e5753387d..32f850c6be11 100644
--- a/drivers/scsi/sd.c
+++ b/drivers/scsi/sd.c
@@ -920,6 +920,7 @@ static blk_status_t sd_setup_unmap_cmnd(struct scsi_cmnd *cmd)
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
unsigned int data_len = 24;
char *buf;
+ blk_status_t ret;
rq->special_vec.bv_page = mempool_alloc(sd_page_pool, GFP_ATOMIC);
if (!rq->special_vec.bv_page)
@@ -943,7 +944,10 @@ static blk_status_t sd_setup_unmap_cmnd(struct scsi_cmnd *cmd)
cmd->transfersize = data_len;
rq->timeout = SD_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd,
@@ -955,6 +959,7 @@ static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd,
u64 lba = sectors_to_logical(sdp, blk_rq_pos(rq));
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
u32 data_len = sdp->sector_size;
+ blk_status_t ret;
rq->special_vec.bv_page = mempool_alloc(sd_page_pool, GFP_ATOMIC);
if (!rq->special_vec.bv_page)
@@ -975,7 +980,10 @@ static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd,
cmd->transfersize = data_len;
rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
@@ -987,6 +995,7 @@ static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
u64 lba = sectors_to_logical(sdp, blk_rq_pos(rq));
u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq));
u32 data_len = sdp->sector_size;
+ blk_status_t ret;
rq->special_vec.bv_page = mempool_alloc(sd_page_pool, GFP_ATOMIC);
if (!rq->special_vec.bv_page)
@@ -1007,7 +1016,10 @@ static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd,
cmd->transfersize = data_len;
rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT;
- return scsi_alloc_sgtables(cmd);
+ ret = scsi_alloc_sgtables(cmd);
+ if (ret != BLK_STS_OK)
+ sd_uninit_command(cmd);
+ return ret;
}
static blk_status_t sd_setup_write_zeroes_cmnd(struct scsi_cmnd *cmd)
@@ -1352,8 +1364,11 @@ static void sd_uninit_command(struct scsi_cmnd *SCpnt)
{
struct request *rq = scsi_cmd_to_rq(SCpnt);
- if (rq->rq_flags & RQF_SPECIAL_PAYLOAD)
- mempool_free(rq->special_vec.bv_page, sd_page_pool);
+ if (!(rq->rq_flags & RQF_SPECIAL_PAYLOAD))
+ return;
+
+ mempool_free(rq->special_vec.bv_page, sd_page_pool);
+ rq->rq_flags &= ~RQF_SPECIAL_PAYLOAD;
}
static bool sd_need_revalidate(struct block_device *bdev,
diff --git a/drivers/scsi/st.c b/drivers/scsi/st.c
index 9ba5ad106b65..2ad1703240e4 100644
--- a/drivers/scsi/st.c
+++ b/drivers/scsi/st.c
@@ -3187,14 +3187,20 @@ static int set_location(struct scsi_tape *STp, unsigned int block, int partition
partition >= ST_NBR_PARTITIONS)
return (-EINVAL);
if (partition != STp->partition) {
- if (get_location(STp, &blk, &p, 1))
+ /*
+ * Save the location for the partition the drive reports,
+ * not for STp->partition: after a reset STp->partition may
+ * not match the partition the drive is in.
+ */
+ if (get_location(STp, &blk, &p, 1) ||
+ p < 0 || p >= ST_NBR_PARTITIONS)
STps->last_block_valid = 0;
else {
- STps->last_block_valid = 1;
- STps->last_block_visited = blk;
+ STp->ps[p].last_block_valid = 1;
+ STp->ps[p].last_block_visited = blk;
DEBC_printk(STp, "Visited block %d for "
"partition %d saved.\n",
- blk, STp->partition);
+ blk, p);
}
}
@@ -4434,11 +4440,11 @@ static int st_remove(struct device *dev)
remove_cdevs(tpnt);
mutex_lock(&st_ref_mutex);
- kref_put(&tpnt->kref, scsi_tape_release);
- mutex_unlock(&st_ref_mutex);
spin_lock(&st_index_lock);
idr_remove(&st_index_idr, index);
spin_unlock(&st_index_lock);
+ kref_put(&tpnt->kref, scsi_tape_release);
+ mutex_unlock(&st_ref_mutex);
return 0;
}
diff --git a/drivers/scsi/xen-scsifront.c b/drivers/scsi/xen-scsifront.c
index 71a3bb83984c..7db0d9f48936 100644
--- a/drivers/scsi/xen-scsifront.c
+++ b/drivers/scsi/xen-scsifront.c
@@ -367,6 +367,7 @@ static void scsifront_do_response(struct vscsifrnt_info *info,
struct vscsifrnt_shadow *shadow;
if (ring_rsp->rqid >= VSCSIIF_MAX_REQS ||
+ !info->shadow[ring_rsp->rqid] ||
!info->shadow[ring_rsp->rqid]->inflight) {
scsifront_set_error(info, "illegal rqid returned by backend!");
return;
diff --git a/drivers/soc/aspeed/aspeed-lpc-snoop.c b/drivers/soc/aspeed/aspeed-lpc-snoop.c
index 19780e37d8ea..95978904c75d 100644
--- a/drivers/soc/aspeed/aspeed-lpc-snoop.c
+++ b/drivers/soc/aspeed/aspeed-lpc-snoop.c
@@ -11,6 +11,7 @@
*/
#include <linux/bitops.h>
+#include <linux/cleanup.h>
#include <linux/clk.h>
#include <linux/interrupt.h>
#include <linux/fs.h>
@@ -60,6 +61,7 @@ struct aspeed_lpc_snoop_model_data {
struct aspeed_lpc_snoop_channel {
bool enabled;
+ spinlock_t lock;
struct kfifo fifo;
wait_queue_head_t wq;
struct miscdevice miscdev;
@@ -83,6 +85,7 @@ static ssize_t snoop_file_read(struct file *file, char __user *buffer,
size_t count, loff_t *ppos)
{
struct aspeed_lpc_snoop_channel *chan = snoop_file_to_chan(file);
+ u8 *buf __free(kfree) = NULL;
unsigned int copied;
int ret = 0;
@@ -94,9 +97,16 @@ static ssize_t snoop_file_read(struct file *file, char __user *buffer,
if (ret == -ERESTARTSYS)
return -EINTR;
}
- ret = kfifo_to_user(&chan->fifo, buffer, count, &copied);
- if (ret)
- return ret;
+
+ count = min_t(size_t, count, SNOOP_FIFO_SIZE);
+
+ buf = kmalloc(count, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ copied = kfifo_out_spinlocked(&chan->fifo, buf, count, &chan->lock);
+ if (copied && copy_to_user(buffer, buf, copied))
+ return -EFAULT;
return copied;
}
@@ -120,11 +130,13 @@ static const struct file_operations snoop_fops = {
/* Save a byte to a FIFO and discard the oldest byte if FIFO is full */
static void put_fifo_with_discard(struct aspeed_lpc_snoop_channel *chan, u8 val)
{
- if (!kfifo_initialized(&chan->fifo))
- return;
- if (kfifo_is_full(&chan->fifo))
- kfifo_skip(&chan->fifo);
- kfifo_put(&chan->fifo, val);
+ scoped_guard(spinlock, &chan->lock) {
+ if (!kfifo_initialized(&chan->fifo))
+ return;
+ if (kfifo_is_full(&chan->fifo))
+ kfifo_skip(&chan->fifo);
+ kfifo_put(&chan->fifo, val);
+ }
wake_up_interruptible(&chan->wq);
}
@@ -285,7 +297,7 @@ static int aspeed_lpc_snoop_probe(struct platform_device *pdev)
struct device *dev;
struct device_node *np;
u32 port;
- int rc;
+ int rc, i;
dev = &pdev->dev;
@@ -293,6 +305,9 @@ static int aspeed_lpc_snoop_probe(struct platform_device *pdev)
if (!lpc_snoop)
return -ENOMEM;
+ for (i = 0; i < NUM_SNOOP_CHANNELS; i++)
+ spin_lock_init(&lpc_snoop->chan[i].lock);
+
np = pdev->dev.parent->of_node;
if (!of_device_is_compatible(np, "aspeed,ast2400-lpc-v2") &&
!of_device_is_compatible(np, "aspeed,ast2500-lpc-v2") &&
diff --git a/drivers/soc/imx/imx8m-blk-ctrl.c b/drivers/soc/imx/imx8m-blk-ctrl.c
index 1b87823d835d..8a80b2d1035b 100644
--- a/drivers/soc/imx/imx8m-blk-ctrl.c
+++ b/drivers/soc/imx/imx8m-blk-ctrl.c
@@ -13,6 +13,7 @@
#include <linux/pm_runtime.h>
#include <linux/regmap.h>
#include <linux/clk.h>
+#include <linux/mutex.h>
#include <dt-bindings/power/imx8mm-power.h>
#include <dt-bindings/power/imx8mn-power.h>
@@ -32,6 +33,12 @@ struct imx8m_blk_ctrl {
struct regmap *regmap;
struct imx8m_blk_ctrl_domain *domains;
struct genpd_onecell_data onecell_data;
+ /*
+ * Serializes the blk-ctrl reset/clock sequence across sibling domains;
+ * their transitions interact through the shared VPUMIX bus domain,
+ * VPU_NOC and the not-ack-verified ADB400 handshake (ERR050531).
+ */
+ struct mutex power_lock;
};
struct imx8m_blk_ctrl_domain_data {
@@ -52,6 +59,15 @@ struct imx8m_blk_ctrl_domain_data {
* register.
*/
u32 mipi_phy_rst_mask;
+
+ /*
+ * VC8000E reset de-assertion edge and AXI clock may have a timing issue.
+ * Workaround: Set bit2 (vc8000e_clk_en) of BLK_CLK_EN_CSR to 0 to gate off
+ * both AXI clock and VC8000E clock sent to VC8000E and AXI clock sent to
+ * VPU_NOC m_v_2 interface during VC8000E power up(VC8000E reset is
+ * de-asserted by HW)
+ */
+ bool is_errata_err050531;
};
#define DOMAIN_MAX_CLKS 4
@@ -87,6 +103,8 @@ static int imx8m_blk_ctrl_power_on(struct generic_pm_domain *genpd)
struct imx8m_blk_ctrl *bc = domain->bc;
int ret;
+ guard(mutex)(&bc->power_lock);
+
/* make sure bus domain is awake */
ret = pm_runtime_get_sync(bc->bus_power_dev);
if (ret < 0) {
@@ -106,7 +124,11 @@ static int imx8m_blk_ctrl_power_on(struct generic_pm_domain *genpd)
dev_err(bc->dev, "failed to enable clocks\n");
goto bus_put;
}
- regmap_set_bits(bc->regmap, BLK_CLK_EN, data->clk_mask);
+
+ if (data->is_errata_err050531)
+ regmap_clear_bits(bc->regmap, BLK_CLK_EN, data->clk_mask);
+ else
+ regmap_set_bits(bc->regmap, BLK_CLK_EN, data->clk_mask);
/* power up upstream GPC domain */
ret = pm_runtime_get_sync(domain->power_dev);
@@ -115,6 +137,9 @@ static int imx8m_blk_ctrl_power_on(struct generic_pm_domain *genpd)
goto clk_disable;
}
+ if (data->is_errata_err050531)
+ regmap_set_bits(bc->regmap, BLK_CLK_EN, data->clk_mask);
+
/* wait for reset to propagate */
udelay(5);
@@ -146,6 +171,8 @@ static int imx8m_blk_ctrl_power_off(struct generic_pm_domain *genpd)
const struct imx8m_blk_ctrl_domain_data *data = domain->data;
struct imx8m_blk_ctrl *bc = domain->bc;
+ guard(mutex)(&bc->power_lock);
+
/* put devices into reset and disable clocks */
if (data->mipi_phy_rst_mask)
regmap_clear_bits(bc->regmap, BLK_MIPI_RESET_DIV, data->mipi_phy_rst_mask);
@@ -184,6 +211,10 @@ static int imx8m_blk_ctrl_probe(struct platform_device *pdev)
bc->dev = dev;
+ ret = devm_mutex_init(dev, &bc->power_lock);
+ if (ret)
+ return ret;
+
bc_data = of_device_get_match_data(dev);
base = devm_platform_ioremap_resource(pdev, 0);
@@ -499,6 +530,7 @@ static const struct imx8m_blk_ctrl_domain_data imx8mp_vpu_blk_ctl_domain_data[]
.clk_mask = BIT(2),
.path_names = (const char *[]){"vc8000e"},
.num_paths = 1,
+ .is_errata_err050531 = true,
},
};
diff --git a/drivers/soc/imx/imx8mp-blk-ctrl.c b/drivers/soc/imx/imx8mp-blk-ctrl.c
index 575ed8e3a520..5357711b2584 100644
--- a/drivers/soc/imx/imx8mp-blk-ctrl.c
+++ b/drivers/soc/imx/imx8mp-blk-ctrl.c
@@ -45,7 +45,7 @@ struct imx8mp_blk_ctrl_domain_data {
const unsigned int flags;
};
-#define DOMAIN_MAX_CLKS 2
+#define DOMAIN_MAX_CLKS 3
#define DOMAIN_MAX_PATHS 3
struct imx8mp_blk_ctrl_domain {
@@ -250,9 +250,6 @@ static void imx8mp_hdmi_blk_ctrl_power_on(struct imx8mp_blk_ctrl *bc,
regmap_set_bits(bc->regmap, HDMI_RTX_RESET_CTL0, BIT(12));
regmap_clear_bits(bc->regmap, HDMI_TX_CONTROL0, BIT(3));
break;
- case IMX8MP_HDMIBLK_PD_HDCP:
- regmap_set_bits(bc->regmap, HDMI_RTX_CLK_CTL0, BIT(11));
- break;
case IMX8MP_HDMIBLK_PD_HRV:
regmap_set_bits(bc->regmap, HDMI_RTX_CLK_CTL1, BIT(3) | BIT(4) | BIT(5));
regmap_set_bits(bc->regmap, HDMI_RTX_RESET_CTL0, BIT(15));
@@ -306,9 +303,6 @@ static void imx8mp_hdmi_blk_ctrl_power_off(struct imx8mp_blk_ctrl *bc,
regmap_clear_bits(bc->regmap, HDMI_RTX_CLK_CTL0, BIT(7));
regmap_clear_bits(bc->regmap, HDMI_RTX_CLK_CTL1, BIT(22) | BIT(24));
break;
- case IMX8MP_HDMIBLK_PD_HDCP:
- regmap_clear_bits(bc->regmap, HDMI_RTX_CLK_CTL0, BIT(11));
- break;
case IMX8MP_HDMIBLK_PD_HRV:
regmap_clear_bits(bc->regmap, HDMI_RTX_RESET_CTL0, BIT(15));
regmap_clear_bits(bc->regmap, HDMI_RTX_CLK_CTL1, BIT(3) | BIT(4) | BIT(5));
@@ -337,7 +331,7 @@ static int imx8mp_hdmi_power_notifier(struct notifier_block *nb,
regmap_write(bc->regmap, HDMI_RTX_CLK_CTL0, 0x0);
regmap_write(bc->regmap, HDMI_RTX_CLK_CTL1, 0x0);
regmap_set_bits(bc->regmap, HDMI_RTX_CLK_CTL0,
- BIT(0) | BIT(1) | BIT(10));
+ BIT(0) | BIT(1) | BIT(10) | BIT(11));
regmap_set_bits(bc->regmap, HDMI_RTX_RESET_CTL0, BIT(0));
/*
@@ -359,8 +353,8 @@ static const struct imx8mp_blk_ctrl_domain_data imx8mp_hdmi_domain_data[] = {
},
[IMX8MP_HDMIBLK_PD_LCDIF] = {
.name = "hdmiblk-lcdif",
- .clk_names = (const char *[]){ "axi", "apb" },
- .num_clks = 2,
+ .clk_names = (const char *[]){ "axi", "apb", "fdcc" },
+ .num_clks = 3,
.gpc_name = "lcdif",
.path_names = (const char *[]){"lcdif-hdmi"},
.num_paths = 1,
@@ -385,8 +379,8 @@ static const struct imx8mp_blk_ctrl_domain_data imx8mp_hdmi_domain_data[] = {
},
[IMX8MP_HDMIBLK_PD_HDMI_TX] = {
.name = "hdmiblk-hdmi-tx",
- .clk_names = (const char *[]){ "apb", "ref_266m" },
- .num_clks = 2,
+ .clk_names = (const char *[]){ "apb", "ref_266m", "fdcc" },
+ .num_clks = 3,
.gpc_name = "hdmi-tx",
},
[IMX8MP_HDMIBLK_PD_HDMI_TX_PHY] = {
diff --git a/drivers/soc/imx/imx93-blk-ctrl.c b/drivers/soc/imx/imx93-blk-ctrl.c
index 2c600329436c..ef15562ded14 100644
--- a/drivers/soc/imx/imx93-blk-ctrl.c
+++ b/drivers/soc/imx/imx93-blk-ctrl.c
@@ -47,6 +47,8 @@
#define PRIO(X) (X)
+#define BLK_CTRL_NO_PARENT UINT_MAX
+
struct imx93_blk_ctrl_domain;
struct imx93_blk_ctrl {
@@ -73,6 +75,7 @@ struct imx93_blk_ctrl_domain_data {
int num_clks;
u32 rst_mask;
u32 clk_mask;
+ u32 parent;
int num_qos;
struct imx93_blk_ctrl_qos qos[DOMAIN_MAX_QOS];
};
@@ -272,18 +275,46 @@ static int imx93_blk_ctrl_probe(struct platform_device *pdev)
bc->onecell_data.domains[i] = &domain->genpd;
}
+ for (i = 0; i < bc_data->num_domains; i++) {
+ struct imx93_blk_ctrl_domain *domain = &bc->domains[i];
+ const struct imx93_blk_ctrl_domain_data *data = domain->data;
+
+ if (data->parent == BLK_CTRL_NO_PARENT)
+ continue;
+
+ ret = pm_genpd_add_subdomain(&bc->domains[data->parent].genpd,
+ &domain->genpd);
+ if (ret) {
+ dev_err_probe(dev, ret, "failed to add subdomain %s\n",
+ domain->genpd.name);
+ goto cleanup_links;
+ }
+ }
+
pm_runtime_enable(dev);
ret = of_genpd_add_provider_onecell(dev->of_node, &bc->onecell_data);
if (ret) {
dev_err_probe(dev, ret, "failed to add power domain provider\n");
- goto cleanup_pds;
+ goto cleanup_links;
}
dev_set_drvdata(dev, bc);
return 0;
+cleanup_links:
+ for (i = 0; i < bc_data->num_domains; i++) {
+ struct imx93_blk_ctrl_domain *domain = &bc->domains[i];
+
+ if (domain->data->parent == BLK_CTRL_NO_PARENT)
+ continue;
+
+ pm_genpd_remove_subdomain(&bc->domains[domain->data->parent].genpd,
+ &domain->genpd);
+ }
+ i = bc_data->num_domains;
+
cleanup_pds:
for (i--; i >= 0; i--)
pm_genpd_remove(&bc->domains[i].genpd);
@@ -298,7 +329,9 @@ static int imx93_blk_ctrl_remove(struct platform_device *pdev)
of_genpd_del_provider(pdev->dev.of_node);
- for (i = 0; bc->onecell_data.num_domains; i++) {
+ pm_runtime_disable(&pdev->dev);
+
+ for (i = 0; i < bc->onecell_data.num_domains; i++) {
struct imx93_blk_ctrl_domain *domain = &bc->domains[i];
pm_genpd_remove(&domain->genpd);
@@ -312,8 +345,9 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.name = "mediablk-mipi-dsi",
.clk_names = (const char *[]){ "dsi" },
.num_clks = 1,
- .rst_mask = BIT(11) | BIT(12),
- .clk_mask = BIT(11) | BIT(12),
+ .rst_mask = BIT(11),
+ .clk_mask = BIT(11),
+ .parent = IMX93_MEDIABLK_PD_MIPI_PHY,
},
[IMX93_MEDIABLK_PD_MIPI_CSI] = {
.name = "mediablk-mipi-csi",
@@ -321,6 +355,7 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.num_clks = 2,
.rst_mask = BIT(9) | BIT(10),
.clk_mask = BIT(9) | BIT(10),
+ .parent = IMX93_MEDIABLK_PD_MIPI_PHY,
},
[IMX93_MEDIABLK_PD_PXP] = {
.name = "mediablk-pxp",
@@ -328,6 +363,7 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.num_clks = 1,
.rst_mask = BIT(7) | BIT(8),
.clk_mask = BIT(7) | BIT(8),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 2,
.qos = {
{
@@ -349,6 +385,7 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.num_clks = 2,
.rst_mask = BIT(4) | BIT(5) | BIT(6),
.clk_mask = BIT(4) | BIT(5) | BIT(6),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 1,
.qos = {
{
@@ -365,6 +402,7 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
.num_clks = 1,
.rst_mask = BIT(2) | BIT(3),
.clk_mask = BIT(2) | BIT(3),
+ .parent = BLK_CTRL_NO_PARENT,
.num_qos = 4,
.qos = {
{
@@ -390,6 +428,14 @@ static const struct imx93_blk_ctrl_domain_data imx93_media_blk_ctl_domain_data[]
}
}
},
+ [IMX93_MEDIABLK_PD_MIPI_PHY] = {
+ .name = "mediablk-mipi-phy",
+ .clk_names = NULL,
+ .num_clks = 0,
+ .rst_mask = BIT(12),
+ .clk_mask = BIT(12),
+ .parent = BLK_CTRL_NO_PARENT,
+ },
};
static const struct regmap_range imx93_media_blk_ctl_yes_ranges[] = {
diff --git a/drivers/soc/qcom/icc-bwmon.c b/drivers/soc/qcom/icc-bwmon.c
index 7829549993c9..2e0a45cd6921 100644
--- a/drivers/soc/qcom/icc-bwmon.c
+++ b/drivers/soc/qcom/icc-bwmon.c
@@ -604,11 +604,13 @@ static int bwmon_probe(struct platform_device *pdev)
opp = dev_pm_opp_find_bw_floor(dev, &bwmon->max_bw_kbps, 0);
if (IS_ERR(opp))
return dev_err_probe(dev, PTR_ERR(opp), "failed to find max peak bandwidth\n");
+ dev_pm_opp_put(opp);
bwmon->min_bw_kbps = 0;
opp = dev_pm_opp_find_bw_ceil(dev, &bwmon->min_bw_kbps, 0);
if (IS_ERR(opp))
return dev_err_probe(dev, PTR_ERR(opp), "failed to find min peak bandwidth\n");
+ dev_pm_opp_put(opp);
bwmon->dev = dev;
diff --git a/drivers/soc/rockchip/pm_domains.c b/drivers/soc/rockchip/pm_domains.c
index 5ba582ffaa1a..2697977cf24c 100644
--- a/drivers/soc/rockchip/pm_domains.c
+++ b/drivers/soc/rockchip/pm_domains.c
@@ -507,7 +507,19 @@ static int rockchip_pd_attach_dev(struct generic_pm_domain *genpd,
}
i = 0;
- while ((clk = of_clk_get(dev->of_node, i++)) && !IS_ERR(clk)) {
+ while (1) {
+ clk = of_clk_get(dev->of_node, i++);
+ if (IS_ERR(clk)) {
+ error = PTR_ERR(clk);
+ if (error == -ENOENT)
+ break;
+
+ dev_err(dev, "failed to get clock %d: %d\n", i - 1,
+ error);
+ pm_clk_destroy(dev);
+ return error;
+ }
+
dev_dbg(dev, "adding clock '%pC' to list of PM clocks\n", clk);
error = pm_clk_add_clk(dev, clk);
if (error) {
@@ -589,7 +601,7 @@ static int rockchip_pm_add_one_domain(struct rockchip_pmu *pmu,
dev_err(pmu->dev,
"%pOFn: failed to get clk at index %d: %d\n",
node, i, error);
- return error;
+ goto err_put_clocks;
}
}
@@ -755,7 +767,9 @@ static int rockchip_pm_add_subdomain(struct rockchip_pmu *pmu,
parent_domain->name, child_domain->name);
}
- rockchip_pm_add_subdomain(pmu, np);
+ error = rockchip_pm_add_subdomain(pmu, np);
+ if (error)
+ goto err_out;
}
return 0;
diff --git a/drivers/soc/xilinx/zynqmp_power.c b/drivers/soc/xilinx/zynqmp_power.c
index 5d41763414ad..b2e36a06921f 100644
--- a/drivers/soc/xilinx/zynqmp_power.c
+++ b/drivers/soc/xilinx/zynqmp_power.c
@@ -276,16 +276,16 @@ static int zynqmp_pm_probe(struct platform_device *pdev)
return 0;
}
-static int zynqmp_pm_remove(struct platform_device *pdev)
+static void zynqmp_pm_remove(struct platform_device *pdev)
{
sysfs_remove_file(&pdev->dev.kobj, &dev_attr_suspend_mode.attr);
if (event_registered)
xlnx_unregister_event(PM_INIT_SUSPEND_CB, 0, 0, suspend_event_callback, NULL);
- if (!rx_chan)
+ if (rx_chan) {
mbox_free_channel(rx_chan);
-
- return 0;
+ rx_chan = NULL;
+ }
}
static const struct of_device_id pm_of_match[] = {
@@ -296,7 +296,7 @@ MODULE_DEVICE_TABLE(of, pm_of_match);
static struct platform_driver zynqmp_pm_platform_driver = {
.probe = zynqmp_pm_probe,
- .remove = zynqmp_pm_remove,
+ .remove_new = zynqmp_pm_remove,
.driver = {
.name = "zynqmp_power",
.of_match_table = pm_of_match,
diff --git a/drivers/soundwire/generic_bandwidth_allocation.c b/drivers/soundwire/generic_bandwidth_allocation.c
index f7c66083a4dd..36bb3f8133aa 100644
--- a/drivers/soundwire/generic_bandwidth_allocation.c
+++ b/drivers/soundwire/generic_bandwidth_allocation.c
@@ -210,28 +210,25 @@ static int sdw_add_element_group_count(struct sdw_group *group,
int num = group->count;
int i;
- for (i = 0; i <= num; i++) {
+ for (i = 0; i < num; i++) {
if (rate == group->rates[i])
- break;
-
- if (i != num)
- continue;
-
- if (group->count >= group->max_size) {
- unsigned int *rates;
+ return 0;
+ }
- group->max_size += 1;
- rates = krealloc(group->rates,
- (sizeof(int) * group->max_size),
- GFP_KERNEL);
- if (!rates)
- return -ENOMEM;
- group->rates = rates;
- }
+ if (group->count >= group->max_size) {
+ unsigned int *rates;
- group->rates[group->count++] = rate;
+ group->max_size += 1;
+ rates = krealloc(group->rates,
+ (sizeof(int) * group->max_size),
+ GFP_KERNEL);
+ if (!rates)
+ return -ENOMEM;
+ group->rates = rates;
}
+ group->rates[group->count++] = rate;
+
return 0;
}
diff --git a/drivers/spi/spi-bcm63xx.c b/drivers/spi/spi-bcm63xx.c
index e57e1f75cd9f..cf0456f41a37 100644
--- a/drivers/spi/spi-bcm63xx.c
+++ b/drivers/spi/spi-bcm63xx.c
@@ -196,7 +196,7 @@ static const unsigned int bcm63xx_spi_freq_table[SPI_CLK_MASK][2] = {
static void bcm63xx_spi_setup_transfer(struct spi_device *spi,
struct spi_transfer *t)
{
- struct bcm63xx_spi *bs = spi_master_get_devdata(spi->master);
+ struct bcm63xx_spi *bs = spi_controller_get_devdata(spi->controller);
u8 clk_cfg, reg;
int i;
@@ -227,7 +227,7 @@ static void bcm63xx_spi_setup_transfer(struct spi_device *spi,
static int bcm63xx_txrx_bufs(struct spi_device *spi, struct spi_transfer *first,
unsigned int num_transfers)
{
- struct bcm63xx_spi *bs = spi_master_get_devdata(spi->master);
+ struct bcm63xx_spi *bs = spi_controller_get_devdata(spi->controller);
u16 msg_ctl;
u16 cmd;
unsigned int i, timeout = 0, prepend_len = 0, len = 0;
@@ -336,10 +336,10 @@ static int bcm63xx_txrx_bufs(struct spi_device *spi, struct spi_transfer *first,
return 0;
}
-static int bcm63xx_spi_transfer_one(struct spi_master *master,
+static int bcm63xx_spi_transfer_one(struct spi_controller *host,
struct spi_message *m)
{
- struct bcm63xx_spi *bs = spi_master_get_devdata(master);
+ struct bcm63xx_spi *bs = spi_controller_get_devdata(host);
struct spi_transfer *t, *first = NULL;
struct spi_device *spi = m->spi;
int status = 0;
@@ -409,18 +409,18 @@ static int bcm63xx_spi_transfer_one(struct spi_master *master,
}
exit:
m->status = status;
- spi_finalize_current_message(master);
+ spi_finalize_current_message(host);
return 0;
}
-/* This driver supports single master mode only. Hence
+/* This driver supports single host mode only. Hence
* CMD_DONE is the only interrupt we care about
*/
static irqreturn_t bcm63xx_spi_interrupt(int irq, void *dev_id)
{
- struct spi_master *master = (struct spi_master *)dev_id;
- struct bcm63xx_spi *bs = spi_master_get_devdata(master);
+ struct spi_controller *host = (struct spi_controller *)dev_id;
+ struct bcm63xx_spi *bs = spi_controller_get_devdata(host);
u8 intr;
/* Read interupts and clear them immediately */
@@ -437,7 +437,7 @@ static irqreturn_t bcm63xx_spi_interrupt(int irq, void *dev_id)
static size_t bcm63xx_spi_max_length(struct spi_device *spi)
{
- struct bcm63xx_spi *bs = spi_master_get_devdata(spi->master);
+ struct bcm63xx_spi *bs = spi_controller_get_devdata(spi->controller);
return bs->fifo_size;
}
@@ -505,7 +505,7 @@ static int bcm63xx_spi_probe(struct platform_device *pdev)
const unsigned long *bcm63xx_spireg;
struct device *dev = &pdev->dev;
int irq, bus_num;
- struct spi_master *master;
+ struct spi_controller *host;
struct clk *clk;
struct bcm63xx_spi *bs;
int ret;
@@ -551,16 +551,16 @@ static int bcm63xx_spi_probe(struct platform_device *pdev)
if (IS_ERR(reset))
return PTR_ERR(reset);
- master = spi_alloc_master(dev, sizeof(*bs));
- if (!master) {
+ host = spi_alloc_host(dev, sizeof(*bs));
+ if (!host) {
dev_err(dev, "out of memory\n");
return -ENOMEM;
}
- bs = spi_master_get_devdata(master);
+ bs = spi_controller_get_devdata(host);
init_completion(&bs->done);
- platform_set_drvdata(pdev, master);
+ platform_set_drvdata(pdev, host);
bs->pdev = pdev;
r = platform_get_resource(pdev, IORESOURCE_MEM, 0);
@@ -576,21 +576,21 @@ static int bcm63xx_spi_probe(struct platform_device *pdev)
bs->fifo_size = bs->reg_offsets[SPI_MSG_DATA_SIZE];
ret = devm_request_irq(&pdev->dev, irq, bcm63xx_spi_interrupt, 0,
- pdev->name, master);
+ pdev->name, host);
if (ret) {
dev_err(dev, "unable to request irq\n");
goto out_err;
}
- master->dev.of_node = dev->of_node;
- master->bus_num = bus_num;
- master->num_chipselect = num_cs;
- master->transfer_one_message = bcm63xx_spi_transfer_one;
- master->mode_bits = MODEBITS;
- master->bits_per_word_mask = SPI_BPW_MASK(8);
- master->max_transfer_size = bcm63xx_spi_max_length;
- master->max_message_size = bcm63xx_spi_max_length;
- master->auto_runtime_pm = true;
+ host->dev.of_node = dev->of_node;
+ host->bus_num = bus_num;
+ host->num_chipselect = num_cs;
+ host->transfer_one_message = bcm63xx_spi_transfer_one;
+ host->mode_bits = MODEBITS;
+ host->bits_per_word_mask = SPI_BPW_MASK(8);
+ host->max_transfer_size = bcm63xx_spi_max_length;
+ host->max_message_size = bcm63xx_spi_max_length;
+ host->auto_runtime_pm = true;
bs->msg_type_shift = bs->reg_offsets[SPI_MSG_TYPE_SHIFT];
bs->msg_ctl_width = bs->reg_offsets[SPI_MSG_CTL_WIDTH];
bs->tx_io = (u8 *)(bs->regs + bs->reg_offsets[SPI_MSG_DATA]);
@@ -614,7 +614,7 @@ static int bcm63xx_spi_probe(struct platform_device *pdev)
goto out_clk_disable;
/* register and we are done */
- ret = devm_spi_register_master(dev, master);
+ ret = devm_spi_register_controller(dev, host);
if (ret) {
dev_err(dev, "spi register failed\n");
goto out_clk_disable;
@@ -628,14 +628,14 @@ static int bcm63xx_spi_probe(struct platform_device *pdev)
out_clk_disable:
clk_disable_unprepare(clk);
out_err:
- spi_master_put(master);
+ spi_controller_put(host);
return ret;
}
static int bcm63xx_spi_remove(struct platform_device *pdev)
{
- struct spi_master *master = platform_get_drvdata(pdev);
- struct bcm63xx_spi *bs = spi_master_get_devdata(master);
+ struct spi_controller *host = platform_get_drvdata(pdev);
+ struct bcm63xx_spi *bs = spi_controller_get_devdata(host);
/* reset spi block */
bcm_spi_writeb(bs, 0, SPI_INT_MASK);
@@ -649,10 +649,10 @@ static int bcm63xx_spi_remove(struct platform_device *pdev)
#ifdef CONFIG_PM_SLEEP
static int bcm63xx_spi_suspend(struct device *dev)
{
- struct spi_master *master = dev_get_drvdata(dev);
- struct bcm63xx_spi *bs = spi_master_get_devdata(master);
+ struct spi_controller *host = dev_get_drvdata(dev);
+ struct bcm63xx_spi *bs = spi_controller_get_devdata(host);
- spi_master_suspend(master);
+ spi_controller_suspend(host);
clk_disable_unprepare(bs->clk);
@@ -661,15 +661,19 @@ static int bcm63xx_spi_suspend(struct device *dev)
static int bcm63xx_spi_resume(struct device *dev)
{
- struct spi_master *master = dev_get_drvdata(dev);
- struct bcm63xx_spi *bs = spi_master_get_devdata(master);
+ struct spi_controller *host = dev_get_drvdata(dev);
+ struct bcm63xx_spi *bs = spi_controller_get_devdata(host);
int ret;
ret = clk_prepare_enable(bs->clk);
if (ret)
return ret;
- spi_master_resume(master);
+ ret = spi_controller_resume(host);
+ if (ret) {
+ clk_disable_unprepare(bs->clk);
+ return ret;
+ }
return 0;
}
diff --git a/drivers/spi/spi-davinci.c b/drivers/spi/spi-davinci.c
index d112c2cac042..ce4689e96bad 100644
--- a/drivers/spi/spi-davinci.c
+++ b/drivers/spi/spi-davinci.c
@@ -863,7 +863,7 @@ static int davinci_spi_probe(struct platform_device *pdev)
int ret = 0;
u32 spipc0;
- master = spi_alloc_master(&pdev->dev, sizeof(struct davinci_spi));
+ master = devm_spi_alloc_master(&pdev->dev, sizeof(struct davinci_spi));
if (master == NULL) {
ret = -ENOMEM;
goto err;
@@ -1004,7 +1004,6 @@ static int davinci_spi_probe(struct platform_device *pdev)
free_clk:
clk_disable_unprepare(dspi->clk);
free_master:
- spi_master_put(master);
err:
return ret;
}
@@ -1028,6 +1027,8 @@ static int davinci_spi_remove(struct platform_device *pdev)
spi_bitbang_stop(&dspi->bitbang);
+ devm_free_irq(&pdev->dev, dspi->irq, dspi);
+
clk_disable_unprepare(dspi->clk);
if (dspi->dma_rx) {
diff --git a/drivers/spi/spi-dw-dma.c b/drivers/spi/spi-dw-dma.c
index a795df52c5fe..6ace5ff7675b 100644
--- a/drivers/spi/spi-dw-dma.c
+++ b/drivers/spi/spi-dw-dma.c
@@ -231,7 +231,7 @@ static int dw_spi_dma_wait(struct dw_spi *dws, unsigned int len, u32 speed)
msecs_to_jiffies(ms));
if (ms == 0) {
- dev_err(&dws->master->cur_msg->spi->dev,
+ dev_err(&dws->master->dev,
"DMA transaction timed out\n");
return -ETIMEDOUT;
}
diff --git a/drivers/spi/spi-fsl-lpspi.c b/drivers/spi/spi-fsl-lpspi.c
index 0daba232f474..df49b52df7c2 100644
--- a/drivers/spi/spi-fsl-lpspi.c
+++ b/drivers/spi/spi-fsl-lpspi.c
@@ -827,11 +827,11 @@ static int fsl_lpspi_probe(struct platform_device *pdev)
is_slave = of_property_read_bool((&pdev->dev)->of_node, "spi-slave");
if (is_slave)
- controller = spi_alloc_slave(&pdev->dev,
- sizeof(struct fsl_lpspi_data));
+ controller = devm_spi_alloc_target(&pdev->dev,
+ sizeof(struct fsl_lpspi_data));
else
- controller = spi_alloc_master(&pdev->dev,
- sizeof(struct fsl_lpspi_data));
+ controller = devm_spi_alloc_host(&pdev->dev,
+ sizeof(struct fsl_lpspi_data));
if (!controller)
return -ENOMEM;
@@ -931,7 +931,6 @@ static int fsl_lpspi_probe(struct platform_device *pdev)
pm_runtime_put_sync(fsl_lpspi->dev);
pm_runtime_disable(fsl_lpspi->dev);
out_controller_put:
- spi_controller_put(controller);
return ret;
}
diff --git a/drivers/spi/spi-fsl-qspi.c b/drivers/spi/spi-fsl-qspi.c
index aee9993fe09d..665ce836c72f 100644
--- a/drivers/spi/spi-fsl-qspi.c
+++ b/drivers/spi/spi-fsl-qspi.c
@@ -840,6 +840,19 @@ static const struct spi_controller_mem_ops fsl_qspi_mem_ops = {
.get_name = fsl_qspi_get_name,
};
+static void fsl_qspi_cleanup(void *data)
+{
+ struct fsl_qspi *q = data;
+
+ /* disable the hardware */
+ qspi_writel(q, QUADSPI_MCR_MDIS_MASK, q->iobase + QUADSPI_MCR);
+ qspi_writel(q, 0x0, q->iobase + QUADSPI_RSER);
+
+ fsl_qspi_clk_disable_unprep(q);
+
+ mutex_destroy(&q->lock);
+}
+
static int fsl_qspi_probe(struct platform_device *pdev)
{
struct spi_controller *ctlr;
@@ -849,7 +862,7 @@ static int fsl_qspi_probe(struct platform_device *pdev)
struct fsl_qspi *q;
int ret;
- ctlr = spi_alloc_master(&pdev->dev, sizeof(*q));
+ ctlr = devm_spi_alloc_master(&pdev->dev, sizeof(*q));
if (!ctlr)
return -ENOMEM;
@@ -930,40 +943,24 @@ static int fsl_qspi_probe(struct platform_device *pdev)
ctlr->dev.of_node = np;
+ ret = devm_add_action_or_reset(dev, fsl_qspi_cleanup, q);
+ if (ret)
+ goto err_put_ctrl;
+
ret = devm_spi_register_controller(dev, ctlr);
if (ret)
- goto err_destroy_mutex;
+ goto err_put_ctrl;
return 0;
-err_destroy_mutex:
- mutex_destroy(&q->lock);
-
err_disable_clk:
fsl_qspi_clk_disable_unprep(q);
err_put_ctrl:
- spi_controller_put(ctlr);
-
dev_err(dev, "Freescale QuadSPI probe failed\n");
return ret;
}
-static int fsl_qspi_remove(struct platform_device *pdev)
-{
- struct fsl_qspi *q = platform_get_drvdata(pdev);
-
- /* disable the hardware */
- qspi_writel(q, QUADSPI_MCR_MDIS_MASK, q->iobase + QUADSPI_MCR);
- qspi_writel(q, 0x0, q->iobase + QUADSPI_RSER);
-
- fsl_qspi_clk_disable_unprep(q);
-
- mutex_destroy(&q->lock);
-
- return 0;
-}
-
static int fsl_qspi_suspend(struct device *dev)
{
return 0;
@@ -1001,7 +998,6 @@ static struct platform_driver fsl_qspi_driver = {
.pm = &fsl_qspi_pm_ops,
},
.probe = fsl_qspi_probe,
- .remove = fsl_qspi_remove,
};
module_platform_driver(fsl_qspi_driver);
diff --git a/drivers/spi/spi-fsl-spi.c b/drivers/spi/spi-fsl-spi.c
index 1a32c4f6f1bf..a92671faca2a 100644
--- a/drivers/spi/spi-fsl-spi.c
+++ b/drivers/spi/spi-fsl-spi.c
@@ -88,7 +88,7 @@ static int fsl_spi_get_type(struct device *dev)
static void fsl_spi_change_mode(struct spi_device *spi)
{
- struct mpc8xxx_spi *mspi = spi_master_get_devdata(spi->master);
+ struct mpc8xxx_spi *mspi = spi_controller_get_devdata(spi->controller);
struct spi_mpc8xxx_cs *cs = spi->controller_state;
struct fsl_spi_reg __iomem *reg_base = mspi->reg_base;
__be32 __iomem *mode = ®_base->mode;
@@ -186,7 +186,7 @@ static int fsl_spi_setup_transfer(struct spi_device *spi,
u32 hz = 0;
struct spi_mpc8xxx_cs *cs = spi->controller_state;
- mpc8xxx_spi = spi_master_get_devdata(spi->master);
+ mpc8xxx_spi = spi_controller_get_devdata(spi->controller);
if (t) {
bits_per_word = t->bits_per_word;
@@ -260,7 +260,7 @@ static int fsl_spi_cpu_bufs(struct mpc8xxx_spi *mspi,
static int fsl_spi_bufs(struct spi_device *spi, struct spi_transfer *t,
bool is_dma_mapped)
{
- struct mpc8xxx_spi *mpc8xxx_spi = spi_master_get_devdata(spi->master);
+ struct mpc8xxx_spi *mpc8xxx_spi = spi_controller_get_devdata(spi->controller);
struct fsl_spi_reg __iomem *reg_base;
unsigned int len = t->len;
u8 bits_per_word;
@@ -401,7 +401,7 @@ static int fsl_spi_setup(struct spi_device *spi)
spi_set_ctldata(spi, cs);
initial_setup = true;
}
- mpc8xxx_spi = spi_master_get_devdata(spi->master);
+ mpc8xxx_spi = spi_controller_get_devdata(spi->controller);
reg_base = mpc8xxx_spi->reg_base;
@@ -495,7 +495,7 @@ static irqreturn_t fsl_spi_irq(s32 irq, void *context_data)
static void fsl_spi_grlib_cs_control(struct spi_device *spi, bool on)
{
- struct mpc8xxx_spi *mpc8xxx_spi = spi_master_get_devdata(spi->master);
+ struct mpc8xxx_spi *mpc8xxx_spi = spi_controller_get_devdata(spi->controller);
struct fsl_spi_reg __iomem *reg_base = mpc8xxx_spi->reg_base;
u32 slvsel;
u16 cs = spi->chip_select;
@@ -509,8 +509,8 @@ static void fsl_spi_grlib_cs_control(struct spi_device *spi, bool on)
static void fsl_spi_grlib_probe(struct device *dev)
{
- struct spi_master *master = dev_get_drvdata(dev);
- struct mpc8xxx_spi *mpc8xxx_spi = spi_master_get_devdata(master);
+ struct spi_controller *host = dev_get_drvdata(dev);
+ struct mpc8xxx_spi *mpc8xxx_spi = spi_controller_get_devdata(host);
struct fsl_spi_reg __iomem *reg_base = mpc8xxx_spi->reg_base;
int mbits;
u32 capabilities;
@@ -527,8 +527,8 @@ static void fsl_spi_grlib_probe(struct device *dev)
mpc8xxx_spi->native_chipselects = SPCAP_SSSZ(capabilities);
mpc8xxx_spi_write_reg(®_base->slvsel, 0xffffffff);
}
- master->num_chipselect = mpc8xxx_spi->native_chipselects;
- master->set_cs = fsl_spi_grlib_cs_control;
+ host->num_chipselect = mpc8xxx_spi->native_chipselects;
+ host->set_cs = fsl_spi_grlib_cs_control;
}
static void fsl_spi_cs_control(struct spi_device *spi, bool on)
@@ -542,35 +542,35 @@ static void fsl_spi_cs_control(struct spi_device *spi, bool on)
iowrite32be(on ? 0 : SPI_BOOT_SEL_BIT, pinfo->immr_spi_cs);
}
-static struct spi_master *fsl_spi_probe(struct device *dev,
+static struct spi_controller *fsl_spi_probe(struct device *dev,
struct resource *mem, unsigned int irq)
{
struct fsl_spi_platform_data *pdata = dev_get_platdata(dev);
- struct spi_master *master;
+ struct spi_controller *host;
struct mpc8xxx_spi *mpc8xxx_spi;
struct fsl_spi_reg __iomem *reg_base;
u32 regval;
int ret = 0;
- master = spi_alloc_master(dev, sizeof(struct mpc8xxx_spi));
- if (master == NULL) {
+ host = spi_alloc_host(dev, sizeof(struct mpc8xxx_spi));
+ if (host == NULL) {
ret = -ENOMEM;
goto err;
}
- dev_set_drvdata(dev, master);
+ dev_set_drvdata(dev, host);
mpc8xxx_spi_probe(dev, mem, irq);
- master->setup = fsl_spi_setup;
- master->cleanup = fsl_spi_cleanup;
- master->prepare_message = fsl_spi_prepare_message;
- master->transfer_one = fsl_spi_transfer_one;
- master->unprepare_message = fsl_spi_unprepare_message;
- master->use_gpio_descriptors = true;
- master->set_cs = fsl_spi_cs_control;
+ host->setup = fsl_spi_setup;
+ host->cleanup = fsl_spi_cleanup;
+ host->prepare_message = fsl_spi_prepare_message;
+ host->transfer_one = fsl_spi_transfer_one;
+ host->unprepare_message = fsl_spi_unprepare_message;
+ host->use_gpio_descriptors = true;
+ host->set_cs = fsl_spi_cs_control;
- mpc8xxx_spi = spi_master_get_devdata(master);
+ mpc8xxx_spi = spi_controller_get_devdata(host);
mpc8xxx_spi->max_bits_per_word = 32;
mpc8xxx_spi->type = fsl_spi_get_type(dev);
@@ -588,13 +588,13 @@ static struct spi_master *fsl_spi_probe(struct device *dev,
fsl_spi_grlib_probe(dev);
if (mpc8xxx_spi->flags & SPI_CPM_MODE)
- master->bits_per_word_mask =
+ host->bits_per_word_mask =
(SPI_BPW_RANGE_MASK(4, 8) | SPI_BPW_MASK(16) | SPI_BPW_MASK(32));
else
- master->bits_per_word_mask =
+ host->bits_per_word_mask =
(SPI_BPW_RANGE_MASK(4, 16) | SPI_BPW_MASK(32));
- master->bits_per_word_mask &=
+ host->bits_per_word_mask &=
SPI_BPW_RANGE_MASK(1, mpc8xxx_spi->max_bits_per_word);
if (mpc8xxx_spi->flags & SPI_QE_CPU_MODE)
@@ -631,19 +631,19 @@ static struct spi_master *fsl_spi_probe(struct device *dev,
mpc8xxx_spi_write_reg(®_base->mode, regval);
- ret = devm_spi_register_master(dev, master);
+ ret = spi_register_controller(host);
if (ret < 0)
goto err_probe;
dev_info(dev, "at 0x%p (irq = %d), %s mode\n", reg_base,
mpc8xxx_spi->irq, mpc8xxx_spi_strmode(mpc8xxx_spi->flags));
- return master;
+ return host;
err_probe:
fsl_spi_cpm_free(mpc8xxx_spi);
err_cpm_init:
- spi_master_put(master);
+ spi_controller_put(host);
err:
return ERR_PTR(ret);
}
@@ -652,7 +652,7 @@ static int of_fsl_spi_probe(struct platform_device *ofdev)
{
struct device *dev = &ofdev->dev;
struct device_node *np = ofdev->dev.of_node;
- struct spi_master *master;
+ struct spi_controller *host;
struct resource mem;
int irq, type;
int ret;
@@ -705,9 +705,9 @@ static int of_fsl_spi_probe(struct platform_device *ofdev)
goto unmap_out;
}
- master = fsl_spi_probe(dev, &mem, irq);
+ host = fsl_spi_probe(dev, &mem, irq);
- return PTR_ERR_OR_ZERO(master);
+ return PTR_ERR_OR_ZERO(host);
unmap_out:
#if IS_ENABLED(CONFIG_FSL_SOC)
@@ -717,13 +717,18 @@ static int of_fsl_spi_probe(struct platform_device *ofdev)
return ret;
}
-static int of_fsl_spi_remove(struct platform_device *ofdev)
+static void of_fsl_spi_remove(struct platform_device *ofdev)
{
- struct spi_master *master = platform_get_drvdata(ofdev);
- struct mpc8xxx_spi *mpc8xxx_spi = spi_master_get_devdata(master);
+ struct spi_controller *host = platform_get_drvdata(ofdev);
+ struct mpc8xxx_spi *mpc8xxx_spi = spi_controller_get_devdata(host);
+
+ spi_controller_get(host);
+
+ spi_unregister_controller(host);
fsl_spi_cpm_free(mpc8xxx_spi);
- return 0;
+
+ spi_controller_put(host);
}
static struct platform_driver of_fsl_spi_driver = {
@@ -732,7 +737,7 @@ static struct platform_driver of_fsl_spi_driver = {
.of_match_table = of_fsl_spi_match,
},
.probe = of_fsl_spi_probe,
- .remove = of_fsl_spi_remove,
+ .remove_new = of_fsl_spi_remove,
};
#ifdef CONFIG_MPC832x_RDB
@@ -747,7 +752,7 @@ static int plat_mpc8xxx_spi_probe(struct platform_device *pdev)
{
struct resource *mem;
int irq;
- struct spi_master *master;
+ struct spi_controller *host;
if (!dev_get_platdata(&pdev->dev))
return -EINVAL;
@@ -760,24 +765,28 @@ static int plat_mpc8xxx_spi_probe(struct platform_device *pdev)
if (irq <= 0)
return -EINVAL;
- master = fsl_spi_probe(&pdev->dev, mem, irq);
- return PTR_ERR_OR_ZERO(master);
+ host = fsl_spi_probe(&pdev->dev, mem, irq);
+ return PTR_ERR_OR_ZERO(host);
}
-static int plat_mpc8xxx_spi_remove(struct platform_device *pdev)
+static void plat_mpc8xxx_spi_remove(struct platform_device *pdev)
{
- struct spi_master *master = platform_get_drvdata(pdev);
- struct mpc8xxx_spi *mpc8xxx_spi = spi_master_get_devdata(master);
+ struct spi_controller *host = platform_get_drvdata(pdev);
+ struct mpc8xxx_spi *mpc8xxx_spi = spi_controller_get_devdata(host);
+
+ spi_controller_get(host);
+
+ spi_unregister_controller(host);
fsl_spi_cpm_free(mpc8xxx_spi);
- return 0;
+ spi_controller_put(host);
}
MODULE_ALIAS("platform:mpc8xxx_spi");
static struct platform_driver mpc8xxx_spi_driver = {
.probe = plat_mpc8xxx_spi_probe,
- .remove = plat_mpc8xxx_spi_remove,
+ .remove_new = plat_mpc8xxx_spi_remove,
.driver = {
.name = "mpc8xxx_spi",
},
diff --git a/drivers/spi/spi-hisi-kunpeng.c b/drivers/spi/spi-hisi-kunpeng.c
index 06c8893243b7..562e23d66d13 100644
--- a/drivers/spi/spi-hisi-kunpeng.c
+++ b/drivers/spi/spi-hisi-kunpeng.c
@@ -162,10 +162,10 @@ static int hisi_spi_debugfs_init(struct hisi_spi *hs)
{
char name[32];
- struct spi_controller *master;
+ struct spi_controller *host;
- master = container_of(hs->dev, struct spi_controller, dev);
- snprintf(name, 32, "hisi_spi%d", master->bus_num);
+ host = container_of(hs->dev, struct spi_controller, dev);
+ snprintf(name, 32, "hisi_spi%d", host->bus_num);
hs->debugfs = debugfs_create_dir(name, NULL);
if (!hs->debugfs)
return -ENOMEM;
@@ -299,18 +299,18 @@ static void __hisi_calc_div_reg(struct hisi_chip_data *chip)
chip->div_post = (chip->clk_div / chip->div_pre) - 1;
}
-static u32 hisi_calc_effective_speed(struct spi_controller *master,
+static u32 hisi_calc_effective_speed(struct spi_controller *host,
struct hisi_chip_data *chip, u32 speed_hz)
{
u32 effective_speed;
/* Note clock divider doesn't support odd numbers */
- chip->clk_div = DIV_ROUND_UP(master->max_speed_hz, speed_hz) + 1;
+ chip->clk_div = DIV_ROUND_UP(host->max_speed_hz, speed_hz) + 1;
chip->clk_div &= 0xfffe;
if (chip->clk_div > CLK_DIV_MAX)
chip->clk_div = CLK_DIV_MAX;
- effective_speed = master->max_speed_hz / chip->clk_div;
+ effective_speed = host->max_speed_hz / chip->clk_div;
if (chip->speed_hz != effective_speed) {
__hisi_calc_div_reg(chip);
chip->speed_hz = effective_speed;
@@ -344,20 +344,20 @@ static void hisi_spi_hw_init(struct hisi_spi *hs)
static irqreturn_t hisi_spi_irq(int irq, void *dev_id)
{
- struct spi_controller *master = dev_id;
- struct hisi_spi *hs = spi_controller_get_devdata(master);
+ struct spi_controller *host = dev_id;
+ struct hisi_spi *hs = spi_controller_get_devdata(host);
u32 irq_status = readl(hs->regs + HISI_SPI_ISR) & ISR_MASK;
if (!irq_status)
return IRQ_NONE;
- if (!master->cur_msg)
+ if (!host->cur_msg)
return IRQ_HANDLED;
/* Error handling */
if (irq_status & ISR_RXOF) {
dev_err(hs->dev, "interrupt_transfer: fifo overflow\n");
- master->cur_msg->status = -EIO;
+ host->cur_msg->status = -EIO;
goto finalize_transfer;
}
@@ -377,20 +377,20 @@ static irqreturn_t hisi_spi_irq(int irq, void *dev_id)
finalize_transfer:
hisi_spi_disable(hs);
- spi_finalize_current_transfer(master);
+ spi_finalize_current_transfer(host);
return IRQ_HANDLED;
}
-static int hisi_spi_transfer_one(struct spi_controller *master,
+static int hisi_spi_transfer_one(struct spi_controller *host,
struct spi_device *spi, struct spi_transfer *transfer)
{
- struct hisi_spi *hs = spi_controller_get_devdata(master);
+ struct hisi_spi *hs = spi_controller_get_devdata(host);
struct hisi_chip_data *chip = spi_get_ctldata(spi);
u32 cr = chip->cr;
/* Update per transfer options for speed and bpw */
transfer->effective_speed_hz =
- hisi_calc_effective_speed(master, chip, transfer->speed_hz);
+ hisi_calc_effective_speed(host, chip, transfer->speed_hz);
cr |= FIELD_PREP(CR_DIV_PRE_MASK, chip->div_pre);
cr |= FIELD_PREP(CR_DIV_POST_MASK, chip->div_post);
cr |= FIELD_PREP(CR_BPW_MASK, transfer->bits_per_word - 1);
@@ -417,10 +417,10 @@ static int hisi_spi_transfer_one(struct spi_controller *master,
return 1;
}
-static void hisi_spi_handle_err(struct spi_controller *master,
+static void hisi_spi_handle_err(struct spi_controller *host,
struct spi_message *msg)
{
- struct hisi_spi *hs = spi_controller_get_devdata(master);
+ struct hisi_spi *hs = spi_controller_get_devdata(host);
hisi_spi_disable(hs);
@@ -460,7 +460,7 @@ static void hisi_spi_cleanup(struct spi_device *spi)
static int hisi_spi_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
- struct spi_controller *master;
+ struct spi_controller *host;
struct hisi_spi *hs;
int ret, irq;
@@ -468,13 +468,13 @@ static int hisi_spi_probe(struct platform_device *pdev)
if (irq < 0)
return irq;
- master = devm_spi_alloc_master(dev, sizeof(*hs));
- if (!master)
+ host = devm_spi_alloc_host(dev, sizeof(*hs));
+ if (!host)
return -ENOMEM;
- platform_set_drvdata(pdev, master);
+ platform_set_drvdata(pdev, host);
- hs = spi_controller_get_devdata(master);
+ hs = spi_controller_get_devdata(host);
hs->dev = dev;
hs->irq = irq;
@@ -482,42 +482,45 @@ static int hisi_spi_probe(struct platform_device *pdev)
if (IS_ERR(hs->regs))
return PTR_ERR(hs->regs);
- /* Specify maximum SPI clocking speed (master only) by firmware */
+ /* Specify maximum SPI clocking speed (host only) by firmware */
ret = device_property_read_u32(dev, "spi-max-frequency",
- &master->max_speed_hz);
+ &host->max_speed_hz);
if (ret) {
dev_err(dev, "failed to get max SPI clocking speed, ret=%d\n",
ret);
return -EINVAL;
}
+ if (host->max_speed_hz == 0)
+ return dev_err_probe(dev, -EINVAL, "spi-max-frequency can't be 0\n");
+
ret = device_property_read_u16(dev, "num-cs",
- &master->num_chipselect);
+ &host->num_chipselect);
if (ret)
- master->num_chipselect = DEFAULT_NUM_CS;
-
- master->use_gpio_descriptors = true;
- master->mode_bits = SPI_CPOL | SPI_CPHA | SPI_CS_HIGH | SPI_LOOP;
- master->bits_per_word_mask = SPI_BPW_RANGE_MASK(4, 32);
- master->bus_num = pdev->id;
- master->setup = hisi_spi_setup;
- master->cleanup = hisi_spi_cleanup;
- master->transfer_one = hisi_spi_transfer_one;
- master->handle_err = hisi_spi_handle_err;
- master->dev.fwnode = dev->fwnode;
+ host->num_chipselect = DEFAULT_NUM_CS;
+
+ host->use_gpio_descriptors = true;
+ host->mode_bits = SPI_CPOL | SPI_CPHA | SPI_CS_HIGH | SPI_LOOP;
+ host->bits_per_word_mask = SPI_BPW_RANGE_MASK(4, 32);
+ host->bus_num = pdev->id;
+ host->setup = hisi_spi_setup;
+ host->cleanup = hisi_spi_cleanup;
+ host->transfer_one = hisi_spi_transfer_one;
+ host->handle_err = hisi_spi_handle_err;
+ host->dev.fwnode = dev->fwnode;
hisi_spi_hw_init(hs);
ret = devm_request_irq(dev, hs->irq, hisi_spi_irq, 0, dev_name(dev),
- master);
+ host);
if (ret < 0) {
dev_err(dev, "failed to get IRQ=%d, ret=%d\n", hs->irq, ret);
return ret;
}
- ret = spi_register_controller(master);
+ ret = spi_register_controller(host);
if (ret) {
- dev_err(dev, "failed to register spi master, ret=%d\n", ret);
+ dev_err(dev, "failed to register spi host, ret=%d\n", ret);
return ret;
}
@@ -526,20 +529,18 @@ static int hisi_spi_probe(struct platform_device *pdev)
dev_info(dev, "hw version:0x%x max-freq:%u kHz\n",
readl(hs->regs + HISI_SPI_VERSION),
- master->max_speed_hz / 1000);
+ host->max_speed_hz / 1000);
return 0;
}
-static int hisi_spi_remove(struct platform_device *pdev)
+static void hisi_spi_remove(struct platform_device *pdev)
{
- struct spi_controller *master = platform_get_drvdata(pdev);
- struct hisi_spi *hs = spi_controller_get_devdata(master);
+ struct spi_controller *host = platform_get_drvdata(pdev);
+ struct hisi_spi *hs = spi_controller_get_devdata(host);
debugfs_remove_recursive(hs->debugfs);
- spi_unregister_controller(master);
-
- return 0;
+ spi_unregister_controller(host);
}
static const struct acpi_device_id hisi_spi_acpi_match[] = {
@@ -550,7 +551,7 @@ MODULE_DEVICE_TABLE(acpi, hisi_spi_acpi_match);
static struct platform_driver hisi_spi_driver = {
.probe = hisi_spi_probe,
- .remove = hisi_spi_remove,
+ .remove_new = hisi_spi_remove,
.driver = {
.name = "hisi-kunpeng-spi",
.acpi_match_table = hisi_spi_acpi_match,
diff --git a/drivers/spi/spi-microchip-core-qspi.c b/drivers/spi/spi-microchip-core-qspi.c
index 160861891b46..f8cee68a9347 100644
--- a/drivers/spi/spi-microchip-core-qspi.c
+++ b/drivers/spi/spi-microchip-core-qspi.c
@@ -74,6 +74,13 @@
#define STATUS_FLAGSX4 BIT(8)
#define STATUS_MASK GENMASK(8, 0)
+/*
+ * QSPI Direct Access register defines
+ */
+#define DIRECT_ACCESS_EN_SSEL BIT(0)
+#define DIRECT_ACCESS_OP_SSEL BIT(1)
+#define DIRECT_ACCESS_OP_SSEL_SHIFT 1
+
#define BYTESUPPER_MASK GENMASK(31, 16)
#define BYTESLOWER_MASK GENMASK(15, 0)
@@ -158,6 +165,38 @@ static int mchp_coreqspi_set_mode(struct mchp_coreqspi *qspi, const struct spi_m
return 0;
}
+static void mchp_coreqspi_set_cs(struct spi_device *spi, bool enable)
+{
+ struct mchp_coreqspi *qspi = spi_controller_get_devdata(spi->controller);
+ u32 val;
+
+ val = readl(qspi->regs + REG_DIRECT_ACCESS);
+
+ val &= ~DIRECT_ACCESS_OP_SSEL;
+ val |= !enable << DIRECT_ACCESS_OP_SSEL_SHIFT;
+
+ writel(val, qspi->regs + REG_DIRECT_ACCESS);
+}
+
+static int mchp_coreqspi_setup(struct spi_device *spi)
+{
+ struct mchp_coreqspi *qspi = spi_controller_get_devdata(spi->controller);
+ u32 val;
+
+ /*
+ * Active low devices need to be specifically set to their inactive
+ * states during probe.
+ */
+ if (spi->mode & SPI_CS_HIGH)
+ return 0;
+
+ val = readl(qspi->regs + REG_DIRECT_ACCESS);
+ val |= DIRECT_ACCESS_OP_SSEL;
+ writel(val, qspi->regs + REG_DIRECT_ACCESS);
+
+ return 0;
+}
+
static inline void mchp_coreqspi_read_op(struct mchp_coreqspi *qspi)
{
u32 control, data;
@@ -298,19 +337,6 @@ static int mchp_coreqspi_setup_clock(struct mchp_coreqspi *qspi, struct spi_devi
return 0;
}
-static int mchp_coreqspi_setup_op(struct spi_device *spi_dev)
-{
- struct spi_controller *ctlr = spi_dev->master;
- struct mchp_coreqspi *qspi = spi_controller_get_devdata(ctlr);
- u32 control = readl_relaxed(qspi->regs + REG_CONTROL);
-
- control |= (CONTROL_MASTER | CONTROL_ENABLE);
- control &= ~CONTROL_CLKIDLE;
- writel_relaxed(control, qspi->regs + REG_CONTROL);
-
- return 0;
-}
-
static inline void mchp_coreqspi_config_op(struct mchp_coreqspi *qspi, const struct spi_mem_op *op)
{
u32 idle_cycles = 0;
@@ -409,6 +435,7 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o
reinit_completion(&qspi->data_completion);
mchp_coreqspi_config_op(qspi, op);
+ mchp_coreqspi_set_cs(mem->spi, true);
if (op->cmd.opcode) {
qspi->txbuf = &opcode;
qspi->rxbuf = NULL;
@@ -449,6 +476,7 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o
err = -ETIMEDOUT;
error:
+ mchp_coreqspi_set_cs(mem->spi, false);
mutex_unlock(&qspi->op_lock);
mchp_coreqspi_disable_ints(qspi);
@@ -505,6 +533,7 @@ static int mchp_coreqspi_probe(struct platform_device *pdev)
struct device *dev = &pdev->dev;
struct device_node *np = dev->of_node;
int ret;
+ u32 num_cs, val;
ctlr = devm_spi_alloc_master(&pdev->dev, sizeof(*qspi));
if (!ctlr)
@@ -538,12 +567,34 @@ static int mchp_coreqspi_probe(struct platform_device *pdev)
return ret;
}
+ /*
+ * The IP core only has a single CS, any more have to be provided via
+ * gpios
+ */
+ if (of_property_read_u32(pdev->dev.of_node, "num-cs", &num_cs))
+ num_cs = 1;
+
+ ctlr->num_chipselect = num_cs;
+
ctlr->bits_per_word_mask = SPI_BPW_MASK(8);
ctlr->mem_ops = &mchp_coreqspi_mem_ops;
- ctlr->setup = mchp_coreqspi_setup_op;
ctlr->mode_bits = SPI_CPOL | SPI_CPHA | SPI_RX_DUAL | SPI_RX_QUAD |
SPI_TX_DUAL | SPI_TX_QUAD;
ctlr->dev.of_node = np;
+ ctlr->setup = mchp_coreqspi_setup;
+ ctlr->set_cs = mchp_coreqspi_set_cs;
+ ctlr->use_gpio_descriptors = true;
+
+ val = readl_relaxed(qspi->regs + REG_CONTROL);
+ val |= (CONTROL_MASTER | CONTROL_ENABLE);
+ writel_relaxed(val, qspi->regs + REG_CONTROL);
+
+ /*
+ * Put cs into software controlled mode
+ */
+ val = readl_relaxed(qspi->regs + REG_DIRECT_ACCESS);
+ val |= DIRECT_ACCESS_EN_SSEL;
+ writel(val, qspi->regs + REG_DIRECT_ACCESS);
ret = spi_register_controller(ctlr);
if (ret)
diff --git a/drivers/spi/spi-microchip-core.c b/drivers/spi/spi-microchip-core.c
index bfad0fe743ad..acc05f5a929e 100644
--- a/drivers/spi/spi-microchip-core.c
+++ b/drivers/spi/spi-microchip-core.c
@@ -91,6 +91,8 @@
#define REG_CONTROL2 (0x28)
#define REG_COMMAND (0x2c)
#define COMMAND_CLRFRAMECNT BIT(4)
+#define COMMAND_TXFIFORST BIT(3)
+#define COMMAND_RXFIFORST BIT(2)
#define REG_PKTSIZE (0x30)
#define REG_CMD_SIZE (0x34)
#define REG_HWSTATUS (0x38)
@@ -489,6 +491,8 @@ static int mchp_corespi_transfer_one(struct spi_controller *host,
mchp_corespi_set_xfer_size(spi, (spi->tx_len > FIFO_DEPTH)
? FIFO_DEPTH : spi->tx_len);
+ mchp_corespi_write(spi, REG_COMMAND, COMMAND_RXFIFORST | COMMAND_TXFIFORST);
+
while (spi->tx_len)
mchp_corespi_write_fifo(spi);
diff --git a/drivers/spi/spi-mpc52xx.c b/drivers/spi/spi-mpc52xx.c
index c4d005c3363e..3845862bf67f 100644
--- a/drivers/spi/spi-mpc52xx.c
+++ b/drivers/spi/spi-mpc52xx.c
@@ -62,7 +62,7 @@ MODULE_LICENSE("GPL");
/* Driver internal data */
struct mpc52xx_spi {
- struct spi_master *master;
+ struct spi_controller *host;
void __iomem *regs;
int irq0; /* MODF irq */
int irq1; /* SPIF irq */
@@ -152,7 +152,7 @@ mpc52xx_spi_fsmstate_idle(int irq, struct mpc52xx_spi *ms, u8 status, u8 data)
u8 ctrl1;
if (status && irq)
- dev_err(&ms->master->dev, "spurious irq, status=0x%.2x\n",
+ dev_err(&ms->host->dev, "spurious irq, status=0x%.2x\n",
status);
/* Check if there is another transfer waiting. */
@@ -235,7 +235,7 @@ static int mpc52xx_spi_fsmstate_transfer(int irq, struct mpc52xx_spi *ms,
return FSM_CONTINUE;
} else if (status & SPI_STATUS_MODF) {
ms->modf_count++;
- dev_err(&ms->master->dev, "mode fault\n");
+ dev_err(&ms->host->dev, "mode fault\n");
mpc52xx_spi_chipsel(ms, 0);
ms->message->status = -EIO;
if (ms->message->complete)
@@ -280,7 +280,7 @@ static int
mpc52xx_spi_fsmstate_wait(int irq, struct mpc52xx_spi *ms, u8 status, u8 data)
{
if (status && irq)
- dev_err(&ms->master->dev, "spurious irq, status=0x%.2x\n",
+ dev_err(&ms->host->dev, "spurious irq, status=0x%.2x\n",
status);
if (((int)mftb()) - ms->timestamp < 0)
@@ -361,12 +361,12 @@ static void mpc52xx_spi_wq(struct work_struct *work)
}
/*
- * spi_master ops
+ * spi_controller ops
*/
static int mpc52xx_spi_transfer(struct spi_device *spi, struct spi_message *m)
{
- struct mpc52xx_spi *ms = spi_master_get_devdata(spi->master);
+ struct mpc52xx_spi *ms = spi_controller_get_devdata(spi->controller);
unsigned long flags;
m->actual_length = 0;
@@ -385,7 +385,7 @@ static int mpc52xx_spi_transfer(struct spi_device *spi, struct spi_message *m)
*/
static int mpc52xx_spi_probe(struct platform_device *op)
{
- struct spi_master *master;
+ struct spi_controller *host;
struct mpc52xx_spi *ms;
struct gpio_desc *gpio_cs;
void __iomem *regs;
@@ -406,7 +406,7 @@ static int mpc52xx_spi_probe(struct platform_device *op)
out_8(regs + SPI_PORTDATA, 0x8); /* Deassert /SS signal */
/* Clear the status register and re-read it to check for a MODF
- * failure. This driver cannot currently handle multiple masters
+ * failure. This driver cannot currently handle multiple hosts
* on the SPI bus. This fault will also occur if the SPI signals
* are not connected to any pins (port_config setting) */
in_8(regs + SPI_STATUS);
@@ -419,22 +419,22 @@ static int mpc52xx_spi_probe(struct platform_device *op)
goto err_init;
}
- dev_dbg(&op->dev, "allocating spi_master struct\n");
- master = spi_alloc_master(&op->dev, sizeof(*ms));
- if (!master) {
+ dev_dbg(&op->dev, "allocating spi_controller struct\n");
+ host = spi_alloc_host(&op->dev, sizeof(*ms));
+ if (!host) {
rc = -ENOMEM;
goto err_alloc;
}
- master->transfer = mpc52xx_spi_transfer;
- master->mode_bits = SPI_CPOL | SPI_CPHA | SPI_LSB_FIRST;
- master->bits_per_word_mask = SPI_BPW_MASK(8);
- master->dev.of_node = op->dev.of_node;
+ host->transfer = mpc52xx_spi_transfer;
+ host->mode_bits = SPI_CPOL | SPI_CPHA | SPI_LSB_FIRST;
+ host->bits_per_word_mask = SPI_BPW_MASK(8);
+ host->dev.of_node = op->dev.of_node;
- platform_set_drvdata(op, master);
+ platform_set_drvdata(op, host);
- ms = spi_master_get_devdata(master);
- ms->master = master;
+ ms = spi_controller_get_devdata(host);
+ ms->host = host;
ms->regs = regs;
ms->irq0 = irq_of_parse_and_map(op->dev.of_node, 0);
ms->irq1 = irq_of_parse_and_map(op->dev.of_node, 1);
@@ -442,7 +442,7 @@ static int mpc52xx_spi_probe(struct platform_device *op)
ms->ipb_freq = mpc5xxx_get_bus_frequency(&op->dev);
ms->gpio_cs_count = gpiod_count(&op->dev, NULL);
if (ms->gpio_cs_count > 0) {
- master->num_chipselect = ms->gpio_cs_count;
+ host->num_chipselect = ms->gpio_cs_count;
ms->gpio_cs = kmalloc_array(ms->gpio_cs_count,
sizeof(*ms->gpio_cs),
GFP_KERNEL);
@@ -489,36 +489,41 @@ static int mpc52xx_spi_probe(struct platform_device *op)
if (!ms->irq0)
dev_info(&op->dev, "using polled mode\n");
- dev_dbg(&op->dev, "registering spi_master struct\n");
- rc = spi_register_master(master);
+ dev_dbg(&op->dev, "registering spi_controller struct\n");
+ rc = spi_register_controller(host);
if (rc)
goto err_register;
- dev_info(&ms->master->dev, "registered MPC5200 SPI bus\n");
+ dev_info(&ms->host->dev, "registered MPC5200 SPI bus\n");
return rc;
err_register:
- dev_err(&ms->master->dev, "initialization failed\n");
+ dev_err(&ms->host->dev, "initialization failed\n");
+ free_irq(ms->irq0, ms);
+ free_irq(ms->irq1, ms);
+ cancel_work_sync(&ms->work);
err_gpio:
while (i-- > 0)
gpiod_put(ms->gpio_cs[i]);
kfree(ms->gpio_cs);
err_alloc_gpio:
- spi_master_put(master);
+ spi_controller_put(host);
err_alloc:
err_init:
iounmap(regs);
return rc;
}
-static int mpc52xx_spi_remove(struct platform_device *op)
+static void mpc52xx_spi_remove(struct platform_device *op)
{
- struct spi_master *master = spi_master_get(platform_get_drvdata(op));
- struct mpc52xx_spi *ms = spi_master_get_devdata(master);
+ struct spi_controller *host = spi_controller_get(platform_get_drvdata(op));
+ struct mpc52xx_spi *ms = spi_controller_get_devdata(host);
int i;
+ spi_unregister_controller(host);
+
free_irq(ms->irq0, ms);
free_irq(ms->irq1, ms);
@@ -528,11 +533,8 @@ static int mpc52xx_spi_remove(struct platform_device *op)
gpiod_put(ms->gpio_cs[i]);
kfree(ms->gpio_cs);
- spi_unregister_master(master);
iounmap(ms->regs);
- spi_master_put(master);
-
- return 0;
+ spi_controller_put(host);
}
static const struct of_device_id mpc52xx_spi_match[] = {
@@ -547,6 +549,6 @@ static struct platform_driver mpc52xx_spi_of_driver = {
.of_match_table = mpc52xx_spi_match,
},
.probe = mpc52xx_spi_probe,
- .remove = mpc52xx_spi_remove,
+ .remove_new = mpc52xx_spi_remove,
};
module_platform_driver(mpc52xx_spi_of_driver);
diff --git a/drivers/spi/spi-oc-tiny.c b/drivers/spi/spi-oc-tiny.c
index 38c14c4e4e21..e1c984a80ed2 100644
--- a/drivers/spi/spi-oc-tiny.c
+++ b/drivers/spi/spi-oc-tiny.c
@@ -213,11 +213,11 @@ static int tiny_spi_probe(struct platform_device *pdev)
struct tiny_spi_platform_data *platp = dev_get_platdata(&pdev->dev);
struct tiny_spi *hw;
struct spi_master *master;
- int err = -ENODEV;
+ int err;
- master = spi_alloc_master(&pdev->dev, sizeof(struct tiny_spi));
+ master = devm_spi_alloc_master(&pdev->dev, sizeof(struct tiny_spi));
if (!master)
- return err;
+ return -ENOMEM;
/* setup the master state. */
master->bus_num = pdev->id;
@@ -235,10 +235,8 @@ static int tiny_spi_probe(struct platform_device *pdev)
/* find and map our resources */
hw->base = devm_platform_ioremap_resource(pdev, 0);
- if (IS_ERR(hw->base)) {
- err = PTR_ERR(hw->base);
- goto exit;
- }
+ if (IS_ERR(hw->base))
+ return PTR_ERR(hw->base);
/* irq is optional */
hw->irq = platform_get_irq(pdev, 0);
if (hw->irq >= 0) {
@@ -246,7 +244,7 @@ static int tiny_spi_probe(struct platform_device *pdev)
err = devm_request_irq(&pdev->dev, hw->irq, tiny_spi_irq, 0,
pdev->name, hw);
if (err)
- goto exit;
+ return err;
}
/* find platform data */
if (platp) {
@@ -255,29 +253,23 @@ static int tiny_spi_probe(struct platform_device *pdev)
} else {
err = tiny_spi_of_probe(pdev);
if (err)
- goto exit;
+ return err;
}
/* register our spi controller */
err = spi_bitbang_start(&hw->bitbang);
if (err)
- goto exit;
+ return err;
dev_info(&pdev->dev, "base %p, irq %d\n", hw->base, hw->irq);
return 0;
-
-exit:
- spi_master_put(master);
- return err;
}
static int tiny_spi_remove(struct platform_device *pdev)
{
struct tiny_spi *hw = platform_get_drvdata(pdev);
- struct spi_master *master = hw->bitbang.master;
spi_bitbang_stop(&hw->bitbang);
- spi_master_put(master);
return 0;
}
diff --git a/drivers/spi/spi-qcom-qspi.c b/drivers/spi/spi-qcom-qspi.c
index a6d5f287eddc..4db034da54b7 100644
--- a/drivers/spi/spi-qcom-qspi.c
+++ b/drivers/spi/spi-qcom-qspi.c
@@ -8,6 +8,7 @@
#include <linux/module.h>
#include <linux/of.h>
#include <linux/of_platform.h>
+#include <linux/pinctrl/consumer.h>
#include <linux/pm_runtime.h>
#include <linux/pm_opp.h>
#include <linux/spi/spi.h>
@@ -570,18 +571,34 @@ static int __maybe_unused qcom_qspi_runtime_suspend(struct device *dev)
struct qcom_qspi *ctrl = spi_master_get_devdata(master);
int ret;
- /* Drop the performance state vote */
- dev_pm_opp_set_rate(dev, 0);
clk_bulk_disable_unprepare(QSPI_NUM_CLKS, ctrl->clks);
ret = icc_disable(ctrl->icc_path_cpu_to_qspi);
if (ret) {
dev_err_ratelimited(ctrl->dev, "%s: ICC disable failed for cpu: %d\n",
__func__, ret);
- return ret;
+ goto err_enable_clk;
}
+ ret = pinctrl_pm_select_sleep_state(dev);
+ if (ret)
+ goto err_enable_icc;
+
+ /* Drop the performance state vote */
+ ret = dev_pm_opp_set_rate(dev, 0);
+ if (ret)
+ goto err_select_default_state;
+
return 0;
+
+err_select_default_state:
+ pinctrl_pm_select_default_state(dev);
+err_enable_icc:
+ icc_enable(ctrl->icc_path_cpu_to_qspi);
+err_enable_clk:
+ if (clk_bulk_prepare_enable(QSPI_NUM_CLKS, ctrl->clks))
+ dev_err_ratelimited(ctrl->dev, "Failed to re-enable clocks\n");
+ return ret;
}
static int __maybe_unused qcom_qspi_runtime_resume(struct device *dev)
@@ -590,18 +607,34 @@ static int __maybe_unused qcom_qspi_runtime_resume(struct device *dev)
struct qcom_qspi *ctrl = spi_master_get_devdata(master);
int ret;
+ ret = dev_pm_opp_set_rate(dev, ctrl->last_speed * 4);
+ if (ret)
+ return ret;
+
+ ret = pinctrl_pm_select_default_state(dev);
+ if (ret)
+ goto err_opp_set_rate_zero;
+
ret = icc_enable(ctrl->icc_path_cpu_to_qspi);
if (ret) {
dev_err_ratelimited(ctrl->dev, "%s: ICC enable failed for cpu: %d\n",
__func__, ret);
- return ret;
+ goto err_select_sleep_state;
}
ret = clk_bulk_prepare_enable(QSPI_NUM_CLKS, ctrl->clks);
if (ret)
- return ret;
+ goto err_disable_icc;
+
+ return 0;
- return dev_pm_opp_set_rate(dev, ctrl->last_speed * 4);
+err_disable_icc:
+ icc_disable(ctrl->icc_path_cpu_to_qspi);
+err_select_sleep_state:
+ pinctrl_pm_select_sleep_state(dev);
+err_opp_set_rate_zero:
+ dev_pm_opp_set_rate(dev, 0);
+ return ret;
}
static int __maybe_unused qcom_qspi_suspend(struct device *dev)
diff --git a/drivers/spi/spi-rockchip.c b/drivers/spi/spi-rockchip.c
index c38d515e4399..ccf602522973 100644
--- a/drivers/spi/spi-rockchip.c
+++ b/drivers/spi/spi-rockchip.c
@@ -541,7 +541,7 @@ static int rockchip_spi_config(struct rockchip_spi *rs,
cr0 |= (spi->mode & 0x3U) << CR0_SCPH_OFFSET;
if (spi->mode & SPI_LSB_FIRST)
cr0 |= CR0_FBM_LSB << CR0_FBM_OFFSET;
- if (spi->mode & SPI_CS_HIGH)
+ if ((spi->mode & SPI_CS_HIGH) && !spi->cs_gpiod)
cr0 |= BIT(spi->chip_select) << CR0_SOI_OFFSET;
if (xfer->rx_buf && xfer->tx_buf)
diff --git a/drivers/spi/spi-rspi.c b/drivers/spi/spi-rspi.c
index 411b1307b7fd..96b4037babba 100644
--- a/drivers/spi/spi-rspi.c
+++ b/drivers/spi/spi-rspi.c
@@ -1172,14 +1172,18 @@ static void rspi_release_dma(struct spi_controller *ctlr)
dma_release_channel(ctlr->dma_rx);
}
-static int rspi_remove(struct platform_device *pdev)
+static void rspi_remove(struct platform_device *pdev)
{
struct rspi_data *rspi = platform_get_drvdata(pdev);
+ spi_controller_get(rspi->ctlr);
+
+ spi_unregister_controller(rspi->ctlr);
+
rspi_release_dma(rspi->ctlr);
pm_runtime_disable(&pdev->dev);
- return 0;
+ spi_controller_put(rspi->ctlr);
}
static const struct spi_ops rspi_ops = {
@@ -1390,9 +1394,9 @@ static int rspi_probe(struct platform_device *pdev)
if (ret < 0)
dev_warn(&pdev->dev, "DMA not available, using PIO\n");
- ret = devm_spi_register_controller(&pdev->dev, ctlr);
+ ret = spi_register_controller(ctlr);
if (ret < 0) {
- dev_err(&pdev->dev, "devm_spi_register_controller error.\n");
+ dev_err(&pdev->dev, "failed to register controller\n");
goto error3;
}
@@ -1440,7 +1444,7 @@ static SIMPLE_DEV_PM_OPS(rspi_pm_ops, rspi_suspend, rspi_resume);
static struct platform_driver rspi_driver = {
.probe = rspi_probe,
- .remove = rspi_remove,
+ .remove_new = rspi_remove,
.id_table = spi_driver_ids,
.driver = {
.name = "renesas_spi",
diff --git a/drivers/spi/spi-tegra114.c b/drivers/spi/spi-tegra114.c
index b2c9b90fc81c..2489a2dbf0be 100644
--- a/drivers/spi/spi-tegra114.c
+++ b/drivers/spi/spi-tegra114.c
@@ -860,6 +860,13 @@ static u32 tegra_spi_setup_transfer_one(struct spi_device *spi,
command1 |= SPI_BIT_LENGTH(bits_per_word - 1);
}
+ /*
+ * Re-assert the CS GPIO for every transfer: transfer_end() drops it
+ * on cs_change, and the assert above only covers is_first_of_msg.
+ */
+ if (spi_get_csgpiod(spi, 0))
+ gpiod_set_value(spi_get_csgpiod(spi, 0), 1);
+
return command1;
}
diff --git a/drivers/spi/spi-tegra210-quad.c b/drivers/spi/spi-tegra210-quad.c
index 022cc0ebee77..cb1b7f251eac 100644
--- a/drivers/spi/spi-tegra210-quad.c
+++ b/drivers/spi/spi-tegra210-quad.c
@@ -1342,6 +1342,11 @@ static irqreturn_t handle_cpu_based_xfer(struct tegra_qspi *tqspi)
spin_lock_irqsave(&tqspi->lock, flags);
t = tqspi->curr_xfer;
+ if (!t) {
+ spin_unlock_irqrestore(&tqspi->lock, flags);
+ return IRQ_HANDLED;
+ }
+
if (tqspi->tx_status || tqspi->rx_status) {
tegra_qspi_handle_error(tqspi);
complete(&tqspi->xfer_completion);
@@ -1410,6 +1415,11 @@ static irqreturn_t handle_dma_based_xfer(struct tegra_qspi *tqspi)
spin_lock_irqsave(&tqspi->lock, flags);
t = tqspi->curr_xfer;
+ if (!t) {
+ spin_unlock_irqrestore(&tqspi->lock, flags);
+ return IRQ_HANDLED;
+ }
+
if (err) {
tegra_qspi_dma_unmap_xfer(tqspi, t);
tegra_qspi_handle_error(tqspi);
@@ -1448,6 +1458,7 @@ static irqreturn_t handle_dma_based_xfer(struct tegra_qspi *tqspi)
static irqreturn_t tegra_qspi_isr_thread(int irq, void *context_data)
{
struct tegra_qspi *tqspi = context_data;
+ unsigned long flags;
u32 status;
/*
@@ -1465,7 +1476,9 @@ static irqreturn_t tegra_qspi_isr_thread(int irq, void *context_data)
* If no transfer is in progress, check if this was a real interrupt
* that the timeout handler already processed, or a spurious one.
*/
+ spin_lock_irqsave(&tqspi->lock, flags);
if (!tqspi->curr_xfer) {
+ spin_unlock_irqrestore(&tqspi->lock, flags);
/* Spurious interrupt - transfer not ready */
if (!(status & QSPI_RDY))
return IRQ_NONE;
@@ -1482,7 +1495,14 @@ static irqreturn_t tegra_qspi_isr_thread(int irq, void *context_data)
tqspi->rx_status = tqspi->status_reg & (QSPI_RX_FIFO_OVF | QSPI_RX_FIFO_UNF);
tegra_qspi_mask_clear_irq(tqspi);
+ spin_unlock_irqrestore(&tqspi->lock, flags);
+ /*
+ * Lock is released here but handlers safely re-check curr_xfer under
+ * lock before dereferencing.
+ * DMA handler also needs to sleep in wait_for_completion_*(), which
+ * cannot be done while holding spinlock.
+ */
if (!tqspi->is_curr_dma_xfer)
return handle_cpu_based_xfer(tqspi);
diff --git a/drivers/staging/greybus/raw.c b/drivers/staging/greybus/raw.c
index 2a375f407d38..ade49fbe9978 100644
--- a/drivers/staging/greybus/raw.c
+++ b/drivers/staging/greybus/raw.c
@@ -21,9 +21,10 @@ struct gb_raw {
struct list_head list;
int list_data;
struct mutex list_lock;
- dev_t dev;
+ struct rw_semaphore disconnect_lock;
+ bool disconnected;
struct cdev cdev;
- struct device *device;
+ struct device dev;
};
struct raw_data {
@@ -145,6 +146,15 @@ static int gb_raw_send(struct gb_raw *raw, u32 len, const char __user *data)
return retval;
}
+static void raw_dev_release(struct device *dev)
+{
+ struct gb_raw *raw = container_of(dev, struct gb_raw, dev);
+
+ ida_free(&minors, MINOR(raw->dev.devt));
+
+ kfree(raw);
+}
+
static int gb_raw_probe(struct gb_bundle *bundle,
const struct greybus_bundle_id *id)
{
@@ -161,63 +171,59 @@ static int gb_raw_probe(struct gb_bundle *bundle,
if (cport_desc->protocol_id != GREYBUS_PROTOCOL_RAW)
return -ENODEV;
+ minor = ida_alloc(&minors, GFP_KERNEL);
+ if (minor < 0)
+ return minor;
+
raw = kzalloc(sizeof(*raw), GFP_KERNEL);
- if (!raw)
+ if (!raw) {
+ ida_free(&minors, minor);
return -ENOMEM;
+ }
+
+ device_initialize(&raw->dev);
+ raw->dev.devt = MKDEV(raw_major, minor);
+ raw->dev.class = raw_class;
+ raw->dev.parent = &bundle->dev;
+ raw->dev.release = raw_dev_release;
+ retval = dev_set_name(&raw->dev, "gb!raw%d", minor);
+ if (retval)
+ goto error_put_device;
connection = gb_connection_create(bundle, le16_to_cpu(cport_desc->id),
gb_raw_request_handler);
if (IS_ERR(connection)) {
retval = PTR_ERR(connection);
- goto error_free;
+ goto error_put_device;
}
INIT_LIST_HEAD(&raw->list);
mutex_init(&raw->list_lock);
+ init_rwsem(&raw->disconnect_lock);
raw->connection = connection;
greybus_set_drvdata(bundle, raw);
- minor = ida_simple_get(&minors, 0, 0, GFP_KERNEL);
- if (minor < 0) {
- retval = minor;
- goto error_connection_destroy;
- }
-
- raw->dev = MKDEV(raw_major, minor);
cdev_init(&raw->cdev, &raw_fops);
retval = gb_connection_enable(connection);
if (retval)
- goto error_remove_ida;
+ goto error_connection_destroy;
- retval = cdev_add(&raw->cdev, raw->dev, 1);
+ retval = cdev_device_add(&raw->cdev, &raw->dev);
if (retval)
goto error_connection_disable;
- raw->device = device_create(raw_class, &connection->bundle->dev,
- raw->dev, raw, "gb!raw%d", minor);
- if (IS_ERR(raw->device)) {
- retval = PTR_ERR(raw->device);
- goto error_del_cdev;
- }
-
return 0;
-error_del_cdev:
- cdev_del(&raw->cdev);
-
error_connection_disable:
gb_connection_disable(connection);
-error_remove_ida:
- ida_simple_remove(&minors, minor);
-
error_connection_destroy:
gb_connection_destroy(connection);
-error_free:
- kfree(raw);
+error_put_device:
+ put_device(&raw->dev);
return retval;
}
@@ -228,11 +234,13 @@ static void gb_raw_disconnect(struct gb_bundle *bundle)
struct raw_data *raw_data;
struct raw_data *temp;
- // FIXME - handle removing a connection when the char device node is open.
- device_destroy(raw_class, raw->dev);
- cdev_del(&raw->cdev);
+ cdev_device_del(&raw->cdev, &raw->dev);
+
+ down_write(&raw->disconnect_lock);
+ raw->disconnected = true;
+ up_write(&raw->disconnect_lock);
+
gb_connection_disable(connection);
- ida_simple_remove(&minors, MINOR(raw->dev));
gb_connection_destroy(connection);
mutex_lock(&raw->list_lock);
@@ -241,8 +249,7 @@ static void gb_raw_disconnect(struct gb_bundle *bundle)
kfree(raw_data);
}
mutex_unlock(&raw->list_lock);
-
- kfree(raw);
+ put_device(&raw->dev);
}
/*
@@ -275,11 +282,22 @@ static ssize_t raw_write(struct file *file, const char __user *buf,
if (count > MAX_PACKET_SIZE)
return -E2BIG;
+ down_read(&raw->disconnect_lock);
+
+ if (raw->disconnected) {
+ retval = -ENODEV;
+ goto exit;
+ }
+
retval = gb_raw_send(raw, count, buf);
if (retval)
- return retval;
+ goto exit;
+
+ retval = count;
+exit:
+ up_read(&raw->disconnect_lock);
- return count;
+ return retval;
}
static ssize_t raw_read(struct file *file, char __user *buf, size_t count,
diff --git a/drivers/staging/greybus/uart.c b/drivers/staging/greybus/uart.c
index 074c02f123e2..0bdef38a15d7 100644
--- a/drivers/staging/greybus/uart.c
+++ b/drivers/staging/greybus/uart.c
@@ -954,7 +954,8 @@ static int gb_tty_init(void)
gb_tty_driver->minor_start = 0;
gb_tty_driver->type = TTY_DRIVER_TYPE_SERIAL;
gb_tty_driver->subtype = SERIAL_TYPE_NORMAL;
- gb_tty_driver->flags = TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV;
+ gb_tty_driver->flags = TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV |
+ TTY_DRIVER_RESET_SAVED_TERMIOS;
gb_tty_driver->init_termios = tty_std_termios;
gb_tty_driver->init_termios.c_cflag = B9600 | CS8 |
CREAD | HUPCL | CLOCAL;
diff --git a/drivers/staging/media/atomisp/i2c/atomisp-gc2235.c b/drivers/staging/media/atomisp/i2c/atomisp-gc2235.c
index 4d5a7e335f85..5b8c7eafbf45 100644
--- a/drivers/staging/media/atomisp/i2c/atomisp-gc2235.c
+++ b/drivers/staging/media/atomisp/i2c/atomisp-gc2235.c
@@ -989,7 +989,7 @@ static int gc2235_probe(struct i2c_client *client)
ret = gc2235_s_config(&dev->sd, client->irq, gcpdev);
if (ret)
- goto out_free;
+ goto err_unregister_subdev;
dev->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;
dev->pad.flags = MEDIA_PAD_FL_SOURCE;
@@ -998,18 +998,16 @@ static int gc2235_probe(struct i2c_client *client)
ret =
v4l2_ctrl_handler_init(&dev->ctrl_handler,
ARRAY_SIZE(gc2235_controls));
- if (ret) {
- gc2235_remove(client);
- return ret;
- }
+ if (ret)
+ goto err_csi_cfg;
for (i = 0; i < ARRAY_SIZE(gc2235_controls); i++)
v4l2_ctrl_new_custom(&dev->ctrl_handler, &gc2235_controls[i],
NULL);
if (dev->ctrl_handler.error) {
- gc2235_remove(client);
- return dev->ctrl_handler.error;
+ ret = dev->ctrl_handler.error;
+ goto err_ctrl_handler;
}
/* Use same lock for controls as for everything else. */
@@ -1018,14 +1016,23 @@ static int gc2235_probe(struct i2c_client *client)
ret = media_entity_pads_init(&dev->sd.entity, 1, &dev->pad);
if (ret)
- gc2235_remove(client);
+ goto err_ctrl_handler;
+
+ ret = atomisp_register_i2c_module(&dev->sd, gcpdev, RAW_CAMERA);
+ if (ret)
+ goto err_media_cleanup;
- return atomisp_register_i2c_module(&dev->sd, gcpdev, RAW_CAMERA);
+ return 0;
-out_free:
+err_media_cleanup:
+ media_entity_cleanup(&dev->sd.entity);
+err_ctrl_handler:
+ v4l2_ctrl_handler_free(&dev->ctrl_handler);
+err_csi_cfg:
+ dev->platform_data->csi_cfg(&dev->sd, 0);
+err_unregister_subdev:
v4l2_device_unregister_subdev(&dev->sd);
kfree(dev);
-
return ret;
}
diff --git a/drivers/staging/media/deprecated/saa7146/av7110/av7110.h b/drivers/staging/media/deprecated/saa7146/av7110/av7110.h
index 9fde69b38f1c..c2956c23d9c1 100644
--- a/drivers/staging/media/deprecated/saa7146/av7110/av7110.h
+++ b/drivers/staging/media/deprecated/saa7146/av7110/av7110.h
@@ -86,6 +86,8 @@ struct infrared {
u32 ir_config;
};
+#define MAX_CI_SLOTS 2
+
/* place to store all the necessary device information */
struct av7110 {
@@ -166,7 +168,7 @@ struct av7110 {
/* CA */
- struct ca_slot_info ci_slot[2];
+ struct ca_slot_info ci_slot[MAX_CI_SLOTS];
enum av7110_video_mode vidmode;
struct dmxdev dmxdev;
diff --git a/drivers/staging/media/deprecated/saa7146/av7110/av7110_ca.c b/drivers/staging/media/deprecated/saa7146/av7110/av7110_ca.c
index c1338e074a3d..642af03fbac8 100644
--- a/drivers/staging/media/deprecated/saa7146/av7110/av7110_ca.c
+++ b/drivers/staging/media/deprecated/saa7146/av7110/av7110_ca.c
@@ -12,6 +12,7 @@
*/
#include <linux/kernel.h>
+#include <linux/nospec.h>
#include <linux/types.h>
#include <linux/delay.h>
#include <linux/fs.h>
@@ -26,23 +27,28 @@
void CI_handle(struct av7110 *av7110, u8 *data, u16 len)
{
+ unsigned slot_num;
+
dprintk(8, "av7110:%p\n",av7110);
if (len < 3)
return;
switch (data[0]) {
case CI_MSG_CI_INFO:
- if (data[2] != 1 && data[2] != 2)
+ if (data[2] != 1 && data[2] != MAX_CI_SLOTS)
break;
+
+ slot_num = array_index_nospec(data[2] - 1, MAX_CI_SLOTS);
+
switch (data[1]) {
case 0:
- av7110->ci_slot[data[2] - 1].flags = 0;
+ av7110->ci_slot[slot_num].flags = 0;
break;
case 1:
- av7110->ci_slot[data[2] - 1].flags |= CA_CI_MODULE_PRESENT;
+ av7110->ci_slot[slot_num].flags |= CA_CI_MODULE_PRESENT;
break;
case 2:
- av7110->ci_slot[data[2] - 1].flags |= CA_CI_MODULE_READY;
+ av7110->ci_slot[slot_num].flags |= CA_CI_MODULE_READY;
break;
}
break;
@@ -264,15 +270,19 @@ static int dvb_ca_ioctl(struct file *file, unsigned int cmd, void *parg)
case CA_GET_SLOT_INFO:
{
struct ca_slot_info *info=(struct ca_slot_info *)parg;
+ unsigned int slot_num;
if (info->num < 0 || info->num > 1) {
mutex_unlock(&av7110->ioctl_mutex);
return -EINVAL;
}
- av7110->ci_slot[info->num].num = info->num;
- av7110->ci_slot[info->num].type = FW_CI_LL_SUPPORT(av7110->arm_app) ?
- CA_CI_LINK : CA_CI;
- memcpy(info, &av7110->ci_slot[info->num], sizeof(struct ca_slot_info));
+ slot_num = array_index_nospec(info->num, MAX_CI_SLOTS);
+
+ av7110->ci_slot[slot_num].num = info->num;
+ av7110->ci_slot[slot_num].type = FW_CI_LL_SUPPORT(av7110->arm_app) ?
+ CA_CI_LINK : CA_CI;
+ memcpy(info, &av7110->ci_slot[slot_num],
+ sizeof(struct ca_slot_info));
break;
}
diff --git a/drivers/staging/media/rkvdec/rkvdec.c b/drivers/staging/media/rkvdec/rkvdec.c
index b5847259f454..0b19a089e56c 100644
--- a/drivers/staging/media/rkvdec/rkvdec.c
+++ b/drivers/staging/media/rkvdec/rkvdec.c
@@ -1044,8 +1044,8 @@ static int rkvdec_probe(struct platform_device *pdev)
vb2_dma_contig_set_max_seg_size(&pdev->dev, DMA_BIT_MASK(32));
irq = platform_get_irq(pdev, 0);
- if (irq <= 0)
- return -ENXIO;
+ if (irq < 0)
+ return irq;
ret = devm_request_threaded_irq(&pdev->dev, irq, NULL,
rkvdec_irq_handler, IRQF_ONESHOT,
diff --git a/drivers/staging/most/Kconfig b/drivers/staging/most/Kconfig
index 6f420cbcdcff..e89658df6f12 100644
--- a/drivers/staging/most/Kconfig
+++ b/drivers/staging/most/Kconfig
@@ -24,6 +24,4 @@ source "drivers/staging/most/video/Kconfig"
source "drivers/staging/most/dim2/Kconfig"
-source "drivers/staging/most/i2c/Kconfig"
-
endif
diff --git a/drivers/staging/most/Makefile b/drivers/staging/most/Makefile
index 8b3fc5a7af51..e45084df7803 100644
--- a/drivers/staging/most/Makefile
+++ b/drivers/staging/most/Makefile
@@ -3,4 +3,3 @@
obj-$(CONFIG_MOST_NET) += net/
obj-$(CONFIG_MOST_VIDEO) += video/
obj-$(CONFIG_MOST_DIM2) += dim2/
-obj-$(CONFIG_MOST_I2C) += i2c/
diff --git a/drivers/staging/most/i2c/Kconfig b/drivers/staging/most/i2c/Kconfig
deleted file mode 100644
index ff64283cbad1..000000000000
--- a/drivers/staging/most/i2c/Kconfig
+++ /dev/null
@@ -1,13 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-#
-# MOST I2C configuration
-#
-
-config MOST_I2C
- tristate "I2C"
- depends on I2C
- help
- Say Y here if you want to connect via I2C to network transceiver.
-
- To compile this driver as a module, choose M here: the
- module will be called most_i2c.
diff --git a/drivers/staging/most/i2c/Makefile b/drivers/staging/most/i2c/Makefile
deleted file mode 100644
index 71099dd0f85b..000000000000
--- a/drivers/staging/most/i2c/Makefile
+++ /dev/null
@@ -1,4 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-obj-$(CONFIG_MOST_I2C) += most_i2c.o
-
-most_i2c-objs := i2c.o
diff --git a/drivers/staging/most/i2c/i2c.c b/drivers/staging/most/i2c/i2c.c
deleted file mode 100644
index 285a071f02be..000000000000
--- a/drivers/staging/most/i2c/i2c.c
+++ /dev/null
@@ -1,371 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * i2c.c - Hardware Dependent Module for I2C Interface
- *
- * Copyright (C) 2013-2015, Microchip Technology Germany II GmbH & Co. KG
- */
-
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/slab.h>
-#include <linux/i2c.h>
-#include <linux/interrupt.h>
-#include <linux/err.h>
-#include <linux/most.h>
-
-enum { CH_RX, CH_TX, NUM_CHANNELS };
-
-#define MAX_BUFFERS_CONTROL 32
-#define MAX_BUF_SIZE_CONTROL 256
-
-/**
- * list_first_mbo - get the first mbo from a list
- * @ptr: the list head to take the mbo from.
- */
-#define list_first_mbo(ptr) \
- list_first_entry(ptr, struct mbo, list)
-
-static unsigned int polling_rate;
-module_param(polling_rate, uint, 0644);
-MODULE_PARM_DESC(polling_rate, "Polling rate [Hz]. Default = 0 (use IRQ)");
-
-struct hdm_i2c {
- struct most_interface most_iface;
- struct most_channel_capability capabilities[NUM_CHANNELS];
- struct i2c_client *client;
- struct rx {
- struct delayed_work dwork;
- struct list_head list;
- bool int_disabled;
- unsigned int delay;
- } rx;
- char name[64];
-};
-
-#define to_hdm(iface) container_of(iface, struct hdm_i2c, most_iface)
-
-static irqreturn_t most_irq_handler(int, void *);
-static void pending_rx_work(struct work_struct *);
-
-/**
- * configure_channel - called from MOST core to configure a channel
- * @most_iface: interface the channel belongs to
- * @ch_idx: channel to be configured
- * @channel_config: structure that holds the configuration information
- *
- * Return 0 on success, negative on failure.
- *
- * Receives configuration information from MOST core and initialize the
- * corresponding channel.
- */
-static int configure_channel(struct most_interface *most_iface,
- int ch_idx,
- struct most_channel_config *channel_config)
-{
- int ret;
- struct hdm_i2c *dev = to_hdm(most_iface);
- unsigned int delay, pr;
-
- BUG_ON(ch_idx < 0 || ch_idx >= NUM_CHANNELS);
-
- if (channel_config->data_type != MOST_CH_CONTROL) {
- pr_err("bad data type for channel %d\n", ch_idx);
- return -EPERM;
- }
-
- if (channel_config->direction != dev->capabilities[ch_idx].direction) {
- pr_err("bad direction for channel %d\n", ch_idx);
- return -EPERM;
- }
-
- if (channel_config->direction == MOST_CH_RX) {
- if (!polling_rate) {
- if (dev->client->irq <= 0) {
- pr_err("bad irq: %d\n", dev->client->irq);
- return -ENOENT;
- }
- dev->rx.int_disabled = false;
- ret = request_irq(dev->client->irq, most_irq_handler, 0,
- dev->client->name, dev);
- if (ret) {
- pr_err("request_irq(%d) failed: %d\n",
- dev->client->irq, ret);
- return ret;
- }
- } else {
- delay = msecs_to_jiffies(MSEC_PER_SEC / polling_rate);
- dev->rx.delay = delay ? delay : 1;
- pr = MSEC_PER_SEC / jiffies_to_msecs(dev->rx.delay);
- pr_info("polling rate is %u Hz\n", pr);
- }
- }
-
- return 0;
-}
-
-/**
- * enqueue - called from MOST core to enqueue a buffer for data transfer
- * @most_iface: intended interface
- * @ch_idx: ID of the channel the buffer is intended for
- * @mbo: pointer to the buffer object
- *
- * Return 0 on success, negative on failure.
- *
- * Transmit the data over I2C if it is a "write" request or push the buffer into
- * list if it is an "read" request
- */
-static int enqueue(struct most_interface *most_iface,
- int ch_idx, struct mbo *mbo)
-{
- struct hdm_i2c *dev = to_hdm(most_iface);
- int ret;
-
- BUG_ON(ch_idx < 0 || ch_idx >= NUM_CHANNELS);
-
- if (ch_idx == CH_RX) {
- /* RX */
- if (!polling_rate)
- disable_irq(dev->client->irq);
- cancel_delayed_work_sync(&dev->rx.dwork);
- list_add_tail(&mbo->list, &dev->rx.list);
- if (dev->rx.int_disabled || polling_rate)
- pending_rx_work(&dev->rx.dwork.work);
- if (!polling_rate)
- enable_irq(dev->client->irq);
- } else {
- /* TX */
- ret = i2c_master_send(dev->client, mbo->virt_address,
- mbo->buffer_length);
- if (ret <= 0) {
- mbo->processed_length = 0;
- mbo->status = MBO_E_INVAL;
- } else {
- mbo->processed_length = mbo->buffer_length;
- mbo->status = MBO_SUCCESS;
- }
- mbo->complete(mbo);
- }
-
- return 0;
-}
-
-/**
- * poison_channel - called from MOST core to poison buffers of a channel
- * @most_iface: pointer to the interface the channel to be poisoned belongs to
- * @ch_idx: corresponding channel ID
- *
- * Return 0 on success, negative on failure.
- *
- * If channel direction is RX, complete the buffers in list with
- * status MBO_E_CLOSE
- */
-static int poison_channel(struct most_interface *most_iface,
- int ch_idx)
-{
- struct hdm_i2c *dev = to_hdm(most_iface);
- struct mbo *mbo;
-
- BUG_ON(ch_idx < 0 || ch_idx >= NUM_CHANNELS);
-
- if (ch_idx == CH_RX) {
- if (!polling_rate)
- free_irq(dev->client->irq, dev);
- cancel_delayed_work_sync(&dev->rx.dwork);
-
- while (!list_empty(&dev->rx.list)) {
- mbo = list_first_mbo(&dev->rx.list);
- list_del(&mbo->list);
-
- mbo->processed_length = 0;
- mbo->status = MBO_E_CLOSE;
- mbo->complete(mbo);
- }
- }
-
- return 0;
-}
-
-static void do_rx_work(struct hdm_i2c *dev)
-{
- struct mbo *mbo;
- unsigned char msg[MAX_BUF_SIZE_CONTROL];
- int ret;
- u16 pml, data_size;
-
- /* Read PML (2 bytes) */
- ret = i2c_master_recv(dev->client, msg, 2);
- if (ret <= 0) {
- pr_err("Failed to receive PML\n");
- return;
- }
-
- pml = (msg[0] << 8) | msg[1];
- if (!pml)
- return;
-
- data_size = pml + 2;
-
- /* Read the whole message, including PML */
- ret = i2c_master_recv(dev->client, msg, data_size);
- if (ret <= 0) {
- pr_err("Failed to receive a Port Message\n");
- return;
- }
-
- mbo = list_first_mbo(&dev->rx.list);
- list_del(&mbo->list);
-
- mbo->processed_length = min(data_size, mbo->buffer_length);
- memcpy(mbo->virt_address, msg, mbo->processed_length);
- mbo->status = MBO_SUCCESS;
- mbo->complete(mbo);
-}
-
-/**
- * pending_rx_work - Read pending messages through I2C
- * @work: definition of this work item
- *
- * Invoked by the Interrupt Service Routine, most_irq_handler()
- */
-static void pending_rx_work(struct work_struct *work)
-{
- struct hdm_i2c *dev = container_of(work, struct hdm_i2c, rx.dwork.work);
-
- if (list_empty(&dev->rx.list))
- return;
-
- do_rx_work(dev);
-
- if (polling_rate) {
- schedule_delayed_work(&dev->rx.dwork, dev->rx.delay);
- } else {
- dev->rx.int_disabled = false;
- enable_irq(dev->client->irq);
- }
-}
-
-/*
- * most_irq_handler - Interrupt Service Routine
- * @irq: irq number
- * @_dev: private data
- *
- * Schedules a delayed work
- *
- * By default the interrupt line behavior is Active Low. Once an interrupt is
- * generated by the device, until driver clears the interrupt (by reading
- * the PMP message), device keeps the interrupt line in low state. Since i2c
- * read is done in work queue, the interrupt line must be disabled temporarily
- * to avoid ISR being called repeatedly. Re-enable the interrupt in workqueue,
- * after reading the message.
- *
- * Note: If we use the interrupt line in Falling edge mode, there is a
- * possibility to miss interrupts when ISR is getting executed.
- *
- */
-static irqreturn_t most_irq_handler(int irq, void *_dev)
-{
- struct hdm_i2c *dev = _dev;
-
- disable_irq_nosync(irq);
- dev->rx.int_disabled = true;
- schedule_delayed_work(&dev->rx.dwork, 0);
-
- return IRQ_HANDLED;
-}
-
-/*
- * i2c_probe - i2c probe handler
- * @client: i2c client device structure
- * @id: i2c client device id
- *
- * Return 0 on success, negative on failure.
- *
- * Register the i2c client device as a MOST interface
- */
-static int i2c_probe(struct i2c_client *client, const struct i2c_device_id *id)
-{
- struct hdm_i2c *dev;
- int ret, i;
-
- dev = kzalloc(sizeof(*dev), GFP_KERNEL);
- if (!dev)
- return -ENOMEM;
-
- /* ID format: i2c-<bus>-<address> */
- snprintf(dev->name, sizeof(dev->name), "i2c-%d-%04x",
- client->adapter->nr, client->addr);
-
- for (i = 0; i < NUM_CHANNELS; i++) {
- dev->capabilities[i].data_type = MOST_CH_CONTROL;
- dev->capabilities[i].num_buffers_packet = MAX_BUFFERS_CONTROL;
- dev->capabilities[i].buffer_size_packet = MAX_BUF_SIZE_CONTROL;
- }
- dev->capabilities[CH_RX].direction = MOST_CH_RX;
- dev->capabilities[CH_RX].name_suffix = "rx";
- dev->capabilities[CH_TX].direction = MOST_CH_TX;
- dev->capabilities[CH_TX].name_suffix = "tx";
-
- dev->most_iface.interface = ITYPE_I2C;
- dev->most_iface.description = dev->name;
- dev->most_iface.num_channels = NUM_CHANNELS;
- dev->most_iface.channel_vector = dev->capabilities;
- dev->most_iface.configure = configure_channel;
- dev->most_iface.enqueue = enqueue;
- dev->most_iface.poison_channel = poison_channel;
-
- INIT_LIST_HEAD(&dev->rx.list);
-
- INIT_DELAYED_WORK(&dev->rx.dwork, pending_rx_work);
-
- dev->client = client;
- i2c_set_clientdata(client, dev);
-
- ret = most_register_interface(&dev->most_iface);
- if (ret) {
- pr_err("Failed to register i2c as a MOST interface\n");
- kfree(dev);
- return ret;
- }
-
- return 0;
-}
-
-/*
- * i2c_remove - i2c remove handler
- * @client: i2c client device structure
- *
- * Return 0 on success.
- *
- * Unregister the i2c client device as a MOST interface
- */
-static void i2c_remove(struct i2c_client *client)
-{
- struct hdm_i2c *dev = i2c_get_clientdata(client);
-
- most_deregister_interface(&dev->most_iface);
- kfree(dev);
-}
-
-static const struct i2c_device_id i2c_id[] = {
- { "most_i2c", 0 },
- { }, /* Terminating entry */
-};
-
-MODULE_DEVICE_TABLE(i2c, i2c_id);
-
-static struct i2c_driver i2c_driver = {
- .driver = {
- .name = "hdm_i2c",
- },
- .probe = i2c_probe,
- .remove = i2c_remove,
- .id_table = i2c_id,
-};
-
-module_i2c_driver(i2c_driver);
-
-MODULE_AUTHOR("Andrey Shvetsov <andrey.shvetsov@k2l.de>");
-MODULE_DESCRIPTION("I2C Hardware Dependent Module");
-MODULE_LICENSE("GPL");
diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme.c b/drivers/staging/rtl8723bs/core/rtw_mlme.c
index d7fdac14e73d..258972f6de3f 100644
--- a/drivers/staging/rtl8723bs/core/rtw_mlme.c
+++ b/drivers/staging/rtl8723bs/core/rtw_mlme.c
@@ -2025,7 +2025,10 @@ int rtw_restruct_wmm_ie(struct adapter *adapter, u8 *in_ie, u8 *out_ie, uint in_
break;
}
- i += (in_ie[i+1]+2); /* to the next IE element */
+ if (i + 1 >= in_len)
+ break;
+
+ i += (in_ie[i + 1] + 2); /* to the next IE element */
}
return ielength;
diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c b/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c
index 78c9425cd5f4..32fd3cd0a5f7 100644
--- a/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c
+++ b/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c
@@ -2261,8 +2261,11 @@ void issue_beacon(struct adapter *padapter, int timeout_ms)
spin_unlock_bh(&pmlmepriv->bcn_update_lock);
- if ((pattrib->pktlen + TXDESC_SIZE) > 512)
+ if ((pattrib->pktlen + TXDESC_SIZE) > 512) {
+ rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf);
+ rtw_free_xmitframe(pxmitpriv, pmgntframe);
return;
+ }
pattrib->last_txcmdsz = pattrib->pktlen;
@@ -2325,8 +2328,11 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p
pframe += pattrib->hdrlen;
- if (cur_network->ie_length > MAX_IE_SZ)
+ if (cur_network->ie_length > MAX_IE_SZ) {
+ rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf);
+ rtw_free_xmitframe(pxmitpriv, pmgntframe);
return;
+ }
if ((pmlmeinfo->state&0x03) == WIFI_FW_AP_STATE) {
pwps_ie = rtw_get_wps_ie(cur_network->ies+_FIXED_IE_LENGTH_, cur_network->ie_length-_FIXED_IE_LENGTH_, NULL, &wps_ielen);
@@ -2373,8 +2379,11 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p
u8 *ies = pmgntframe->buf_addr+TXDESC_OFFSET+sizeof(struct ieee80211_hdr_3addr);
buf = rtw_zmalloc(MAX_IE_SZ);
- if (!buf)
+ if (!buf) {
+ rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf);
+ rtw_free_xmitframe(pxmitpriv, pmgntframe);
return;
+ }
ssid_ie = rtw_get_ie(ies+_FIXED_IE_LENGTH_, WLAN_EID_SSID, &ssid_ielen,
(pframe-ies)-_FIXED_IE_LENGTH_);
@@ -2767,10 +2776,13 @@ void issue_asocrsp(struct adapter *padapter, unsigned short status, struct sta_i
SetSeqNum(pwlanhdr, pmlmeext->mgnt_seq);
pmlmeext->mgnt_seq++;
- if ((pkt_type == WIFI_ASSOCRSP) || (pkt_type == WIFI_REASSOCRSP))
+ if ((pkt_type == WIFI_ASSOCRSP) || (pkt_type == WIFI_REASSOCRSP)) {
SetFrameSubType(pwlanhdr, pkt_type);
- else
+ } else {
+ rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf);
+ rtw_free_xmitframe(pxmitpriv, pmgntframe);
return;
+ }
pattrib->hdrlen = sizeof(struct ieee80211_hdr_3addr);
pattrib->pktlen += pattrib->hdrlen;
diff --git a/drivers/staging/sm750fb/sm750.c b/drivers/staging/sm750fb/sm750.c
index 758759b706d4..ad41ba284a3a 100644
--- a/drivers/staging/sm750fb/sm750.c
+++ b/drivers/staging/sm750fb/sm750.c
@@ -276,7 +276,7 @@ static void lynxfb_ops_imageblit(struct fb_info *info,
spin_lock(&sm750_dev->slock);
sm750_dev->accel.de_imageblit(&sm750_dev->accel,
- image->data, image->width >> 3, 0,
+ image->data, 0,
base, pitch, Bpp,
image->dx, image->dy,
image->width, image->height,
diff --git a/drivers/staging/sm750fb/sm750.h b/drivers/staging/sm750fb/sm750.h
index aff69661c8e6..a693d9f1d0d5 100644
--- a/drivers/staging/sm750fb/sm750.h
+++ b/drivers/staging/sm750fb/sm750.h
@@ -73,7 +73,7 @@ struct lynx_accel {
u32 rop2);
int (*de_imageblit)(struct lynx_accel *accel, const char *p_srcbuf,
- u32 src_delta, u32 start_bit, u32 d_base, u32 d_pitch,
+ u32 start_bit, u32 d_base, u32 d_pitch,
u32 byte_per_pixel, u32 dx, u32 dy, u32 width,
u32 height, u32 f_color, u32 b_color, u32 rop2);
diff --git a/drivers/staging/sm750fb/sm750_accel.c b/drivers/staging/sm750fb/sm750_accel.c
index 24b9077a634a..eb47c18a585d 100644
--- a/drivers/staging/sm750fb/sm750_accel.c
+++ b/drivers/staging/sm750fb/sm750_accel.c
@@ -301,8 +301,6 @@ static unsigned int deGetTransparency(struct lynx_accel *accel)
* sm750_hw_imageblit
* @accel: Acceleration device data
* @pSrcbuf: pointer to start of source buffer in system memory
- * @srcDelta: Pitch value (in bytes) of the source buffer, +ive means top down
- * and -ive mean button up
* @startBit: Mono data can start at any bit in a byte, this value should be
* 0 to 7
* @dBase: Address of destination: offset in frame buffer
@@ -317,7 +315,7 @@ static unsigned int deGetTransparency(struct lynx_accel *accel)
* @rop2: ROP value
*/
int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf,
- u32 srcDelta, u32 startBit, u32 dBase, u32 dPitch,
+ u32 startBit, u32 dBase, u32 dPitch,
u32 bytePerPixel, u32 dx, u32 dy, u32 width,
u32 height, u32 fColor, u32 bColor, u32 rop2)
{
@@ -406,7 +404,7 @@ int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf,
write_dpPort(accel, *(unsigned int *)ajRemain);
}
- pSrcbuf += srcDelta;
+ pSrcbuf += ulBytesPerScan;
}
return 0;
diff --git a/drivers/staging/sm750fb/sm750_accel.h b/drivers/staging/sm750fb/sm750_accel.h
index 2c79cb730a0a..df48ed57a054 100644
--- a/drivers/staging/sm750fb/sm750_accel.h
+++ b/drivers/staging/sm750fb/sm750_accel.h
@@ -220,8 +220,6 @@ int sm750_hw_copyarea(struct lynx_accel *accel,
/**
* sm750_hw_imageblit
* @pSrcbuf: pointer to start of source buffer in system memory
- * @srcDelta: Pitch value (in bytes) of the source buffer, +ive means top down
- *>----- and -ive mean button up
* @startBit: Mono data can start at any bit in a byte, this value should be
*>----- 0 to 7
* @dBase: Address of destination: offset in frame buffer
@@ -236,7 +234,7 @@ int sm750_hw_copyarea(struct lynx_accel *accel,
* @rop2: ROP value
*/
int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf,
- u32 srcDelta, u32 startBit, u32 dBase, u32 dPitch,
+ u32 startBit, u32 dBase, u32 dPitch,
u32 bytePerPixel, u32 dx, u32 dy, u32 width,
u32 height, u32 fColor, u32 bColor, u32 rop2);
diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c
index c142a67dc7cc..1210975624df 100644
--- a/drivers/target/iscsi/iscsi_target_configfs.c
+++ b/drivers/target/iscsi/iscsi_target_configfs.c
@@ -1308,17 +1308,24 @@ static ssize_t iscsi_disc_enforce_discovery_auth_store(struct config_item *item,
return -EINVAL;
}
+ if (iscsit_get_tpg(discovery_tpg) < 0)
+ return -EINVAL;
+
param = iscsi_find_param_from_key(AUTHMETHOD,
discovery_tpg->param_list);
- if (!param)
- return -EINVAL;
+ if (!param) {
+ err = -EINVAL;
+ goto out;
+ }
if (op) {
/*
* Reset the AuthMethod key to CHAP.
*/
- if (iscsi_update_param_value(param, CHAP) < 0)
- return -EINVAL;
+ if (iscsi_update_param_value(param, CHAP) < 0) {
+ err = -EINVAL;
+ goto out;
+ }
discovery_tpg->tpg_attrib.authentication = 1;
iscsit_global->discovery_acl.node_auth.enforce_discovery_auth = 1;
@@ -1329,8 +1336,10 @@ static ssize_t iscsi_disc_enforce_discovery_auth_store(struct config_item *item,
/*
* Reset the AuthMethod key to CHAP,None
*/
- if (iscsi_update_param_value(param, "CHAP,None") < 0)
- return -EINVAL;
+ if (iscsi_update_param_value(param, "CHAP,None") < 0) {
+ err = -EINVAL;
+ goto out;
+ }
discovery_tpg->tpg_attrib.authentication = 0;
iscsit_global->discovery_acl.node_auth.enforce_discovery_auth = 0;
@@ -1339,7 +1348,10 @@ static ssize_t iscsi_disc_enforce_discovery_auth_store(struct config_item *item,
" Discovery TPG\n");
}
- return count;
+ err = count;
+out:
+ iscsit_put_tpg(discovery_tpg);
+ return err;
}
CONFIGFS_ATTR(iscsi_disc_, enforce_discovery_auth);
diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c
index 927c3d7947f9..f0ecd931035e 100644
--- a/drivers/tee/optee/ffa_abi.c
+++ b/drivers/tee/optee/ffa_abi.c
@@ -844,14 +844,6 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
}
optee->supp_teedev = teedev;
- rc = tee_device_register(optee->teedev);
- if (rc)
- goto err_unreg_supp_teedev;
-
- rc = tee_device_register(optee->supp_teedev);
- if (rc)
- goto err_unreg_supp_teedev;
-
rc = rhashtable_init(&optee->ffa.global_ids, &shm_rhash_params);
if (rc)
goto err_unreg_supp_teedev;
@@ -871,6 +863,14 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev)
if (rc)
goto err_close_ctx;
+ rc = tee_device_register(optee->teedev);
+ if (rc)
+ goto err_unregister_devices;
+
+ rc = tee_device_register(optee->supp_teedev);
+ if (rc)
+ goto err_unregister_devices;
+
rc = optee_enumerate_devices(PTA_CMD_GET_DEVICES);
if (rc)
goto err_unregister_devices;
diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c
index e6e0428f8e7b..89ab5323f9fe 100644
--- a/drivers/tee/optee/smc_abi.c
+++ b/drivers/tee/optee/smc_abi.c
@@ -1467,14 +1467,6 @@ static int optee_probe(struct platform_device *pdev)
}
optee->supp_teedev = teedev;
- rc = tee_device_register(optee->teedev);
- if (rc)
- goto err_unreg_supp_teedev;
-
- rc = tee_device_register(optee->supp_teedev);
- if (rc)
- goto err_unreg_supp_teedev;
-
mutex_init(&optee->call_queue.mutex);
INIT_LIST_HEAD(&optee->call_queue.waiters);
optee_supp_init(&optee->supp);
@@ -1531,6 +1523,14 @@ static int optee_probe(struct platform_device *pdev)
if (optee->smc.sec_caps & OPTEE_SMC_SEC_CAP_DYNAMIC_SHM)
pr_info("dynamic shared memory is enabled\n");
+ rc = tee_device_register(optee->teedev);
+ if (rc)
+ goto err_disable_shm_cache;
+
+ rc = tee_device_register(optee->supp_teedev);
+ if (rc)
+ goto err_disable_shm_cache;
+
rc = optee_enumerate_devices(PTA_CMD_GET_DEVICES);
if (rc)
goto err_disable_shm_cache;
@@ -1551,7 +1551,6 @@ static int optee_probe(struct platform_device *pdev)
optee_shm_arg_cache_uninit(optee);
optee_supp_uninit(&optee->supp);
mutex_destroy(&optee->call_queue.mutex);
-err_unreg_supp_teedev:
tee_device_unregister(optee->supp_teedev);
err_unreg_teedev:
tee_device_unregister(optee->teedev);
diff --git a/drivers/thermal/thermal_hwmon.c b/drivers/thermal/thermal_hwmon.c
index 1013e0221e37..8edc59454d62 100644
--- a/drivers/thermal/thermal_hwmon.c
+++ b/drivers/thermal/thermal_hwmon.c
@@ -92,34 +92,12 @@ thermal_hwmon_lookup_by_type(const struct thermal_zone_device *tz)
struct thermal_hwmon_device *hwmon;
char type[THERMAL_NAME_LENGTH];
- mutex_lock(&thermal_hwmon_list_lock);
list_for_each_entry(hwmon, &thermal_hwmon_list, node) {
strcpy(type, tz->type);
strreplace(type, '-', '_');
- if (!strcmp(hwmon->type, type)) {
- mutex_unlock(&thermal_hwmon_list_lock);
+ if (!strcmp(hwmon->type, type))
return hwmon;
- }
}
- mutex_unlock(&thermal_hwmon_list_lock);
-
- return NULL;
-}
-
-/* Find the temperature input matching a given thermal zone */
-static struct thermal_hwmon_temp *
-thermal_hwmon_lookup_temp(const struct thermal_hwmon_device *hwmon,
- const struct thermal_zone_device *tz)
-{
- struct thermal_hwmon_temp *temp;
-
- mutex_lock(&thermal_hwmon_list_lock);
- list_for_each_entry(temp, &hwmon->tz_list, hwmon_node)
- if (temp->tz == tz) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return temp;
- }
- mutex_unlock(&thermal_hwmon_list_lock);
return NULL;
}
@@ -135,7 +113,9 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
struct thermal_hwmon_device *hwmon;
struct thermal_hwmon_temp *temp;
int new_hwmon_device = 1;
- int result;
+ int result = 0;
+
+ mutex_lock(&thermal_hwmon_list_lock);
hwmon = thermal_hwmon_lookup_by_type(tz);
if (hwmon) {
@@ -144,8 +124,10 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
}
hwmon = kzalloc(sizeof(*hwmon), GFP_KERNEL);
- if (!hwmon)
- return -ENOMEM;
+ if (!hwmon) {
+ result = -ENOMEM;
+ goto unlock;
+ }
INIT_LIST_HEAD(&hwmon->tz_list);
strscpy(hwmon->type, tz->type, THERMAL_NAME_LENGTH);
@@ -193,24 +175,24 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz)
temp->temp_crit_present = true;
}
- mutex_lock(&thermal_hwmon_list_lock);
if (new_hwmon_device)
list_add_tail(&hwmon->node, &thermal_hwmon_list);
list_add_tail(&temp->hwmon_node, &hwmon->tz_list);
- mutex_unlock(&thermal_hwmon_list_lock);
- return 0;
+ goto unlock;
- unregister_input:
+unregister_input:
device_remove_file(hwmon->device, &temp->temp_input.attr);
- free_temp_mem:
+free_temp_mem:
kfree(temp);
- unregister_name:
+unregister_name:
if (new_hwmon_device)
hwmon_device_unregister(hwmon->device);
- free_mem:
+free_mem:
if (new_hwmon_device)
kfree(hwmon);
+unlock:
+ mutex_unlock(&thermal_hwmon_list_lock);
return result;
}
@@ -218,8 +200,11 @@ EXPORT_SYMBOL_GPL(thermal_add_hwmon_sysfs);
void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz)
{
+ struct thermal_hwmon_temp *temp, *entry;
struct thermal_hwmon_device *hwmon;
- struct thermal_hwmon_temp *temp;
+ bool unregister;
+
+ guard(mutex)(&thermal_hwmon_list_lock);
hwmon = thermal_hwmon_lookup_by_type(tz);
if (unlikely(!hwmon)) {
@@ -228,29 +213,25 @@ void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz)
return;
}
- temp = thermal_hwmon_lookup_temp(hwmon, tz);
- if (unlikely(!temp)) {
- /* Should never happen... */
- dev_dbg(&tz->device, "temperature input lookup failed!\n");
- return;
- }
+ unregister = hwmon->device->parent == &tz->device;
- device_remove_file(hwmon->device, &temp->temp_input.attr);
- if (temp->temp_crit_present)
- device_remove_file(hwmon->device, &temp->temp_crit.attr);
+ list_for_each_entry_safe_reverse(temp, entry, &hwmon->tz_list, hwmon_node) {
+ if (!unregister && temp->tz != tz)
+ continue;
- mutex_lock(&thermal_hwmon_list_lock);
- list_del(&temp->hwmon_node);
- kfree(temp);
- if (!list_empty(&hwmon->tz_list)) {
- mutex_unlock(&thermal_hwmon_list_lock);
- return;
+ device_remove_file(hwmon->device, &temp->temp_input.attr);
+ if (temp->temp_crit_present)
+ device_remove_file(hwmon->device, &temp->temp_crit.attr);
+
+ list_del(&temp->hwmon_node);
+ kfree(temp);
}
- list_del(&hwmon->node);
- mutex_unlock(&thermal_hwmon_list_lock);
- hwmon_device_unregister(hwmon->device);
- kfree(hwmon);
+ if (unregister) {
+ list_del(&hwmon->node);
+ hwmon_device_unregister(hwmon->device);
+ kfree(hwmon);
+ }
}
EXPORT_SYMBOL_GPL(thermal_remove_hwmon_sysfs);
diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c
index 62c6c9325243..fa8115a145f6 100644
--- a/drivers/thunderbolt/ctl.c
+++ b/drivers/thunderbolt/ctl.c
@@ -65,7 +65,6 @@ struct tb_ctl {
#define tb_ctl_dbg(ctl, format, arg...) \
dev_dbg(&(ctl)->nhi->pdev->dev, format, ## arg)
-static DECLARE_WAIT_QUEUE_HEAD(tb_cfg_request_cancel_queue);
/* Serializes access to request kref_get/put */
static DEFINE_MUTEX(tb_cfg_request_lock);
@@ -123,41 +122,42 @@ void tb_cfg_request_put(struct tb_cfg_request *req)
static int tb_cfg_request_enqueue(struct tb_ctl *ctl,
struct tb_cfg_request *req)
{
+ tb_cfg_request_get(req);
+
WARN_ON(test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags));
WARN_ON(req->ctl);
- mutex_lock(&ctl->request_queue_lock);
+ guard(mutex)(&ctl->request_queue_lock);
if (!ctl->running) {
- mutex_unlock(&ctl->request_queue_lock);
+ tb_cfg_request_put(req);
return -ENOTCONN;
}
req->ctl = ctl;
list_add_tail(&req->list, &ctl->request_queue);
set_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
- mutex_unlock(&ctl->request_queue_lock);
return 0;
}
-static void tb_cfg_request_dequeue(struct tb_cfg_request *req)
+static bool tb_cfg_request_is_active(struct tb_cfg_request *req)
{
- struct tb_ctl *ctl = req->ctl;
-
- mutex_lock(&ctl->request_queue_lock);
- if (!test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags)) {
- mutex_unlock(&ctl->request_queue_lock);
- return;
- }
+ return test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+}
- list_del(&req->list);
- clear_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
- if (test_bit(TB_CFG_REQUEST_CANCELED, &req->flags))
- wake_up(&tb_cfg_request_cancel_queue);
- mutex_unlock(&ctl->request_queue_lock);
+static bool tb_cfg_request_is_canceled(struct tb_cfg_request *req)
+{
+ return test_bit(TB_CFG_REQUEST_CANCELED, &req->flags);
}
-static bool tb_cfg_request_is_active(struct tb_cfg_request *req)
+static void tb_cfg_request_dequeue(struct tb_cfg_request *req)
{
- return test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+ struct tb_ctl *ctl = req->ctl;
+
+ guard(mutex)(&ctl->request_queue_lock);
+ if (tb_cfg_request_is_active(req)) {
+ list_del(&req->list);
+ clear_bit(TB_CFG_REQUEST_ACTIVE, &req->flags);
+ tb_cfg_request_put(req);
+ }
}
static struct tb_cfg_request *
@@ -168,7 +168,7 @@ tb_cfg_request_find(struct tb_ctl *ctl, struct ctl_pkg *pkg)
mutex_lock(&pkg->ctl->request_queue_lock);
list_for_each_entry(iter, &pkg->ctl->request_queue, list) {
tb_cfg_request_get(iter);
- if (iter->match(iter, pkg)) {
+ if (!tb_cfg_request_is_canceled(iter) && iter->match(iter, pkg)) {
req = iter;
break;
}
@@ -495,8 +495,11 @@ static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame,
*/
req = tb_cfg_request_find(pkg->ctl, pkg);
if (req) {
- if (req->copy(req, pkg))
- schedule_work(&req->work);
+ scoped_guard(mutex, &pkg->ctl->request_queue_lock) {
+ if (!tb_cfg_request_is_canceled(req) &&
+ req->copy(req, pkg))
+ schedule_work(&req->work);
+ }
tb_cfg_request_put(req);
}
@@ -508,11 +511,10 @@ static void tb_cfg_request_work(struct work_struct *work)
{
struct tb_cfg_request *req = container_of(work, typeof(*req), work);
- if (!test_bit(TB_CFG_REQUEST_CANCELED, &req->flags))
+ if (!tb_cfg_request_is_canceled(req))
req->callback(req->callback_data);
tb_cfg_request_dequeue(req);
- tb_cfg_request_put(req);
}
/**
@@ -536,10 +538,9 @@ int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req,
INIT_WORK(&req->work, tb_cfg_request_work);
INIT_LIST_HEAD(&req->list);
- tb_cfg_request_get(req);
ret = tb_cfg_request_enqueue(ctl, req);
if (ret)
- goto err_put;
+ return ret;
ret = tb_ctl_tx(ctl, req->request, req->request_size,
req->request_type);
@@ -553,9 +554,6 @@ int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req,
err_dequeue:
tb_cfg_request_dequeue(req);
-err_put:
- tb_cfg_request_put(req);
-
return ret;
}
@@ -569,9 +567,10 @@ int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req,
*/
void tb_cfg_request_cancel(struct tb_cfg_request *req, int err)
{
- set_bit(TB_CFG_REQUEST_CANCELED, &req->flags);
- schedule_work(&req->work);
- wait_event(tb_cfg_request_cancel_queue, !tb_cfg_request_is_active(req));
+ scoped_guard(mutex, &req->ctl->request_queue_lock)
+ set_bit(TB_CFG_REQUEST_CANCELED, &req->flags);
+ cancel_work_sync(&req->work);
+ tb_cfg_request_dequeue(req);
req->result.err = err;
}
diff --git a/drivers/thunderbolt/quirks.c b/drivers/thunderbolt/quirks.c
index e81de9c30eac..520afebebd42 100644
--- a/drivers/thunderbolt/quirks.c
+++ b/drivers/thunderbolt/quirks.c
@@ -92,6 +92,9 @@ static const struct tb_quirk tb_quirks[] = {
quirk_usb3_maximum_bandwidth },
{ 0x8087, PCI_DEVICE_ID_INTEL_BARLOW_RIDGE_HUB_40G_BRIDGE, 0x0000, 0x0000,
quirk_usb3_maximum_bandwidth },
+ /* CLx is unstable on the Anker Prime TB5 dock (DROM 01ea:83b5) */
+ { 0x8087, PCI_DEVICE_ID_INTEL_BARLOW_RIDGE_HUB_80G_BRIDGE, 0x01ea, 0x83b5,
+ quirk_clx_disable },
/*
* Block Runtime PM in DP redrive mode for Intel Barlow Ridge host
* controllers.
diff --git a/drivers/thunderbolt/xdomain.c b/drivers/thunderbolt/xdomain.c
index 6dac354de4bb..98bd2d859037 100644
--- a/drivers/thunderbolt/xdomain.c
+++ b/drivers/thunderbolt/xdomain.c
@@ -377,6 +377,10 @@ static int tb_xdp_properties_request(struct tb_ctl *ctl, u64 route,
len += sizeof(res->hdr.xd_hdr) / 4;
len -= sizeof(*res) / 4;
+ if (len > TB_XDP_PROPERTIES_MAX_DATA_LENGTH) {
+ ret = -EINVAL;
+ goto err;
+ }
if (res->offset != req.offset) {
ret = -EINVAL;
diff --git a/drivers/tty/amiserial.c b/drivers/tty/amiserial.c
index 50a8c6d987d5..b893ea9fe264 100644
--- a/drivers/tty/amiserial.c
+++ b/drivers/tty/amiserial.c
@@ -1140,11 +1140,6 @@ static int rs_ioctl(struct tty_struct *tty,
local_irq_save(flags);
cnow = info->icount; /* atomic copy */
local_irq_restore(flags);
- if (cnow.rng == cprev.rng && cnow.dsr == cprev.dsr &&
- cnow.dcd == cprev.dcd && cnow.cts == cprev.cts) {
- ret = -EIO; /* no change => error */
- break;
- }
if ( ((arg & TIOCM_RNG) && (cnow.rng != cprev.rng)) ||
((arg & TIOCM_DSR) && (cnow.dsr != cprev.dsr)) ||
((arg & TIOCM_CD) && (cnow.dcd != cprev.dcd)) ||
@@ -1152,6 +1147,10 @@ static int rs_ioctl(struct tty_struct *tty,
ret = 0;
break;
}
+ if (tty_io_error(tty)) {
+ ret = -EIO;
+ break;
+ }
schedule();
/* see if a signal did it */
if (signal_pending(current)) {
diff --git a/drivers/tty/n_gsm.c b/drivers/tty/n_gsm.c
index fc52a7a38df3..db9382b57300 100644
--- a/drivers/tty/n_gsm.c
+++ b/drivers/tty/n_gsm.c
@@ -40,6 +40,7 @@
#include <linux/tty.h>
#include <linux/ctype.h>
#include <linux/mm.h>
+#include <linux/math.h>
#include <linux/string.h>
#include <linux/slab.h>
#include <linux/poll.h>
@@ -75,7 +76,12 @@ module_param(debug, int, 0600);
#define T1 10 /* 100mS */
#define T2 34 /* 333mS */
+#define T3 10 /* 10s */
#define N2 3 /* Retry 3 times */
+#define K 2 /* outstanding I frames */
+
+#define MAX_T3 255 /* In seconds. */
+#define MAX_WINDOW_SIZE 7 /* Limit of K in error recovery mode. */
/* Use long timers for testing at low speed with debug on */
#ifdef DEBUG_TIMING
@@ -89,6 +95,7 @@ module_param(debug, int, 0600);
*/
#define MAX_MRU 1500
#define MAX_MTU 1500
+#define MIN_MTU (PROT_OVERHEAD + 1)
/* SOF, ADDR, CTRL, LEN1, LEN2, ..., FCS, EOF */
#define PROT_OVERHEAD 7
#define GSM_NET_TX_TIMEOUT (HZ*10)
@@ -159,7 +166,12 @@ struct gsm_dlci {
int prev_adaption;
u32 modem_rx; /* Our incoming virtual modem lines */
u32 modem_tx; /* Our outgoing modem lines */
+ unsigned int mtu;
bool dead; /* Refuse re-open */
+ /* Configuration */
+ u8 prio; /* Priority */
+ u8 ftype; /* Frame type */
+ u8 k; /* Window size */
/* Flow control */
bool throttled; /* Private copy of throttle state */
bool constipated; /* Throttle status for outgoing */
@@ -284,7 +296,9 @@ struct gsm_mux {
int adaption; /* 1 or 2 supported */
u8 ftype; /* UI or UIH */
int t1, t2; /* Timers in 1/100th of a sec */
+ unsigned int t3; /* Power wake-up timer in seconds. */
int n2; /* Retry count */
+ u8 k; /* Window size */
/* Statistics (not currently exposed) */
unsigned long bad_fcs;
@@ -406,6 +420,7 @@ static struct gsm_msg *gsm_data_alloc(struct gsm_mux *gsm, u8 addr, int len,
static int gsm_send_packet(struct gsm_mux *gsm, struct gsm_msg *msg);
static void gsmld_write_trigger(struct gsm_mux *gsm);
static void gsmld_write_task(struct work_struct *work);
+static int gsm_modem_send_initial_msc(struct gsm_dlci *dlci);
/**
* gsm_fcs_add - update FCS
@@ -1078,12 +1093,12 @@ static int gsm_dlci_data_output(struct gsm_mux *gsm, struct gsm_dlci *dlci)
return 0;
/* MTU/MRU count only the data bits but watch adaption mode */
- if ((len + h) > gsm->mtu)
- len = gsm->mtu - h;
+ if ((len + h) > dlci->mtu)
+ len = dlci->mtu - h;
size = len + h;
- msg = gsm_data_alloc(gsm, dlci->addr, size, gsm->ftype);
+ msg = gsm_data_alloc(gsm, dlci->addr, size, dlci->ftype);
if (!msg)
return -ENOMEM;
dp = msg->data;
@@ -1147,19 +1162,19 @@ static int gsm_dlci_data_output_framed(struct gsm_mux *gsm,
len = dlci->skb->len + overhead;
/* MTU/MRU count only the data bits */
- if (len > gsm->mtu) {
+ if (len > dlci->mtu) {
if (dlci->adaption == 3) {
/* Over long frame, bin it */
dev_kfree_skb_any(dlci->skb);
dlci->skb = NULL;
return 0;
}
- len = gsm->mtu;
+ len = dlci->mtu;
} else
last = 1;
size = len + overhead;
- msg = gsm_data_alloc(gsm, dlci->addr, size, gsm->ftype);
+ msg = gsm_data_alloc(gsm, dlci->addr, size, dlci->ftype);
if (msg == NULL) {
skb_queue_tail(&dlci->skb_list, dlci->skb);
dlci->skb = NULL;
@@ -1216,7 +1231,7 @@ static int gsm_dlci_modem_output(struct gsm_mux *gsm, struct gsm_dlci *dlci,
return -EINVAL;
}
- msg = gsm_data_alloc(gsm, dlci->addr, size, gsm->ftype);
+ msg = gsm_data_alloc(gsm, dlci->addr, size, dlci->ftype);
if (!msg) {
pr_err("%s: gsm_data_alloc error", __func__);
return -ENOMEM;
@@ -1342,8 +1357,9 @@ static void gsm_dlci_data_kick(struct gsm_dlci *dlci)
static int gsm_control_command(struct gsm_mux *gsm, int cmd, const u8 *data,
int dlen)
{
- struct gsm_msg *msg = gsm_data_alloc(gsm, 0, dlen + 2, gsm->ftype);
+ struct gsm_msg *msg;
+ msg = gsm_data_alloc(gsm, 0, dlen + 2, gsm->dlci[0]->ftype);
if (msg == NULL)
return -ENOMEM;
@@ -1369,7 +1385,8 @@ static void gsm_control_reply(struct gsm_mux *gsm, int cmd, const u8 *data,
int dlen)
{
struct gsm_msg *msg;
- msg = gsm_data_alloc(gsm, 0, dlen + 2, gsm->ftype);
+
+ msg = gsm_data_alloc(gsm, 0, dlen + 2, gsm->dlci[0]->ftype);
if (msg == NULL)
return;
msg->data[0] = (cmd & 0xFE) << 1 | EA; /* Clear C/R */
@@ -1818,7 +1835,7 @@ static void gsm_dlci_open(struct gsm_dlci *dlci)
pr_debug("DLCI %d goes open.\n", dlci->addr);
/* Send current modem state */
if (dlci->addr)
- gsm_modem_update(dlci, 0);
+ gsm_modem_send_initial_msc(dlci);
gsm_dlci_data_kick(dlci);
wake_up(&dlci->gsm->event);
}
@@ -2080,6 +2097,13 @@ static struct gsm_dlci *gsm_dlci_alloc(struct gsm_mux *gsm, int addr)
dlci->gsm = gsm;
dlci->addr = addr;
dlci->adaption = gsm->adaption;
+ dlci->mtu = gsm->mtu;
+ if (addr == 0)
+ dlci->prio = 0;
+ else
+ dlci->prio = roundup(addr + 1, 8) - 1;
+ dlci->ftype = gsm->ftype;
+ dlci->k = gsm->k;
dlci->state = DLCI_CLOSED;
if (addr) {
dlci->data = gsm_dlci_data;
@@ -2709,7 +2733,9 @@ static struct gsm_mux *gsm_alloc_mux(void)
gsm->t1 = T1;
gsm->t2 = T2;
+ gsm->t3 = T3;
gsm->n2 = N2;
+ gsm->k = K;
gsm->ftype = UIH;
gsm->adaption = 1;
gsm->encoding = GSM_ADV_OPT;
@@ -2749,7 +2775,7 @@ static void gsm_copy_config_values(struct gsm_mux *gsm,
c->initiator = gsm->initiator;
c->t1 = gsm->t1;
c->t2 = gsm->t2;
- c->t3 = 0; /* Not supported */
+ c->t3 = gsm->t3;
c->n2 = gsm->n2;
if (gsm->ftype == UIH)
c->i = 1;
@@ -2758,7 +2784,7 @@ static void gsm_copy_config_values(struct gsm_mux *gsm,
pr_debug("Ftype %d i %d\n", gsm->ftype, c->i);
c->mru = gsm->mru;
c->mtu = gsm->mtu;
- c->k = 0;
+ c->k = gsm->k;
}
static int gsm_config(struct gsm_mux *gsm, struct gsm_config *c)
@@ -2771,7 +2797,11 @@ static int gsm_config(struct gsm_mux *gsm, struct gsm_config *c)
if ((c->adaption != 1 && c->adaption != 2) || c->k)
return -EOPNOTSUPP;
/* Check the MRU/MTU range looks sane */
- if (c->mru > MAX_MRU || c->mtu > MAX_MTU || c->mru < 8 || c->mtu < 8)
+ if (c->mru < MIN_MTU || c->mtu < MIN_MTU)
+ return -EINVAL;
+ if (c->mru > MAX_MRU || c->mtu > MAX_MTU)
+ return -EINVAL;
+ if (c->t3 > MAX_T3)
return -EINVAL;
if (c->n2 > 255)
return -EINVAL;
@@ -2779,6 +2809,8 @@ static int gsm_config(struct gsm_mux *gsm, struct gsm_config *c)
return -EINVAL;
if (c->initiator > 1)
return -EINVAL;
+ if (c->k > MAX_WINDOW_SIZE)
+ return -EINVAL;
if (c->i == 0 || c->i > 2) /* UIH and UI only */
return -EINVAL;
/*
@@ -2826,6 +2858,10 @@ static int gsm_config(struct gsm_mux *gsm, struct gsm_config *c)
gsm->t1 = c->t1;
if (c->t2)
gsm->t2 = c->t2;
+ if (c->t3)
+ gsm->t3 = c->t3;
+ if (c->k)
+ gsm->k = c->k;
/*
* FIXME: We need to separate activation/deactivation from adding
@@ -3359,9 +3395,9 @@ static int gsm_create_network(struct gsm_dlci *dlci, struct gsm_netconfig *nc)
pr_err("alloc_netdev failed\n");
return -ENOMEM;
}
- net->mtu = dlci->gsm->mtu;
- net->min_mtu = 8;
- net->max_mtu = dlci->gsm->mtu;
+ net->mtu = dlci->mtu;
+ net->min_mtu = MIN_MTU;
+ net->max_mtu = dlci->mtu;
mux_net = netdev_priv(net);
mux_net->dlci = dlci;
kref_init(&mux_net->ref);
@@ -3455,6 +3491,28 @@ static int gsm_modem_upd_via_msc(struct gsm_dlci *dlci, u8 brk)
return gsm_control_wait(dlci->gsm, ctrl);
}
+/**
+ * gsm_modem_send_initial_msc - Send initial modem status message
+ *
+ * @dlci channel
+ *
+ * Send an initial MSC message after DLCI open to set the initial
+ * modem status lines. This is only done for basic mode.
+ * Does not wait for a response as we cannot block the input queue
+ * processing.
+ */
+static int gsm_modem_send_initial_msc(struct gsm_dlci *dlci)
+{
+ u8 modembits[2];
+
+ if (dlci->adaption != 1 || dlci->gsm->encoding != GSM_BASIC_OPT)
+ return 0;
+
+ modembits[0] = (dlci->addr << 2) | 2 | EA; /* DLCI, Valid, EA */
+ modembits[1] = (gsm_encode_modem(dlci) << 1) | EA;
+ return gsm_control_command(dlci->gsm, CMD_MSC, (const u8 *)&modembits, 2);
+}
+
/**
* gsm_modem_update - send modem status line state
* @dlci: channel
diff --git a/drivers/tty/pty.c b/drivers/tty/pty.c
index c658cacdb933..335f5744f320 100644
--- a/drivers/tty/pty.c
+++ b/drivers/tty/pty.c
@@ -931,7 +931,7 @@ static void __init unix98_pty_init(void)
if (cdev_add(&ptmx_cdev, MKDEV(TTYAUX_MAJOR, 2), 1) ||
register_chrdev_region(MKDEV(TTYAUX_MAJOR, 2), 1, "/dev/ptmx") < 0)
panic("Couldn't register /dev/ptmx driver");
- device_create(tty_class, NULL, MKDEV(TTYAUX_MAJOR, 2), NULL, "ptmx");
+ device_create(&tty_class, NULL, MKDEV(TTYAUX_MAJOR, 2), NULL, "ptmx");
}
#else
diff --git a/drivers/tty/serial/8250/8250_bcm7271.c b/drivers/tty/serial/8250/8250_bcm7271.c
index a28f115f6194..c4927ad86a86 100644
--- a/drivers/tty/serial/8250/8250_bcm7271.c
+++ b/drivers/tty/serial/8250/8250_bcm7271.c
@@ -1153,8 +1153,8 @@ static int brcmuart_remove(struct platform_device *pdev)
struct brcmuart_priv *priv = platform_get_drvdata(pdev);
debugfs_remove_recursive(priv->debugfs_dir);
- hrtimer_cancel(&priv->hrt);
serial8250_unregister_port(priv->line);
+ hrtimer_cancel(&priv->hrt);
brcmuart_free_bufs(&pdev->dev, priv);
clk_disable_unprepare(priv->baud_mux_clk);
if (priv->dma_enabled)
diff --git a/drivers/tty/serial/8250/8250_core.c b/drivers/tty/serial/8250/8250_core.c
index 81a5dab1a828..6ba421d97d30 100644
--- a/drivers/tty/serial/8250/8250_core.c
+++ b/drivers/tty/serial/8250/8250_core.c
@@ -1038,7 +1038,7 @@ int serial8250_register_8250_port(const struct uart_8250_port *up)
* Only call mctrl_gpio_init(), if the device has no ACPI
* companion device
*/
- if (!has_acpi_companion(uart->port.dev)) {
+ if (uart->port.dev && !has_acpi_companion(uart->port.dev)) {
gpios = mctrl_gpio_init(&uart->port, 0);
if (IS_ERR(gpios)) {
ret = PTR_ERR(gpios);
diff --git a/drivers/tty/serial/imx.c b/drivers/tty/serial/imx.c
index 47e59664dbde..fccba3cb9d5a 100644
--- a/drivers/tty/serial/imx.c
+++ b/drivers/tty/serial/imx.c
@@ -21,6 +21,7 @@
#include <linux/clk.h>
#include <linux/delay.h>
#include <linux/ktime.h>
+#include <linux/mutex.h>
#include <linux/pinctrl/consumer.h>
#include <linux/rational.h>
#include <linux/slab.h>
@@ -2037,6 +2038,9 @@ static const struct uart_ops imx_uart_pops = {
static struct imx_port *imx_uart_ports[UART_NR];
+/* Held across uart_add/remove_one_port(); console callbacks must not take it. */
+static DEFINE_MUTEX(imx_uart_ports_lock);
+
#if IS_ENABLED(CONFIG_SERIAL_IMX_CONSOLE)
static void imx_uart_console_putchar(struct uart_port *port, unsigned char ch)
{
@@ -2522,11 +2526,19 @@ static int imx_uart_probe(struct platform_device *pdev)
}
}
- imx_uart_ports[sport->port.line] = sport;
-
platform_set_drvdata(pdev, sport);
- ret = uart_add_one_port(&imx_uart_uart_driver, &sport->port);
+ scoped_guard(mutex, &imx_uart_ports_lock) {
+ if (imx_uart_ports[sport->port.line]) {
+ ret = -EBUSY;
+ } else {
+ imx_uart_ports[sport->port.line] = sport;
+ ret = uart_add_one_port(&imx_uart_uart_driver,
+ &sport->port);
+ if (ret)
+ imx_uart_ports[sport->port.line] = NULL;
+ }
+ }
err_clk:
clk_disable_unprepare(sport->clk_ipg);
@@ -2534,13 +2546,13 @@ static int imx_uart_probe(struct platform_device *pdev)
return ret;
}
-static int imx_uart_remove(struct platform_device *pdev)
+static void imx_uart_remove(struct platform_device *pdev)
{
struct imx_port *sport = platform_get_drvdata(pdev);
+ guard(mutex)(&imx_uart_ports_lock);
uart_remove_one_port(&imx_uart_uart_driver, &sport->port);
-
- return 0;
+ imx_uart_ports[sport->port.line] = NULL;
}
static void imx_uart_restore_context(struct imx_port *sport)
@@ -2709,7 +2721,7 @@ static const struct dev_pm_ops imx_uart_pm_ops = {
static struct platform_driver imx_uart_platform_driver = {
.probe = imx_uart_probe,
- .remove = imx_uart_remove,
+ .remove_new = imx_uart_remove,
.driver = {
.name = "imx-uart",
diff --git a/drivers/tty/serial/kgdboc.c b/drivers/tty/serial/kgdboc.c
index 86433e3c3409..970070bc5ba9 100644
--- a/drivers/tty/serial/kgdboc.c
+++ b/drivers/tty/serial/kgdboc.c
@@ -190,6 +190,11 @@ static void cleanup_kgdboc(void)
return;
kgdboc_unregister_kbd();
kgdb_unregister_io_module(&kgdboc_io_ops);
+
+ if (kgdb_tty_driver) {
+ tty_driver_kref_put(kgdb_tty_driver);
+ kgdb_tty_driver = NULL;
+ }
}
static int configure_kgdboc(void)
@@ -250,6 +255,10 @@ static int configure_kgdboc(void)
kgdb_unregister_io_module(&kgdboc_io_ops);
noconfig:
kgdboc_unregister_kbd();
+ if (kgdb_tty_driver) {
+ tty_driver_kref_put(kgdb_tty_driver);
+ kgdb_tty_driver = NULL;
+ }
configured = 0;
return err;
diff --git a/drivers/tty/serial/max3100.c b/drivers/tty/serial/max3100.c
index 67803242a70c..4ba444460753 100644
--- a/drivers/tty/serial/max3100.c
+++ b/drivers/tty/serial/max3100.c
@@ -568,11 +568,11 @@ static void max3100_shutdown(struct uart_port *port)
del_timer_sync(&s->timer);
if (s->workqueue) {
+ free_irq(s->irq, s);
+ cancel_work_sync(&s->work);
destroy_workqueue(s->workqueue);
s->workqueue = NULL;
}
- if (s->irq)
- free_irq(s->irq, s);
/* set shutdown mode to save power */
if (s->max3100_hw_suspend)
@@ -831,6 +831,15 @@ static void max3100_remove(struct spi_device *spi)
if (max3100s[i] == s) {
dev_dbg(&spi->dev, "%s: removing port %d\n", __func__, i);
uart_remove_one_port(&max3100_uart_driver, &max3100s[i]->port);
+
+ s->force_end_work = 1;
+ timer_shutdown_sync(&s->timer);
+ if (s->workqueue) {
+ free_irq(s->irq, s);
+ cancel_work_sync(&s->work);
+ destroy_workqueue(s->workqueue);
+ s->workqueue = NULL;
+ }
kfree(max3100s[i]);
max3100s[i] = NULL;
break;
diff --git a/drivers/tty/serial/qcom_geni_serial.c b/drivers/tty/serial/qcom_geni_serial.c
index c5fcdd5667b4..2e35894a5ecd 100644
--- a/drivers/tty/serial/qcom_geni_serial.c
+++ b/drivers/tty/serial/qcom_geni_serial.c
@@ -1056,9 +1056,8 @@ static void qcom_geni_serial_set_termios(struct uart_port *uport,
* Bump up BW vote on CPU and CORE path as driver supports FIFO mode
* only.
*/
- avg_bw_core = (baud > 115200) ? Bps_to_icc(CORE_2X_50_MHZ)
- : GENI_DEFAULT_BW;
- port->se.icc_paths[GENI_TO_CORE].avg_bw = avg_bw_core;
+ avg_bw_core = baud > 115200 ? CORE_2X_50_MHZ : CORE_2X_19_2_MHZ;
+ port->se.icc_paths[GENI_TO_CORE].avg_bw = Bps_to_icc(avg_bw_core);
port->se.icc_paths[CPU_TO_GENI].avg_bw = Bps_to_icc(baud);
geni_icc_set_bw(&port->se);
diff --git a/drivers/tty/serial/sc16is7xx.c b/drivers/tty/serial/sc16is7xx.c
index f0ba5a930b2c..7e2f48ea0f33 100644
--- a/drivers/tty/serial/sc16is7xx.c
+++ b/drivers/tty/serial/sc16is7xx.c
@@ -336,6 +336,7 @@ struct sc16is7xx_one {
struct kthread_work reg_work;
struct kthread_delayed_work ms_work;
struct sc16is7xx_one_config config;
+ unsigned char buf[SC16IS7XX_FIFO_SIZE]; /* Rx buffer. */
unsigned int old_mctrl;
u8 old_lcr; /* Value before EFR access. */
bool irda_mode;
@@ -349,7 +350,6 @@ struct sc16is7xx_port {
unsigned long gpio_valid_mask;
#endif
u8 mctrl_mask;
- unsigned char buf[SC16IS7XX_FIFO_SIZE];
struct kthread_worker kworker;
struct task_struct *kworker_task;
struct sc16is7xx_one p[];
@@ -384,15 +384,13 @@ static void sc16is7xx_port_write(struct uart_port *port, u8 reg, u8 val)
static void sc16is7xx_fifo_read(struct uart_port *port, unsigned int rxlen)
{
- struct sc16is7xx_port *s = dev_get_drvdata(port->dev);
struct sc16is7xx_one *one = to_sc16is7xx_one(port, port);
- regmap_noinc_read(one->regmap, SC16IS7XX_RHR_REG, s->buf, rxlen);
+ regmap_noinc_read(one->regmap, SC16IS7XX_RHR_REG, one->buf, rxlen);
}
static void sc16is7xx_fifo_write(struct uart_port *port, u8 to_send)
{
- struct sc16is7xx_port *s = dev_get_drvdata(port->dev);
struct sc16is7xx_one *one = to_sc16is7xx_one(port, port);
/*
@@ -402,7 +400,7 @@ static void sc16is7xx_fifo_write(struct uart_port *port, u8 to_send)
if (unlikely(!to_send))
return;
- regmap_noinc_write(one->regmap, SC16IS7XX_THR_REG, s->buf, to_send);
+ regmap_noinc_write(one->regmap, SC16IS7XX_THR_REG, one->buf, to_send);
}
static void sc16is7xx_port_update(struct uart_port *port, u8 reg,
@@ -619,17 +617,17 @@ static int sc16is7xx_set_baud(struct uart_port *port, int baud)
static void sc16is7xx_handle_rx(struct uart_port *port, unsigned int rxlen,
unsigned int iir)
{
- struct sc16is7xx_port *s = dev_get_drvdata(port->dev);
+ struct sc16is7xx_one *one = to_sc16is7xx_one(port, port);
unsigned int lsr = 0, ch, flag, bytes_read, i;
bool read_lsr = (iir == SC16IS7XX_IIR_RLSE_SRC) ? true : false;
- if (unlikely(rxlen >= sizeof(s->buf))) {
+ if (unlikely(rxlen >= sizeof(one->buf))) {
dev_warn_ratelimited(port->dev,
"ttySC%i: Possible RX FIFO overrun: %d\n",
port->line, rxlen);
port->icount.buf_overrun++;
/* Ensure sanity of RX level */
- rxlen = sizeof(s->buf);
+ rxlen = sizeof(one->buf);
}
while (rxlen) {
@@ -642,7 +640,7 @@ static void sc16is7xx_handle_rx(struct uart_port *port, unsigned int rxlen,
lsr = 0;
if (read_lsr) {
- s->buf[0] = sc16is7xx_port_read(port, SC16IS7XX_RHR_REG);
+ one->buf[0] = sc16is7xx_port_read(port, SC16IS7XX_RHR_REG);
bytes_read = 1;
} else {
sc16is7xx_fifo_read(port, rxlen);
@@ -678,7 +676,7 @@ static void sc16is7xx_handle_rx(struct uart_port *port, unsigned int rxlen,
}
for (i = 0; i < bytes_read; ++i) {
- ch = s->buf[i];
+ ch = one->buf[i];
if (uart_handle_sysrq_char(port, ch))
continue;
@@ -696,7 +694,7 @@ static void sc16is7xx_handle_rx(struct uart_port *port, unsigned int rxlen,
static void sc16is7xx_handle_tx(struct uart_port *port)
{
- struct sc16is7xx_port *s = dev_get_drvdata(port->dev);
+ struct sc16is7xx_one *one = to_sc16is7xx_one(port, port);
struct circ_buf *xmit = &port->state->xmit;
unsigned int txlen, to_send, i;
unsigned long flags;
@@ -733,7 +731,7 @@ static void sc16is7xx_handle_tx(struct uart_port *port)
/* Convert to linear buffer */
for (i = 0; i < to_send; ++i) {
- s->buf[i] = xmit->buf[xmit->tail];
+ one->buf[i] = xmit->buf[xmit->tail];
xmit->tail = (xmit->tail + 1) & (UART_XMIT_SIZE - 1);
}
diff --git a/drivers/tty/serial/serial-tegra.c b/drivers/tty/serial/serial-tegra.c
index 7aa2b5b67001..3fed5212b176 100644
--- a/drivers/tty/serial/serial-tegra.c
+++ b/drivers/tty/serial/serial-tegra.c
@@ -307,11 +307,19 @@ static void tegra_uart_fifo_reset(struct tegra_uart_port *tup, u8 fcr_bits)
{
unsigned long fcr = tup->fcr_shadow;
unsigned int lsr, tmout = 10000;
+ bool clear_tx = !!(fcr_bits & UART_FCR_CLEAR_XMIT);
if (tup->rts_active)
set_rts(tup, false);
- if (tup->cdata->allow_txfifo_reset_fifo_mode) {
+ /*
+ * Leaving FIFO mode below is a workaround for a Tegra30
+ * restriction on clearing the Tx FIFO while FIFO mode is
+ * enabled. It empties both FIFOs, so applying it to an
+ * Rx-only reset would destroy an in-flight transmission.
+ * Only take that path when the caller asked for CLEAR_XMIT.
+ */
+ if (tup->cdata->allow_txfifo_reset_fifo_mode || !clear_tx) {
fcr |= fcr_bits & (UART_FCR_CLEAR_RCVR | UART_FCR_CLEAR_XMIT);
tegra_uart_write(tup, fcr, UART_FCR);
} else {
@@ -336,9 +344,15 @@ static void tegra_uart_fifo_reset(struct tegra_uart_port *tup, u8 fcr_bits)
*/
tegra_uart_wait_cycle_time(tup, 32);
+ /*
+ * Only wait for the transmitter to drain when the Tx FIFO was
+ * part of the reset. For an Rx-only reset it is left intact,
+ * and waiting for TEMT here would spin for a full frame time.
+ */
do {
lsr = tegra_uart_read(tup, UART_LSR);
- if ((lsr & UART_LSR_TEMT) && !(lsr & UART_LSR_DR))
+ if ((!clear_tx || (lsr & UART_LSR_TEMT)) &&
+ !(lsr & UART_LSR_DR))
break;
udelay(1);
} while (--tmout);
diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c
index 2704eb8af588..69845f9e7aea 100644
--- a/drivers/tty/serial/serial_core.c
+++ b/drivers/tty/serial/serial_core.c
@@ -610,7 +610,10 @@ static unsigned int uart_write_room(struct tty_struct *tty)
unsigned int ret;
port = uart_port_lock(state, flags);
- ret = uart_circ_chars_free(&state->xmit);
+ if (!state->xmit.buf)
+ ret = 0;
+ else
+ ret = uart_circ_chars_free(&state->xmit);
uart_port_unlock(port, flags);
return ret;
}
@@ -1222,12 +1225,12 @@ static int uart_wait_modem_status(struct uart_state *state, unsigned long arg)
add_wait_queue(&port->delta_msr_wait, &wait);
for (;;) {
+ set_current_state(TASK_INTERRUPTIBLE);
+
spin_lock_irq(&uport->lock);
memcpy(&cnow, &uport->icount, sizeof(struct uart_icount));
spin_unlock_irq(&uport->lock);
- set_current_state(TASK_INTERRUPTIBLE);
-
if (((arg & TIOCM_RNG) && (cnow.rng != cprev.rng)) ||
((arg & TIOCM_DSR) && (cnow.dsr != cprev.dsr)) ||
((arg & TIOCM_CD) && (cnow.dcd != cprev.dcd)) ||
@@ -2809,8 +2812,10 @@ int uart_register_driver(struct uart_driver *drv)
for (i = 0; i < drv->nr; i++)
tty_port_destroy(&drv->state[i].port);
tty_driver_kref_put(normal);
+ drv->tty_driver = NULL;
out_kfree:
kfree(drv->state);
+ drv->state = NULL;
out:
return retval;
}
@@ -3089,7 +3094,6 @@ int uart_add_one_port(struct uart_driver *drv, struct uart_port *uport)
struct tty_port *port;
int ret = 0;
struct device *tty_dev;
- int num_groups;
if (uport->line >= drv->nr)
return -EINVAL;
@@ -3104,6 +3108,25 @@ int uart_add_one_port(struct uart_driver *drv, struct uart_port *uport)
goto out;
}
+ uport->name = kasprintf(GFP_KERNEL, "%s%d", drv->dev_name,
+ drv->tty_driver->name_base + uport->line);
+ if (!uport->name) {
+ ret = -ENOMEM;
+ goto out;
+ }
+
+ /*
+ * uart_configure_port() may set uport->attr_group and register the
+ * console. Allocate room for both groups and a NULL terminator first.
+ */
+ uport->tty_groups = kcalloc(3, sizeof(*uport->tty_groups), GFP_KERNEL);
+ if (!uport->tty_groups) {
+ kfree(uport->name);
+ ret = -ENOMEM;
+ goto out;
+ }
+ uport->tty_groups[0] = &tty_dev_attr_group;
+
/* Link the port to the driver state table and vice versa */
atomic_set(&state->refcount, 1);
init_waitqueue_head(&state->remove_wait);
@@ -3113,12 +3136,6 @@ int uart_add_one_port(struct uart_driver *drv, struct uart_port *uport)
state->pm_state = UART_PM_STATE_UNDEFINED;
uport->cons = drv->cons;
uport->minor = drv->tty_driver->minor_start + uport->line;
- uport->name = kasprintf(GFP_KERNEL, "%s%d", drv->dev_name,
- drv->tty_driver->name_base + uport->line);
- if (!uport->name) {
- ret = -ENOMEM;
- goto out;
- }
/*
* If this port is in use as a console then the spinlock is already
@@ -3135,17 +3152,6 @@ int uart_add_one_port(struct uart_driver *drv, struct uart_port *uport)
port->console = uart_console(uport);
- num_groups = 2;
- if (uport->attr_group)
- num_groups++;
-
- uport->tty_groups = kcalloc(num_groups, sizeof(*uport->tty_groups),
- GFP_KERNEL);
- if (!uport->tty_groups) {
- ret = -ENOMEM;
- goto out;
- }
- uport->tty_groups[0] = &tty_dev_attr_group;
if (uport->attr_group)
uport->tty_groups[1] = uport->attr_group;
diff --git a/drivers/tty/serial/vt8500_serial.c b/drivers/tty/serial/vt8500_serial.c
index 10fbdb09965f..2c760fd6d71c 100644
--- a/drivers/tty/serial/vt8500_serial.c
+++ b/drivers/tty/serial/vt8500_serial.c
@@ -682,6 +682,7 @@ static int vt8500_serial_probe(struct platform_device *pdev)
ret = clk_prepare_enable(vt8500_port->clk);
if (ret) {
dev_err(&pdev->dev, "failed to enable clock\n");
+ clk_put(vt8500_port->clk);
return ret;
}
diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c
index 9eceb168137f..1e36eda7fd3b 100644
--- a/drivers/tty/tty_io.c
+++ b/drivers/tty/tty_io.c
@@ -630,6 +630,8 @@ static void __tty_hangup(struct tty_struct *tty, int exit_session)
tty_ldisc_hangup(tty, cons_filp != NULL);
+ wake_up_interruptible(&tty->break_wait);
+
spin_lock_irq(&tty->ctrl.lock);
clear_bit(TTY_THROTTLED, &tty->flags);
clear_bit(TTY_DO_WRITE_WAKEUP, &tty->flags);
@@ -2457,6 +2459,7 @@ static int tiocgetd(struct tty_struct *tty, int __user *p)
* send_break - performed time break
* @tty: device to break on
* @duration: timeout in mS
+ * @file: file object
*
* Perform a timed break on hardware that lacks its own driver level timed
* break functionality.
@@ -2464,8 +2467,9 @@ static int tiocgetd(struct tty_struct *tty, int __user *p)
* Locking:
* @tty->atomic_write_lock serializes
*/
-static int send_break(struct tty_struct *tty, unsigned int duration)
+static int send_break(struct file *file, struct tty_struct *tty, unsigned int duration)
{
+ long timeout;
int retval;
if (tty->ops->break_ctl == NULL)
@@ -2479,13 +2483,26 @@ static int send_break(struct tty_struct *tty, unsigned int duration)
return -EINTR;
retval = tty->ops->break_ctl(tty, -1);
- if (!retval) {
- msleep_interruptible(duration);
- retval = tty->ops->break_ctl(tty, 0);
- } else if (retval == -EOPNOTSUPP) {
- /* some drivers can tell only dynamically */
- retval = 0;
+ if (retval) {
+ if (retval == -EOPNOTSUPP) {
+ /* some drivers can tell only dynamically */
+ retval = 0;
+ }
+ goto out_unlock;
}
+
+ timeout = msecs_to_jiffies(duration);
+ timeout = wait_event_interruptible_timeout(tty->break_wait,
+ tty_hung_up_p(file),
+ timeout);
+ /* return early on hangup only */
+ if (timeout > 0) {
+ retval = -EIO;
+ goto out_unlock;
+ }
+
+ retval = tty->ops->break_ctl(tty, 0);
+out_unlock:
tty_write_unlock(tty);
if (signal_pending(current))
@@ -2738,10 +2755,10 @@ long tty_ioctl(struct file *file, unsigned int cmd, unsigned long arg)
* This is used by the tcdrain() termios function.
*/
if (!arg)
- return send_break(tty, 250);
+ return send_break(file, tty, 250);
return 0;
case TCSBRKP: /* support for POSIX tcsendbreak() */
- return send_break(tty, arg ? arg*100 : 250);
+ return send_break(file, tty, arg ? arg * 100 : 250);
case TIOCMGET:
return tty_tiocmget(tty, p);
@@ -3069,7 +3086,7 @@ static struct device *tty_get_device(struct tty_struct *tty)
{
dev_t devt = tty_devnum(tty);
- return class_find_device_by_devt(tty_class, devt);
+ return class_find_device_by_devt(&tty_class, devt);
}
@@ -3104,6 +3121,7 @@ struct tty_struct *alloc_tty_struct(struct tty_driver *driver, int idx)
init_ldsem(&tty->ldisc_sem);
init_waitqueue_head(&tty->write_wait);
init_waitqueue_head(&tty->read_wait);
+ init_waitqueue_head(&tty->break_wait);
INIT_WORK(&tty->hangup_work, do_tty_hangup);
mutex_init(&tty->atomic_write_lock);
spin_lock_init(&tty->ctrl.lock);
@@ -3142,8 +3160,6 @@ int tty_put_char(struct tty_struct *tty, unsigned char ch)
}
EXPORT_SYMBOL_GPL(tty_put_char);
-struct class *tty_class;
-
static int tty_cdev_add(struct tty_driver *driver, dev_t dev,
unsigned int index, unsigned int count)
{
@@ -3240,7 +3256,7 @@ struct device *tty_register_device_attr(struct tty_driver *driver,
return ERR_PTR(-ENOMEM);
dev->devt = devt;
- dev->class = tty_class;
+ dev->class = &tty_class;
dev->parent = device;
dev->release = tty_device_create_release;
dev_set_name(dev, "%s", name);
@@ -3254,14 +3270,16 @@ struct device *tty_register_device_attr(struct tty_driver *driver,
goto err_put;
if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC)) {
- /*
- * Free any saved termios data so that the termios state is
- * reset when reusing a minor number.
- */
- tp = driver->termios[index];
- if (tp) {
- driver->termios[index] = NULL;
- kfree(tp);
+ if (driver->flags & TTY_DRIVER_RESET_SAVED_TERMIOS) {
+ /*
+ * Free any saved termios data so that the termios
+ * state is reset when reusing a minor number.
+ */
+ tp = driver->termios[index];
+ if (tp) {
+ driver->termios[index] = NULL;
+ kfree(tp);
+ }
}
retval = tty_cdev_add(driver, devt, index, 1);
@@ -3295,9 +3313,8 @@ EXPORT_SYMBOL_GPL(tty_register_device_attr);
*/
void tty_unregister_device(struct tty_driver *driver, unsigned index)
{
- device_destroy(tty_class,
- MKDEV(driver->major, driver->minor_start) + index);
- if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC)) {
+ device_destroy(&tty_class, MKDEV(driver->major, driver->minor_start) + index);
+ if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC) && driver->cdevs[index]) {
cdev_del(driver->cdevs[index]);
driver->cdevs[index] = NULL;
}
@@ -3511,13 +3528,14 @@ static char *tty_devnode(struct device *dev, umode_t *mode)
return NULL;
}
+struct class tty_class = {
+ .name = "tty",
+ .devnode = tty_devnode,
+};
+
static int __init tty_class_init(void)
{
- tty_class = class_create(THIS_MODULE, "tty");
- if (IS_ERR(tty_class))
- return PTR_ERR(tty_class);
- tty_class->devnode = tty_devnode;
- return 0;
+ return class_register(&tty_class);
}
postcore_initcall(tty_class_init);
@@ -3590,13 +3608,13 @@ int __init tty_init(void)
if (cdev_add(&tty_cdev, MKDEV(TTYAUX_MAJOR, 0), 1) ||
register_chrdev_region(MKDEV(TTYAUX_MAJOR, 0), 1, "/dev/tty") < 0)
panic("Couldn't register /dev/tty driver\n");
- device_create(tty_class, NULL, MKDEV(TTYAUX_MAJOR, 0), NULL, "tty");
+ device_create(&tty_class, NULL, MKDEV(TTYAUX_MAJOR, 0), NULL, "tty");
cdev_init(&console_cdev, &console_fops);
if (cdev_add(&console_cdev, MKDEV(TTYAUX_MAJOR, 1), 1) ||
register_chrdev_region(MKDEV(TTYAUX_MAJOR, 1), 1, "/dev/console") < 0)
panic("Couldn't register /dev/console driver\n");
- consdev = device_create_with_groups(tty_class, NULL,
+ consdev = device_create_with_groups(&tty_class, NULL,
MKDEV(TTYAUX_MAJOR, 1), NULL,
cons_dev_groups, "console");
if (IS_ERR(consdev))
diff --git a/drivers/tty/tty_port.c b/drivers/tty/tty_port.c
index 494500c453cc..82f7ddf449c0 100644
--- a/drivers/tty/tty_port.c
+++ b/drivers/tty/tty_port.c
@@ -350,21 +350,12 @@ void tty_port_tty_set(struct tty_port *port, struct tty_struct *tty)
}
EXPORT_SYMBOL(tty_port_tty_set);
-/**
- * tty_port_shutdown - internal helper to shutdown the device
- * @port: tty port to be shut down
- * @tty: the associated tty
- *
- * It is used by tty_port_hangup() and tty_port_close(). Its task is to
- * shutdown the device if it was initialized (note consoles remain
- * functioning). It lowers DTR/RTS (if @tty has HUPCL set) and invokes
- * @port->ops->shutdown().
- */
-static void tty_port_shutdown(struct tty_port *port, struct tty_struct *tty)
+static void tty_port_shutdown_locked(struct tty_port *port, struct tty_struct *tty)
{
- mutex_lock(&port->mutex);
+ lockdep_assert_held(&port->mutex);
+
if (port->console)
- goto out;
+ return;
if (tty_port_initialized(port)) {
tty_port_set_initialized(port, 0);
@@ -378,7 +369,22 @@ static void tty_port_shutdown(struct tty_port *port, struct tty_struct *tty)
if (port->ops->shutdown)
port->ops->shutdown(port);
}
-out:
+}
+
+/**
+ * tty_port_shutdown - internal helper to shutdown the device
+ * @port: tty port to be shut down
+ * @tty: the associated tty
+ *
+ * It is used by tty_port_hangup() and tty_port_close(). Its task is to
+ * shutdown the device if it was initialized (note consoles remain
+ * functioning). It lowers DTR/RTS (if @tty has HUPCL set) and invokes
+ * @port->ops->shutdown().
+ */
+static void tty_port_shutdown(struct tty_port *port, struct tty_struct *tty)
+{
+ mutex_lock(&port->mutex);
+ tty_port_shutdown_locked(port, tty);
mutex_unlock(&port->mutex);
}
@@ -396,6 +402,7 @@ void tty_port_hangup(struct tty_port *port)
struct tty_struct *tty;
unsigned long flags;
+ mutex_lock(&port->mutex);
spin_lock_irqsave(&port->lock, flags);
port->count = 0;
tty = port->tty;
@@ -404,26 +411,47 @@ void tty_port_hangup(struct tty_port *port)
port->tty = NULL;
spin_unlock_irqrestore(&port->lock, flags);
tty_port_set_active(port, 0);
- tty_port_shutdown(port, tty);
+ tty_port_shutdown_locked(port, tty);
+ mutex_unlock(&port->mutex);
tty_kref_put(tty);
wake_up_interruptible(&port->open_wait);
wake_up_interruptible(&port->delta_msr_wait);
}
EXPORT_SYMBOL(tty_port_hangup);
-void __tty_port_tty_hangup(struct tty_port *port, bool check_clocal, bool async)
+/**
+ * tty_port_tty_hangup - helper to hang up a tty asynchronously
+ * @port: tty port
+ * @check_clocal: hang only ttys with %CLOCAL unset?
+ */
+void tty_port_tty_hangup(struct tty_port *port, bool check_clocal)
{
struct tty_struct *tty = tty_port_tty_get(port);
- if (tty && (!check_clocal || !C_CLOCAL(tty))) {
- if (async)
- tty_hangup(tty);
- else
- tty_vhangup(tty);
- }
+ if (tty && (!check_clocal || !C_CLOCAL(tty)))
+ tty_hangup(tty);
tty_kref_put(tty);
}
-EXPORT_SYMBOL_GPL(__tty_port_tty_hangup);
+EXPORT_SYMBOL_GPL(tty_port_tty_hangup);
+
+/**
+ * tty_port_tty_vhangup - helper to hang up a tty synchronously
+ * @port: tty port
+ */
+void tty_port_tty_vhangup(struct tty_port *port)
+{
+ struct tty_struct *tty;
+
+ mutex_lock(&port->mutex);
+ tty = tty_port_tty_get(port);
+ mutex_unlock(&port->mutex);
+
+ if (tty) {
+ tty_vhangup(tty);
+ tty_kref_put(tty);
+ }
+}
+EXPORT_SYMBOL_GPL(tty_port_tty_vhangup);
/**
* tty_port_tty_wakeup - helper to wake up a tty
@@ -783,6 +811,7 @@ int tty_port_open(struct tty_port *port, struct tty_struct *tty,
int retval = port->ops->activate(port, tty);
if (retval) {
+ set_bit(TTY_IO_ERROR, &tty->flags);
mutex_unlock(&port->mutex);
return retval;
}
diff --git a/drivers/tty/vcc.c b/drivers/tty/vcc.c
index cac23f67d2f1..12667342e74a 100644
--- a/drivers/tty/vcc.c
+++ b/drivers/tty/vcc.c
@@ -492,7 +492,7 @@ static ssize_t domain_show(struct device *dev,
static int vcc_send_ctl(struct vcc_port *port, int ctl)
{
- struct vio_vcc pkt;
+ struct vio_vcc pkt = {};
int rv;
pkt.tag.type = VIO_TYPE_CTRL;
@@ -985,7 +985,8 @@ static void vcc_cleanup(struct tty_struct *tty)
port = vcc_get(tty->index, true);
if (port) {
- port->tty = NULL;
+ scoped_guard(spinlock_irqsave, &port->lock)
+ port->tty = NULL;
if (port->removed) {
vcc_table_remove(tty->index);
diff --git a/drivers/tty/vt/vc_screen.c b/drivers/tty/vt/vc_screen.c
index 27b44dc12385..22735ca14395 100644
--- a/drivers/tty/vt/vc_screen.c
+++ b/drivers/tty/vt/vc_screen.c
@@ -643,6 +643,19 @@ vcs_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos)
ret = copy_from_user(con_buf, buf, this_round);
console_lock();
+ /* The vc might have been freed or vcs_size might have changed
+ * while we slept to grab the user buffer; recheck here, before
+ * if (ret), so every break path below passes a fresh vc to the
+ * post-loop vcs_scr_updated(). Return data written so far.
+ */
+ vc = vcs_vc(inode, &viewed);
+ if (!vc) {
+ if (written)
+ break;
+ ret = -ENXIO;
+ goto unlock_out;
+ }
+
if (ret) {
this_round -= ret;
if (!this_round) {
@@ -655,18 +668,6 @@ vcs_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos)
goto unlock_out;
}
}
-
- /* The vc might have been freed or vcs_size might have changed
- * while we slept to grab the user buffer, so recheck.
- * Return data written up to now on failure.
- */
- vc = vcs_vc(inode, &viewed);
- if (!vc) {
- if (written)
- break;
- ret = -ENXIO;
- goto unlock_out;
- }
size = vcs_size(vc, attr, false);
if (size < 0) {
if (written)
diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c
index 9dd527f521c8..bdfbf6a1efe2 100644
--- a/drivers/tty/vt/vt.c
+++ b/drivers/tty/vt/vt.c
@@ -1160,6 +1160,8 @@ int vc_allocate(unsigned int currcons) /* return 0 on success */
return 0;
err_free:
visual_deinit(vc);
+ if (*vc->uni_pagedict_loc)
+ con_free_unimap(vc);
kfree(vc);
vc_cons[currcons].d = NULL;
return err;
@@ -2543,7 +2545,8 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, int c)
csi_J(vc, 2);
vc->vc_video_erase_char =
(vc->vc_video_erase_char & 0xff00) | ' ';
- do_update_region(vc, vc->vc_origin, vc->vc_screenbuf_size / 2);
+ if (con_should_update(vc))
+ do_update_region(vc, vc->vc_origin, vc->vc_screenbuf_size / 2);
}
return;
case ESsetG0:
@@ -3600,7 +3603,7 @@ int __init vty_init(const struct file_operations *console_fops)
if (cdev_add(&vc0_cdev, MKDEV(TTY_MAJOR, 0), 1) ||
register_chrdev_region(MKDEV(TTY_MAJOR, 0), 1, "/dev/vc/0") < 0)
panic("Couldn't register /dev/tty0 driver\n");
- tty0dev = device_create_with_groups(tty_class, NULL,
+ tty0dev = device_create_with_groups(&tty_class, NULL,
MKDEV(TTY_MAJOR, 0), NULL,
vt_dev_groups, "tty0");
if (IS_ERR(tty0dev))
diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c
index f8f64b8f111c..f441fc0903a8 100644
--- a/drivers/ufs/core/ufshcd.c
+++ b/drivers/ufs/core/ufshcd.c
@@ -3530,7 +3530,7 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
{
struct uc_string_id *uc_str;
u8 *str;
- int ret;
+ int ret, uc_len;
if (!buf)
return -EINVAL;
@@ -3555,11 +3555,19 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
goto out;
}
+ uc_len = uc_str->len - QUERY_DESC_HDR_SIZE;
+ if (uc_len % sizeof(*uc_str->uc)) {
+ dev_err(hba->dev, "String Desc has an odd UTF-16 payload length\n");
+ str = NULL;
+ ret = -EINVAL;
+ goto out;
+ }
+
if (ascii) {
ssize_t ascii_len;
int i;
- /* remove header and divide by 2 to move from UTF16 to UTF8 */
- ascii_len = (uc_str->len - QUERY_DESC_HDR_SIZE) / 2 + 1;
+ /* Allow up to three UTF-8 bytes per UTF-16 code unit plus a NUL. */
+ ascii_len = uc_len / sizeof(*uc_str->uc) * 3 + 1;
str = kzalloc(ascii_len, GFP_KERNEL);
if (!str) {
ret = -ENOMEM;
@@ -3571,7 +3579,7 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
* we need to convert to utf-8 so it can be displayed
*/
ret = utf16s_to_utf8s(uc_str->uc,
- uc_str->len - QUERY_DESC_HDR_SIZE,
+ uc_len / sizeof(*uc_str->uc),
UTF16_BIG_ENDIAN, str, ascii_len - 1);
/* replace non-printable or non-ASCII characters with spaces */
@@ -3581,11 +3589,17 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index,
str[ret++] = '\0';
} else {
- str = kmemdup(uc_str->uc, uc_str->len, GFP_KERNEL);
+ /*
+ * Keep the bLength-sized raw output for the RPMB device ID ABI.
+ * The two bytes beyond the UTF-16 payload are explicitly zeroed
+ * instead of being read past the descriptor buffer.
+ */
+ str = kzalloc(uc_str->len, GFP_KERNEL);
if (!str) {
ret = -ENOMEM;
goto out;
}
+ memcpy(str, uc_str->uc, uc_len);
ret = uc_str->len;
}
out:
@@ -8838,7 +8852,7 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
* we are functional while we are here, skip host resume in error
* handling context.
*/
- hba->host->eh_noresume = 1;
+ WRITE_ONCE(hba->host->eh_noresume, 1);
cmd[4] = pwr_mode << 4;
@@ -8872,7 +8886,7 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba,
hba->curr_dev_pwr_mode = pwr_mode;
scsi_device_put(sdp);
- hba->host->eh_noresume = 0;
+ WRITE_ONCE(hba->host->eh_noresume, 0);
return ret;
}
diff --git a/drivers/ufs/host/ufs-exynos.c b/drivers/ufs/host/ufs-exynos.c
index 17db3f8c8c51..6bb8da143686 100644
--- a/drivers/ufs/host/ufs-exynos.c
+++ b/drivers/ufs/host/ufs-exynos.c
@@ -200,8 +200,8 @@ static int exynosauto_ufs_drv_init(struct device *dev, struct exynos_ufs *ufs)
/* IO Coherency setting */
if (ufs->sysreg) {
return regmap_update_bits(ufs->sysreg,
- ufs->shareability_reg_offset,
- UFS_SHARABLE, UFS_SHARABLE);
+ ufs->iocc_offset,
+ UFS_SHARABLE, ufs->iocc_val);
}
attr->tx_dif_p_nsec = 3200000;
@@ -1102,12 +1102,21 @@ static int exynos_ufs_parse_dt(struct device *dev, struct exynos_ufs *ufs)
ufs->sysreg = NULL;
else {
if (of_property_read_u32_index(np, "samsung,sysreg", 1,
- &ufs->shareability_reg_offset)) {
+ &ufs->iocc_offset)) {
dev_warn(dev, "can't get an offset from sysreg. Set to default value\n");
- ufs->shareability_reg_offset = UFS_SHAREABILITY_OFFSET;
+ ufs->iocc_offset = UFS_SHAREABILITY_OFFSET;
}
}
+ /*
+ * no 'dma-coherent' property means the descriptors are
+ * non-cacheable so iocc shareability should be disabled.
+ */
+ if (of_dma_is_coherent(dev->of_node))
+ ufs->iocc_val = UFS_SHARABLE;
+ else
+ ufs->iocc_val = 0;
+
ufs->pclk_avail_min = PCLK_AVAIL_MIN;
ufs->pclk_avail_max = PCLK_AVAIL_MAX;
diff --git a/drivers/ufs/host/ufs-exynos.h b/drivers/ufs/host/ufs-exynos.h
index a4bd6646d7f1..65aa04d41c3e 100644
--- a/drivers/ufs/host/ufs-exynos.h
+++ b/drivers/ufs/host/ufs-exynos.h
@@ -212,7 +212,8 @@ struct exynos_ufs {
ktime_t entry_hibern8_t;
const struct exynos_ufs_drv_data *drv_data;
struct regmap *sysreg;
- u32 shareability_reg_offset;
+ u32 iocc_offset;
+ u32 iocc_val;
u32 opts;
#define EXYNOS_UFS_OPT_HAS_APB_CLK_CTRL BIT(0)
diff --git a/drivers/ufs/host/ufs-qcom.c b/drivers/ufs/host/ufs-qcom.c
index ecd5939f4c9a..3d5d10612bfd 100644
--- a/drivers/ufs/host/ufs-qcom.c
+++ b/drivers/ufs/host/ufs-qcom.c
@@ -590,7 +590,18 @@ static int ufs_qcom_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op,
/* reset the connected UFS device during power down */
ufs_qcom_device_reset_ctrl(hba, true);
-
+ /*
+ * After sending the SSU command, asserting the rst_n
+ * line causes the device firmware to wake up and
+ * execute its reset routine.
+ *
+ * During this process, the device may draw current
+ * beyond the permissible limit for low-power mode (LPM).
+ * A 10ms delay, based on experimental observations,
+ * allows the UFS device to complete its hardware reset
+ * before transitioning the power rail to LPM.
+ */
+ usleep_range(10000, 11000);
} else if (!ufs_qcom_is_link_active(hba)) {
ufs_qcom_disable_lane_clks(host);
}
diff --git a/drivers/usb/cdns3/cdns3-gadget.c b/drivers/usb/cdns3/cdns3-gadget.c
index ff10668b0f13..c8872be1edf2 100644
--- a/drivers/usb/cdns3/cdns3-gadget.c
+++ b/drivers/usb/cdns3/cdns3-gadget.c
@@ -3265,6 +3265,10 @@ static void cdns3_gadget_exit(struct cdns *cdns)
usb_del_gadget(&priv_dev->gadget);
devm_free_irq(cdns->dev, cdns->dev_irq, priv_dev);
+ /* The works can still be queued until the IRQ is freed. */
+ cancel_work_sync(&priv_dev->pending_status_wq);
+ cancel_work_sync(&priv_dev->aligned_buf_wq);
+
cdns3_free_all_eps(priv_dev);
while (!list_empty(&priv_dev->aligned_buf_list)) {
diff --git a/drivers/usb/cdns3/cdns3-pci-wrap.c b/drivers/usb/cdns3/cdns3-pci-wrap.c
index 92ae5d443350..2f321ce033be 100644
--- a/drivers/usb/cdns3/cdns3-pci-wrap.c
+++ b/drivers/usb/cdns3/cdns3-pci-wrap.c
@@ -99,6 +99,11 @@ static int cdns3_pci_probe(struct pci_dev *pdev,
if (pci_is_enabled(func)) {
wrap = pci_get_drvdata(func);
+ if (!wrap) {
+ dev_err(&pdev->dev,
+ "second function not initialized, retrying\n");
+ return -EPROBE_DEFER;
+ }
} else {
wrap = kzalloc(sizeof(*wrap), GFP_KERNEL);
if (!wrap)
diff --git a/drivers/usb/chipidea/ci_hdrc_tegra.c b/drivers/usb/chipidea/ci_hdrc_tegra.c
index a72a9474afea..d7ffbe5b4d1b 100644
--- a/drivers/usb/chipidea/ci_hdrc_tegra.c
+++ b/drivers/usb/chipidea/ci_hdrc_tegra.c
@@ -307,8 +307,10 @@ static int tegra_usb_probe(struct platform_device *pdev)
pm_runtime_enable(&pdev->dev);
err = pm_runtime_resume_and_get(&pdev->dev);
- if (err)
+ if (err) {
+ pm_runtime_disable(&pdev->dev);
return err;
+ }
if (device_property_present(&pdev->dev, "nvidia,needs-double-reset"))
usb->needs_double_reset = true;
diff --git a/drivers/usb/chipidea/udc.c b/drivers/usb/chipidea/udc.c
index e7a02d9e1c07..b1b5f8a13559 100644
--- a/drivers/usb/chipidea/udc.c
+++ b/drivers/usb/chipidea/udc.c
@@ -773,6 +773,10 @@ __acquires(hwep->lock)
list_del_init(&hwreq->queue);
hwreq->req.status = -ESHUTDOWN;
+ /* Unmap DMA before giving back */
+ usb_gadget_unmap_request_by_dev(hwep->ci->dev->parent,
+ &hwreq->req, hwep->dir);
+
if (hwreq->req.complete != NULL) {
spin_unlock(hwep->lock);
usb_gadget_giveback_request(&hwep->ep, &hwreq->req);
diff --git a/drivers/usb/class/cdc-acm.c b/drivers/usb/class/cdc-acm.c
index 9ff5feceadad..80e86fa38eab 100644
--- a/drivers/usb/class/cdc-acm.c
+++ b/drivers/usb/class/cdc-acm.c
@@ -320,7 +320,6 @@ static void acm_process_notification(struct acm *acm, unsigned char *buf)
difference = acm->ctrlin ^ newctrl;
spin_lock_irqsave(&acm->read_lock, flags);
acm->ctrlin = newctrl;
- acm->oldcount = acm->iocount;
if (difference & USB_CDC_SERIAL_STATE_DSR)
acm->iocount.dsr++;
@@ -777,6 +776,9 @@ static void acm_port_shutdown(struct tty_port *port)
usb_autopm_put_interface_async(acm->control);
}
+ if (acm->disconnected)
+ return;
+
acm_unpoison_urbs(acm);
if (acm->quirks & ALWAYS_POLL_CTRL) {
@@ -1002,11 +1004,16 @@ static int wait_serial_change(struct acm *acm, unsigned long arg)
DECLARE_WAITQUEUE(wait, current);
struct async_icount old, new;
- do {
+ spin_lock_irq(&acm->read_lock);
+ old = acm->iocount;
+ spin_unlock_irq(&acm->read_lock);
+
+ add_wait_queue(&acm->wioctl, &wait);
+ for (;;) {
+ set_current_state(TASK_INTERRUPTIBLE);
+
spin_lock_irq(&acm->read_lock);
- old = acm->oldcount;
new = acm->iocount;
- acm->oldcount = new;
spin_unlock_irq(&acm->read_lock);
if ((arg & TIOCM_DSR) &&
@@ -1019,22 +1026,20 @@ static int wait_serial_change(struct acm *acm, unsigned long arg)
old.rng != new.rng)
break;
- add_wait_queue(&acm->wioctl, &wait);
- set_current_state(TASK_INTERRUPTIBLE);
- schedule();
- remove_wait_queue(&acm->wioctl, &wait);
if (acm->disconnected) {
- if (arg & TIOCM_CD)
- break;
- else
- rv = -ENODEV;
- } else {
- if (signal_pending(current))
- rv = -ERESTARTSYS;
+ rv = -ENODEV;
+ break;
}
- } while (!rv);
-
+ schedule();
+
+ if (signal_pending(current)) {
+ rv = -ERESTARTSYS;
+ break;
+ }
+ }
+ __set_current_state(TASK_RUNNING);
+ remove_wait_queue(&acm->wioctl, &wait);
return rv;
}
@@ -2114,7 +2119,7 @@ static int __init acm_init(void)
{
int retval;
acm_tty_driver = tty_alloc_driver(ACM_TTY_MINORS, TTY_DRIVER_REAL_RAW |
- TTY_DRIVER_DYNAMIC_DEV);
+ TTY_DRIVER_DYNAMIC_DEV | TTY_DRIVER_RESET_SAVED_TERMIOS);
if (IS_ERR(acm_tty_driver))
return PTR_ERR(acm_tty_driver);
acm_tty_driver->driver_name = "acm",
diff --git a/drivers/usb/class/cdc-acm.h b/drivers/usb/class/cdc-acm.h
index 01f448a783c0..d245e60eb9f5 100644
--- a/drivers/usb/class/cdc-acm.h
+++ b/drivers/usb/class/cdc-acm.h
@@ -90,7 +90,6 @@ struct acm {
unsigned int ctrlin; /* input control lines (DCD, DSR, RI, break, overruns) */
unsigned int ctrlout; /* output control lines (DTR, RTS) */
struct async_icount iocount; /* counters for control line changes */
- struct async_icount oldcount; /* for comparison of counter */
wait_queue_head_t wioctl; /* for ioctl */
unsigned int writesize; /* max packet size for the output bulk endpoint */
unsigned int readsize,ctrlsize; /* buffer sizes for freeing */
diff --git a/drivers/usb/core/hub.c b/drivers/usb/core/hub.c
index d7ae3eabc534..787d415c699e 100644
--- a/drivers/usb/core/hub.c
+++ b/drivers/usb/core/hub.c
@@ -69,8 +69,11 @@
/*
* Give SS hubs 200ms time after wake to train downstream links before
* assuming no port activity and allowing hub to runtime suspend back.
+ * Root hubs have no upstream hub whose wake propagation needs to be
+ * accounted for, so they need less time, use 120ms for them.
*/
#define USB_SS_PORT_U0_WAKE_TIME 200 /* ms */
+#define USB_SS_RH_PORT_U0_WAKE_TIME 120 /* ms */
/* Protect struct usb_device->state and ->children members
* Note: Both are also protected by ->dev.sem, except that ->state can
@@ -1332,7 +1335,9 @@ static void hub_activate(struct usb_hub *hub, enum hub_activation_type type)
queue_delayed_work(system_power_efficient_wq,
&hub->post_resume_work,
- msecs_to_jiffies(USB_SS_PORT_U0_WAKE_TIME));
+ msecs_to_jiffies(hdev->parent ?
+ USB_SS_PORT_U0_WAKE_TIME :
+ USB_SS_RH_PORT_U0_WAKE_TIME));
return;
}
diff --git a/drivers/usb/core/message.c b/drivers/usb/core/message.c
index 94c01050aeb7..58046946411f 100644
--- a/drivers/usb/core/message.c
+++ b/drivers/usb/core/message.c
@@ -1331,7 +1331,8 @@ static void remove_intf_ep_devs(struct usb_interface *intf)
*
* Disables the endpoint for URB submission and nukes all pending URBs.
* If @reset_hardware is set then also deallocates hcd/hardware state
- * for the endpoint.
+ * for the endpoint (clearing both ep_in and ep_out pointers for
+ * bidirectional non-ep0 control endpoints).
*/
void usb_disable_endpoint(struct usb_device *dev, unsigned int epaddr,
bool reset_hardware)
@@ -1352,6 +1353,11 @@ void usb_disable_endpoint(struct usb_device *dev, unsigned int epaddr,
dev->ep_in[epnum] = NULL;
}
if (ep) {
+ if (reset_hardware && epnum != 0 &&
+ usb_endpoint_xfer_control(&ep->desc)) {
+ dev->ep_out[epnum] = NULL;
+ dev->ep_in[epnum] = NULL;
+ }
ep->enabled = 0;
usb_hcd_flush_endpoint(dev, ep);
if (reset_hardware)
diff --git a/drivers/usb/dwc2/hcd.c b/drivers/usb/dwc2/hcd.c
index e0f84726f4b1..5e71094a22b2 100644
--- a/drivers/usb/dwc2/hcd.c
+++ b/drivers/usb/dwc2/hcd.c
@@ -5082,14 +5082,13 @@ static void dwc2_hcd_free(struct dwc2_hsotg *hsotg)
}
cancel_work_sync(&hsotg->phy_reset_work);
-
- del_timer(&hsotg->wkp_timer);
}
static void dwc2_hcd_release(struct dwc2_hsotg *hsotg)
{
/* Turn off all host-specific interrupts */
dwc2_disable_host_interrupts(hsotg);
+ timer_shutdown_sync(&hsotg->wkp_timer);
dwc2_hcd_free(hsotg);
}
diff --git a/drivers/usb/dwc3/core.c b/drivers/usb/dwc3/core.c
index 7793936757ac..dbddc1d45fdc 100644
--- a/drivers/usb/dwc3/core.c
+++ b/drivers/usb/dwc3/core.c
@@ -1855,6 +1855,20 @@ static struct extcon_dev *dwc3_get_extcon(struct dwc3 *dwc)
return edev;
}
+static void dwc3_vbus_draw_work(struct work_struct *work)
+{
+ struct dwc3 *dwc = container_of(work, struct dwc3, vbus_draw_work);
+ union power_supply_propval val = {0};
+ int ret;
+
+ val.intval = 1000 * (dwc->current_limit);
+ ret = power_supply_set_property(dwc->usb_psy, POWER_SUPPLY_PROP_INPUT_CURRENT_LIMIT, &val);
+
+ if (ret < 0)
+ dev_dbg(dwc->dev, "Error (%d) setting vbus draw (%d mA)\n",
+ ret, dwc->current_limit);
+}
+
static struct power_supply *dwc3_get_usb_power_supply(struct dwc3 *dwc)
{
struct power_supply *usb_psy;
@@ -1869,6 +1883,7 @@ static struct power_supply *dwc3_get_usb_power_supply(struct dwc3 *dwc)
if (!usb_psy)
return ERR_PTR(-EPROBE_DEFER);
+ INIT_WORK(&dwc->vbus_draw_work, dwc3_vbus_draw_work);
return usb_psy;
}
@@ -2120,8 +2135,10 @@ static int dwc3_remove(struct platform_device *pdev)
dwc3_free_event_buffers(dwc);
dwc3_free_scratch_buffers(dwc);
- if (dwc->usb_psy)
+ if (dwc->usb_psy) {
+ cancel_work_sync(&dwc->vbus_draw_work);
power_supply_put(dwc->usb_psy);
+ }
return 0;
}
diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h
index 933a208c3aab..0119a8940c8d 100644
--- a/drivers/usb/dwc3/core.h
+++ b/drivers/usb/dwc3/core.h
@@ -42,6 +42,7 @@
#define DWC3_ENDPOINTS_NUM 32
#define DWC3_XHCI_RESOURCES_NUM 2
#define DWC3_ISOC_MAX_RETRIES 5
+#define DWC3_ERR_RECOVERY_MAX 3
#define DWC3_SCRATCHBUF_SIZE 4096 /* each buffer is assumed to be 4KiB */
#define DWC3_EVENT_BUFFERS_SIZE 4096
@@ -817,6 +818,12 @@ enum dwc3_link_state {
DWC3_LINK_STATE_MASK = 0x0f,
};
+enum dwc3_err_state {
+ DWC3_ERR_NONE = 0,
+ DWC3_ERR_RECOVERY,
+ DWC3_ERR_UNRECOVERABLE,
+};
+
/* TRB Length, PCM and Status */
#define DWC3_TRB_SIZE_MASK (0x00ffffff)
#define DWC3_TRB_SIZE_LENGTH(n) ((n) & DWC3_TRB_SIZE_MASK)
@@ -972,6 +979,7 @@ struct dwc3_scratchpad_array {
/**
* struct dwc3 - representation of our controller
* @drd_work: workqueue used for role swapping
+ * @err_recovery_work: workqueue used for controller error recovery
* @ep0_trb: trb which is used for the ctrl_req
* @bounce: address of bounce buffer
* @scratchbuf: address of scratch buffer
@@ -983,6 +991,7 @@ struct dwc3_scratchpad_array {
* @ep0_in_setup: one control transfer is completed and enter setup phase
* @lock: for synchronizing
* @mutex: for mode switching
+ * @connect_mutex: for the pull-up and err_recovery_work
* @dev: pointer to our struct device
* @sysdev: pointer to the DMA-capable device
* @xhci: pointer to our xHCI child
@@ -1026,6 +1035,8 @@ struct dwc3_scratchpad_array {
* @role_switch_default_mode: default operation mode of controller while
* usb role is USB_ROLE_NONE.
* @usb_psy: pointer to power supply interface.
+ * @vbus_draw_work: Work to set the vbus drawing limit
+ * @current_limit: How much current to draw from vbus, in milliAmperes.
* @usb2_phy: pointer to USB2 PHY
* @usb3_phy: pointer to USB3 PHY
* @usb2_generic_phy: pointer to USB2 PHY
@@ -1041,6 +1052,9 @@ struct dwc3_scratchpad_array {
* @ep0_next_event: hold the next expected event
* @ep0state: state of endpoint zero
* @link_state: link state
+ * @err_state: current error recovery state.
+ * @err_recovery_count: number of consecutive error recovery attempts until
+ * confirmed healthy and reset to 0 on reset event.
* @speed: device speed (super, high, full, low)
* @hwparams: copy of hwparams registers
* @regset: debugfs pointer to regdump file
@@ -1133,6 +1147,7 @@ struct dwc3_scratchpad_array {
*/
struct dwc3 {
struct work_struct drd_work;
+ struct work_struct err_recovery_work;
struct dwc3_trb *ep0_trb;
void *bounce;
void *scratchbuf;
@@ -1149,6 +1164,9 @@ struct dwc3 {
/* mode switching lock */
struct mutex mutex;
+ /* serializes pull-up run/stop vs error recovery */
+ struct mutex connect_mutex;
+
struct device *dev;
struct device *sysdev;
@@ -1191,6 +1209,8 @@ struct dwc3 {
enum usb_dr_mode role_switch_default_mode;
struct power_supply *usb_psy;
+ struct work_struct vbus_draw_work;
+ unsigned int current_limit;
u32 fladj;
u32 ref_clk_per;
@@ -1265,6 +1285,9 @@ struct dwc3 {
enum dwc3_ep0_next ep0_next_event;
enum dwc3_ep0_state ep0state;
enum dwc3_link_state link_state;
+ enum dwc3_err_state err_state;
+
+ u32 err_recovery_count;
u16 u2sel;
u16 u2pel;
diff --git a/drivers/usb/dwc3/dwc3-am62.c b/drivers/usb/dwc3/dwc3-am62.c
index b0bb8edbfef7..22c93e6aecae 100644
--- a/drivers/usb/dwc3/dwc3-am62.c
+++ b/drivers/usb/dwc3/dwc3-am62.c
@@ -243,6 +243,7 @@ static int dwc3_ti_probe(struct platform_device *pdev)
err_pm_disable:
clk_disable_unprepare(am62->usb2_refclk);
+ pm_runtime_put_noidle(dev);
pm_runtime_disable(dev);
pm_runtime_set_suspended(dev);
return ret;
diff --git a/drivers/usb/dwc3/ep0.c b/drivers/usb/dwc3/ep0.c
index f30ad37ea8a1..c4aa925bcd7c 100644
--- a/drivers/usb/dwc3/ep0.c
+++ b/drivers/usb/dwc3/ep0.c
@@ -198,6 +198,11 @@ int dwc3_gadget_ep0_queue(struct usb_ep *ep, struct usb_request *request,
int ret;
spin_lock_irqsave(&dwc->lock, flags);
+ if (dwc->err_state != DWC3_ERR_NONE) {
+ ret = -ESHUTDOWN;
+ goto out;
+ }
+
if (!dep->endpoint.desc || !dwc->pullups_connected || !dwc->connected) {
dev_err(dwc->dev, "%s: can't queue to disabled endpoint\n",
dep->name);
@@ -269,6 +274,10 @@ int dwc3_gadget_ep0_set_halt(struct usb_ep *ep, int value)
int ret;
spin_lock_irqsave(&dwc->lock, flags);
+ if (dwc->err_state != DWC3_ERR_NONE) {
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return -ESHUTDOWN;
+ }
ret = __dwc3_gadget_ep0_set_halt(ep, value);
spin_unlock_irqrestore(&dwc->lock, flags);
diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c
index 25f7ab01c3c3..18fab84af369 100644
--- a/drivers/usb/dwc3/gadget.c
+++ b/drivers/usb/dwc3/gadget.c
@@ -1108,6 +1108,10 @@ static int dwc3_gadget_ep_enable(struct usb_ep *ep,
return 0;
spin_lock_irqsave(&dwc->lock, flags);
+ if (dwc->err_state != DWC3_ERR_NONE) {
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return -ESHUTDOWN;
+ }
ret = __dwc3_gadget_ep_enable(dep, DWC3_DEPCFG_ACTION_INIT);
spin_unlock_irqrestore(&dwc->lock, flags);
@@ -2006,6 +2010,10 @@ static int dwc3_gadget_ep_queue(struct usb_ep *ep, struct usb_request *request,
int ret;
spin_lock_irqsave(&dwc->lock, flags);
+ if (dwc->err_state != DWC3_ERR_NONE) {
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return -ESHUTDOWN;
+ }
ret = __dwc3_gadget_ep_queue(dep, req);
spin_unlock_irqrestore(&dwc->lock, flags);
@@ -2238,6 +2246,10 @@ static int dwc3_gadget_ep_set_halt(struct usb_ep *ep, int value)
int ret;
spin_lock_irqsave(&dwc->lock, flags);
+ if (dwc->err_state != DWC3_ERR_NONE) {
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return -ESHUTDOWN;
+ }
ret = __dwc3_gadget_ep_set_halt(dep, value, false);
spin_unlock_irqrestore(&dwc->lock, flags);
@@ -2252,6 +2264,10 @@ static int dwc3_gadget_ep_set_wedge(struct usb_ep *ep)
int ret;
spin_lock_irqsave(&dwc->lock, flags);
+ if (dwc->err_state != DWC3_ERR_NONE) {
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return -ESHUTDOWN;
+ }
dep->flags |= DWC3_EP_WEDGE;
if (dep->number == 0 || dep->number == 1)
@@ -2373,6 +2389,10 @@ static int dwc3_gadget_wakeup(struct usb_gadget *g)
int ret;
spin_lock_irqsave(&dwc->lock, flags);
+ if (dwc->err_state != DWC3_ERR_NONE) {
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return -ESHUTDOWN;
+ }
ret = __dwc3_gadget_wakeup(dwc);
spin_unlock_irqrestore(&dwc->lock, flags);
@@ -2706,10 +2726,13 @@ static int dwc3_gadget_pullup(struct usb_gadget *g, int is_on)
synchronize_irq(dwc->irq_gadget);
+ /* Serialize against dwc3_err_recovery_work() */
+ mutex_lock(&dwc->connect_mutex);
if (!is_on)
ret = dwc3_gadget_soft_disconnect(dwc);
else
ret = dwc3_gadget_soft_connect(dwc);
+ mutex_unlock(&dwc->connect_mutex);
pm_runtime_put(dwc->dev);
@@ -2996,8 +3019,6 @@ static void dwc3_gadget_set_ssp_rate(struct usb_gadget *g,
static int dwc3_gadget_vbus_draw(struct usb_gadget *g, unsigned int mA)
{
struct dwc3 *dwc = gadget_to_dwc(g);
- union power_supply_propval val = {0};
- int ret;
if (dwc->usb2_phy)
return usb_phy_set_power(dwc->usb2_phy, mA);
@@ -3005,10 +3026,10 @@ static int dwc3_gadget_vbus_draw(struct usb_gadget *g, unsigned int mA)
if (!dwc->usb_psy)
return -EOPNOTSUPP;
- val.intval = 1000 * mA;
- ret = power_supply_set_property(dwc->usb_psy, POWER_SUPPLY_PROP_INPUT_CURRENT_LIMIT, &val);
+ dwc->current_limit = mA;
+ schedule_work(&dwc->vbus_draw_work);
- return ret;
+ return 0;
}
/**
@@ -3984,6 +4005,10 @@ static void dwc3_gadget_reset_interrupt(struct dwc3 *dwc)
dwc->suspended = false;
+ /* The controller is recovered. */
+ if (dwc->err_state == DWC3_ERR_NONE)
+ dwc->err_recovery_count = 0;
+
/*
* Ideally, dwc3_reset_gadget() would trigger the function
* drivers to stop any active transfers through ep disable.
@@ -4446,6 +4471,12 @@ static irqreturn_t dwc3_thread_interrupt(int irq, void *_evt)
return ret;
}
+static void dwc3_schedule_err_recovery(struct dwc3 *dwc)
+{
+ dwc->err_state = DWC3_ERR_RECOVERY;
+ schedule_work(&dwc->err_recovery_work);
+}
+
static irqreturn_t dwc3_check_event_buf(struct dwc3_event_buffer *evt)
{
struct dwc3 *dwc = evt->dwc;
@@ -4479,9 +4510,21 @@ static irqreturn_t dwc3_check_event_buf(struct dwc3_event_buffer *evt)
return IRQ_NONE;
if (count > evt->length) {
- dev_err_ratelimited(dwc->dev, "invalid count(%u) > evt->length(%u)\n",
+ dev_err(dwc->dev, "invalid count(%u) > evt->length(%u)\n",
count, evt->length);
- return IRQ_NONE;
+ /*
+ * This is a fatal error - the driver and controller are out of
+ * sync on which event has been consumed. Reinitializing the
+ * controller is required to recover. Write the bogus count back
+ * to GEVNTCOUNT to clear the IRQ source, consistent with the
+ * stale event clearing in dwc3_event_buffers_setup(), then
+ * schedule error recovery.
+ */
+ spin_lock(&dwc->lock);
+ dwc3_schedule_err_recovery(dwc);
+ spin_unlock(&dwc->lock);
+ dwc3_writel(dwc->regs, DWC3_GEVNTCOUNT(0), count);
+ return IRQ_HANDLED;
}
evt->count = count;
@@ -4546,6 +4589,57 @@ static void dwc_gadget_release(struct device *dev)
kfree(gadget);
}
+static void dwc3_err_recovery_work(struct work_struct *work)
+{
+ struct dwc3 *dwc = container_of(work, struct dwc3, err_recovery_work);
+ unsigned long flags;
+ int ret;
+
+ /* serializes against dwc3_gadget_pullup() */
+ mutex_lock(&dwc->connect_mutex);
+
+ ret = dwc3_gadget_soft_disconnect(dwc);
+
+ dwc3_disconnect_gadget_sleepable(dwc);
+
+ if (ret)
+ goto err_unrecoverable;
+
+ if (dwc->softconnect) {
+ u32 count;
+ /*
+ * Wait irq to finish before soft_connect resets evt->lpos and
+ * the event buffer registers to avoid racing with
+ * dwc3_process_event_buf().
+ */
+ synchronize_irq(dwc->irq_gadget);
+
+ spin_lock_irqsave(&dwc->lock, flags);
+ count = ++dwc->err_recovery_count;
+ spin_unlock_irqrestore(&dwc->lock, flags);
+
+ if (count > DWC3_ERR_RECOVERY_MAX)
+ goto err_unrecoverable;
+
+ ret = dwc3_gadget_soft_connect(dwc);
+ if (ret)
+ goto err_unrecoverable;
+ }
+ mutex_unlock(&dwc->connect_mutex);
+
+ spin_lock_irqsave(&dwc->lock, flags);
+ dwc->err_state = DWC3_ERR_NONE;
+ spin_unlock_irqrestore(&dwc->lock, flags);
+ return;
+
+err_unrecoverable:
+ dev_err(dwc->dev, "Unable to recover the controller\n");
+ mutex_unlock(&dwc->connect_mutex);
+ spin_lock_irqsave(&dwc->lock, flags);
+ dwc->err_state = DWC3_ERR_UNRECOVERABLE;
+ spin_unlock_irqrestore(&dwc->lock, flags);
+}
+
/**
* dwc3_gadget_init - initializes gadget related registers
* @dwc: pointer to our controller context structure
@@ -4589,6 +4683,8 @@ int dwc3_gadget_init(struct dwc3 *dwc)
}
init_completion(&dwc->ep0_in_setup);
+ INIT_WORK(&dwc->err_recovery_work, dwc3_err_recovery_work);
+ mutex_init(&dwc->connect_mutex);
dwc->gadget = kzalloc(sizeof(struct usb_gadget), GFP_KERNEL);
if (!dwc->gadget) {
ret = -ENOMEM;
@@ -4679,6 +4775,8 @@ void dwc3_gadget_exit(struct dwc3 *dwc)
if (!dwc->gadget)
return;
+ cancel_work_sync(&dwc->err_recovery_work);
+ mutex_destroy(&dwc->connect_mutex);
dwc3_enable_susphy(dwc, true);
usb_del_gadget(dwc->gadget);
dwc3_gadget_free_endpoints(dwc);
diff --git a/drivers/usb/gadget/function/f_ecm.c b/drivers/usb/gadget/function/f_ecm.c
index 35ec50685b49..4ea8787734f9 100644
--- a/drivers/usb/gadget/function/f_ecm.c
+++ b/drivers/usb/gadget/function/f_ecm.c
@@ -692,6 +692,7 @@ ecm_bind(struct usb_configuration *c, struct usb_function *f)
struct usb_ep *ep;
struct f_ecm_opts *ecm_opts;
+ struct net_device *net __free(detach_gadget) = NULL;
struct usb_request *request __free(free_usb_request) = NULL;
if (!can_support_ecm(cdev->gadget))
@@ -699,22 +700,18 @@ ecm_bind(struct usb_configuration *c, struct usb_function *f)
ecm_opts = container_of(f->fi, struct f_ecm_opts, func_inst);
- /*
- * in drivers/usb/gadget/configfs.c:configfs_composite_bind()
- * configurations are bound in sequence with list_for_each_entry,
- * in each configuration its functions are bound in sequence
- * with list_for_each_entry, so we assume no race condition
- * with regard to ecm_opts->bound access
- */
- if (!ecm_opts->bound) {
- mutex_lock(&ecm_opts->lock);
- gether_set_gadget(ecm_opts->net, cdev->gadget);
- status = gether_register_netdev(ecm_opts->net);
- mutex_unlock(&ecm_opts->lock);
- if (status)
- return status;
- ecm_opts->bound = true;
- }
+ scoped_guard(mutex, &ecm_opts->lock)
+ if (ecm_opts->bind_count == 0 && !ecm_opts->bound) {
+ if (!device_is_registered(&ecm_opts->net->dev)) {
+ gether_set_gadget(ecm_opts->net, cdev->gadget);
+ status = gether_register_netdev(ecm_opts->net);
+ } else
+ status = gether_attach_gadget(ecm_opts->net, cdev->gadget);
+
+ if (status)
+ return status;
+ net = ecm_opts->net;
+ }
ecm_string_defs[1].s = ecm->ethaddr;
@@ -805,6 +802,9 @@ ecm_bind(struct usb_configuration *c, struct usb_function *f)
ecm->notify_req = no_free_ptr(request);
+ ecm_opts->bind_count++;
+ no_free_ptr(net);
+
DBG(cdev, "CDC Ethernet: %s speed IN/%s OUT/%s NOTIFY/%s\n",
gadget_is_superspeed(c->cdev->gadget) ? "super" :
gadget_is_dualspeed(c->cdev->gadget) ? "dual" : "full",
@@ -853,7 +853,7 @@ static void ecm_free_inst(struct usb_function_instance *f)
struct f_ecm_opts *opts;
opts = container_of(f, struct f_ecm_opts, func_inst);
- if (opts->bound)
+ if (device_is_registered(&opts->net->dev))
gether_cleanup(netdev_priv(opts->net));
else
free_netdev(opts->net);
@@ -897,9 +897,12 @@ static void ecm_free(struct usb_function *f)
static void ecm_unbind(struct usb_configuration *c, struct usb_function *f)
{
struct f_ecm *ecm = func_to_ecm(f);
+ struct f_ecm_opts *ecm_opts;
DBG(c->cdev, "ecm unbind\n");
+ ecm_opts = container_of(f->fi, struct f_ecm_opts, func_inst);
+
usb_free_all_descriptors(f);
if (atomic_read(&ecm->notify_count)) {
@@ -909,6 +912,10 @@ static void ecm_unbind(struct usb_configuration *c, struct usb_function *f)
kfree(ecm->notify_req->buf);
usb_ep_free_request(ecm->notify, ecm->notify_req);
+
+ ecm_opts->bind_count--;
+ if (ecm_opts->bind_count == 0 && !ecm_opts->bound)
+ gether_detach_gadget(ecm_opts->net);
}
static struct usb_function *ecm_alloc(struct usb_function_instance *fi)
diff --git a/drivers/usb/gadget/function/f_eem.c b/drivers/usb/gadget/function/f_eem.c
index a308163fc8bc..803fe047c512 100644
--- a/drivers/usb/gadget/function/f_eem.c
+++ b/drivers/usb/gadget/function/f_eem.c
@@ -7,6 +7,7 @@
* Copyright (C) 2009 EF Johnson Technologies
*/
+#include <linux/cleanup.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/device.h>
@@ -251,24 +252,22 @@ static int eem_bind(struct usb_configuration *c, struct usb_function *f)
struct usb_ep *ep;
struct f_eem_opts *eem_opts;
+ struct net_device *net __free(detach_gadget) = NULL;
eem_opts = container_of(f->fi, struct f_eem_opts, func_inst);
- /*
- * in drivers/usb/gadget/configfs.c:configfs_composite_bind()
- * configurations are bound in sequence with list_for_each_entry,
- * in each configuration its functions are bound in sequence
- * with list_for_each_entry, so we assume no race condition
- * with regard to eem_opts->bound access
- */
- if (!eem_opts->bound) {
- mutex_lock(&eem_opts->lock);
- gether_set_gadget(eem_opts->net, cdev->gadget);
- status = gether_register_netdev(eem_opts->net);
- mutex_unlock(&eem_opts->lock);
- if (status)
- return status;
- eem_opts->bound = true;
- }
+
+ scoped_guard(mutex, &eem_opts->lock)
+ if (eem_opts->bind_count == 0 && !eem_opts->bound) {
+ if (!device_is_registered(&eem_opts->net->dev)) {
+ gether_set_gadget(eem_opts->net, cdev->gadget);
+ status = gether_register_netdev(eem_opts->net);
+ } else
+ status = gether_attach_gadget(eem_opts->net, cdev->gadget);
+
+ if (status)
+ return status;
+ net = eem_opts->net;
+ }
us = usb_gstrings_attach(cdev, eem_strings,
ARRAY_SIZE(eem_string_defs));
@@ -279,21 +278,19 @@ static int eem_bind(struct usb_configuration *c, struct usb_function *f)
/* allocate instance-specific interface IDs */
status = usb_interface_id(c, f);
if (status < 0)
- goto fail;
+ return status;
eem->ctrl_id = status;
eem_intf.bInterfaceNumber = status;
- status = -ENODEV;
-
/* allocate instance-specific endpoints */
ep = usb_ep_autoconfig(cdev->gadget, &eem_fs_in_desc);
if (!ep)
- goto fail;
+ return -ENODEV;
eem->port.in_ep = ep;
ep = usb_ep_autoconfig(cdev->gadget, &eem_fs_out_desc);
if (!ep)
- goto fail;
+ return -ENODEV;
eem->port.out_ep = ep;
/* support all relevant hardware speeds... we expect that when
@@ -309,18 +306,16 @@ static int eem_bind(struct usb_configuration *c, struct usb_function *f)
status = usb_assign_descriptors(f, eem_fs_function, eem_hs_function,
eem_ss_function, eem_ss_function);
if (status)
- goto fail;
+ return status;
+
+ eem_opts->bind_count++;
+ no_free_ptr(net);
DBG(cdev, "CDC Ethernet (EEM): %s speed IN/%s OUT/%s\n",
gadget_is_superspeed(c->cdev->gadget) ? "super" :
gadget_is_dualspeed(c->cdev->gadget) ? "dual" : "full",
eem->port.in_ep->name, eem->port.out_ep->name);
return 0;
-
-fail:
- ERROR(cdev, "%s: can't bind, err %d\n", f->name, status);
-
- return status;
}
static void eem_cmd_complete(struct usb_ep *ep, struct usb_request *req)
@@ -599,7 +594,7 @@ static void eem_free_inst(struct usb_function_instance *f)
struct f_eem_opts *opts;
opts = container_of(f, struct f_eem_opts, func_inst);
- if (opts->bound)
+ if (device_is_registered(&opts->net->dev))
gether_cleanup(netdev_priv(opts->net));
else
free_netdev(opts->net);
@@ -642,9 +637,17 @@ static void eem_free(struct usb_function *f)
static void eem_unbind(struct usb_configuration *c, struct usb_function *f)
{
+ struct f_eem_opts *opts;
+
DBG(c->cdev, "eem unbind\n");
+ opts = container_of(f->fi, struct f_eem_opts, func_inst);
+
usb_free_all_descriptors(f);
+
+ opts->bind_count--;
+ if (opts->bind_count == 0 && !opts->bound)
+ gether_detach_gadget(opts->net);
}
static struct usb_function *eem_alloc(struct usb_function_instance *fi)
diff --git a/drivers/usb/gadget/function/f_fs.c b/drivers/usb/gadget/function/f_fs.c
index 56350ce567cd..b3979b24e318 100644
--- a/drivers/usb/gadget/function/f_fs.c
+++ b/drivers/usb/gadget/function/f_fs.c
@@ -336,8 +336,6 @@ static ssize_t ffs_ep0_write(struct file *file, const char __user *buf,
ssize_t ret;
char *data;
- ENTER();
-
/* Fast check if setup was canceled */
if (ffs_setup_state_clear_cancelled(ffs) == FFS_SETUP_CANCELLED)
return -EIDRM;
@@ -513,8 +511,6 @@ static ssize_t ffs_ep0_read(struct file *file, char __user *buf,
size_t n;
int ret;
- ENTER();
-
/* Fast check if setup was canceled */
if (ffs_setup_state_clear_cancelled(ffs) == FFS_SETUP_CANCELLED)
return -EIDRM;
@@ -617,15 +613,22 @@ static ssize_t ffs_ep0_read(struct file *file, char __user *buf,
static int ffs_ep0_open(struct inode *inode, struct file *file)
{
- struct ffs_data *ffs = inode->i_private;
+ struct ffs_data *ffs = inode->i_sb->s_fs_info;
+ int ret;
- ENTER();
+ /* Acquire mutex */
+ ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK);
+ if (ret < 0)
+ return ret;
- if (ffs->state == FFS_CLOSING)
+ ffs_data_opened(ffs);
+ if (ffs->state == FFS_CLOSING) {
+ ffs_data_closed(ffs);
+ mutex_unlock(&ffs->mutex);
return -EBUSY;
-
+ }
+ mutex_unlock(&ffs->mutex);
file->private_data = ffs;
- ffs_data_opened(ffs);
return stream_open(inode, file);
}
@@ -634,8 +637,6 @@ static int ffs_ep0_release(struct inode *inode, struct file *file)
{
struct ffs_data *ffs = file->private_data;
- ENTER();
-
ffs_data_closed(ffs);
return 0;
@@ -647,8 +648,6 @@ static long ffs_ep0_ioctl(struct file *file, unsigned code, unsigned long value)
struct usb_gadget *gadget = ffs->gadget;
long ret;
- ENTER();
-
if (code == FUNCTIONFS_INTERFACE_REVMAP) {
struct ffs_function *func = ffs->func;
ret = func ? ffs_func_revmap_intf(func, value) : -ENODEV;
@@ -722,7 +721,6 @@ static void ffs_epfile_io_complete(struct usb_ep *_ep, struct usb_request *req)
{
struct ffs_io_data *io_data = req->context;
- ENTER();
if (req->status)
io_data->status = req->status;
else
@@ -842,9 +840,15 @@ static void ffs_user_copy_worker(struct work_struct *work)
unsigned long flags;
if (io_data->read && ret > 0) {
- kthread_use_mm(io_data->mm);
- ret = ffs_copy_to_iter(io_data->buf, ret, &io_data->data);
- kthread_unuse_mm(io_data->mm);
+ if (mmget_not_zero(io_data->mm)) {
+ kthread_use_mm(io_data->mm);
+ ret = ffs_copy_to_iter(io_data->buf, ret, &io_data->data);
+ kthread_unuse_mm(io_data->mm);
+ mmput(io_data->mm);
+ } else {
+ ret = -EFAULT;
+ }
+ mmdrop(io_data->mm);
}
io_data->kiocb->ki_complete(io_data->kiocb, ret);
@@ -869,8 +873,6 @@ static void ffs_epfile_async_io_complete(struct usb_ep *_ep,
struct ffs_io_data *io_data = req->context;
struct ffs_data *ffs = io_data->ffs;
- ENTER();
-
io_data->status = req->status ? req->status : req->actual;
INIT_WORK(&io_data->work, ffs_user_copy_worker);
@@ -959,31 +961,44 @@ static ssize_t __ffs_epfile_read_data(struct ffs_epfile *epfile,
return ret;
}
-static ssize_t ffs_epfile_io(struct file *file, struct ffs_io_data *io_data)
+static struct ffs_ep *ffs_epfile_wait_ep(struct file *file)
{
struct ffs_epfile *epfile = file->private_data;
- struct usb_request *req;
struct ffs_ep *ep;
- char *data = NULL;
- ssize_t ret, data_len = -EINVAL;
- int halt;
-
- /* Are we still active? */
- if (WARN_ON(epfile->ffs->state != FFS_ACTIVE))
- return -ENODEV;
+ int ret;
/* Wait for endpoint to be enabled */
ep = epfile->ep;
if (!ep) {
if (file->f_flags & O_NONBLOCK)
- return -EAGAIN;
+ return ERR_PTR(-EAGAIN);
ret = wait_event_interruptible(
epfile->ffs->wait, (ep = epfile->ep));
if (ret)
- return -EINTR;
+ return ERR_PTR(-EINTR);
}
+ return ep;
+}
+
+static ssize_t ffs_epfile_io(struct file *file, struct ffs_io_data *io_data)
+{
+ struct ffs_epfile *epfile = file->private_data;
+ struct usb_request *req;
+ struct ffs_ep *ep;
+ char *data = NULL;
+ ssize_t ret, data_len = -EINVAL;
+ int halt;
+
+ /* Are we still active? */
+ if (WARN_ON(epfile->ffs->state != FFS_ACTIVE))
+ return -ENODEV;
+
+ ep = ffs_epfile_wait_ep(file);
+ if (IS_ERR(ep))
+ return PTR_ERR(ep);
+
/* Do we halt? */
halt = (!io_data->read == !epfile->in);
if (halt && epfile->isoc)
@@ -1171,16 +1186,34 @@ static ssize_t ffs_epfile_io(struct file *file, struct ffs_io_data *io_data)
static int
ffs_epfile_open(struct inode *inode, struct file *file)
{
- struct ffs_epfile *epfile = inode->i_private;
+ struct ffs_data *ffs = inode->i_sb->s_fs_info;
+ struct ffs_epfile *epfile;
+ int ret;
- ENTER();
+ /* Acquire mutex */
+ ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK);
+ if (ret < 0)
+ return ret;
- if (WARN_ON(epfile->ffs->state != FFS_ACTIVE))
+ if (!atomic_inc_not_zero(&ffs->opened)) {
+ mutex_unlock(&ffs->mutex);
return -ENODEV;
+ }
+ refcount_inc(&ffs->ref);
+ /*
+ * we want the state to be FFS_ACTIVE; FFS_ACTIVE alone is
+ * not enough, though - we might have been through FFS_CLOSING
+ * and back to FFS_ACTIVE, with our file already removed.
+ */
+ epfile = smp_load_acquire(&inode->i_private);
+ if (unlikely(ffs->state != FFS_ACTIVE || !epfile)) {
+ mutex_unlock(&ffs->mutex);
+ ffs_data_closed(ffs);
+ return -ENODEV;
+ }
+ mutex_unlock(&ffs->mutex);
file->private_data = epfile;
- ffs_data_opened(epfile->ffs);
-
return stream_open(inode, file);
}
@@ -1191,8 +1224,6 @@ static int ffs_aio_cancel(struct kiocb *kiocb)
unsigned long flags;
int value;
- ENTER();
-
spin_lock_irqsave(&epfile->ffs->eps_lock, flags);
if (io_data && io_data->ep && io_data->req)
@@ -1210,8 +1241,6 @@ static ssize_t ffs_epfile_write_iter(struct kiocb *kiocb, struct iov_iter *from)
struct ffs_io_data io_data, *p = &io_data;
ssize_t res;
- ENTER();
-
if (!is_sync_kiocb(kiocb)) {
p = kzalloc(sizeof(io_data), GFP_KERNEL);
if (!p)
@@ -1229,16 +1258,22 @@ static ssize_t ffs_epfile_write_iter(struct kiocb *kiocb, struct iov_iter *from)
kiocb->private = p;
- if (p->aio)
+ if (p->aio) {
+ mmgrab(p->mm);
kiocb_set_cancel_fn(kiocb, ffs_aio_cancel);
+ }
res = ffs_epfile_io(kiocb->ki_filp, p);
if (res == -EIOCBQUEUED)
return res;
- if (p->aio)
+ if (p->aio) {
+ kiocb->ki_complete(kiocb, res);
+ mmdrop(p->mm);
kfree(p);
- else
+ return -EIOCBQUEUED;
+ } else {
*from = p->data;
+ }
return res;
}
@@ -1247,8 +1282,6 @@ static ssize_t ffs_epfile_read_iter(struct kiocb *kiocb, struct iov_iter *to)
struct ffs_io_data io_data, *p = &io_data;
ssize_t res;
- ENTER();
-
if (!is_sync_kiocb(kiocb)) {
p = kzalloc(sizeof(io_data), GFP_KERNEL);
if (!p)
@@ -1275,16 +1308,21 @@ static ssize_t ffs_epfile_read_iter(struct kiocb *kiocb, struct iov_iter *to)
kiocb->private = p;
- if (p->aio)
+ if (p->aio) {
+ mmgrab(p->mm);
kiocb_set_cancel_fn(kiocb, ffs_aio_cancel);
+ }
res = ffs_epfile_io(kiocb->ki_filp, p);
if (res == -EIOCBQUEUED)
return res;
if (p->aio) {
+ kiocb->ki_complete(kiocb, res);
+ mmdrop(p->mm);
kfree(p->to_free);
kfree(p);
+ return -EIOCBQUEUED;
} else {
*to = p->data;
}
@@ -1294,9 +1332,7 @@ static ssize_t ffs_epfile_read_iter(struct kiocb *kiocb, struct iov_iter *to)
static int
ffs_epfile_release(struct inode *inode, struct file *file)
{
- struct ffs_epfile *epfile = inode->i_private;
-
- ENTER();
+ struct ffs_epfile *epfile = file->private_data;
ffs_data_closed(epfile->ffs);
@@ -1310,22 +1346,13 @@ static long ffs_epfile_ioctl(struct file *file, unsigned code,
struct ffs_ep *ep;
int ret;
- ENTER();
-
if (WARN_ON(epfile->ffs->state != FFS_ACTIVE))
return -ENODEV;
/* Wait for endpoint to be enabled */
- ep = epfile->ep;
- if (!ep) {
- if (file->f_flags & O_NONBLOCK)
- return -EAGAIN;
-
- ret = wait_event_interruptible(
- epfile->ffs->wait, (ep = epfile->ep));
- if (ret)
- return -EINTR;
- }
+ ep = ffs_epfile_wait_ep(file);
+ if (IS_ERR(ep))
+ return PTR_ERR(ep);
spin_lock_irq(&epfile->ffs->eps_lock);
@@ -1351,8 +1378,9 @@ static long ffs_epfile_ioctl(struct file *file, unsigned code,
break;
case FUNCTIONFS_ENDPOINT_DESC:
{
+ const struct usb_endpoint_descriptor *desc;
+ struct usb_endpoint_descriptor desc1;
int desc_idx;
- struct usb_endpoint_descriptor desc1, *desc;
switch (epfile->ffs->gadget->speed) {
case USB_SPEED_SUPER:
@@ -1366,7 +1394,17 @@ static long ffs_epfile_ioctl(struct file *file, unsigned code,
desc_idx = 0;
}
- desc = epfile->ep->descs[desc_idx];
+ do {
+ desc = epfile->ep->descs[desc_idx];
+ } while (!desc && --desc_idx >= 0);
+
+ if (!desc)
+ desc = epfile->ep->ep->desc;
+ if (!desc) {
+ ret = -EINVAL;
+ break;
+ }
+
memcpy(&desc1, desc, desc->bLength);
spin_unlock_irq(&epfile->ffs->eps_lock);
@@ -1410,8 +1448,6 @@ ffs_sb_make_inode(struct super_block *sb, void *data,
{
struct inode *inode;
- ENTER();
-
inode = new_inode(sb);
if (inode) {
@@ -1443,8 +1479,6 @@ static struct dentry *ffs_sb_create_file(struct super_block *sb,
struct dentry *dentry;
struct inode *inode;
- ENTER();
-
dentry = d_alloc_name(sb->s_root, name);
if (!dentry)
return NULL;
@@ -1479,8 +1513,6 @@ static int ffs_sb_fill(struct super_block *sb, struct fs_context *fc)
struct inode *inode;
struct ffs_data *ffs = data->ffs_data;
- ENTER();
-
ffs->sb = sb;
data->ffs_data = NULL;
sb->s_fs_info = ffs;
@@ -1532,8 +1564,6 @@ static int ffs_fs_parse_param(struct fs_context *fc, struct fs_parameter *param)
struct fs_parse_result result;
int opt;
- ENTER();
-
opt = fs_parse(fc, ffs_fs_fs_parameters, param, &result);
if (opt < 0)
return opt;
@@ -1583,8 +1613,6 @@ static int ffs_fs_get_tree(struct fs_context *fc)
struct ffs_data *ffs;
int ret;
- ENTER();
-
if (!fc->source)
return invalf(fc, "No source specified");
@@ -1651,8 +1679,6 @@ static int ffs_fs_init_fs_context(struct fs_context *fc)
static void
ffs_fs_kill_sb(struct super_block *sb)
{
- ENTER();
-
kill_litter_super(sb);
if (sb->s_fs_info)
ffs_data_closed(sb->s_fs_info);
@@ -1674,8 +1700,6 @@ static int functionfs_init(void)
{
int ret;
- ENTER();
-
ret = register_filesystem(&ffs_fs_type);
if (!ret)
pr_info("file system registered\n");
@@ -1687,8 +1711,6 @@ static int functionfs_init(void)
static void functionfs_cleanup(void)
{
- ENTER();
-
pr_info("unloading\n");
unregister_filesystem(&ffs_fs_type);
}
@@ -1701,15 +1723,11 @@ static void ffs_data_reset(struct ffs_data *ffs);
static void ffs_data_get(struct ffs_data *ffs)
{
- ENTER();
-
refcount_inc(&ffs->ref);
}
static void ffs_data_opened(struct ffs_data *ffs)
{
- ENTER();
-
refcount_inc(&ffs->ref);
if (atomic_add_return(1, &ffs->opened) == 1 &&
ffs->state == FFS_DEACTIVATED) {
@@ -1720,8 +1738,6 @@ static void ffs_data_opened(struct ffs_data *ffs)
static void ffs_data_put(struct ffs_data *ffs)
{
- ENTER();
-
if (refcount_dec_and_test(&ffs->ref)) {
pr_info("%s(): freeing\n", __func__);
ffs_data_clear(ffs);
@@ -1740,8 +1756,6 @@ static void ffs_data_closed(struct ffs_data *ffs)
struct ffs_epfile *epfiles;
unsigned long flags;
- ENTER();
-
if (atomic_dec_and_test(&ffs->opened)) {
if (ffs->no_disconnect) {
ffs->state = FFS_DEACTIVATED;
@@ -1776,8 +1790,6 @@ static struct ffs_data *ffs_data_new(const char *dev_name)
if (!ffs)
return NULL;
- ENTER();
-
ffs->io_completion_wq = alloc_ordered_workqueue("%s", 0, dev_name);
if (!ffs->io_completion_wq) {
kfree(ffs);
@@ -1805,8 +1817,6 @@ static void ffs_data_clear(struct ffs_data *ffs)
struct ffs_epfile *epfiles;
unsigned long flags;
- ENTER();
-
ffs_closed(ffs);
BUG_ON(ffs->gadget);
@@ -1838,8 +1848,6 @@ static void ffs_data_clear(struct ffs_data *ffs)
static void ffs_data_reset(struct ffs_data *ffs)
{
- ENTER();
-
ffs_data_clear(ffs);
ffs->raw_descs_data = NULL;
@@ -1873,8 +1881,6 @@ static int functionfs_bind(struct ffs_data *ffs, struct usb_composite_dev *cdev)
struct usb_gadget_strings **lang;
int first_id;
- ENTER();
-
if ((ffs->state != FFS_ACTIVE
|| test_and_set_bit(FFS_FL_BOUND, &ffs->flags)))
return -EBADFD;
@@ -1906,8 +1912,6 @@ static int functionfs_bind(struct ffs_data *ffs, struct usb_composite_dev *cdev)
static void functionfs_unbind(struct ffs_data *ffs)
{
- ENTER();
-
if (!WARN_ON(!ffs->gadget)) {
/* dequeue before freeing ep0req */
usb_ep_dequeue(ffs->gadget->ep0, ffs->ep0req);
@@ -1926,8 +1930,6 @@ static int ffs_epfiles_create(struct ffs_data *ffs)
struct ffs_epfile *epfile, *epfiles;
unsigned i, count;
- ENTER();
-
count = ffs->eps_count;
epfiles = kcalloc(count, sizeof(*epfiles), GFP_KERNEL);
if (!epfiles)
@@ -1954,18 +1956,20 @@ static int ffs_epfiles_create(struct ffs_data *ffs)
return 0;
}
+static void clear_one(struct dentry *dentry)
+{
+ smp_store_release(&dentry->d_inode->i_private, NULL);
+}
+
static void ffs_epfiles_destroy(struct ffs_epfile *epfiles, unsigned count)
{
struct ffs_epfile *epfile = epfiles;
- ENTER();
-
for (; count; --count, ++epfile) {
BUG_ON(mutex_is_locked(&epfile->mutex));
__ffs_epfile_read_buffer_free(epfile);
if (epfile->dentry) {
- d_delete(epfile->dentry);
- dput(epfile->dentry);
+ simple_recursive_removal(epfile->dentry, clear_one);
epfile->dentry = NULL;
}
}
@@ -2083,8 +2087,6 @@ static int __must_check ffs_do_single_desc(char *data, unsigned len,
u8 length;
int ret;
- ENTER();
-
/* At least two bytes are required: length and type */
if (len < 2) {
pr_vdebug("descriptor too short\n");
@@ -2223,8 +2225,6 @@ static int __must_check ffs_do_descs(unsigned count, char *data, unsigned len,
unsigned long num = 0;
int current_class = -1;
- ENTER();
-
for (;;) {
int ret;
@@ -2262,8 +2262,6 @@ static int __ffs_data_do_entity(enum ffs_entity_type type,
struct ffs_desc_helper *helper = priv;
struct usb_endpoint_descriptor *d;
- ENTER();
-
switch (type) {
case FFS_DESCRIPTOR:
break;
@@ -2345,8 +2343,6 @@ static int __must_check ffs_do_single_os_desc(char *data, unsigned len,
int ret;
const unsigned _len = len;
- ENTER();
-
/* loop over all ext compat/ext prop descriptors */
while (feature_count--) {
ret = entity(type, h, data, len, priv);
@@ -2368,8 +2364,6 @@ static int __must_check ffs_do_os_descs(unsigned count,
const unsigned _len = len;
unsigned long num = 0;
- ENTER();
-
for (num = 0; num < count; ++num) {
int ret;
enum ffs_os_desc_type type;
@@ -2432,8 +2426,6 @@ static int __ffs_data_do_os_desc(enum ffs_os_desc_type type,
struct ffs_data *ffs = priv;
u8 length;
- ENTER();
-
switch (type) {
case FFS_OS_DESC_EXT_COMPAT: {
struct usb_ext_compat_desc *d = data;
@@ -2509,8 +2501,6 @@ static int __ffs_data_got_descs(struct ffs_data *ffs,
int ret = -EINVAL, i;
struct ffs_desc_helper helper;
- ENTER();
-
if (get_unaligned_le32(data + 4) != len)
goto error;
@@ -2641,8 +2631,6 @@ static int __ffs_data_got_strings(struct ffs_data *ffs,
const char *data = _data;
struct usb_string *s;
- ENTER();
-
if (len < 16 ||
get_unaligned_le32(data) != FUNCTIONFS_STRINGS_MAGIC ||
get_unaligned_le32(data + 4) != len)
@@ -3101,8 +3089,6 @@ static inline struct f_fs_opts *ffs_do_functionfs_bind(struct usb_function *f,
struct ffs_data *ffs_data;
int ret;
- ENTER();
-
/*
* Legacy gadget triggers binding in functionfs_ready_callback,
* which already uses locking; taking the same lock here would
@@ -3179,8 +3165,6 @@ static int _ffs_func_bind(struct usb_configuration *c,
vla_item_with_sz(d, char, raw_descs, ffs->raw_descs_length);
char *vlabuf;
- ENTER();
-
/* Has descriptors only for speeds gadget does not support */
if (!(full | high | super))
return -ENOTSUPP;
@@ -3383,8 +3367,6 @@ static int ffs_func_setup(struct usb_function *f,
unsigned long flags;
int ret;
- ENTER();
-
pr_vdebug("creq->bRequestType = %02x\n", creq->bRequestType);
pr_vdebug("creq->bRequest = %02x\n", creq->bRequest);
pr_vdebug("creq->wValue = %04x\n", le16_to_cpu(creq->wValue));
@@ -3459,13 +3441,11 @@ static bool ffs_func_req_match(struct usb_function *f,
static void ffs_func_suspend(struct usb_function *f)
{
- ENTER();
ffs_event_add(ffs_func_from_usb(f)->ffs, FUNCTIONFS_SUSPEND);
}
static void ffs_func_resume(struct usb_function *f)
{
- ENTER();
ffs_event_add(ffs_func_from_usb(f)->ffs, FUNCTIONFS_RESUME);
}
@@ -3629,7 +3609,6 @@ static void ffs_func_unbind(struct usb_configuration *c,
unsigned count = ffs->eps_count;
unsigned long flags;
- ENTER();
if (ffs->func == func) {
ffs_func_eps_disable(func);
ffs->func = NULL;
@@ -3669,8 +3648,6 @@ static struct usb_function *ffs_alloc(struct usb_function_instance *fi)
{
struct ffs_function *func;
- ENTER();
-
func = kzalloc(sizeof(*func), GFP_KERNEL);
if (!func)
return ERR_PTR(-ENOMEM);
@@ -3771,7 +3748,6 @@ static int ffs_acquire_dev(const char *dev_name, struct ffs_data *ffs_data)
int ret = 0;
struct ffs_dev *ffs_dev;
- ENTER();
ffs_dev_lock();
ffs_dev = _ffs_find_dev(dev_name);
@@ -3794,7 +3770,6 @@ static int ffs_acquire_dev(const char *dev_name, struct ffs_data *ffs_data)
static void ffs_release_dev(struct ffs_dev *ffs_dev)
{
- ENTER();
ffs_dev_lock();
if (ffs_dev && ffs_dev->mounted) {
@@ -3816,7 +3791,6 @@ static int ffs_ready(struct ffs_data *ffs)
struct ffs_dev *ffs_obj;
int ret = 0;
- ENTER();
ffs_dev_lock();
ffs_obj = ffs->private_data;
@@ -3849,7 +3823,6 @@ static void ffs_closed(struct ffs_data *ffs)
struct f_fs_opts *opts;
struct config_item *ci;
- ENTER();
ffs_dev_lock();
ffs_obj = ffs->private_data;
diff --git a/drivers/usb/gadget/function/f_hid.c b/drivers/usb/gadget/function/f_hid.c
index a3e79121d9eb..6d12e30f820d 100644
--- a/drivers/usb/gadget/function/f_hid.c
+++ b/drivers/usb/gadget/function/f_hid.c
@@ -72,7 +72,7 @@ struct f_hidg {
struct usb_request *req;
struct device dev;
- struct cdev cdev;
+ struct cdev *cdev;
struct usb_function func;
struct usb_ep *in_ep;
@@ -554,8 +554,9 @@ static int f_hidg_release(struct inode *inode, struct file *fd)
static int f_hidg_open(struct inode *inode, struct file *fd)
{
+ struct kobject *parent = inode->i_cdev->kobj.parent;
struct f_hidg *hidg =
- container_of(inode->i_cdev, struct f_hidg, cdev);
+ container_of(parent, struct f_hidg, dev.kobj);
fd->private_data = hidg;
@@ -1000,8 +1001,14 @@ static int hidg_bind(struct usb_configuration *c, struct usb_function *f)
hidg->req = NULL;
/* create char device */
- cdev_init(&hidg->cdev, &f_hidg_fops);
- status = cdev_device_add(&hidg->cdev, &hidg->dev);
+ hidg->cdev = cdev_alloc();
+ if (!hidg->cdev) {
+ status = -ENOMEM;
+ goto fail_free_descs;
+ }
+ hidg->cdev->ops = &f_hidg_fops;
+
+ status = cdev_device_add(hidg->cdev, &hidg->dev);
if (status)
goto fail_free_descs;
@@ -1246,7 +1253,7 @@ static void hidg_unbind(struct usb_configuration *c, struct usb_function *f)
{
struct f_hidg *hidg = func_to_hidg(f);
- cdev_device_del(&hidg->cdev, &hidg->dev);
+ cdev_device_del(hidg->cdev, &hidg->dev);
usb_free_all_descriptors(f);
}
diff --git a/drivers/usb/gadget/function/f_mass_storage.c b/drivers/usb/gadget/function/f_mass_storage.c
index a40be63aca29..5618ca904f5d 100644
--- a/drivers/usb/gadget/function/f_mass_storage.c
+++ b/drivers/usb/gadget/function/f_mass_storage.c
@@ -2747,6 +2747,9 @@ int fsg_common_set_num_buffers(struct fsg_common *common, unsigned int n)
struct fsg_buffhd *bh, *buffhds;
int i;
+ if (n < 2)
+ return -EINVAL;
+
buffhds = kcalloc(n, sizeof(*buffhds), GFP_KERNEL);
if (!buffhds)
return -ENOMEM;
diff --git a/drivers/usb/gadget/function/f_midi.c b/drivers/usb/gadget/function/f_midi.c
index 3b4ec6c0a39b..f556fe7163c0 100644
--- a/drivers/usb/gadget/function/f_midi.c
+++ b/drivers/usb/gadget/function/f_midi.c
@@ -99,7 +99,7 @@ struct f_midi {
unsigned int in_last_port;
unsigned char free_ref;
- struct gmidi_in_port in_ports_array[/* in_ports */];
+ struct gmidi_in_port in_ports_array[] __counted_by(in_ports);
};
static inline struct f_midi *func_to_midi(struct usb_function *f)
@@ -877,7 +877,6 @@ static int f_midi_bind(struct usb_configuration *c, struct usb_function *f)
int status, n, jack = 1, i = 0, endpoint_descriptor_index = 0;
midi->gadget = cdev->gadget;
- INIT_WORK(&midi->work, f_midi_in_work);
status = f_midi_register_card(midi);
if (status < 0)
goto fail_register;
@@ -1360,6 +1359,7 @@ static struct usb_function *f_midi_alloc(struct usb_function_instance *fi)
status = -ENOMEM;
goto setup_fail;
}
+ midi->in_ports = opts->in_ports;
for (i = 0; i < opts->in_ports; i++)
midi->in_ports_array[i].cable = i;
@@ -1370,7 +1370,7 @@ static struct usb_function *f_midi_alloc(struct usb_function_instance *fi)
status = -ENOMEM;
goto midi_free;
}
- midi->in_ports = opts->in_ports;
+ INIT_WORK(&midi->work, f_midi_in_work);
midi->out_ports = opts->out_ports;
midi->index = opts->index;
midi->buflen = opts->buflen;
diff --git a/drivers/usb/gadget/function/f_printer.c b/drivers/usb/gadget/function/f_printer.c
index e4d35f592694..427d80849fa5 100644
--- a/drivers/usb/gadget/function/f_printer.c
+++ b/drivers/usb/gadget/function/f_printer.c
@@ -360,12 +360,11 @@ printer_open(struct inode *inode, struct file *fd)
ret = 0;
/* Change the printer status to show that it's on-line. */
dev->printer_status |= PRINTER_SELECTED;
+ kref_get(&dev->kref);
}
spin_unlock_irqrestore(&dev->lock, flags);
- kref_get(&dev->kref);
- DBG(dev, "printer_open returned %x\n", ret);
return ret;
}
@@ -383,7 +382,6 @@ printer_close(struct inode *inode, struct file *fd)
spin_unlock_irqrestore(&dev->lock, flags);
kref_put(&dev->kref, printer_dev_free);
- DBG(dev, "printer_close\n");
return 0;
}
@@ -881,8 +879,6 @@ static void printer_reset_interface(struct printer_dev *dev)
if (dev->interface < 0)
return;
- DBG(dev, "%s\n", __func__);
-
if (dev->in_ep->desc)
usb_ep_disable(dev->in_ep);
@@ -920,8 +916,6 @@ static void printer_soft_reset(struct printer_dev *dev)
{
struct usb_request *req;
- INFO(dev, "Received Printer Reset Request\n");
-
if (usb_ep_disable(dev->in_ep))
DBG(dev, "Failed to disable USB in_ep\n");
if (usb_ep_disable(dev->out_ep))
@@ -1218,8 +1212,6 @@ static void printer_func_disable(struct usb_function *f)
{
struct printer_dev *dev = func_to_printer(f);
- DBG(dev, "%s\n", __func__);
-
printer_reset_interface(dev);
}
diff --git a/drivers/usb/gadget/function/f_rndis.c b/drivers/usb/gadget/function/f_rndis.c
index 57c2246d0a4a..8cdb2f408895 100644
--- a/drivers/usb/gadget/function/f_rndis.c
+++ b/drivers/usb/gadget/function/f_rndis.c
@@ -679,6 +679,7 @@ rndis_bind(struct usb_configuration *c, struct usb_function *f)
struct f_rndis_opts *rndis_opts;
struct usb_os_desc_table *os_desc_table __free(kfree) = NULL;
+ struct net_device *net __free(detach_gadget) = NULL;
struct usb_request *request __free(free_usb_request) = NULL;
if (!can_support_rndis(c))
@@ -696,21 +697,18 @@ rndis_bind(struct usb_configuration *c, struct usb_function *f)
rndis_iad_descriptor.bFunctionClass = rndis_opts->class;
rndis_iad_descriptor.bFunctionSubClass = rndis_opts->subclass;
rndis_iad_descriptor.bFunctionProtocol = rndis_opts->protocol;
- }
- /*
- * in drivers/usb/gadget/configfs.c:configfs_composite_bind()
- * configurations are bound in sequence with list_for_each_entry,
- * in each configuration its functions are bound in sequence
- * with list_for_each_entry, so we assume no race condition
- * with regard to rndis_opts->bound access
- */
- if (!rndis_opts->bound) {
- gether_set_gadget(rndis_opts->net, cdev->gadget);
- status = gether_register_netdev(rndis_opts->net);
- if (status)
- return status;
- rndis_opts->bound = true;
+ if (rndis_opts->bind_count == 0 && !rndis_opts->borrowed_net) {
+ if (!device_is_registered(&rndis_opts->net->dev)) {
+ gether_set_gadget(rndis_opts->net, cdev->gadget);
+ status = gether_register_netdev(rndis_opts->net);
+ } else
+ status = gether_attach_gadget(rndis_opts->net, cdev->gadget);
+
+ if (status)
+ return status;
+ net = rndis_opts->net;
+ }
}
us = usb_gstrings_attach(cdev, rndis_strings,
@@ -809,6 +807,9 @@ rndis_bind(struct usb_configuration *c, struct usb_function *f)
}
rndis->notify_req = no_free_ptr(request);
+ rndis_opts->bind_count++;
+ no_free_ptr(net);
+
/* NOTE: all that is done without knowing or caring about
* the network link ... which is unavailable to this code
* until we're activated via set_alt().
@@ -827,11 +828,11 @@ void rndis_borrow_net(struct usb_function_instance *f, struct net_device *net)
struct f_rndis_opts *opts;
opts = container_of(f, struct f_rndis_opts, func_inst);
- if (opts->bound)
+ if (device_is_registered(&opts->net->dev))
gether_cleanup(netdev_priv(opts->net));
else
free_netdev(opts->net);
- opts->borrowed_net = opts->bound = true;
+ opts->borrowed_net = true;
opts->net = net;
}
EXPORT_SYMBOL_GPL(rndis_borrow_net);
@@ -889,7 +890,7 @@ static void rndis_free_inst(struct usb_function_instance *f)
opts = container_of(f, struct f_rndis_opts, func_inst);
if (!opts->borrowed_net) {
- if (opts->bound)
+ if (device_is_registered(&opts->net->dev))
gether_cleanup(netdev_priv(opts->net));
else
free_netdev(opts->net);
@@ -958,6 +959,9 @@ static void rndis_free(struct usb_function *f)
static void rndis_unbind(struct usb_configuration *c, struct usb_function *f)
{
struct f_rndis *rndis = func_to_rndis(f);
+ struct f_rndis_opts *rndis_opts;
+
+ rndis_opts = container_of(f->fi, struct f_rndis_opts, func_inst);
kfree(f->os_desc_table);
f->os_desc_n = 0;
@@ -965,6 +969,10 @@ static void rndis_unbind(struct usb_configuration *c, struct usb_function *f)
kfree(rndis->notify_req->buf);
usb_ep_free_request(rndis->notify, rndis->notify_req);
+
+ rndis_opts->bind_count--;
+ if (rndis_opts->bind_count == 0 && !rndis_opts->borrowed_net)
+ gether_detach_gadget(rndis_opts->net);
}
static struct usb_function *rndis_alloc(struct usb_function_instance *fi)
diff --git a/drivers/usb/gadget/function/f_subset.c b/drivers/usb/gadget/function/f_subset.c
index 447e94c83c7b..bd8e0996d6a8 100644
--- a/drivers/usb/gadget/function/f_subset.c
+++ b/drivers/usb/gadget/function/f_subset.c
@@ -299,25 +299,22 @@ geth_bind(struct usb_configuration *c, struct usb_function *f)
struct usb_ep *ep;
struct f_gether_opts *gether_opts;
+ struct net_device *net __free(detach_gadget) = NULL;
gether_opts = container_of(f->fi, struct f_gether_opts, func_inst);
- /*
- * in drivers/usb/gadget/configfs.c:configfs_composite_bind()
- * configurations are bound in sequence with list_for_each_entry,
- * in each configuration its functions are bound in sequence
- * with list_for_each_entry, so we assume no race condition
- * with regard to gether_opts->bound access
- */
- if (!gether_opts->bound) {
- mutex_lock(&gether_opts->lock);
- gether_set_gadget(gether_opts->net, cdev->gadget);
- status = gether_register_netdev(gether_opts->net);
- mutex_unlock(&gether_opts->lock);
- if (status)
- return status;
- gether_opts->bound = true;
- }
+ scoped_guard(mutex, &gether_opts->lock)
+ if (gether_opts->bind_count == 0 && !gether_opts->bound) {
+ if (!device_is_registered(&gether_opts->net->dev)) {
+ gether_set_gadget(gether_opts->net, cdev->gadget);
+ status = gether_register_netdev(gether_opts->net);
+ } else
+ status = gether_attach_gadget(gether_opts->net, cdev->gadget);
+
+ if (status)
+ return status;
+ net = gether_opts->net;
+ }
us = usb_gstrings_attach(cdev, geth_strings,
ARRAY_SIZE(geth_string_defs));
@@ -330,20 +327,18 @@ geth_bind(struct usb_configuration *c, struct usb_function *f)
/* allocate instance-specific interface IDs */
status = usb_interface_id(c, f);
if (status < 0)
- goto fail;
+ return status;
subset_data_intf.bInterfaceNumber = status;
- status = -ENODEV;
-
/* allocate instance-specific endpoints */
ep = usb_ep_autoconfig(cdev->gadget, &fs_subset_in_desc);
if (!ep)
- goto fail;
+ return -ENODEV;
geth->port.in_ep = ep;
ep = usb_ep_autoconfig(cdev->gadget, &fs_subset_out_desc);
if (!ep)
- goto fail;
+ return -ENODEV;
geth->port.out_ep = ep;
/* support all relevant hardware speeds... we expect that when
@@ -361,23 +356,21 @@ geth_bind(struct usb_configuration *c, struct usb_function *f)
status = usb_assign_descriptors(f, fs_eth_function, hs_eth_function,
ss_eth_function, ss_eth_function);
if (status)
- goto fail;
+ return status;
/* NOTE: all that is done without knowing or caring about
* the network link ... which is unavailable to this code
* until we're activated via set_alt().
*/
+ gether_opts->bind_count++;
+ no_free_ptr(net);
+
DBG(cdev, "CDC Subset: %s speed IN/%s OUT/%s\n",
gadget_is_superspeed(c->cdev->gadget) ? "super" :
gadget_is_dualspeed(c->cdev->gadget) ? "dual" : "full",
geth->port.in_ep->name, geth->port.out_ep->name);
return 0;
-
-fail:
- ERROR(cdev, "%s: can't bind, err %d\n", f->name, status);
-
- return status;
}
static inline struct f_gether_opts *to_f_gether_opts(struct config_item *item)
@@ -420,7 +413,7 @@ static void geth_free_inst(struct usb_function_instance *f)
struct f_gether_opts *opts;
opts = container_of(f, struct f_gether_opts, func_inst);
- if (opts->bound)
+ if (device_is_registered(&opts->net->dev))
gether_cleanup(netdev_priv(opts->net));
else
free_netdev(opts->net);
@@ -464,8 +457,16 @@ static void geth_free(struct usb_function *f)
static void geth_unbind(struct usb_configuration *c, struct usb_function *f)
{
+ struct f_gether_opts *opts;
+
+ opts = container_of(f->fi, struct f_gether_opts, func_inst);
+
geth_string_defs[0].id = 0;
usb_free_all_descriptors(f);
+
+ opts->bind_count--;
+ if (opts->bind_count == 0 && !opts->bound)
+ gether_detach_gadget(opts->net);
}
static struct usb_function *geth_alloc(struct usb_function_instance *fi)
diff --git a/drivers/usb/gadget/function/f_uac1.c b/drivers/usb/gadget/function/f_uac1.c
index 6f0e1d803dc2..7939dadd52a5 100644
--- a/drivers/usb/gadget/function/f_uac1.c
+++ b/drivers/usb/gadget/function/f_uac1.c
@@ -1521,7 +1521,8 @@ static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
const char *page, size_t len) \
{ \
struct f_uac1_opts *opts = to_f_uac1_opts(item); \
- char *split_page = NULL; \
+ char *buf = NULL; \
+ char *split_page; \
int ret = -EINVAL; \
char *token; \
u32 num; \
@@ -1535,18 +1536,22 @@ static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \
\
i = 0; \
memset(opts->name##s, 0x00, sizeof(opts->name##s)); \
- split_page = kstrdup(page, GFP_KERNEL); \
+ buf = kstrdup(page, GFP_KERNEL); \
+ split_page = buf; \
while ((token = strsep(&split_page, ",")) != NULL) { \
ret = kstrtou32(token, 0, &num); \
if (ret) \
goto end; \
- \
+ if (i >= UAC_MAX_RATES) { \
+ ret = -EINVAL; \
+ goto end; \
+ } \
opts->name##s[i++] = num; \
ret = len; \
}; \
\
end: \
- kfree(split_page); \
+ kfree(buf); \
mutex_unlock(&opts->lock); \
return ret; \
} \
diff --git a/drivers/usb/gadget/function/f_uac1_legacy.c b/drivers/usb/gadget/function/f_uac1_legacy.c
index 6b1efbb57072..6724228da7ab 100644
--- a/drivers/usb/gadget/function/f_uac1_legacy.c
+++ b/drivers/usb/gadget/function/f_uac1_legacy.c
@@ -797,6 +797,9 @@ f_audio_bind(struct usb_configuration *c, struct usb_function *f)
static int generic_set_cmd(struct usb_audio_control *con, u8 cmd, int value)
{
+ if (cmd >= ARRAY_SIZE(con->data))
+ return -EINVAL;
+
con->data[cmd] = value;
return 0;
@@ -804,6 +807,9 @@ static int generic_set_cmd(struct usb_audio_control *con, u8 cmd, int value)
static int generic_get_cmd(struct usb_audio_control *con, u8 cmd)
{
+ if (cmd >= ARRAY_SIZE(con->data))
+ return -EINVAL;
+
return con->data[cmd];
}
diff --git a/drivers/usb/gadget/function/f_uac2.c b/drivers/usb/gadget/function/f_uac2.c
index eabd10817e1e..b7d3988caa9c 100644
--- a/drivers/usb/gadget/function/f_uac2.c
+++ b/drivers/usb/gadget/function/f_uac2.c
@@ -2003,7 +2003,8 @@ static ssize_t f_uac2_opts_##name##_store(struct config_item *item, \
const char *page, size_t len) \
{ \
struct f_uac2_opts *opts = to_f_uac2_opts(item); \
- char *split_page = NULL; \
+ char *buf = NULL; \
+ char *split_page; \
int ret = -EINVAL; \
char *token; \
u32 num; \
@@ -2017,18 +2018,22 @@ static ssize_t f_uac2_opts_##name##_store(struct config_item *item, \
\
i = 0; \
memset(opts->name##s, 0x00, sizeof(opts->name##s)); \
- split_page = kstrdup(page, GFP_KERNEL); \
+ buf = kstrdup(page, GFP_KERNEL); \
+ split_page = buf; \
while ((token = strsep(&split_page, ",")) != NULL) { \
ret = kstrtou32(token, 0, &num); \
if (ret) \
goto end; \
- \
+ if (i >= UAC_MAX_RATES) { \
+ ret = -EINVAL; \
+ goto end; \
+ } \
opts->name##s[i++] = num; \
ret = len; \
}; \
\
end: \
- kfree(split_page); \
+ kfree(buf); \
mutex_unlock(&opts->lock); \
return ret; \
} \
diff --git a/drivers/usb/gadget/function/u_ecm.h b/drivers/usb/gadget/function/u_ecm.h
index 77cfb89932be..7f666b9dea02 100644
--- a/drivers/usb/gadget/function/u_ecm.h
+++ b/drivers/usb/gadget/function/u_ecm.h
@@ -15,17 +15,26 @@
#include <linux/usb/composite.h>
+/**
+ * struct f_ecm_opts - ECM function options
+ * @func_inst: USB function instance.
+ * @net: The net_device associated with the ECM function.
+ * @bound: True if the net_device is shared and pre-registered during the
+ * legacy composite driver's bind phase (e.g., multi.c). If false,
+ * the ECM function will register the net_device during its own
+ * bind phase.
+ * @bind_count: Tracks the number of configurations the ECM function is
+ * bound to, preventing double-registration of the @net device.
+ * @lock: Protects the data from concurrent access by configfs read/write
+ * and create symlink/remove symlink operations.
+ * @refcnt: Reference counter for the function instance.
+ */
struct f_ecm_opts {
struct usb_function_instance func_inst;
struct net_device *net;
bool bound;
+ int bind_count;
- /*
- * Read/write access to configfs attributes is handled by configfs.
- *
- * This is to protect the data from concurrent access by read/write
- * and create symlink/remove symlink.
- */
struct mutex lock;
int refcnt;
};
diff --git a/drivers/usb/gadget/function/u_eem.h b/drivers/usb/gadget/function/u_eem.h
index 3bd85dfcd71c..78ef55815219 100644
--- a/drivers/usb/gadget/function/u_eem.h
+++ b/drivers/usb/gadget/function/u_eem.h
@@ -15,17 +15,26 @@
#include <linux/usb/composite.h>
+/**
+ * struct f_eem_opts - EEM function options
+ * @func_inst: USB function instance.
+ * @net: The net_device associated with the EEM function.
+ * @bound: True if the net_device is shared and pre-registered during the
+ * legacy composite driver's bind phase (e.g., multi.c). If false,
+ * the EEM function will register the net_device during its own
+ * bind phase.
+ * @bind_count: Tracks the number of configurations the EEM function is
+ * bound to, preventing double-registration of the @net device.
+ * @lock: Protects the data from concurrent access by configfs read/write
+ * and create symlink/remove symlink operations.
+ * @refcnt: Reference counter for the function instance.
+ */
struct f_eem_opts {
struct usb_function_instance func_inst;
struct net_device *net;
bool bound;
+ int bind_count;
- /*
- * Read/write access to configfs attributes is handled by configfs.
- *
- * This is to protect the data from concurrent access by read/write
- * and create symlink/remove symlink.
- */
struct mutex lock;
int refcnt;
};
diff --git a/drivers/usb/gadget/function/u_fs.h b/drivers/usb/gadget/function/u_fs.h
index f102ec23f3af..4b3365f23fd7 100644
--- a/drivers/usb/gadget/function/u_fs.h
+++ b/drivers/usb/gadget/function/u_fs.h
@@ -32,8 +32,6 @@
# define ffs_dump_mem(prefix, ptr, len) do { } while (0)
#endif /* VERBOSE_DEBUG */
-#define ENTER() pr_vdebug("%s()\n", __func__)
-
struct f_fs_opts;
struct ffs_dev {
diff --git a/drivers/usb/gadget/function/u_gether.h b/drivers/usb/gadget/function/u_gether.h
index 2f7a373ed449..e7b6b51f69c1 100644
--- a/drivers/usb/gadget/function/u_gether.h
+++ b/drivers/usb/gadget/function/u_gether.h
@@ -15,17 +15,25 @@
#include <linux/usb/composite.h>
+/**
+ * struct f_gether_opts - subset function options
+ * @func_inst: USB function instance.
+ * @net: The net_device associated with the subset function.
+ * @bound: True if the net_device is shared and pre-registered during the
+ * legacy composite driver's bind phase (e.g., multi.c). If false,
+ * the subset function will register the net_device during its own
+ * bind phase.
+ * @bind_count: Tracks the number of configurations the subset function is
+ * bound to, preventing double-registration of the @net device.
+ * @lock: Protects the data from concurrent access by configfs read/write
+ * and create symlink/remove symlink operations.
+ * @refcnt: Reference counter for the function instance.
+ */
struct f_gether_opts {
struct usb_function_instance func_inst;
struct net_device *net;
bool bound;
-
- /*
- * Read/write access to configfs attributes is handled by configfs.
- *
- * This is to protect the data from concurrent access by read/write
- * and create symlink/remove symlink.
- */
+ int bind_count;
struct mutex lock;
int refcnt;
};
diff --git a/drivers/usb/gadget/function/u_rndis.h b/drivers/usb/gadget/function/u_rndis.h
index a8c409b2f52f..4e64619714dc 100644
--- a/drivers/usb/gadget/function/u_rndis.h
+++ b/drivers/usb/gadget/function/u_rndis.h
@@ -15,12 +15,34 @@
#include <linux/usb/composite.h>
+/**
+ * struct f_rndis_opts - RNDIS function options
+ * @func_inst: USB function instance.
+ * @vendor_id: Vendor ID.
+ * @manufacturer: Manufacturer string.
+ * @net: The net_device associated with the RNDIS function.
+ * @bind_count: Tracks the number of configurations the RNDIS function is
+ * bound to, preventing double-registration of the @net device.
+ * @borrowed_net: True if the net_device is shared and pre-registered during
+ * the legacy composite driver's bind phase (e.g., multi.c).
+ * If false, the RNDIS function will register the net_device
+ * during its own bind phase.
+ * @rndis_interf_group: ConfigFS group for RNDIS interface.
+ * @rndis_os_desc: USB OS descriptor for RNDIS.
+ * @rndis_ext_compat_id: Extended compatibility ID.
+ * @class: USB class.
+ * @subclass: USB subclass.
+ * @protocol: USB protocol.
+ * @lock: Protects the data from concurrent access by configfs read/write
+ * and create symlink/remove symlink operations.
+ * @refcnt: Reference counter for the function instance.
+ */
struct f_rndis_opts {
struct usb_function_instance func_inst;
u32 vendor_id;
const char *manufacturer;
struct net_device *net;
- bool bound;
+ int bind_count;
bool borrowed_net;
struct config_group *rndis_interf_group;
@@ -30,13 +52,6 @@ struct f_rndis_opts {
u8 class;
u8 subclass;
u8 protocol;
-
- /*
- * Read/write access to configfs attributes is handled by configfs.
- *
- * This is to protect the data from concurrent access by read/write
- * and create symlink/remove symlink.
- */
struct mutex lock;
int refcnt;
};
diff --git a/drivers/usb/gadget/legacy/g_ffs.c b/drivers/usb/gadget/legacy/g_ffs.c
index ae6d8f7092b8..097854683e5b 100644
--- a/drivers/usb/gadget/legacy/g_ffs.c
+++ b/drivers/usb/gadget/legacy/g_ffs.c
@@ -180,8 +180,6 @@ static int __init gfs_init(void)
int i;
int ret = 0;
- ENTER();
-
if (func_num < 2) {
gfs_single_func = true;
func_num = 1;
@@ -242,8 +240,6 @@ static void __exit gfs_exit(void)
{
int i;
- ENTER();
-
if (gfs_registered)
usb_composite_unregister(&gfs_driver);
gfs_registered = false;
@@ -316,8 +312,6 @@ static int gfs_bind(struct usb_composite_dev *cdev)
#endif
int ret, i;
- ENTER();
-
if (missing_funcs)
return -ENODEV;
#if defined CONFIG_USB_FUNCTIONFS_ETH
@@ -445,9 +439,6 @@ static int gfs_unbind(struct usb_composite_dev *cdev)
{
int i;
- ENTER();
-
-
#ifdef CONFIG_USB_FUNCTIONFS_RNDIS
usb_put_function(f_rndis);
usb_put_function_instance(fi_rndis);
diff --git a/drivers/usb/gadget/legacy/inode.c b/drivers/usb/gadget/legacy/inode.c
index 471f3f83753b..9ef922c537b4 100644
--- a/drivers/usb/gadget/legacy/inode.c
+++ b/drivers/usb/gadget/legacy/inode.c
@@ -1547,7 +1547,6 @@ static void destroy_ep_files (struct dev_data *dev)
spin_lock_irq (&dev->lock);
while (!list_empty(&dev->epfiles)) {
struct ep_data *ep;
- struct inode *parent;
struct dentry *dentry;
/* break link to FS */
@@ -1557,7 +1556,6 @@ static void destroy_ep_files (struct dev_data *dev)
dentry = ep->dentry;
ep->dentry = NULL;
- parent = d_inode(dentry->d_parent);
/* break link to controller */
mutex_lock(&ep->lock);
@@ -1572,10 +1570,7 @@ static void destroy_ep_files (struct dev_data *dev)
put_ep (ep);
/* break link to dcache */
- inode_lock(parent);
- d_delete (dentry);
- dput (dentry);
- inode_unlock(parent);
+ simple_recursive_removal(dentry, NULL);
spin_lock_irq (&dev->lock);
}
diff --git a/drivers/usb/gadget/udc/aspeed-vhub/core.c b/drivers/usb/gadget/udc/aspeed-vhub/core.c
index 7a635c499777..90d56ac19a88 100644
--- a/drivers/usb/gadget/udc/aspeed-vhub/core.c
+++ b/drivers/usb/gadget/udc/aspeed-vhub/core.c
@@ -267,6 +267,9 @@ static int ast_vhub_remove(struct platform_device *pdev)
for (i = 0; i < vhub->max_ports; i++)
ast_vhub_del_dev(&vhub->ports[i].dev);
+ /* Final drain; the worker takes vhub->lock, so stay outside of it */
+ cancel_work_sync(&vhub->wake_work);
+
spin_lock_irqsave(&vhub->lock, flags);
/* Mask & ack all interrupts */
@@ -327,6 +330,7 @@ static int ast_vhub_probe(struct platform_device *pdev)
return -ENOMEM;
spin_lock_init(&vhub->lock);
+ INIT_WORK(&vhub->wake_work, ast_vhub_wake_work);
vhub->pdev = pdev;
vhub->port_irq_mask = GENMASK(VHUB_IRQ_DEV1_BIT + vhub->max_ports - 1,
VHUB_IRQ_DEV1_BIT);
diff --git a/drivers/usb/gadget/udc/aspeed-vhub/dev.c b/drivers/usb/gadget/udc/aspeed-vhub/dev.c
index 73664a123c7a..acf159ae25bc 100644
--- a/drivers/usb/gadget/udc/aspeed-vhub/dev.c
+++ b/drivers/usb/gadget/udc/aspeed-vhub/dev.c
@@ -281,7 +281,7 @@ static int ast_vhub_udc_wakeup(struct usb_gadget* gadget)
int rc = -EINVAL;
spin_lock_irqsave(&d->vhub->lock, flags);
- if (!d->wakeup_en)
+ if (!d->wakeup_en || !d->registered)
goto err;
DDBG(d, "Device initiated wakeup\n");
diff --git a/drivers/usb/gadget/udc/aspeed-vhub/hub.c b/drivers/usb/gadget/udc/aspeed-vhub/hub.c
index e2207d014620..e5d67f8eb034 100644
--- a/drivers/usb/gadget/udc/aspeed-vhub/hub.c
+++ b/drivers/usb/gadget/udc/aspeed-vhub/hub.c
@@ -558,7 +558,7 @@ void ast_vhub_device_connect(struct ast_vhub *vhub,
ast_vhub_send_host_wakeup(vhub);
}
-static void ast_vhub_wake_work(struct work_struct *work)
+void ast_vhub_wake_work(struct work_struct *work)
{
struct ast_vhub *vhub = container_of(work,
struct ast_vhub,
@@ -588,6 +588,8 @@ static void ast_vhub_wake_work(struct work_struct *work)
void ast_vhub_hub_wake_all(struct ast_vhub *vhub)
{
+ lockdep_assert_held(&vhub->lock);
+
/*
* A device is trying to wake the world, because this
* can recurse into the device, we break the call chain
@@ -1076,7 +1078,6 @@ static int ast_vhub_init_desc(struct ast_vhub *vhub)
int ast_vhub_init_hub(struct ast_vhub *vhub)
{
vhub->speed = USB_SPEED_UNKNOWN;
- INIT_WORK(&vhub->wake_work, ast_vhub_wake_work);
return ast_vhub_init_desc(vhub);
}
diff --git a/drivers/usb/gadget/udc/aspeed-vhub/vhub.h b/drivers/usb/gadget/udc/aspeed-vhub/vhub.h
index 6b9dfa6e10eb..0ca168b39d96 100644
--- a/drivers/usb/gadget/udc/aspeed-vhub/vhub.h
+++ b/drivers/usb/gadget/udc/aspeed-vhub/vhub.h
@@ -546,6 +546,7 @@ void ast_vhub_hub_suspend(struct ast_vhub *vhub);
void ast_vhub_hub_resume(struct ast_vhub *vhub);
void ast_vhub_hub_reset(struct ast_vhub *vhub);
void ast_vhub_hub_wake_all(struct ast_vhub *vhub);
+void ast_vhub_wake_work(struct work_struct *work);
/* dev.c */
int ast_vhub_init_dev(struct ast_vhub *vhub, unsigned int idx);
diff --git a/drivers/usb/gadget/udc/at91_udc.c b/drivers/usb/gadget/udc/at91_udc.c
index a9a7b3fc60ec..2d55454e6115 100644
--- a/drivers/usb/gadget/udc/at91_udc.c
+++ b/drivers/usb/gadget/udc/at91_udc.c
@@ -1796,6 +1796,19 @@ static void at91udc_of_init(struct at91_udc *udc, struct device_node *np)
udc->caps = match->data;
}
+/*
+ * The work handler re-arms this timer, so shut the timer down before
+ * draining the work; otherwise it restarts the polling cycle.
+ */
+static void at91_udc_shutdown_vbus_timer(struct at91_udc *udc)
+{
+ if (!(udc->board.vbus_pin && udc->board.vbus_polled))
+ return;
+
+ timer_shutdown_sync(&udc->vbus_timer);
+ cancel_work_sync(&udc->vbus_timer_work);
+}
+
static int at91udc_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
@@ -1909,7 +1922,7 @@ static int at91udc_probe(struct platform_device *pdev)
}
retval = usb_add_gadget_udc(dev, &udc->gadget);
if (retval)
- goto err_unprepare_iclk;
+ goto err_shutdown_vbus;
dev_set_drvdata(dev, udc);
device_init_wakeup(dev, 1);
create_debug_file(udc);
@@ -1917,6 +1930,8 @@ static int at91udc_probe(struct platform_device *pdev)
INFO("%s version %s\n", driver_name, DRIVER_VERSION);
return 0;
+err_shutdown_vbus:
+ at91_udc_shutdown_vbus_timer(udc);
err_unprepare_iclk:
clk_unprepare(udc->iclk);
err_unprepare_fclk:
@@ -1927,7 +1942,7 @@ static int at91udc_probe(struct platform_device *pdev)
return retval;
}
-static int at91udc_remove(struct platform_device *pdev)
+static void at91udc_remove(struct platform_device *pdev)
{
struct at91_udc *udc = platform_get_drvdata(pdev);
unsigned long flags;
@@ -1935,8 +1950,14 @@ static int at91udc_remove(struct platform_device *pdev)
DBG("remove\n");
usb_del_gadget_udc(&udc->gadget);
- if (udc->driver)
- return -EBUSY;
+
+ at91_udc_shutdown_vbus_timer(udc);
+
+ if (udc->driver) {
+ dev_err(&pdev->dev,
+ "Driver still in use but removing anyhow\n");
+ return;
+ }
spin_lock_irqsave(&udc->lock, flags);
pullup(udc, 0);
@@ -1946,8 +1967,6 @@ static int at91udc_remove(struct platform_device *pdev)
remove_debug_file(udc);
clk_unprepare(udc->fclk);
clk_unprepare(udc->iclk);
-
- return 0;
}
#ifdef CONFIG_PM
@@ -2003,7 +2022,7 @@ static int at91udc_resume(struct platform_device *pdev)
#endif
static struct platform_driver at91_udc_driver = {
- .remove = at91udc_remove,
+ .remove_new = at91udc_remove,
.shutdown = at91udc_shutdown,
.suspend = at91udc_suspend,
.resume = at91udc_resume,
diff --git a/drivers/usb/gadget/udc/core.c b/drivers/usb/gadget/udc/core.c
index addcff2f7711..bc9c060c9e27 100644
--- a/drivers/usb/gadget/udc/core.c
+++ b/drivers/usb/gadget/udc/core.c
@@ -913,6 +913,11 @@ int usb_gadget_map_request_by_dev(struct device *dev,
if (req->length == 0)
return 0;
+ if (req->sg_was_mapped) {
+ req->num_mapped_sgs = req->num_sgs;
+ return 0;
+ }
+
if (req->num_sgs) {
int mapped;
@@ -958,7 +963,7 @@ EXPORT_SYMBOL_GPL(usb_gadget_map_request);
void usb_gadget_unmap_request_by_dev(struct device *dev,
struct usb_request *req, int is_in)
{
- if (req->length == 0)
+ if (req->length == 0 || req->sg_was_mapped)
return;
if (req->num_mapped_sgs) {
diff --git a/drivers/usb/gadget/udc/dummy_hcd.c b/drivers/usb/gadget/udc/dummy_hcd.c
index 77393fc628eb..b105de357543 100644
--- a/drivers/usb/gadget/udc/dummy_hcd.c
+++ b/drivers/usb/gadget/udc/dummy_hcd.c
@@ -342,9 +342,11 @@ static void dummy_giveback(struct dummy *dum, struct usb_ep *_ep,
{
bool fifo = req == &dum->fifo_req;
+ ++dum->callback_usage;
spin_unlock(&dum->lock);
usb_gadget_giveback_request(_ep, &req->req);
spin_lock(&dum->lock);
+ --dum->callback_usage;
if (fifo)
dum->fifo_req_busy = 0;
}
@@ -758,11 +760,13 @@ static int dummy_queue(struct usb_ep *_ep, struct usb_request *_req,
req->req.complete = fifo_complete;
list_add_tail(&req->queue, &ep->queue);
+ ++dum->callback_usage;
spin_unlock(&dum->lock);
_req->actual = _req->length;
_req->status = 0;
usb_gadget_giveback_request(_ep, _req);
spin_lock(&dum->lock);
+ --dum->callback_usage;
} else
list_add_tail(&req->queue, &ep->queue);
spin_unlock_irqrestore(&dum->lock, flags);
diff --git a/drivers/usb/host/ehci-sched.c b/drivers/usb/host/ehci-sched.c
index bd542b6fc46b..d002e70171f6 100644
--- a/drivers/usb/host/ehci-sched.c
+++ b/drivers/usb/host/ehci-sched.c
@@ -1623,6 +1623,7 @@ iso_stream_schedule(
status = 1; /* and give it back immediately */
iso_sched_free(stream, sched);
sched = NULL;
+ urb->hcpriv = NULL;
}
}
urb->error_count = skip / period;
@@ -1653,8 +1654,6 @@ iso_stream_schedule(
return status;
fail:
- iso_sched_free(stream, sched);
- urb->hcpriv = NULL;
return status;
}
@@ -1966,6 +1965,10 @@ static int itd_submit(struct ehci_hcd *ehci, struct urb *urb,
usb_hcd_unlink_urb_from_ep(ehci_to_hcd(ehci), urb);
}
done_not_linked:
+ if (status < 0) {
+ iso_sched_free(stream, urb->hcpriv);
+ urb->hcpriv = NULL;
+ }
spin_unlock_irqrestore(&ehci->lock, flags);
done:
return status;
@@ -2343,6 +2346,10 @@ static int sitd_submit(struct ehci_hcd *ehci, struct urb *urb,
usb_hcd_unlink_urb_from_ep(ehci_to_hcd(ehci), urb);
}
done_not_linked:
+ if (status < 0) {
+ iso_sched_free(stream, urb->hcpriv);
+ urb->hcpriv = NULL;
+ }
spin_unlock_irqrestore(&ehci->lock, flags);
done:
return status;
diff --git a/drivers/usb/host/fotg210-hcd.c b/drivers/usb/host/fotg210-hcd.c
index dc968960769e..5cdae91211ec 100644
--- a/drivers/usb/host/fotg210-hcd.c
+++ b/drivers/usb/host/fotg210-hcd.c
@@ -4272,8 +4272,6 @@ static int iso_stream_schedule(struct fotg210_hcd *fotg210, struct urb *urb,
return 0;
fail:
- iso_sched_free(stream, sched);
- urb->hcpriv = NULL;
return status;
}
@@ -4567,6 +4565,10 @@ static int itd_submit(struct fotg210_hcd *fotg210, struct urb *urb,
else
usb_hcd_unlink_urb_from_ep(fotg210_to_hcd(fotg210), urb);
done_not_linked:
+ if (status < 0) {
+ iso_sched_free(stream, urb->hcpriv);
+ urb->hcpriv = NULL;
+ }
spin_unlock_irqrestore(&fotg210->lock, flags);
done:
return status;
diff --git a/drivers/usb/host/octeon-hcd.c b/drivers/usb/host/octeon-hcd.c
index a1cd81d4a114..16fb72eb3d18 100644
--- a/drivers/usb/host/octeon-hcd.c
+++ b/drivers/usb/host/octeon-hcd.c
@@ -578,7 +578,7 @@ static int cvmx_wait_tx_rx(struct octeon_hcd *usb, int fflsh_type)
int result;
u64 address = CVMX_USBCX_GRSTCTL(usb->index);
u64 done = cvmx_get_cycle() + 100 *
- (u64)octeon_get_clock_rate / 1000000;
+ (u64)octeon_get_clock_rate() / 1000000;
union cvmx_usbcx_grstctl c;
while (1) {
diff --git a/drivers/usb/host/ohci-da8xx.c b/drivers/usb/host/ohci-da8xx.c
index d4818e8d652b..613df2c96885 100644
--- a/drivers/usb/host/ohci-da8xx.c
+++ b/drivers/usb/host/ohci-da8xx.c
@@ -464,6 +464,7 @@ static int ohci_da8xx_probe(struct platform_device *pdev)
err_remove_hcd:
usb_remove_hcd(hcd);
+ device_wakeup_disable(hcd->self.controller);
err:
usb_put_hcd(hcd);
return error;
@@ -474,6 +475,7 @@ static int ohci_da8xx_remove(struct platform_device *pdev)
struct usb_hcd *hcd = platform_get_drvdata(pdev);
usb_remove_hcd(hcd);
+ device_wakeup_disable(hcd->self.controller);
usb_put_hcd(hcd);
return 0;
diff --git a/drivers/usb/host/ohci-s3c2410.c b/drivers/usb/host/ohci-s3c2410.c
index 85a0a9ae0095..40dcc231784b 100644
--- a/drivers/usb/host/ohci-s3c2410.c
+++ b/drivers/usb/host/ohci-s3c2410.c
@@ -335,6 +335,7 @@ ohci_hcd_s3c2410_remove(struct platform_device *dev)
struct usb_hcd *hcd = platform_get_drvdata(dev);
usb_remove_hcd(hcd);
+ device_wakeup_disable(hcd->self.controller);
s3c2410_stop_hc(dev);
usb_put_hcd(hcd);
return 0;
diff --git a/drivers/usb/host/ohci-spear.c b/drivers/usb/host/ohci-spear.c
index 196951a27f3f..47f590bdc1aa 100644
--- a/drivers/usb/host/ohci-spear.c
+++ b/drivers/usb/host/ohci-spear.c
@@ -104,8 +104,8 @@ static int spear_ohci_hcd_drv_remove(struct platform_device *pdev)
struct spear_ohci *sohci_p = to_spear_ohci(hcd);
usb_remove_hcd(hcd);
- if (sohci_p->clk)
- clk_disable_unprepare(sohci_p->clk);
+ device_wakeup_disable(hcd->self.controller);
+ clk_disable_unprepare(sohci_p->clk);
usb_put_hcd(hcd);
return 0;
diff --git a/drivers/usb/host/ohci-st.c b/drivers/usb/host/ohci-st.c
index 82eef3c62e11..b0b0d6a6c2fe 100644
--- a/drivers/usb/host/ohci-st.c
+++ b/drivers/usb/host/ohci-st.c
@@ -241,6 +241,7 @@ static int st_ohci_platform_remove(struct platform_device *dev)
int clk;
usb_remove_hcd(hcd);
+ device_wakeup_disable(hcd->self.controller);
if (pdata->power_off)
pdata->power_off(dev);
diff --git a/drivers/usb/host/xhci.c b/drivers/usb/host/xhci.c
index 104ee26b27aa..6cfc8205d10d 100644
--- a/drivers/usb/host/xhci.c
+++ b/drivers/usb/host/xhci.c
@@ -5365,6 +5365,7 @@ int xhci_gen_setup(struct usb_hcd *hcd, xhci_get_quirks_t get_quirks)
*/
struct device *dev = hcd->self.sysdev;
int retval;
+ u32 hc_capbase;
/* Accept arbitrarily long scatter-gather lists */
hcd->self.sg_tablesize = ~0;
@@ -5385,15 +5386,19 @@ int xhci_gen_setup(struct usb_hcd *hcd, xhci_get_quirks_t get_quirks)
mutex_init(&xhci->mutex);
xhci->main_hcd = hcd;
xhci->cap_regs = hcd->regs;
- xhci->op_regs = hcd->regs +
- HC_LENGTH(readl(&xhci->cap_regs->hc_capbase));
+ hc_capbase = readl(&xhci->cap_regs->hc_capbase);
+ if (hc_capbase == U32_MAX) {
+ xhci_warn(xhci, "Host controller not accessible, removed?\n");
+ return -ENODEV;
+ }
+ xhci->op_regs = hcd->regs + HC_LENGTH(hc_capbase);
xhci->run_regs = hcd->regs +
(readl(&xhci->cap_regs->run_regs_off) & RTSOFF_MASK);
/* Cache read-only capability registers */
xhci->hcs_params1 = readl(&xhci->cap_regs->hcs_params1);
xhci->hcs_params2 = readl(&xhci->cap_regs->hcs_params2);
xhci->hcs_params3 = readl(&xhci->cap_regs->hcs_params3);
- xhci->hci_version = HC_VERSION(readl(&xhci->cap_regs->hc_capbase));
+ xhci->hci_version = HC_VERSION(hc_capbase);
xhci->hcc_params = readl(&xhci->cap_regs->hcc_params);
if (xhci->hci_version > 0x100)
xhci->hcc_params2 = readl(&xhci->cap_regs->hcc_params2);
diff --git a/drivers/usb/misc/iowarrior.c b/drivers/usb/misc/iowarrior.c
index bbd4c1f3adc7..8e9a66ecfd33 100644
--- a/drivers/usb/misc/iowarrior.c
+++ b/drivers/usb/misc/iowarrior.c
@@ -71,6 +71,7 @@ static struct usb_driver iowarrior_driver;
/* Structure to hold all of our device specific stuff */
struct iowarrior {
+ struct kref kref;
struct mutex mutex; /* locks this structure */
struct usb_device *udev; /* save off the usb device pointer */
struct usb_interface *interface; /* the interface for this device */
@@ -243,8 +244,10 @@ static void iowarrior_write_callback(struct urb *urb)
/*
* iowarrior_delete
*/
-static inline void iowarrior_delete(struct iowarrior *dev)
+static inline void iowarrior_delete(struct kref *kref)
{
+ struct iowarrior *dev = container_of(kref, struct iowarrior, kref);
+
kfree(dev->int_in_buffer);
usb_free_urb(dev->int_in_urb);
kfree(dev->read_queue);
@@ -641,6 +644,9 @@ static int iowarrior_open(struct inode *inode, struct file *file)
}
/* increment our usage count for the driver */
++dev->opened;
+
+ kref_get(&dev->kref);
+
/* save our object in the file's private structure */
file->private_data = dev;
retval = 0;
@@ -656,7 +662,6 @@ static int iowarrior_open(struct inode *inode, struct file *file)
static int iowarrior_release(struct inode *inode, struct file *file)
{
struct iowarrior *dev;
- int retval = 0;
dev = file->private_data;
if (!dev)
@@ -664,29 +669,18 @@ static int iowarrior_release(struct inode *inode, struct file *file)
/* lock our device */
mutex_lock(&dev->mutex);
+ dev->opened = 0; /* we're closing now */
- if (dev->opened <= 0) {
- retval = -ENODEV; /* close called more than once */
- mutex_unlock(&dev->mutex);
- } else {
- dev->opened = 0; /* we're closing now */
- retval = 0;
- if (dev->present) {
- /*
- The device is still connected so we only shutdown
- pending read-/write-ops.
- */
- usb_kill_urb(dev->int_in_urb);
- wake_up_interruptible(&dev->read_wait);
- wake_up_interruptible(&dev->write_wait);
- mutex_unlock(&dev->mutex);
- } else {
- /* The device was unplugged, cleanup resources */
- mutex_unlock(&dev->mutex);
- iowarrior_delete(dev);
- }
+ if (dev->present) {
+ usb_kill_urb(dev->int_in_urb);
+ wake_up_interruptible(&dev->read_wait);
+ wake_up_interruptible(&dev->write_wait);
}
- return retval;
+ mutex_unlock(&dev->mutex);
+
+ kref_put(&dev->kref, iowarrior_delete);
+
+ return 0;
}
static __poll_t iowarrior_poll(struct file *file, poll_table * wait)
@@ -771,6 +765,7 @@ static int iowarrior_probe(struct usb_interface *interface,
if (!dev)
return retval;
+ kref_init(&dev->kref);
mutex_init(&dev->mutex);
atomic_set(&dev->intr_idx, 0);
@@ -889,7 +884,8 @@ static int iowarrior_probe(struct usb_interface *interface,
return retval;
error:
- iowarrior_delete(dev);
+ kref_put(&dev->kref, iowarrior_delete);
+
return retval;
}
@@ -913,19 +909,14 @@ static void iowarrior_disconnect(struct usb_interface *interface)
usb_kill_anchored_urbs(&dev->submitted);
if (dev->opened) {
- /* There is a process that holds a filedescriptor to the device ,
- so we only shutdown read-/write-ops going on.
- Deleting the device is postponed until close() was called.
- */
usb_kill_urb(dev->int_in_urb);
wake_up_interruptible(&dev->read_wait);
wake_up_interruptible(&dev->write_wait);
- mutex_unlock(&dev->mutex);
- } else {
- /* no process is using the device, cleanup now */
- mutex_unlock(&dev->mutex);
- iowarrior_delete(dev);
}
+
+ mutex_unlock(&dev->mutex);
+
+ kref_put(&dev->kref, iowarrior_delete);
}
/* usb specific object needed to register this driver with the usb subsystem */
diff --git a/drivers/usb/serial/ark3116.c b/drivers/usb/serial/ark3116.c
index 9452291f1703..67a07cc007f0 100644
--- a/drivers/usb/serial/ark3116.c
+++ b/drivers/usb/serial/ark3116.c
@@ -433,10 +433,11 @@ static int ark3116_tiocmset(struct tty_struct *tty,
return 0;
}
-static void ark3116_break_ctl(struct tty_struct *tty, int break_state)
+static int ark3116_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct ark3116_private *priv = usb_get_serial_port_data(port);
+ int ret;
/* LCR is also used for other things: protect access */
mutex_lock(&priv->hw_lock);
@@ -446,9 +447,11 @@ static void ark3116_break_ctl(struct tty_struct *tty, int break_state)
else
priv->lcr &= ~UART_LCR_SBC;
- ark3116_write_reg(port->serial, UART_LCR, priv->lcr);
+ ret = ark3116_write_reg(port->serial, UART_LCR, priv->lcr);
mutex_unlock(&priv->hw_lock);
+
+ return ret;
}
static void ark3116_update_msr(struct usb_serial_port *port, __u8 msr)
diff --git a/drivers/usb/serial/belkin_sa.c b/drivers/usb/serial/belkin_sa.c
index 0c7f7d090483..cda3295d68cf 100644
--- a/drivers/usb/serial/belkin_sa.c
+++ b/drivers/usb/serial/belkin_sa.c
@@ -46,7 +46,7 @@ static void belkin_sa_process_read_urb(struct urb *urb);
static void belkin_sa_set_termios(struct tty_struct *tty,
struct usb_serial_port *port,
const struct ktermios *old_termios);
-static void belkin_sa_break_ctl(struct tty_struct *tty, int break_state);
+static int belkin_sa_break_ctl(struct tty_struct *tty, int break_state);
static int belkin_sa_tiocmget(struct tty_struct *tty);
static int belkin_sa_tiocmset(struct tty_struct *tty,
unsigned int set, unsigned int clear);
@@ -402,13 +402,19 @@ static void belkin_sa_set_termios(struct tty_struct *tty,
spin_unlock_irqrestore(&priv->lock, flags);
}
-static void belkin_sa_break_ctl(struct tty_struct *tty, int break_state)
+static int belkin_sa_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct usb_serial *serial = port->serial;
+ int ret;
- if (BSA_USB_CMD(BELKIN_SA_SET_BREAK_REQUEST, break_state ? 1 : 0) < 0)
+ ret = BSA_USB_CMD(BELKIN_SA_SET_BREAK_REQUEST, break_state ? 1 : 0);
+ if (ret < 0) {
dev_err(&port->dev, "Set break_ctl %d\n", break_state);
+ return ret;
+ }
+
+ return 0;
}
static int belkin_sa_tiocmget(struct tty_struct *tty)
diff --git a/drivers/usb/serial/bus.c b/drivers/usb/serial/bus.c
index 25027ee64eae..651c9656242b 100644
--- a/drivers/usb/serial/bus.c
+++ b/drivers/usb/serial/bus.c
@@ -165,7 +165,11 @@ int usb_serial_bus_register(struct usb_serial_driver *driver)
void usb_serial_bus_deregister(struct usb_serial_driver *driver)
{
- free_dynids(driver);
driver_unregister(&driver->driver);
+ free_dynids(driver);
}
+void usb_serial_bus_remove_new_id(struct usb_serial_driver *driver)
+{
+ driver_remove_file(&driver->driver, &driver_attr_new_id);
+}
diff --git a/drivers/usb/serial/ch341.c b/drivers/usb/serial/ch341.c
index 6e1b87e67304..612bea504d7a 100644
--- a/drivers/usb/serial/ch341.c
+++ b/drivers/usb/serial/ch341.c
@@ -562,12 +562,12 @@ static void ch341_set_termios(struct tty_struct *tty,
* TCSBRKP. Due to how the simulation is implemented the duration can't be
* controlled. The duration is always about (1s / 46bd * 9bit) = 196ms.
*/
-static void ch341_simulate_break(struct tty_struct *tty, int break_state)
+static int ch341_simulate_break(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct ch341_private *priv = usb_get_serial_port_data(port);
unsigned long now, delay;
- int r;
+ int r, r2;
if (break_state != 0) {
dev_dbg(&port->dev, "enter break state requested\n");
@@ -599,7 +599,7 @@ static void ch341_simulate_break(struct tty_struct *tty, int break_state)
*/
priv->break_end = jiffies + (11 * HZ / CH341_MIN_BPS);
- return;
+ return 0;
}
dev_dbg(&port->dev, "leave break state requested\n");
@@ -615,17 +615,22 @@ static void ch341_simulate_break(struct tty_struct *tty, int break_state)
schedule_timeout_interruptible(delay);
}
+ r = 0;
restore:
/* Restore original baud rate */
- r = ch341_set_baudrate_lcr(port->serial->dev, priv, priv->baud_rate,
- priv->lcr);
- if (r < 0)
+ r2 = ch341_set_baudrate_lcr(port->serial->dev, priv, priv->baud_rate,
+ priv->lcr);
+ if (r2 < 0) {
dev_err(&port->dev,
"restoring original baud rate of %u failed: %d\n",
- priv->baud_rate, r);
+ priv->baud_rate, r2);
+ return r2;
+ }
+
+ return r;
}
-static void ch341_break_ctl(struct tty_struct *tty, int break_state)
+static int ch341_break_ctl(struct tty_struct *tty, int break_state)
{
const uint16_t ch341_break_reg =
((uint16_t) CH341_REG_LCR << 8) | CH341_REG_BREAK;
@@ -635,17 +640,17 @@ static void ch341_break_ctl(struct tty_struct *tty, int break_state)
uint16_t reg_contents;
uint8_t break_reg[2];
- if (priv->quirks & CH341_QUIRK_SIMULATE_BREAK) {
- ch341_simulate_break(tty, break_state);
- return;
- }
+ if (priv->quirks & CH341_QUIRK_SIMULATE_BREAK)
+ return ch341_simulate_break(tty, break_state);
r = ch341_control_in(port->serial->dev, CH341_REQ_READ_REG,
ch341_break_reg, 0, break_reg, 2);
if (r) {
dev_err(&port->dev, "%s - USB control read error (%d)\n",
__func__, r);
- return;
+ if (r > 0)
+ r = -EIO;
+ return r;
}
dev_dbg(&port->dev, "%s - initial ch341 break register contents - reg1: %x, reg2: %x\n",
__func__, break_reg[0], break_reg[1]);
@@ -663,9 +668,13 @@ static void ch341_break_ctl(struct tty_struct *tty, int break_state)
reg_contents = get_unaligned_le16(break_reg);
r = ch341_control_out(port->serial->dev, CH341_REQ_WRITE_REG,
ch341_break_reg, reg_contents);
- if (r < 0)
+ if (r < 0) {
dev_err(&port->dev, "%s - USB control write error (%d)\n",
__func__, r);
+ return r;
+ }
+
+ return 0;
}
static int ch341_tiocmset(struct tty_struct *tty,
diff --git a/drivers/usb/serial/cp210x.c b/drivers/usb/serial/cp210x.c
index c869cddb5471..233c25af0204 100644
--- a/drivers/usb/serial/cp210x.c
+++ b/drivers/usb/serial/cp210x.c
@@ -39,7 +39,7 @@ static int cp210x_tiocmget(struct tty_struct *);
static int cp210x_tiocmset(struct tty_struct *, unsigned int, unsigned int);
static int cp210x_tiocmset_port(struct usb_serial_port *port,
unsigned int, unsigned int);
-static void cp210x_break_ctl(struct tty_struct *, int);
+static int cp210x_break_ctl(struct tty_struct *, int);
static int cp210x_attach(struct usb_serial *);
static void cp210x_disconnect(struct usb_serial *);
static void cp210x_release(struct usb_serial *);
@@ -223,6 +223,7 @@ static const struct usb_device_id id_table[] = {
{ USB_DEVICE(0x19CF, 0x3000) }, /* Parrot NMEA GPS Flight Recorder */
{ USB_DEVICE(0x1ADB, 0x0001) }, /* Schweitzer Engineering C662 Cable */
{ USB_DEVICE(0x1B1C, 0x1C00) }, /* Corsair USB Dongle */
+ { USB_DEVICE(0x1B1C, 0x1C02) }, /* Corsair AX1500i Power Supply */
{ USB_DEVICE(0x1B93, 0x1013) }, /* Phoenix Contact UPS Device */
{ USB_DEVICE(0x1BA4, 0x0002) }, /* Silicon Labs 358x factory default */
{ USB_DEVICE(0x1BE3, 0x07A6) }, /* WAGO 750-923 USB Service Cable */
@@ -1429,7 +1430,7 @@ static int cp210x_tiocmget(struct tty_struct *tty)
return result;
}
-static void cp210x_break_ctl(struct tty_struct *tty, int break_state)
+static int cp210x_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
u16 state;
@@ -1438,9 +1439,11 @@ static void cp210x_break_ctl(struct tty_struct *tty, int break_state)
state = BREAK_OFF;
else
state = BREAK_ON;
+
dev_dbg(&port->dev, "%s - turning break %s\n", __func__,
state == BREAK_OFF ? "off" : "on");
- cp210x_write_u16_reg(port, CP210X_SET_BREAK, state);
+
+ return cp210x_write_u16_reg(port, CP210X_SET_BREAK, state);
}
#ifdef CONFIG_GPIOLIB
diff --git a/drivers/usb/serial/digi_acceleport.c b/drivers/usb/serial/digi_acceleport.c
index 1bfd64425b25..62a75be78dfc 100644
--- a/drivers/usb/serial/digi_acceleport.c
+++ b/drivers/usb/serial/digi_acceleport.c
@@ -217,7 +217,7 @@ static void digi_rx_unthrottle(struct tty_struct *tty);
static void digi_set_termios(struct tty_struct *tty,
struct usb_serial_port *port,
const struct ktermios *old_termios);
-static void digi_break_ctl(struct tty_struct *tty, int break_state);
+static int digi_break_ctl(struct tty_struct *tty, int break_state);
static int digi_tiocmget(struct tty_struct *tty);
static int digi_tiocmset(struct tty_struct *tty, unsigned int set,
unsigned int clear);
@@ -846,7 +846,7 @@ static void digi_set_termios(struct tty_struct *tty,
}
-static void digi_break_ctl(struct tty_struct *tty, int break_state)
+static int digi_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
unsigned char buf[4];
@@ -855,7 +855,8 @@ static void digi_break_ctl(struct tty_struct *tty, int break_state)
buf[1] = 2; /* length */
buf[2] = break_state ? 1 : 0;
buf[3] = 0; /* pad */
- digi_write_inb_command(port, buf, 4, 0);
+
+ return digi_write_inb_command(port, buf, 4, 0);
}
diff --git a/drivers/usb/serial/f81232.c b/drivers/usb/serial/f81232.c
index 5e689fb2b7aa..f397a68bdb53 100644
--- a/drivers/usb/serial/f81232.c
+++ b/drivers/usb/serial/f81232.c
@@ -447,7 +447,7 @@ static void f81534a_process_read_urb(struct urb *urb)
tty_flip_buffer_push(&port->port);
}
-static void f81232_break_ctl(struct tty_struct *tty, int break_state)
+static int f81232_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct f81232_private *priv = usb_get_serial_port_data(port);
@@ -466,6 +466,8 @@ static void f81232_break_ctl(struct tty_struct *tty, int break_state)
dev_err(&port->dev, "set break failed: %d\n", status);
mutex_unlock(&priv->lock);
+
+ return status;
}
static int f81232_find_clk(speed_t baudrate)
diff --git a/drivers/usb/serial/f81534.c b/drivers/usb/serial/f81534.c
index 4083ae961be4..ef126cd3d94f 100644
--- a/drivers/usb/serial/f81534.c
+++ b/drivers/usb/serial/f81534.c
@@ -656,7 +656,7 @@ static int f81534_set_port_config(struct usb_serial_port *port,
return status;
}
-static void f81534_break_ctl(struct tty_struct *tty, int break_state)
+static int f81534_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct f81534_port_private *port_priv = usb_get_serial_port_data(port);
@@ -675,6 +675,8 @@ static void f81534_break_ctl(struct tty_struct *tty, int break_state)
dev_err(&port->dev, "set break failed: %d\n", status);
mutex_unlock(&port_priv->lcr_mutex);
+
+ return status;
}
static int f81534_update_mctrl(struct usb_serial_port *port, unsigned int set,
diff --git a/drivers/usb/serial/ftdi_sio.c b/drivers/usb/serial/ftdi_sio.c
index 8c54dcc746a9..0b54f64e27fd 100644
--- a/drivers/usb/serial/ftdi_sio.c
+++ b/drivers/usb/serial/ftdi_sio.c
@@ -2550,11 +2550,12 @@ static void ftdi_process_read_urb(struct urb *urb)
tty_flip_buffer_push(&port->port);
}
-static void ftdi_break_ctl(struct tty_struct *tty, int break_state)
+static int ftdi_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct ftdi_private *priv = usb_get_serial_port_data(port);
u16 value;
+ int ret;
/* break_state = -1 to turn on break, and 0 to turn off break */
/* see drivers/char/tty_io.c to see it used */
@@ -2565,19 +2566,22 @@ static void ftdi_break_ctl(struct tty_struct *tty, int break_state)
else
value = priv->last_set_data_value;
- if (usb_control_msg(port->serial->dev,
+ ret = usb_control_msg(port->serial->dev,
usb_sndctrlpipe(port->serial->dev, 0),
FTDI_SIO_SET_DATA_REQUEST,
FTDI_SIO_SET_DATA_REQUEST_TYPE,
value, priv->channel,
- NULL, 0, WDR_TIMEOUT) < 0) {
+ NULL, 0, WDR_TIMEOUT);
+ if (ret < 0) {
dev_err(&port->dev, "%s FAILED to enable/disable break state (state was %d)\n",
__func__, break_state);
+ return ret;
}
dev_dbg(&port->dev, "%s break state is %d - urb is %d\n", __func__,
break_state, value);
+ return 0;
}
static bool ftdi_tx_empty(struct usb_serial_port *port)
diff --git a/drivers/usb/serial/generic.c b/drivers/usb/serial/generic.c
index 15b6dee3a8e5..64890f6d78e0 100644
--- a/drivers/usb/serial/generic.c
+++ b/drivers/usb/serial/generic.c
@@ -268,6 +268,7 @@ EXPORT_SYMBOL_GPL(usb_serial_generic_chars_in_buffer);
void usb_serial_generic_wait_until_sent(struct tty_struct *tty, long timeout)
{
struct usb_serial_port *port = tty->driver_data;
+ struct tty_port *tport = &port->port;
unsigned int bps;
unsigned long period;
unsigned long expire;
@@ -287,7 +288,14 @@ void usb_serial_generic_wait_until_sent(struct tty_struct *tty, long timeout)
__func__, jiffies_to_msecs(timeout),
jiffies_to_msecs(period));
expire = jiffies + timeout;
- while (!port->serial->type->tx_empty(port)) {
+ for (;;) {
+ mutex_lock(&tport->mutex);
+ if (tty_io_error(tty) || port->serial->type->tx_empty(port)) {
+ mutex_unlock(&tport->mutex);
+ break;
+ }
+ mutex_unlock(&tport->mutex);
+
schedule_timeout_interruptible(period);
if (signal_pending(current))
break;
diff --git a/drivers/usb/serial/io_edgeport.c b/drivers/usb/serial/io_edgeport.c
index c5ada8debfd6..ab29c867956b 100644
--- a/drivers/usb/serial/io_edgeport.c
+++ b/drivers/usb/serial/io_edgeport.c
@@ -1564,12 +1564,12 @@ static int edge_ioctl(struct tty_struct *tty,
* SerialBreak
* this function sends a break to the port
*****************************************************************************/
-static void edge_break(struct tty_struct *tty, int break_state)
+static int edge_break(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct edgeport_port *edge_port = usb_get_serial_port_data(port);
struct edgeport_serial *edge_serial = usb_get_serial_data(port->serial);
- int status;
+ int status = 0;
if (!edge_serial->is_epic ||
edge_serial->epic_descriptor.Supports.IOSPChase) {
@@ -1601,6 +1601,8 @@ static void edge_break(struct tty_struct *tty, int break_state)
dev_dbg(&port->dev, "%s - error sending break set/clear command.\n",
__func__);
}
+
+ return status;
}
diff --git a/drivers/usb/serial/io_ti.c b/drivers/usb/serial/io_ti.c
index 69f724056847..45184faf02bc 100644
--- a/drivers/usb/serial/io_ti.c
+++ b/drivers/usb/serial/io_ti.c
@@ -2432,7 +2432,7 @@ static int edge_tiocmget(struct tty_struct *tty)
return result;
}
-static void edge_break(struct tty_struct *tty, int break_state)
+static int edge_break(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct edgeport_port *edge_port = usb_get_serial_port_data(port);
@@ -2441,10 +2441,15 @@ static void edge_break(struct tty_struct *tty, int break_state)
if (break_state == -1)
bv = 1; /* On */
+
status = ti_do_config(edge_port, UMPC_SET_CLR_BREAK, bv);
- if (status)
+ if (status) {
dev_dbg(&port->dev, "%s - error %d sending break set/clear command.\n",
__func__, status);
+ return status;
+ }
+
+ return 0;
}
static void edge_heartbeat_schedule(struct edgeport_serial *edge_serial)
diff --git a/drivers/usb/serial/keyspan.c b/drivers/usb/serial/keyspan.c
index 26c2cc372ed3..bb4f047381ff 100644
--- a/drivers/usb/serial/keyspan.c
+++ b/drivers/usb/serial/keyspan.c
@@ -599,7 +599,7 @@ struct keyspan_port_private {
#include "keyspan_usa67msg.h"
-static void keyspan_break_ctl(struct tty_struct *tty, int break_state)
+static int keyspan_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct keyspan_port_private *p_priv;
@@ -611,7 +611,10 @@ static void keyspan_break_ctl(struct tty_struct *tty, int break_state)
else
p_priv->break_on = 0;
+ /* FIXME: return errors */
keyspan_send_setup(port, 0);
+
+ return 0;
}
diff --git a/drivers/usb/serial/keyspan_pda.c b/drivers/usb/serial/keyspan_pda.c
index 264efa6ce8e4..b3964f20b254 100644
--- a/drivers/usb/serial/keyspan_pda.c
+++ b/drivers/usb/serial/keyspan_pda.c
@@ -323,7 +323,7 @@ static speed_t keyspan_pda_setbaud(struct usb_serial *serial, speed_t baud)
return baud;
}
-static void keyspan_pda_break_ctl(struct tty_struct *tty, int break_state)
+static int keyspan_pda_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct usb_serial *serial = port->serial;
@@ -339,9 +339,13 @@ static void keyspan_pda_break_ctl(struct tty_struct *tty, int break_state)
4, /* set break */
USB_TYPE_VENDOR | USB_RECIP_INTERFACE | USB_DIR_OUT,
value, 0, NULL, 0, 2000);
- if (result < 0)
+ if (result < 0) {
dev_dbg(&port->dev, "%s - error %d from usb_control_msg\n",
__func__, result);
+ return result;
+ }
+
+ return 0;
}
static void keyspan_pda_set_termios(struct tty_struct *tty,
diff --git a/drivers/usb/serial/mct_u232.c b/drivers/usb/serial/mct_u232.c
index 3c1351bb7ca3..be63f5f37404 100644
--- a/drivers/usb/serial/mct_u232.c
+++ b/drivers/usb/serial/mct_u232.c
@@ -47,7 +47,7 @@ static void mct_u232_read_int_callback(struct urb *urb);
static void mct_u232_set_termios(struct tty_struct *tty,
struct usb_serial_port *port,
const struct ktermios *old_termios);
-static void mct_u232_break_ctl(struct tty_struct *tty, int break_state);
+static int mct_u232_break_ctl(struct tty_struct *tty, int break_state);
static int mct_u232_tiocmget(struct tty_struct *tty);
static int mct_u232_tiocmset(struct tty_struct *tty,
unsigned int set, unsigned int clear);
@@ -683,7 +683,7 @@ static void mct_u232_set_termios(struct tty_struct *tty,
spin_unlock_irqrestore(&priv->lock, flags);
} /* mct_u232_set_termios */
-static void mct_u232_break_ctl(struct tty_struct *tty, int break_state)
+static int mct_u232_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct mct_u232_private *priv = usb_get_serial_port_data(port);
@@ -697,7 +697,7 @@ static void mct_u232_break_ctl(struct tty_struct *tty, int break_state)
lcr |= MCT_U232_SET_BREAK;
spin_unlock_irqrestore(&priv->lock, flags);
- mct_u232_set_line_ctrl(port, lcr);
+ return mct_u232_set_line_ctrl(port, lcr);
} /* mct_u232_break_ctl */
diff --git a/drivers/usb/serial/mos7720.c b/drivers/usb/serial/mos7720.c
index 1d1f85fabc28..23544074eb1c 100644
--- a/drivers/usb/serial/mos7720.c
+++ b/drivers/usb/serial/mos7720.c
@@ -996,7 +996,7 @@ static void mos7720_close(struct usb_serial_port *port)
mos7720_port->open = 0;
}
-static void mos7720_break(struct tty_struct *tty, int break_state)
+static int mos7720_break(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
unsigned char data;
@@ -1007,7 +1007,7 @@ static void mos7720_break(struct tty_struct *tty, int break_state)
mos7720_port = usb_get_serial_port_data(port);
if (mos7720_port == NULL)
- return;
+ return -ENODEV;
if (break_state == -1)
data = mos7720_port->shadowLCR | UART_LCR_SBC;
@@ -1015,8 +1015,9 @@ static void mos7720_break(struct tty_struct *tty, int break_state)
data = mos7720_port->shadowLCR & ~UART_LCR_SBC;
mos7720_port->shadowLCR = data;
- write_mos_reg(serial, port->port_number, MOS7720_LCR,
- mos7720_port->shadowLCR);
+
+ return write_mos_reg(serial, port->port_number, MOS7720_LCR,
+ mos7720_port->shadowLCR);
}
/*
diff --git a/drivers/usb/serial/mos7840.c b/drivers/usb/serial/mos7840.c
index 26f287180f8a..f90b427e4c3f 100644
--- a/drivers/usb/serial/mos7840.c
+++ b/drivers/usb/serial/mos7840.c
@@ -787,7 +787,7 @@ static void mos7840_close(struct usb_serial_port *port)
* mos7840_break
* this function sends a break to the port
*****************************************************************************/
-static void mos7840_break(struct tty_struct *tty, int break_state)
+static int mos7840_break(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct moschip_port *mos7840_port = usb_get_serial_port_data(port);
@@ -801,8 +801,9 @@ static void mos7840_break(struct tty_struct *tty, int break_state)
/* FIXME: no locking on shadowLCR anywhere in driver */
mos7840_port->shadowLCR = data;
dev_dbg(&port->dev, "%s mos7840_port->shadowLCR is %x\n", __func__, mos7840_port->shadowLCR);
- mos7840_set_uart_reg(port, LINE_CONTROL_REGISTER,
- mos7840_port->shadowLCR);
+
+ return mos7840_set_uart_reg(port, LINE_CONTROL_REGISTER,
+ mos7840_port->shadowLCR);
}
/*****************************************************************************
diff --git a/drivers/usb/serial/mxuport.c b/drivers/usb/serial/mxuport.c
index 3030d53e8b1f..8d20928ce54c 100644
--- a/drivers/usb/serial/mxuport.c
+++ b/drivers/usb/serial/mxuport.c
@@ -1238,7 +1238,7 @@ static void mxuport_close(struct usb_serial_port *port)
}
/* Send a break to the port. */
-static void mxuport_break_ctl(struct tty_struct *tty, int break_state)
+static int mxuport_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct usb_serial *serial = port->serial;
@@ -1252,8 +1252,8 @@ static void mxuport_break_ctl(struct tty_struct *tty, int break_state)
dev_dbg(&port->dev, "%s - clearing break\n", __func__);
}
- mxuport_send_ctrl_urb(serial, RQ_VENDOR_SET_BREAK,
- enable, port->port_number);
+ return mxuport_send_ctrl_urb(serial, RQ_VENDOR_SET_BREAK,
+ enable, port->port_number);
}
static int mxuport_resume(struct usb_serial *serial)
diff --git a/drivers/usb/serial/option.c b/drivers/usb/serial/option.c
index dd018abda6c3..3690fb93a844 100644
--- a/drivers/usb/serial/option.c
+++ b/drivers/usb/serial/option.c
@@ -261,6 +261,7 @@ static void option_instat_callback(struct urb *urb);
#define QUECTEL_PRODUCT_EM060K_12a 0x012a
#define QUECTEL_PRODUCT_EM060K_12b 0x012b
#define QUECTEL_PRODUCT_EM060K_12c 0x012c
+#define QUECTEL_PRODUCT_RG660QB 0x013d
#define QUECTEL_PRODUCT_EG91 0x0191
#define QUECTEL_PRODUCT_EG95 0x0195
#define QUECTEL_PRODUCT_BG96 0x0296
@@ -281,6 +282,7 @@ static void option_instat_callback(struct urb *urb);
#define QUECTEL_PRODUCT_EC200U 0x0901
#define QUECTEL_PRODUCT_EG912Y 0x6001
#define QUECTEL_PRODUCT_EC200S_CN 0x6002
+#define QUECTEL_PRODUCT_EG060W 0x6004
#define QUECTEL_PRODUCT_EC200A 0x6005
#define QUECTEL_PRODUCT_EG916Q 0x6007
#define QUECTEL_PRODUCT_EM061K_LWW 0x6008
@@ -1269,12 +1271,15 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC200A, 0xff, 0, 0) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC200U, 0xff, 0, 0) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC200S_CN, 0xff, 0, 0) },
+ { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EG060W, 0xff, 0, 0) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC200T, 0xff, 0, 0) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EG912Y, 0xff, 0, 0) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EG916Q, 0xff, 0x00, 0x00) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RM500K, 0xff, 0x00, 0x00) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG650V, 0xff, 0xff, 0x30) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG650V, 0xff, 0, 0) },
+ { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG660QB, 0xff, 0xff, 0x30) },
+ { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG660QB, 0xff, 0, 0) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG255C, 0xff, 0xff, 0x30) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG255C, 0xff, 0, 0) },
{ USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG255C, 0xff, 0xff, 0x40) },
@@ -2169,6 +2174,8 @@ static const struct usb_device_id option_ids[] = {
{ USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x9003, 0xff) }, /* Simcom SIM7500/SIM7600 MBIM mode */
{ USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x9011, 0xff), /* Simcom SIM7500/SIM7600 RNDIS mode */
.driver_info = RSVD(7) },
+ { USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x902b, 0xff),
+ .driver_info = RSVD(4) },
{ USB_DEVICE(0x1e0e, 0x9071), /* Simcom SIM8230 RMNET mode */
.driver_info = RSVD(3) | RSVD(4) },
{ USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x9078, 0xff), /* Simcom SIM8230 ECM mode */
@@ -2430,6 +2437,13 @@ static const struct usb_device_id option_ids[] = {
.driver_info = RSVD(5) },
{ USB_DEVICE_INTERFACE_CLASS(0x0489, 0xe167, 0xff), /* Foxconn T99W640 MBIM */
.driver_info = RSVD(3) },
+ { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x4d22, 0xff, 0x10, 0x01) }, /* Compal EXC-T1 */
+ { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x4d22, 0xff, 0x10, 0x02) },
+ { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x4d22, 0xff, 0x10, 0x03) },
+ { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x4d22, 0xff, 0x10, 0x04) },
+ { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x8217, 0xff, 0xff, 0x30) }, /* Compal EXM-G1x */
+ { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x8217, 0xff, 0xff, 0x40) },
+ { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x8217, 0xff, 0xff, 0x60) },
{ USB_DEVICE(0x1508, 0x1001), /* Fibocom NL668 (IOT version) */
.driver_info = RSVD(4) | RSVD(5) | RSVD(6) },
{ USB_DEVICE(0x1782, 0x4d10) }, /* Fibocom L610 (AT mode) */
diff --git a/drivers/usb/serial/pl2303.c b/drivers/usb/serial/pl2303.c
index 103994af4bff..1ba2287d4d45 100644
--- a/drivers/usb/serial/pl2303.c
+++ b/drivers/usb/serial/pl2303.c
@@ -174,7 +174,7 @@ MODULE_DEVICE_TABLE(usb, id_table);
#define PL2303_HXN_FLOWCTRL_RTS_CTS 0x18
#define PL2303_HXN_FLOWCTRL_XON_XOFF 0x0c
-static void pl2303_set_break(struct usb_serial_port *port, bool enable);
+static int pl2303_set_break(struct usb_serial_port *port, bool enable);
enum pl2303_type {
TYPE_H,
@@ -1063,7 +1063,7 @@ static int pl2303_carrier_raised(struct usb_serial_port *port)
return 0;
}
-static void pl2303_set_break(struct usb_serial_port *port, bool enable)
+static int pl2303_set_break(struct usb_serial_port *port, bool enable)
{
struct usb_serial *serial = port->serial;
u16 state;
@@ -1080,15 +1080,19 @@ static void pl2303_set_break(struct usb_serial_port *port, bool enable)
result = usb_control_msg(serial->dev, usb_sndctrlpipe(serial->dev, 0),
BREAK_REQUEST, BREAK_REQUEST_TYPE, state,
0, NULL, 0, 100);
- if (result)
+ if (result) {
dev_err(&port->dev, "error sending break = %d\n", result);
+ return result;
+ }
+
+ return 0;
}
-static void pl2303_break_ctl(struct tty_struct *tty, int state)
+static int pl2303_break_ctl(struct tty_struct *tty, int state)
{
struct usb_serial_port *port = tty->driver_data;
- pl2303_set_break(port, state);
+ return pl2303_set_break(port, state);
}
static void pl2303_update_line_status(struct usb_serial_port *port,
diff --git a/drivers/usb/serial/quatech2.c b/drivers/usb/serial/quatech2.c
index 995313f4687a..cf2ab7cb80f0 100644
--- a/drivers/usb/serial/quatech2.c
+++ b/drivers/usb/serial/quatech2.c
@@ -54,6 +54,7 @@
#define QT2_XMIT_FLUSH 0x05 /* no following info */
#define QT2_CONTROL_ESCAPE 0xff /* pass through previous 2 control bytes */
+#define MIN_BAUD_RATE 50
#define MAX_BAUD_RATE 921600
#define DEFAULT_BAUD_RATE 9600
@@ -158,7 +159,7 @@ static inline int calc_baud_divisor(int baudrate)
static inline int qt2_set_port_config(struct usb_device *dev,
unsigned char port_number,
- u16 baudrate, u16 lcr)
+ speed_t baudrate, u16 lcr)
{
int divisor = calc_baud_divisor(baudrate);
u16 index = ((u16) (lcr << 8) | (u16) (port_number));
@@ -267,9 +268,9 @@ static void qt2_set_termios(struct tty_struct *tty,
struct usb_device *dev = port->serial->dev;
struct qt2_port_private *port_priv;
struct ktermios *termios = &tty->termios;
- u16 baud;
unsigned int cflag = termios->c_cflag;
u16 new_lcr = 0;
+ speed_t baud;
int status;
port_priv = usb_get_serial_port_data(port);
@@ -287,6 +288,18 @@ static void qt2_set_termios(struct tty_struct *tty,
if (!baud)
baud = 9600;
+ if (baud < MIN_BAUD_RATE || baud > MAX_BAUD_RATE) {
+ if (old_termios)
+ baud = tty_termios_baud_rate(old_termios);
+ else
+ baud = clamp(baud, MIN_BAUD_RATE, MAX_BAUD_RATE);
+
+ tty_encode_baud_rate(tty, baud, baud);
+
+ if (!baud)
+ baud = 9600;
+ }
+
status = qt2_set_port_config(dev, port_priv->device_port, baud,
new_lcr);
if (status < 0)
@@ -383,7 +396,7 @@ static int qt2_open(struct tty_struct *tty, struct usb_serial_port *port)
port_priv->device_port = (u8) device_port;
if (tty)
- qt2_set_termios(tty, port, &tty->termios);
+ qt2_set_termios(tty, port, NULL);
return 0;
@@ -749,7 +762,7 @@ static int qt2_tiocmset(struct tty_struct *tty,
return update_mctrl(port_priv, set, clear);
}
-static void qt2_break_ctl(struct tty_struct *tty, int break_state)
+static int qt2_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct qt2_port_private *port_priv;
@@ -762,10 +775,14 @@ static void qt2_break_ctl(struct tty_struct *tty, int break_state)
status = qt2_control_msg(port->serial->dev, QT2_BREAK_CONTROL,
val, port_priv->device_port);
- if (status < 0)
+ if (status < 0) {
dev_warn(&port->dev,
"%s - failed to send control message: %i\n", __func__,
status);
+ return status;
+ }
+
+ return 0;
}
diff --git a/drivers/usb/serial/ssu100.c b/drivers/usb/serial/ssu100.c
index 1e1888b66305..52b239d16a0e 100644
--- a/drivers/usb/serial/ssu100.c
+++ b/drivers/usb/serial/ssu100.c
@@ -34,6 +34,7 @@
#define SERIAL_EVEN_PARITY (UART_LCR_PARITY | UART_LCR_EPAR)
+#define MIN_BAUD_RATE 50
#define MAX_BAUD_RATE 460800
#define ATC_DISABLED 0x00
@@ -219,9 +220,10 @@ static void ssu100_set_termios(struct tty_struct *tty,
{
struct usb_device *dev = port->serial->dev;
struct ktermios *termios = &tty->termios;
- u16 baud, divisor, remainder;
+ speed_t baud, remainder;
unsigned int cflag = termios->c_cflag;
u16 urb_value = 0; /* will hold the new flags */
+ u16 divisor;
int result;
if (cflag & PARENB) {
@@ -237,6 +239,18 @@ static void ssu100_set_termios(struct tty_struct *tty,
if (!baud)
baud = 9600;
+ if (baud < MIN_BAUD_RATE || baud > MAX_BAUD_RATE) {
+ if (old_termios)
+ baud = tty_termios_baud_rate(old_termios);
+ else
+ baud = clamp(baud, MIN_BAUD_RATE, MAX_BAUD_RATE);
+
+ tty_encode_baud_rate(tty, baud, baud);
+
+ if (!baud)
+ baud = 9600;
+ }
+
dev_dbg(&port->dev, "%s - got baud = %d\n", __func__, baud);
@@ -312,7 +326,7 @@ static int ssu100_open(struct tty_struct *tty, struct usb_serial_port *port)
dev_dbg(&port->dev, "%s - set uart failed\n", __func__);
if (tty)
- ssu100_set_termios(tty, port, &tty->termios);
+ ssu100_set_termios(tty, port, NULL);
return usb_serial_generic_open(tty, port);
}
diff --git a/drivers/usb/serial/ti_usb_3410_5052.c b/drivers/usb/serial/ti_usb_3410_5052.c
index b99f78224846..0fba25abf671 100644
--- a/drivers/usb/serial/ti_usb_3410_5052.c
+++ b/drivers/usb/serial/ti_usb_3410_5052.c
@@ -319,7 +319,7 @@ static void ti_set_termios(struct tty_struct *tty,
static int ti_tiocmget(struct tty_struct *tty);
static int ti_tiocmset(struct tty_struct *tty,
unsigned int set, unsigned int clear);
-static void ti_break(struct tty_struct *tty, int break_state);
+static int ti_break(struct tty_struct *tty, int break_state);
static void ti_interrupt_callback(struct urb *urb);
static void ti_bulk_in_callback(struct urb *urb);
static void ti_bulk_out_callback(struct urb *urb);
@@ -1071,7 +1071,7 @@ static int ti_tiocmset(struct tty_struct *tty,
}
-static void ti_break(struct tty_struct *tty, int break_state)
+static int ti_break(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct ti_port *tport = usb_get_serial_port_data(port);
@@ -1083,8 +1083,12 @@ static void ti_break(struct tty_struct *tty, int break_state)
tport->tp_uart_base_addr + TI_UART_OFFSET_LCR,
TI_LCR_BREAK, break_state == -1 ? TI_LCR_BREAK : 0);
- if (status)
+ if (status) {
dev_dbg(&port->dev, "%s - error setting break, %d\n", __func__, status);
+ return status;
+ }
+
+ return 0;
}
static int ti_get_port_from_code(unsigned char code)
diff --git a/drivers/usb/serial/upd78f0730.c b/drivers/usb/serial/upd78f0730.c
index c47439bd90fa..46952182e04f 100644
--- a/drivers/usb/serial/upd78f0730.c
+++ b/drivers/usb/serial/upd78f0730.c
@@ -238,12 +238,13 @@ static int upd78f0730_tiocmset(struct tty_struct *tty,
return res;
}
-static void upd78f0730_break_ctl(struct tty_struct *tty, int break_state)
+static int upd78f0730_break_ctl(struct tty_struct *tty, int break_state)
{
struct upd78f0730_port_private *private;
struct usb_serial_port *port = tty->driver_data;
struct upd78f0730_set_dtr_rts request;
struct device *dev = &port->dev;
+ int res;
private = usb_get_serial_port_data(port);
@@ -258,8 +259,10 @@ static void upd78f0730_break_ctl(struct tty_struct *tty, int break_state)
request.opcode = UPD78F0730_CMD_SET_DTR_RTS;
request.params = private->line_signals;
- upd78f0730_send_ctl(port, &request, sizeof(request));
+ res = upd78f0730_send_ctl(port, &request, sizeof(request));
mutex_unlock(&private->lock);
+
+ return res;
}
static void upd78f0730_dtr_rts(struct usb_serial_port *port, int on)
diff --git a/drivers/usb/serial/usb-serial.c b/drivers/usb/serial/usb-serial.c
index cf5db67ead73..9c6e6618e169 100644
--- a/drivers/usb/serial/usb-serial.c
+++ b/drivers/usb/serial/usb-serial.c
@@ -403,17 +403,13 @@ static unsigned int serial_chars_in_buffer(struct tty_struct *tty)
static void serial_wait_until_sent(struct tty_struct *tty, int timeout)
{
struct usb_serial_port *port = tty->driver_data;
- struct usb_serial *serial = port->serial;
dev_dbg(&port->dev, "%s\n", __func__);
if (!port->serial->type->wait_until_sent)
return;
- mutex_lock(&serial->disc_mutex);
- if (!serial->disconnected)
- port->serial->type->wait_until_sent(tty, timeout);
- mutex_unlock(&serial->disc_mutex);
+ port->serial->type->wait_until_sent(tty, timeout);
}
static void serial_throttle(struct tty_struct *tty)
@@ -441,8 +437,13 @@ static int serial_get_serial(struct tty_struct *tty, struct serial_struct *ss)
struct usb_serial_port *port = tty->driver_data;
struct tty_port *tport = &port->port;
unsigned int close_delay, closing_wait;
+ int ret = 0;
mutex_lock(&tport->mutex);
+ if (tty_io_error(tty)) {
+ ret = -EIO;
+ goto out_unlock;
+ }
close_delay = jiffies_to_msecs(tport->close_delay) / 10;
closing_wait = tport->closing_wait;
@@ -455,10 +456,10 @@ static int serial_get_serial(struct tty_struct *tty, struct serial_struct *ss)
if (port->serial->type->get_serial)
port->serial->type->get_serial(tty, ss);
-
+out_unlock:
mutex_unlock(&tport->mutex);
- return 0;
+ return ret;
}
static int serial_set_serial(struct tty_struct *tty, struct serial_struct *ss)
@@ -474,6 +475,10 @@ static int serial_set_serial(struct tty_struct *tty, struct serial_struct *ss)
closing_wait = msecs_to_jiffies(closing_wait * 10);
mutex_lock(&tport->mutex);
+ if (tty_io_error(tty)) {
+ ret = -EIO;
+ goto out_unlock;
+ }
if (!capable(CAP_SYS_ADMIN)) {
if (close_delay != tport->close_delay ||
@@ -501,6 +506,7 @@ static int serial_ioctl(struct tty_struct *tty,
unsigned int cmd, unsigned long arg)
{
struct usb_serial_port *port = tty->driver_data;
+ struct tty_port *tport = &port->port;
int retval = -ENOIOCTLCMD;
dev_dbg(&port->dev, "%s - cmd 0x%04x\n", __func__, cmd);
@@ -511,8 +517,21 @@ static int serial_ioctl(struct tty_struct *tty,
retval = port->serial->type->tiocmiwait(tty, arg);
break;
default:
- if (port->serial->type->ioctl)
+ if (!port->serial->type->ioctl)
+ break;
+
+ if (cmd == TIOCSRS485)
+ down_write(&tty->termios_rwsem);
+
+ mutex_lock(&tport->mutex);
+ if (tty_io_error(tty))
+ retval = -EIO;
+ else
retval = port->serial->type->ioctl(tty, cmd, arg);
+ mutex_unlock(&tport->mutex);
+
+ if (cmd == TIOCSRS485)
+ up_write(&tty->termios_rwsem);
}
return retval;
@@ -522,25 +541,40 @@ static void serial_set_termios(struct tty_struct *tty,
const struct ktermios *old)
{
struct usb_serial_port *port = tty->driver_data;
+ struct tty_port *tport = &port->port;
dev_dbg(&port->dev, "%s\n", __func__);
- if (port->serial->type->set_termios)
- port->serial->type->set_termios(tty, port, old);
- else
+ if (!port->serial->type->set_termios) {
tty_termios_copy_hw(&tty->termios, old);
+ return;
+ }
+
+ mutex_lock(&tport->mutex);
+ if (!tty_io_error(tty))
+ port->serial->type->set_termios(tty, port, old);
+ mutex_unlock(&tport->mutex);
}
static int serial_break(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
+ struct tty_port *tport = &port->port;
+ int ret;
dev_dbg(&port->dev, "%s\n", __func__);
- if (port->serial->type->break_ctl)
- port->serial->type->break_ctl(tty, break_state);
+ if (!port->serial->type->break_ctl)
+ return -ENOTTY;
- return 0;
+ mutex_lock(&tport->mutex);
+ if (tty_io_error(tty))
+ ret = -EIO;
+ else
+ ret = port->serial->type->break_ctl(tty, break_state);
+ mutex_unlock(&tport->mutex);
+
+ return ret;
}
static int serial_proc_show(struct seq_file *m, void *v)
@@ -581,24 +615,44 @@ static int serial_proc_show(struct seq_file *m, void *v)
static int serial_tiocmget(struct tty_struct *tty)
{
struct usb_serial_port *port = tty->driver_data;
+ struct tty_port *tport = &port->port;
+ int ret;
dev_dbg(&port->dev, "%s\n", __func__);
- if (port->serial->type->tiocmget)
- return port->serial->type->tiocmget(tty);
- return -ENOTTY;
+ if (!port->serial->type->tiocmget)
+ return -ENOTTY;
+
+ mutex_lock(&tport->mutex);
+ if (tty_io_error(tty))
+ ret = -EIO;
+ else
+ ret = port->serial->type->tiocmget(tty);
+ mutex_unlock(&tport->mutex);
+
+ return ret;
}
static int serial_tiocmset(struct tty_struct *tty,
unsigned int set, unsigned int clear)
{
struct usb_serial_port *port = tty->driver_data;
+ struct tty_port *tport = &port->port;
+ int ret;
dev_dbg(&port->dev, "%s\n", __func__);
- if (port->serial->type->tiocmset)
- return port->serial->type->tiocmset(tty, set, clear);
- return -ENOTTY;
+ if (!port->serial->type->tiocmset)
+ return -ENOTTY;
+
+ mutex_lock(&tport->mutex);
+ if (tty_io_error(tty))
+ ret = -EIO;
+ else
+ ret = port->serial->type->tiocmset(tty, set, clear);
+ mutex_unlock(&tport->mutex);
+
+ return ret;
}
static int serial_get_icount(struct tty_struct *tty,
@@ -1194,12 +1248,17 @@ static void usb_serial_disconnect(struct usb_interface *interface)
usb_serial_port_poison_urbs(port);
wake_up_interruptible(&port->port.delta_msr_wait);
cancel_work_sync(&port->work);
- if (device_is_registered(&port->dev))
- device_del(&port->dev);
}
+
if (serial->type->disconnect)
serial->type->disconnect(serial);
+ for (i = 0; i < serial->num_ports; ++i) {
+ port = serial->port[i];
+ if (device_is_registered(&port->dev))
+ device_del(&port->dev);
+ }
+
release_sibling(serial, interface);
/* let the last holder of this object cause it to be cleaned up */
@@ -1316,7 +1375,8 @@ static int __init usb_serial_init(void)
int result;
usb_serial_tty_driver = tty_alloc_driver(USB_SERIAL_TTY_MINORS,
- TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV);
+ TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV |
+ TTY_DRIVER_RESET_SAVED_TERMIOS);
if (IS_ERR(usb_serial_tty_driver))
return PTR_ERR(usb_serial_tty_driver);
@@ -1466,7 +1526,7 @@ int usb_serial_register_drivers(struct usb_serial_driver *const serial_drivers[]
{
int rc;
struct usb_driver *udriver;
- struct usb_serial_driver * const *sd;
+ struct usb_serial_driver * const *sd, * const *s;
/*
* udriver must be registered before any of the serial drivers,
@@ -1518,9 +1578,11 @@ int usb_serial_register_drivers(struct usb_serial_driver *const serial_drivers[]
return 0;
err_deregister_drivers:
+ for (s = serial_drivers; s < sd; ++s)
+ usb_serial_bus_remove_new_id(*s);
+ usb_deregister(udriver);
while (sd-- > serial_drivers)
usb_serial_deregister(*sd);
- usb_deregister(udriver);
err_free_driver:
kfree(udriver);
return rc;
@@ -1538,10 +1600,23 @@ EXPORT_SYMBOL_GPL(usb_serial_register_drivers);
void usb_serial_deregister_drivers(struct usb_serial_driver *const serial_drivers[])
{
struct usb_driver *udriver = (*serial_drivers)->usb_driver;
+ struct usb_serial_driver * const *sd;
+
+ /*
+ * udriver must be deregistered before the serial drivers so that
+ * I/O is stopped before unbinding the ports.
+ *
+ * Remove the new_id attributes to prevent ids from being added and
+ * triggering a probe of udriver after it has been deregistered.
+ */
+ for (sd = serial_drivers; *sd; ++sd)
+ usb_serial_bus_remove_new_id(*sd);
+
+ usb_deregister(udriver);
for (; *serial_drivers; ++serial_drivers)
usb_serial_deregister(*serial_drivers);
- usb_deregister(udriver);
+
kfree(udriver);
}
EXPORT_SYMBOL_GPL(usb_serial_deregister_drivers);
diff --git a/drivers/usb/serial/usb_debug.c b/drivers/usb/serial/usb_debug.c
index 406cb326e812..5a8869cd95d5 100644
--- a/drivers/usb/serial/usb_debug.c
+++ b/drivers/usb/serial/usb_debug.c
@@ -47,12 +47,19 @@ MODULE_DEVICE_TABLE(usb, id_table_combined);
/* This HW really does not support a serial break, so one will be
* emulated when ever the break state is set to true.
*/
-static void usb_debug_break_ctl(struct tty_struct *tty, int break_state)
+static int usb_debug_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
+ int ret;
+
if (!break_state)
- return;
- usb_serial_generic_write(tty, port, USB_DEBUG_BRK, USB_DEBUG_BRK_SIZE);
+ return 0;
+
+ ret = usb_serial_generic_write(tty, port, USB_DEBUG_BRK, USB_DEBUG_BRK_SIZE);
+ if (ret < 0)
+ return ret;
+
+ return 0;
}
static void usb_debug_process_read_urb(struct urb *urb)
diff --git a/drivers/usb/serial/whiteheat.c b/drivers/usb/serial/whiteheat.c
index 7f82d40753ee..ca48e90a8e81 100644
--- a/drivers/usb/serial/whiteheat.c
+++ b/drivers/usb/serial/whiteheat.c
@@ -87,7 +87,7 @@ static void whiteheat_set_termios(struct tty_struct *tty,
static int whiteheat_tiocmget(struct tty_struct *tty);
static int whiteheat_tiocmset(struct tty_struct *tty,
unsigned int set, unsigned int clear);
-static void whiteheat_break_ctl(struct tty_struct *tty, int break_state);
+static int whiteheat_break_ctl(struct tty_struct *tty, int break_state);
static struct usb_serial_driver whiteheat_fake_device = {
.driver = {
@@ -449,10 +449,11 @@ static void whiteheat_set_termios(struct tty_struct *tty,
firm_setup_port(tty);
}
-static void whiteheat_break_ctl(struct tty_struct *tty, int break_state)
+static int whiteheat_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
- firm_set_break(port, break_state);
+
+ return firm_set_break(port, break_state);
}
diff --git a/drivers/usb/serial/xr_serial.c b/drivers/usb/serial/xr_serial.c
index f3811e060a44..60fae9d012a6 100644
--- a/drivers/usb/serial/xr_serial.c
+++ b/drivers/usb/serial/xr_serial.c
@@ -503,7 +503,7 @@ static void xr_dtr_rts(struct usb_serial_port *port, int on)
xr_tiocmset_port(port, 0, TIOCM_DTR | TIOCM_RTS);
}
-static void xr_break_ctl(struct tty_struct *tty, int break_state)
+static int xr_break_ctl(struct tty_struct *tty, int break_state)
{
struct usb_serial_port *port = tty->driver_data;
struct xr_data *data = usb_get_serial_port_data(port);
@@ -517,7 +517,7 @@ static void xr_break_ctl(struct tty_struct *tty, int break_state)
dev_dbg(&port->dev, "Turning break %s\n", state == 0 ? "off" : "on");
- xr_set_reg_uart(port, type->tx_break, state);
+ return xr_set_reg_uart(port, type->tx_break, state);
}
/* Tx and Rx clock mask values obtained from section 3.3.4 of datasheet */
diff --git a/drivers/usb/storage/realtek_cr.c b/drivers/usb/storage/realtek_cr.c
index a026c6cb6e68..de82d8260d72 100644
--- a/drivers/usb/storage/realtek_cr.c
+++ b/drivers/usb/storage/realtek_cr.c
@@ -915,7 +915,6 @@ static int realtek_cr_autosuspend_setup(struct us_data *us)
us->proto_handler = rts51x_invoke_transport;
chip->timer_expires = 0;
- timer_setup(&chip->rts51x_suspend_timer, rts51x_suspend_timer_fn, 0);
fw5895_init(us);
/* enable autosuspend function of the usb device */
@@ -933,10 +932,7 @@ static void realtek_cr_destructor(void *extra)
return;
#ifdef CONFIG_REALTEK_AUTOPM
- if (ss_en) {
- del_timer(&chip->rts51x_suspend_timer);
- chip->timer_expires = 0;
- }
+ timer_shutdown_sync(&chip->rts51x_suspend_timer);
#endif
kfree(chip->status);
}
@@ -981,6 +977,9 @@ static int init_realtek_cr(struct us_data *us)
us->extra = chip;
us->extra_destructor = realtek_cr_destructor;
+#ifdef CONFIG_REALTEK_AUTOPM
+ timer_setup(&chip->rts51x_suspend_timer, rts51x_suspend_timer_fn, 0);
+#endif
us->max_lun = chip->max_lun = rts51x_get_max_lun(us);
chip->us = us;
diff --git a/drivers/usb/storage/sierra_ms.c b/drivers/usb/storage/sierra_ms.c
index 0774ba22fb66..26b98413b85c 100644
--- a/drivers/usb/storage/sierra_ms.c
+++ b/drivers/usb/storage/sierra_ms.c
@@ -77,6 +77,13 @@ static int sierra_get_swoc_info(struct usb_device *udev,
sizeof(struct swoc_info), /* __u16 size */
USB_CTRL_SET_TIMEOUT); /* int timeout */
+ /*
+ * A short IN transfer leaves the tail of swocInfo uninitialized;
+ * only a full transfer is valid.
+ */
+ if (result != sizeof(struct swoc_info))
+ return -EIO;
+
swocInfo->LinuxSKU = le16_to_cpu(swocInfo->LinuxSKU);
swocInfo->LinuxVer = le16_to_cpu(swocInfo->LinuxVer);
return result;
diff --git a/drivers/usb/typec/anx7411.c b/drivers/usb/typec/anx7411.c
index 4ddaeb7ddb34..02d6e44c9481 100644
--- a/drivers/usb/typec/anx7411.c
+++ b/drivers/usb/typec/anx7411.c
@@ -1572,8 +1572,11 @@ static void anx7411_i2c_remove(struct i2c_client *client)
anx7411_partner_unregister_altmode(plat);
anx7411_unregister_partner(plat);
- if (plat->workqueue)
+ if (plat->workqueue) {
+ disable_irq(plat->intp_irq);
+ cancel_work_sync(&plat->work);
destroy_workqueue(plat->workqueue);
+ }
if (plat->spi_client)
i2c_unregister_device(plat->spi_client);
diff --git a/drivers/usb/typec/tcpm/tcpci_maxim.c b/drivers/usb/typec/tcpm/tcpci_maxim.c
index 83f5f1dcd243..5333c609ddef 100644
--- a/drivers/usb/typec/tcpm/tcpci_maxim.c
+++ b/drivers/usb/typec/tcpm/tcpci_maxim.c
@@ -191,6 +191,15 @@ static void process_rx(struct max_tcpci_chip *chip, u16 status)
rx_buf_ptr = rx_buf + TCPC_RECEIVE_BUFFER_RX_BYTE_BUF_OFFSET;
msg.header = cpu_to_le16(*(u16 *)rx_buf_ptr);
rx_buf_ptr = rx_buf_ptr + sizeof(msg.header);
+
+ if (count < TCPC_RECEIVE_BUFFER_RX_BYTE_BUF_OFFSET + sizeof(msg.header) +
+ pd_header_cnt_le(msg.header) * sizeof(msg.payload[0])) {
+ max_tcpci_write16(chip, TCPC_ALERT, TCPC_ALERT_RX_STATUS);
+ dev_err(chip->dev, "Invalid TCPC_RX_BYTE_CNT %d for header cnt %d\n",
+ count, pd_header_cnt_le(msg.header));
+ return;
+ }
+
for (payload_index = 0; payload_index < pd_header_cnt_le(msg.header); payload_index++,
rx_buf_ptr += sizeof(msg.payload[0]))
msg.payload[payload_index] = cpu_to_le32(*(u32 *)rx_buf_ptr);
diff --git a/drivers/usb/typec/tcpm/tcpci_rt1711h.c b/drivers/usb/typec/tcpm/tcpci_rt1711h.c
index d608fefd4f30..8fdffec401e0 100644
--- a/drivers/usb/typec/tcpm/tcpci_rt1711h.c
+++ b/drivers/usb/typec/tcpm/tcpci_rt1711h.c
@@ -327,6 +327,8 @@ static int rt1711h_check_revision(struct i2c_client *i2c, struct rt1711h_chip *c
return ret;
}
+static void rt1711h_unregister_tcpci_port(void *tcpci);
+
static int rt1711h_probe(struct i2c_client *client,
const struct i2c_device_id *i2c_id)
{
@@ -379,6 +381,10 @@ static int rt1711h_probe(struct i2c_client *client,
if (IS_ERR_OR_NULL(chip->tcpci))
return PTR_ERR(chip->tcpci);
+ ret = devm_add_action_or_reset(chip->dev, rt1711h_unregister_tcpci_port, chip->tcpci);
+ if (ret)
+ return ret;
+
ret = devm_request_threaded_irq(chip->dev, client->irq, NULL,
rt1711h_irq,
IRQF_ONESHOT | IRQF_TRIGGER_LOW,
@@ -396,11 +402,9 @@ static int rt1711h_probe(struct i2c_client *client,
return 0;
}
-static void rt1711h_remove(struct i2c_client *client)
+static void rt1711h_unregister_tcpci_port(void *tcpci)
{
- struct rt1711h_chip *chip = i2c_get_clientdata(client);
-
- tcpci_unregister_port(chip->tcpci);
+ tcpci_unregister_port(tcpci);
}
static const struct i2c_device_id rt1711h_id[] = {
@@ -425,7 +429,6 @@ static struct i2c_driver rt1711h_i2c_driver = {
.of_match_table = of_match_ptr(rt1711h_of_match),
},
.probe = rt1711h_probe,
- .remove = rt1711h_remove,
.id_table = rt1711h_id,
};
module_i2c_driver(rt1711h_i2c_driver);
diff --git a/drivers/usb/typec/tcpm/tcpm.c b/drivers/usb/typec/tcpm/tcpm.c
index 6969db16f07c..b7f81db5f371 100644
--- a/drivers/usb/typec/tcpm/tcpm.c
+++ b/drivers/usb/typec/tcpm/tcpm.c
@@ -522,9 +522,12 @@ static const char * const pd_rev[] = {
(tcpm_cc_is_source((port)->cc2) && \
!tcpm_cc_is_source((port)->cc1)))
-#define tcpm_port_is_debug(port) \
+#define tcpm_port_is_debug_source(port) \
(tcpm_cc_is_source((port)->cc1) && tcpm_cc_is_source((port)->cc2))
+#define tcpm_port_is_debug(port) \
+ tcpm_port_is_debug_source(port)
+
#define tcpm_port_is_audio(port) \
(tcpm_cc_is_audio((port)->cc1) && tcpm_cc_is_audio((port)->cc2))
@@ -1473,6 +1476,9 @@ static void svdm_consume_identity(struct tcpm_port *port, const u32 *p, int cnt)
u32 vdo = p[VDO_INDEX_IDH];
u32 product = p[VDO_INDEX_PRODUCT];
+ if (cnt <= VDO_INDEX_PRODUCT)
+ return;
+
memset(&port->mode_data, 0, sizeof(port->mode_data));
port->partner_ident.id_header = vdo;
@@ -1541,17 +1547,16 @@ static void svdm_consume_modes(struct tcpm_port *port, const u32 *p, int cnt)
struct typec_altmode_desc *paltmode;
int i;
- if (pmdata->altmodes >= ARRAY_SIZE(port->partner_altmode)) {
- /* Already logged in svdm_consume_svids() */
- return;
- }
-
if (pmdata->svid_index < 0 || pmdata->svid_index >= pmdata->nsvids) {
tcpm_log(port, "Invalid SVID index %d", pmdata->svid_index);
return;
}
for (i = 1; i < cnt; i++) {
+ if (pmdata->altmodes >= ALTMODE_DISCOVERY_MAX) {
+ /* Already logged in svdm_consume_svids() */
+ return;
+ }
paltmode = &pmdata->altmode_desc[pmdata->altmodes];
memset(paltmode, 0, sizeof(*paltmode));
@@ -4023,7 +4028,7 @@ static void run_state_machine(struct tcpm_port *port)
tcpm_set_state(port, SNK_UNATTACHED, PD_T_DRP_SNK);
break;
case SRC_ATTACH_WAIT:
- if (tcpm_port_is_debug(port))
+ if (tcpm_port_is_debug_source(port))
tcpm_set_state(port, DEBUG_ACC_ATTACHED,
PD_T_CC_DEBOUNCE);
else if (tcpm_port_is_audio(port))
@@ -5035,7 +5040,7 @@ static void _tcpm_cc_change(struct tcpm_port *port, enum typec_cc_status cc1,
switch (port->state) {
case TOGGLING:
- if (tcpm_port_is_debug(port) || tcpm_port_is_audio(port) ||
+ if (tcpm_port_is_debug_source(port) || tcpm_port_is_audio(port) ||
tcpm_port_is_source(port))
tcpm_set_state(port, SRC_ATTACH_WAIT, 0);
else if (tcpm_port_is_sink(port))
@@ -5043,7 +5048,7 @@ static void _tcpm_cc_change(struct tcpm_port *port, enum typec_cc_status cc1,
break;
case SRC_UNATTACHED:
case ACC_UNATTACHED:
- if (tcpm_port_is_debug(port) || tcpm_port_is_audio(port) ||
+ if (tcpm_port_is_debug_source(port) || tcpm_port_is_audio(port) ||
tcpm_port_is_source(port))
tcpm_set_state(port, SRC_ATTACH_WAIT, 0);
break;
@@ -5540,22 +5545,38 @@ static void tcpm_pd_event_handler(struct kthread_work *work)
port->upcoming_state = FR_SWAP_SEND;
ret = tcpm_ams_start(port, FAST_ROLE_SWAP);
if (ret == -EAGAIN)
- port->upcoming_state = INVALID_STATE;
+ tcpm_set_state(port, ERROR_RECOVERY, 0);
} else {
tcpm_log(port, "Discarding FRS_SIGNAL! Not in sink ready");
}
}
if (events & TCPM_SOURCING_VBUS) {
- tcpm_log(port, "sourcing vbus");
/*
* In fast role swap case TCPC autonomously sources vbus. Set vbus_source
- * true as TCPM wouldn't have called tcpm_set_vbus.
+ * true conditionally as TCPM wouldn't have called tcpm_set_vbus.
+ * If TCPM calls tcpm_set_vbus to source vbus, vbus_source would already
+ * be true.
*
- * When vbus is sourced on the command on TCPM i.e. TCPM called
- * tcpm_set_vbus to source vbus, vbus_source would already be true.
+ * When TCPM_FRS_EVENT and TCPM_SOURCING_VBUS arrive simultaneously,
+ * handling TCPM_FRS_EVENT above transitions the state to AMS_START
+ * with upcoming_state FR_SWAP_SEND.
*/
- port->vbus_source = true;
- _tcpm_pd_vbus_on(port);
+
+ if (tcpm_port_is_source(port) ||
+ tcpm_port_is_debug_source(port) ||
+ (port->state == AMS_START && port->upcoming_state == FR_SWAP_SEND) ||
+ port->state == FR_SWAP_SEND ||
+ port->state == FR_SWAP_SEND_TIMEOUT ||
+ port->state == FR_SWAP_SNK_SRC_TRANSITION_TO_OFF ||
+ port->state == FR_SWAP_SNK_SRC_NEW_SINK_READY ||
+ port->state == FR_SWAP_SNK_SRC_SOURCE_VBUS_APPLIED) {
+ tcpm_log(port, "sourcing vbus");
+ port->vbus_source = true;
+ _tcpm_pd_vbus_on(port);
+ } else {
+ tcpm_log(port, "Discarding sourcing vbus! Invalid state %s",
+ tcpm_states[port->state]);
+ }
}
spin_lock(&port->pd_event_lock);
diff --git a/drivers/usb/typec/tipd/core.c b/drivers/usb/typec/tipd/core.c
index c6ebe2b7bad1..44a3ef2bc657 100644
--- a/drivers/usb/typec/tipd/core.c
+++ b/drivers/usb/typec/tipd/core.c
@@ -231,7 +231,7 @@ static void tps6598x_set_data_role(struct tps6598x *tps,
static int tps6598x_connect(struct tps6598x *tps, u32 status)
{
- struct typec_partner_desc desc;
+ struct typec_partner_desc desc = { };
enum typec_pwr_opmode mode;
int ret;
@@ -242,7 +242,6 @@ static int tps6598x_connect(struct tps6598x *tps, u32 status)
desc.usb_pd = mode == TYPEC_PWR_MODE_PD;
desc.accessory = TYPEC_ACCESSORY_NONE; /* XXX: handle accessories */
- desc.identity = NULL;
if (desc.usb_pd) {
ret = tps6598x_read_partner_identity(tps);
diff --git a/drivers/usb/typec/ucsi/displayport.c b/drivers/usb/typec/ucsi/displayport.c
index 5d78f9ba3b02..1d92020b0664 100644
--- a/drivers/usb/typec/ucsi/displayport.c
+++ b/drivers/usb/typec/ucsi/displayport.c
@@ -71,11 +71,14 @@ static int ucsi_displayport_enter(struct typec_altmode *alt, u32 *vdo)
if (ret < 0) {
if (ucsi->version > 0x0100)
goto err_unlock;
- cur = 0xff;
}
- if (cur < UCSI_MAX_ALTMODES) {
- ret = dp->con->port_altmode[cur] == alt ? 0 : -EBUSY;
+ if (cur != 0xff) {
+ if (cur < UCSI_MAX_ALTMODES)
+ ret = dp->con->port_altmode[cur] == alt ? 0 : -EBUSY;
+ else
+ ret = -EINVAL;
+
goto err_unlock;
}
diff --git a/drivers/usb/typec/ucsi/ucsi.c b/drivers/usb/typec/ucsi/ucsi.c
index b4703ce86212..221fe9ade958 100644
--- a/drivers/usb/typec/ucsi/ucsi.c
+++ b/drivers/usb/typec/ucsi/ucsi.c
@@ -36,6 +36,19 @@
*/
#define UCSI_SWAP_TIMEOUT_MS 5000
+static int ucsi_read_message_in(struct ucsi *ucsi, void *buf,
+ size_t buf_size)
+{
+ /*
+ * Below UCSI 2.0, MESSAGE_IN was limited to 16 bytes. Truncate the
+ * reads here.
+ */
+ if (ucsi->version <= UCSI_VERSION_1_2)
+ buf_size = clamp(buf_size, 0, 16);
+
+ return ucsi->ops->read(ucsi, UCSI_MESSAGE_IN, buf, buf_size);
+}
+
static int ucsi_acknowledge(struct ucsi *ucsi, bool conn_ack)
{
u64 ctrl;
@@ -66,7 +79,7 @@ static int ucsi_read_error(struct ucsi *ucsi)
if (ret < 0)
return ret;
- ret = ucsi->ops->read(ucsi, UCSI_MESSAGE_IN, &error, sizeof(error));
+ ret = ucsi_read_message_in(ucsi, &error, sizeof(error));
if (ret)
return ret;
@@ -173,7 +186,7 @@ static int ucsi_send_command_common(struct ucsi *ucsi, u64 command,
length = ret;
if (data) {
- ret = ucsi->ops->read(ucsi, UCSI_MESSAGE_IN, data, size);
+ ret = ucsi_read_message_in(ucsi, data, size);
if (ret)
goto out;
}
@@ -1092,10 +1105,21 @@ static struct fwnode_handle *ucsi_find_fwnode(struct ucsi_connector *con)
{
struct fwnode_handle *fwnode;
int i = 1;
+ int ret;
+ u32 port;
- device_for_each_child_node(con->ucsi->dev, fwnode)
- if (i++ == con->num)
- return fwnode;
+ device_for_each_child_node(con->ucsi->dev, fwnode) {
+ ret = fwnode_property_read_u32(fwnode, "reg", &port);
+ if (ret < 0) {
+ if (i == con->num)
+ return fwnode;
+ } else {
+ if (port == con->num - 1)
+ return fwnode;
+ }
+
+ i++;
+ }
return NULL;
}
@@ -1578,6 +1602,15 @@ int ucsi_register(struct ucsi *ucsi)
if (!ucsi->version)
return -ENODEV;
+ /*
+ * Version format is JJ.M.N (JJ = Major version, M = Minor version,
+ * N = sub-minor version).
+ */
+ dev_dbg(ucsi->dev, "Registered UCSI interface with version %x.%x.%x",
+ UCSI_BCD_GET_MAJOR(ucsi->version),
+ UCSI_BCD_GET_MINOR(ucsi->version),
+ UCSI_BCD_GET_SUBMINOR(ucsi->version));
+
queue_delayed_work(system_long_wq, &ucsi->work, 0);
return 0;
diff --git a/drivers/usb/typec/ucsi/ucsi.h b/drivers/usb/typec/ucsi/ucsi.h
index cee43f19adf5..16ead80d3a4e 100644
--- a/drivers/usb/typec/ucsi/ucsi.h
+++ b/drivers/usb/typec/ucsi/ucsi.h
@@ -23,6 +23,17 @@ struct ucsi_connector;
#define UCSI_CONTROL 8
#define UCSI_MESSAGE_IN 16
#define UCSI_MESSAGE_OUT 32
+#define UCSIv2_MESSAGE_OUT 272
+
+/* UCSI versions */
+#define UCSI_VERSION_1_2 0x0120
+#define UCSI_VERSION_2_0 0x0200
+#define UCSI_VERSION_2_1 0x0210
+#define UCSI_VERSION_3_0 0x0300
+
+#define UCSI_BCD_GET_MAJOR(_v_) (((_v_) >> 8) & 0xFF)
+#define UCSI_BCD_GET_MINOR(_v_) (((_v_) >> 4) & 0x0F)
+#define UCSI_BCD_GET_SUBMINOR(_v_) ((_v_) & 0x0F)
/* Command Status and Connector Change Indication (CCI) bits */
#define UCSI_CCI_CONNECTOR(_c_) (((_c_) & GENMASK(7, 1)) >> 1)
diff --git a/drivers/usb/typec/ucsi/ucsi_acpi.c b/drivers/usb/typec/ucsi/ucsi_acpi.c
index 03c820fe5b0c..eeb457ff255e 100644
--- a/drivers/usb/typec/ucsi/ucsi_acpi.c
+++ b/drivers/usb/typec/ucsi/ucsi_acpi.c
@@ -106,6 +106,31 @@ static const struct ucsi_operations ucsi_acpi_ops = {
.async_write = ucsi_acpi_async_write
};
+static int ucsi_acer_read(struct ucsi *ucsi, unsigned int offset,
+ void *val, size_t val_len)
+{
+ struct ucsi_acpi *ua = ucsi_get_drvdata(ucsi);
+ u16 *version = val;
+ int ret;
+
+ ret = ucsi_acpi_read(ucsi, offset, val, val_len);
+ if (ret)
+ return ret;
+
+ if (offset == UCSI_VERSION && val_len == sizeof(*version) && !*version) {
+ dev_warn(ua->dev, "UCSI version is zero, assuming 1.2\n");
+ *version = 0x0120;
+ }
+
+ return 0;
+}
+
+static const struct ucsi_operations ucsi_acer_ops = {
+ .read = ucsi_acer_read,
+ .sync_write = ucsi_acpi_sync_write,
+ .async_write = ucsi_acpi_async_write
+};
+
static int
ucsi_zenbook_read(struct ucsi *ucsi, unsigned int offset, void *val, size_t val_len)
{
@@ -173,6 +198,13 @@ static const struct ucsi_operations ucsi_dell_ops = {
};
static const struct dmi_system_id ucsi_acpi_quirks[] = {
+ {
+ .matches = {
+ DMI_MATCH(DMI_SYS_VENDOR, "Acer"),
+ DMI_MATCH(DMI_PRODUCT_NAME, "Nitro ANV15-41"),
+ },
+ .driver_data = (void *)&ucsi_acer_ops,
+ },
{
.matches = {
DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."),
diff --git a/drivers/usb/usbip/usbip_common.c b/drivers/usb/usbip/usbip_common.c
index d6dbd038af67..ba9e7c616e12 100644
--- a/drivers/usb/usbip/usbip_common.c
+++ b/drivers/usb/usbip/usbip_common.c
@@ -315,6 +315,7 @@ int usbip_recv(struct socket *sock, void *buf, int size)
do {
sock->sk->sk_allocation = GFP_NOIO;
+ sock->sk->sk_use_task_frag = false;
result = sock_recvmsg(sock, &msg, MSG_WAITALL);
if (result <= 0)
diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
index b53420e874ac..9727b4352c40 100644
--- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h
+++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h
@@ -11,6 +11,8 @@
#define MLX5V_ETH_HARD_MTU (ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN)
+extern int mlx5_vdpa_max_iotlb_entries;
+
struct mlx5_vdpa_direct_mr {
u64 start;
u64 end;
diff --git a/drivers/vdpa/mlx5/core/mr.c b/drivers/vdpa/mlx5/core/mr.c
index 41489d4371f3..2c0e3ae87476 100644
--- a/drivers/vdpa/mlx5/core/mr.c
+++ b/drivers/vdpa/mlx5/core/mr.c
@@ -562,6 +562,9 @@ static int _mlx5_vdpa_create_dvq_mr(struct mlx5_vdpa_dev *mvdev,
if (mr->initialized)
return 0;
+ if (mlx5_vdpa_max_iotlb_entries < 2)
+ return -EINVAL;
+
if (iotlb)
err = create_user_mr(mvdev, iotlb);
else
diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c
index 9800f9bec225..f249f98a3e17 100644
--- a/drivers/vdpa/mlx5/core/resources.c
+++ b/drivers/vdpa/mlx5/core/resources.c
@@ -3,8 +3,14 @@
#include <linux/iova.h>
#include <linux/mlx5/driver.h>
+#include <linux/moduleparam.h>
#include "mlx5_vdpa.h"
+int mlx5_vdpa_max_iotlb_entries = 2048;
+module_param_named(max_iotlb_entries, mlx5_vdpa_max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
static int alloc_pd(struct mlx5_vdpa_dev *dev, u32 *pdn, u16 uid)
{
struct mlx5_core_dev *mdev = dev->mdev;
@@ -229,7 +235,10 @@ int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey)
static int init_ctrl_vq(struct mlx5_vdpa_dev *mvdev)
{
- mvdev->cvq.iotlb = vhost_iotlb_alloc(0, 0);
+ if (mlx5_vdpa_max_iotlb_entries < 2)
+ return -EINVAL;
+
+ mvdev->cvq.iotlb = vhost_iotlb_alloc(mlx5_vdpa_max_iotlb_entries, 0);
if (!mvdev->cvq.iotlb)
return -ENOMEM;
diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
index 74d295312466..fd4a62101a3d 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
@@ -594,6 +594,8 @@ static int cq_create(struct mlx5_vdpa_net *ndev, u16 idx, u32 num_ent)
vcq->mcq.set_ci_db = vcq->db.db;
vcq->mcq.arm_db = vcq->db.db + 1;
vcq->mcq.cqe_sz = 64;
+ vcq->mcq.comp = mlx5_vdpa_cq_comp;
+ vcq->cqe = num_ent;
err = cq_frag_buf_alloc(ndev, &vcq->buf, num_ent);
if (err)
@@ -633,10 +635,6 @@ static int cq_create(struct mlx5_vdpa_net *ndev, u16 idx, u32 num_ent)
if (err)
goto err_vec;
- vcq->mcq.comp = mlx5_vdpa_cq_comp;
- vcq->cqe = num_ent;
- vcq->mcq.set_ci_db = vcq->db.db;
- vcq->mcq.arm_db = vcq->db.db + 1;
mlx5_cq_arm(&mvq->cq.mcq, MLX5_CQ_DB_REQ_NOT, uar_page, mvq->cq.mcq.cons_index);
kfree(in);
return 0;
@@ -2847,9 +2845,6 @@ static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group,
{
struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev);
- if (group >= MLX5_VDPA_NUMVQ_GROUPS)
- return -EINVAL;
-
mvdev->group2asid[group] = asid;
return 0;
}
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index e7fc25bfdd23..68d63bb17cac 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -34,7 +34,7 @@ MODULE_PARM_DESC(batch_mapping, "Batched mapping 1 -Enable; 0 - Disable");
static int max_iotlb_entries = 2048;
module_param(max_iotlb_entries, int, 0444);
MODULE_PARM_DESC(max_iotlb_entries,
- "Maximum number of iotlb entries for each address space. 0 means unlimited. (default: 2048)");
+ "Maximum number of iotlb entries for each address space. (default: 2048)");
#define VDPASIM_QUEUE_ALIGN PAGE_SIZE
#define VDPASIM_QUEUE_MAX 256
@@ -266,6 +266,9 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr,
struct device *dev;
int i, ret = -ENOMEM;
+ if (max_iotlb_entries < 2)
+ return ERR_PTR(-EINVAL);
+
if (config->mask & BIT_ULL(VDPA_ATTR_DEV_FEATURES)) {
if (config->device_features &
~dev_attr->supported_features)
@@ -599,12 +602,6 @@ static int vdpasim_set_group_asid(struct vdpa_device *vdpa, unsigned int group,
struct vhost_iotlb *iommu;
int i;
- if (group > vdpasim->dev_attr.ngroups)
- return -EINVAL;
-
- if (asid >= vdpasim->dev_attr.nas)
- return -EINVAL;
-
iommu = &vdpasim->iommu[asid];
spin_lock(&vdpasim->lock);
diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_user/iova_domain.c
index 82b796332564..e60a2b4065af 100644
--- a/drivers/vdpa/vdpa_user/iova_domain.c
+++ b/drivers/vdpa/vdpa_user/iova_domain.c
@@ -12,11 +12,17 @@
#include <linux/file.h>
#include <linux/anon_inodes.h>
#include <linux/highmem.h>
+#include <linux/moduleparam.h>
#include <linux/vmalloc.h>
#include <linux/vdpa.h>
#include "iova_domain.h"
+static int max_iotlb_entries = 2048;
+module_param(max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
static int vduse_iotlb_add_range(struct vduse_iova_domain *domain,
u64 start, u64 last,
u64 addr, unsigned int perm,
@@ -608,11 +614,14 @@ vduse_domain_create(unsigned long iova_limit, size_t bounce_size)
if (iova_limit <= bounce_size)
return NULL;
+ if (max_iotlb_entries <= 0)
+ return NULL;
+
domain = kzalloc(sizeof(*domain), GFP_KERNEL);
if (!domain)
return NULL;
- domain->iotlb = vhost_iotlb_alloc(0, 0);
+ domain->iotlb = vhost_iotlb_alloc(max_iotlb_entries, 0);
if (!domain->iotlb)
goto err_iotlb;
diff --git a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
index e682c1461499..77ddf681b9dc 100644
--- a/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
+++ b/drivers/vfio/pci/hisilicon/hisi_acc_vfio_pci.c
@@ -426,13 +426,6 @@ static int vf_qm_check_match(struct hisi_acc_vf_core_device *hisi_acc_vdev,
return -EINVAL;
}
- ret = qm_write_regs(vf_qm, QM_VF_STATE, &vf_data->vf_qm_state, 1);
- if (ret) {
- dev_err(dev, "failed to write QM_VF_STATE\n");
- return ret;
- }
-
- hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
return 0;
}
@@ -497,6 +490,20 @@ static int vf_qm_load_data(struct hisi_acc_vf_core_device *hisi_acc_vdev,
if (migf->total_length < sizeof(struct acc_vf_data))
return -EINVAL;
+ if (!vf_data->eqe_dma || !vf_data->aeqe_dma ||
+ !vf_data->sqc_dma || !vf_data->cqc_dma) {
+ dev_info(dev, "resume dma addr is NULL!\n");
+ hisi_acc_vdev->vf_qm_state = QM_NOT_READY;
+ return 0;
+ }
+
+ ret = qm_write_regs(qm, QM_VF_STATE, &vf_data->vf_qm_state, 1);
+ if (ret) {
+ dev_err(dev, "failed to write QM_VF_STATE\n");
+ return -EINVAL;
+ }
+ hisi_acc_vdev->vf_qm_state = vf_data->vf_qm_state;
+
qm->eqe_dma = vf_data->eqe_dma;
qm->aeqe_dma = vf_data->aeqe_dma;
qm->sqc_dma = vf_data->sqc_dma;
@@ -1269,6 +1276,7 @@ static int hisi_acc_vfio_pci_migrn_init_dev(struct vfio_device *core_vdev)
hisi_acc_vdev->vf_id = pci_iov_vf_id(pdev) + 1;
hisi_acc_vdev->pf_qm = pf_qm;
hisi_acc_vdev->vf_dev = pdev;
+ hisi_acc_vdev->vf_qm_state = QM_NOT_READY;
mutex_init(&hisi_acc_vdev->state_mutex);
core_vdev->migration_flags = VFIO_MIGRATION_STOP_COPY;
diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c
index fd6ccb8454a2..191d4c7b686a 100644
--- a/drivers/vfio/pci/mlx5/main.c
+++ b/drivers/vfio/pci/mlx5/main.c
@@ -71,6 +71,7 @@ static int mlx5vf_add_migration_pages(struct mlx5_vf_migration_file *migf,
unsigned long filled;
unsigned int to_fill;
int ret;
+ int i;
to_fill = min_t(unsigned int, npages, PAGE_SIZE / sizeof(*page_list));
page_list = kvzalloc(to_fill * sizeof(*page_list), GFP_KERNEL);
@@ -90,7 +91,7 @@ static int mlx5vf_add_migration_pages(struct mlx5_vf_migration_file *migf,
GFP_KERNEL);
if (ret)
- goto err;
+ goto err_append;
migf->allocated_length += filled * PAGE_SIZE;
/* clean input for another bulk allocation */
memset(page_list, 0, filled * sizeof(*page_list));
@@ -101,6 +102,9 @@ static int mlx5vf_add_migration_pages(struct mlx5_vf_migration_file *migf,
kvfree(page_list);
return 0;
+err_append:
+ for (i = filled - 1; i >= 0; i--)
+ __free_page(page_list[i]);
err:
kvfree(page_list);
return ret;
diff --git a/drivers/vhost/iotlb.c b/drivers/vhost/iotlb.c
index ea61330a3431..a6228dd24931 100644
--- a/drivers/vhost/iotlb.c
+++ b/drivers/vhost/iotlb.c
@@ -20,6 +20,14 @@ INTERVAL_TREE_DEFINE(struct vhost_iotlb_map,
rb, __u64, __subtree_last,
START, LAST, static inline, vhost_iotlb_itree);
+static void vhost_iotlb_map_unlink(struct vhost_iotlb *iotlb,
+ struct vhost_iotlb_map *map)
+{
+ vhost_iotlb_itree_remove(map, &iotlb->root);
+ list_del(&map->link);
+ iotlb->nmaps--;
+}
+
/**
* vhost_iotlb_map_free - remove a map node and free it
* @iotlb: the IOTLB
@@ -28,10 +36,8 @@ INTERVAL_TREE_DEFINE(struct vhost_iotlb_map,
void vhost_iotlb_map_free(struct vhost_iotlb *iotlb,
struct vhost_iotlb_map *map)
{
- vhost_iotlb_itree_remove(map, &iotlb->root);
- list_del(&map->link);
+ vhost_iotlb_map_unlink(iotlb, map);
kfree(map);
- iotlb->nmaps--;
}
EXPORT_SYMBOL_GPL(vhost_iotlb_map_free);
@@ -57,14 +63,25 @@ int vhost_iotlb_add_range_ctx(struct vhost_iotlb *iotlb,
if (last < start)
return -EFAULT;
+ if (!iotlb->limit)
+ return -EINVAL;
+
/* If the range being mapped is [0, ULONG_MAX], split it into two entries
* otherwise its size would overflow u64.
*/
if (start == 0 && last == ULONG_MAX) {
u64 mid = last / 2;
- int err = vhost_iotlb_add_range_ctx(iotlb, start, mid, addr,
- perm, opaque);
+ int err;
+
+ if (iotlb->limit < 2)
+ return -ENOSPC;
+ if (!(iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) &&
+ iotlb->nmaps > iotlb->limit - 2)
+ return -ENOSPC;
+
+ err = vhost_iotlb_add_range_ctx(iotlb, start, mid, addr,
+ perm, opaque);
if (err)
return err;
@@ -72,17 +89,19 @@ int vhost_iotlb_add_range_ctx(struct vhost_iotlb *iotlb,
start = mid + 1;
}
- if (iotlb->limit &&
- iotlb->nmaps == iotlb->limit &&
- iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) {
- map = list_first_entry(&iotlb->list, typeof(*map), link);
- vhost_iotlb_map_free(iotlb, map);
+ if (iotlb->nmaps >= iotlb->limit) {
+ if (iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) {
+ map = list_first_entry(&iotlb->list, typeof(*map), link);
+ vhost_iotlb_map_unlink(iotlb, map);
+ } else {
+ return -ENOSPC;
+ }
+ } else {
+ map = kmalloc(sizeof(*map), GFP_ATOMIC);
+ if (!map)
+ return -ENOMEM;
}
- map = kmalloc(sizeof(*map), GFP_ATOMIC);
- if (!map)
- return -ENOMEM;
-
map->start = start;
map->size = last - start + 1;
map->last = last;
diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
index 40c43e0611ca..03bc085358ad 100644
--- a/drivers/vhost/vdpa.c
+++ b/drivers/vhost/vdpa.c
@@ -34,6 +34,11 @@ enum {
#define VHOST_VDPA_DEV_MAX (1U << MINORBITS)
+static int max_iotlb_entries = 2048;
+module_param(max_iotlb_entries, int, 0444);
+MODULE_PARM_DESC(max_iotlb_entries,
+ "Maximum number of iotlb entries. (default: 2048)");
+
#define VHOST_VDPA_IOTLB_BUCKETS 16
struct vhost_vdpa_as {
@@ -108,12 +113,14 @@ static struct vhost_vdpa_as *vhost_vdpa_alloc_as(struct vhost_vdpa *v, u32 asid)
if (asid >= v->vdpa->nas)
return NULL;
+ if (max_iotlb_entries <= 0)
+ return NULL;
as = kmalloc(sizeof(*as), GFP_KERNEL);
if (!as)
return NULL;
- vhost_iotlb_init(&as->iotlb, 0, 0);
+ vhost_iotlb_init(&as->iotlb, max_iotlb_entries, 0);
as->id = asid;
hlist_add_head(&as->hash_link, head);
@@ -546,7 +553,7 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
case VHOST_VDPA_SET_GROUP_ASID:
if (copy_from_user(&s, argp, sizeof(s)))
return -EFAULT;
- if (s.num >= vdpa->nas)
+ if (idx >= vdpa->ngroups || s.num >= vdpa->nas)
return -EINVAL;
if (!ops->set_group_asid)
return -EOPNOTSUPP;
diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c
index 3cd58e9ae99b..d68f61cef67e 100644
--- a/drivers/vhost/vhost.c
+++ b/drivers/vhost/vhost.c
@@ -622,6 +622,9 @@ EXPORT_SYMBOL_GPL(vhost_dev_set_owner);
static struct vhost_iotlb *iotlb_alloc(void)
{
+ if (max_iotlb_entries <= 0)
+ return NULL;
+
return vhost_iotlb_alloc(max_iotlb_entries,
VHOST_IOTLB_FLAG_RETIRE);
}
@@ -1008,10 +1011,39 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d)
mutex_unlock(&d->vqs[i]->mutex);
}
-static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq,
- __virtio16 *idx)
+static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq)
{
- return vhost_get_avail(vq, *idx, &vq->avail->idx);
+ __virtio16 idx;
+ u16 avail_idx;
+ int r;
+
+ r = vhost_get_avail(vq, idx, &vq->avail->idx);
+ if (unlikely(r < 0)) {
+ vq_err(vq, "Failed to access available index at %p (%d)\n",
+ &vq->avail->idx, r);
+ return r;
+ }
+
+ /* Check it isn't doing very strange thing with available indexes */
+ avail_idx = vhost16_to_cpu(vq, idx);
+ if (unlikely((u16)(avail_idx - vq->last_avail_idx) > vq->num)) {
+ vq_err(vq, "Invalid available index change from %u to %u",
+ vq->last_avail_idx, avail_idx);
+ return -EINVAL;
+ }
+
+ /* We're done if there is nothing new */
+ if (avail_idx == vq->avail_idx)
+ return 0;
+
+ vq->avail_idx = avail_idx;
+
+ /*
+ * We updated vq->avail_idx so we need a memory barrier between
+ * the index read above and the caller reading avail ring entries.
+ */
+ smp_rmb();
+ return 1;
}
static inline int vhost_get_avail_head(struct vhost_virtqueue *vq,
@@ -1441,6 +1473,8 @@ static long vhost_set_memory(struct vhost_dev *d, struct vhost_memory __user *m)
return -EOPNOTSUPP;
if (mem.nregions > max_mem_regions)
return -E2BIG;
+ if (max_iotlb_entries <= 0)
+ return -EINVAL;
newmem = kvzalloc(struct_size(newmem, regions, mem.nregions),
GFP_KERNEL);
if (!newmem)
@@ -1749,6 +1783,9 @@ int vhost_init_device_iotlb(struct vhost_dev *d, bool enabled)
struct vhost_iotlb *niotlb, *oiotlb;
int i;
+ if (max_iotlb_entries <= 0)
+ return -EINVAL;
+
niotlb = iotlb_alloc();
if (!niotlb)
return -ENOMEM;
@@ -2230,38 +2267,17 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq,
{
struct vring_desc desc;
unsigned int i, head, found = 0;
- u16 last_avail_idx;
- __virtio16 avail_idx;
+ u16 last_avail_idx = vq->last_avail_idx;
__virtio16 ring_head;
int ret, access;
- /* Check it isn't doing very strange things with descriptor numbers. */
- last_avail_idx = vq->last_avail_idx;
-
if (vq->avail_idx == vq->last_avail_idx) {
- if (unlikely(vhost_get_avail_idx(vq, &avail_idx))) {
- vq_err(vq, "Failed to access avail idx at %p\n",
- &vq->avail->idx);
- return -EFAULT;
- }
- vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
-
- if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) {
- vq_err(vq, "Guest moved used index from %u to %u",
- last_avail_idx, vq->avail_idx);
- return -EFAULT;
- }
+ ret = vhost_get_avail_idx(vq);
+ if (unlikely(ret < 0))
+ return ret;
- /* If there's nothing new since last we looked, return
- * invalid.
- */
- if (vq->avail_idx == last_avail_idx)
+ if (!ret)
return vq->num;
-
- /* Only get avail ring entries after they have been
- * exposed by guest.
- */
- smp_rmb();
}
/* Grab the next descriptor number they're advertising, and increment
@@ -2525,35 +2541,21 @@ EXPORT_SYMBOL_GPL(vhost_add_used_and_signal_n);
/* return true if we're sure that avaiable ring is empty */
bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq)
{
- __virtio16 avail_idx;
int r;
if (vq->avail_idx != vq->last_avail_idx)
return false;
- r = vhost_get_avail_idx(vq, &avail_idx);
- if (unlikely(r))
- return false;
-
- vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
- if (vq->avail_idx != vq->last_avail_idx) {
- /* Since we have updated avail_idx, the following
- * call to vhost_get_vq_desc() will read available
- * ring entries. Make sure that read happens after
- * the avail_idx read.
- */
- smp_rmb();
- return false;
- }
+ r = vhost_get_avail_idx(vq);
- return true;
+ /* Note: we treat error as non-empty here */
+ return r == 0;
}
EXPORT_SYMBOL_GPL(vhost_vq_avail_empty);
/* OK, now we need to know about added descriptors. */
bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)
{
- __virtio16 avail_idx;
int r;
if (!(vq->used_flags & VRING_USED_F_NO_NOTIFY))
@@ -2577,25 +2579,13 @@ bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq)
/* They could have slipped one in as we were doing that: make
* sure it's written, then check again. */
smp_mb();
- r = vhost_get_avail_idx(vq, &avail_idx);
- if (r) {
- vq_err(vq, "Failed to check avail idx at %p: %d\n",
- &vq->avail->idx, r);
- return false;
- }
- vq->avail_idx = vhost16_to_cpu(vq, avail_idx);
- if (vq->avail_idx != vq->last_avail_idx) {
- /* Since we have updated avail_idx, the following
- * call to vhost_get_vq_desc() will read available
- * ring entries. Make sure that read happens after
- * the avail_idx read.
- */
- smp_rmb();
- return true;
- }
+ r = vhost_get_avail_idx(vq);
+ /* Note: we treat error as empty here */
+ if (unlikely(r < 0))
+ return false;
- return false;
+ return r;
}
EXPORT_SYMBOL_GPL(vhost_enable_notify);
diff --git a/drivers/video/fbdev/arcfb.c b/drivers/video/fbdev/arcfb.c
index 024d0ee4f04f..69a6cda61ceb 100644
--- a/drivers/video/fbdev/arcfb.c
+++ b/drivers/video/fbdev/arcfb.c
@@ -451,6 +451,9 @@ static ssize_t arcfb_write(struct fb_info *info, const char __user *buf,
struct arcfb_par *par;
unsigned int xres;
+ if (!info->screen_buffer)
+ return -ENODEV;
+
p = *ppos;
par = info->par;
xres = info->var.xres;
diff --git a/drivers/video/fbdev/broadsheetfb.c b/drivers/video/fbdev/broadsheetfb.c
index 49296ea9926c..4c6c42e2d32f 100644
--- a/drivers/video/fbdev/broadsheetfb.c
+++ b/drivers/video/fbdev/broadsheetfb.c
@@ -1013,8 +1013,8 @@ static ssize_t broadsheetfb_write(struct fb_info *info, const char __user *buf,
int err = 0;
unsigned long total_size;
- if (info->state != FBINFO_STATE_RUNNING)
- return -EPERM;
+ if (!info->screen_buffer)
+ return -ENODEV;
total_size = info->fix.smem_len;
diff --git a/drivers/video/fbdev/cobalt_lcdfb.c b/drivers/video/fbdev/cobalt_lcdfb.c
index 5f8b6324d2e8..26dbd1c78195 100644
--- a/drivers/video/fbdev/cobalt_lcdfb.c
+++ b/drivers/video/fbdev/cobalt_lcdfb.c
@@ -129,6 +129,9 @@ static ssize_t cobalt_lcdfb_read(struct fb_info *info, char __user *buf,
unsigned long pos;
int len, retval = 0;
+ if (!info->screen_base)
+ return -ENODEV;
+
pos = *ppos;
if (pos >= LCD_CHARS_MAX || count == 0)
return 0;
@@ -175,6 +178,9 @@ static ssize_t cobalt_lcdfb_write(struct fb_info *info, const char __user *buf,
unsigned long pos;
int len, retval = 0;
+ if (!info->screen_base)
+ return -ENODEV;
+
pos = *ppos;
if (pos >= LCD_CHARS_MAX || count == 0)
return 0;
diff --git a/drivers/video/fbdev/core/fb_sys_fops.c b/drivers/video/fbdev/core/fb_sys_fops.c
index ff275d7f3eaf..432fa3fa7fa1 100644
--- a/drivers/video/fbdev/core/fb_sys_fops.c
+++ b/drivers/video/fbdev/core/fb_sys_fops.c
@@ -21,8 +21,8 @@ ssize_t fb_sys_read(struct fb_info *info, char __user *buf, size_t count,
int err = 0;
unsigned long total_size;
- if (info->state != FBINFO_STATE_RUNNING)
- return -EPERM;
+ if (!info->screen_buffer)
+ return -ENODEV;
total_size = info->screen_size;
@@ -61,8 +61,8 @@ ssize_t fb_sys_write(struct fb_info *info, const char __user *buf,
int err = 0;
unsigned long total_size;
- if (info->state != FBINFO_STATE_RUNNING)
- return -EPERM;
+ if (!info->screen_buffer)
+ return -ENODEV;
total_size = info->screen_size;
diff --git a/drivers/video/fbdev/core/fbcon.c b/drivers/video/fbdev/core/fbcon.c
index 1c807473a0a0..a0a9b0b93318 100644
--- a/drivers/video/fbdev/core/fbcon.c
+++ b/drivers/video/fbdev/core/fbcon.c
@@ -2774,7 +2774,6 @@ void fbcon_update_vcs(struct fb_info *info, bool all)
else
fbcon_modechanged(info);
}
-EXPORT_SYMBOL(fbcon_update_vcs);
/* let fbcon check if it supports a new screen resolution */
int fbcon_modechange_possible(struct fb_info *info, struct fb_var_screeninfo *var)
@@ -2802,7 +2801,6 @@ int fbcon_modechange_possible(struct fb_info *info, struct fb_var_screeninfo *va
return 0;
}
-EXPORT_SYMBOL_GPL(fbcon_modechange_possible);
int fbcon_mode_deleted(struct fb_info *info,
struct fb_videomode *mode)
diff --git a/drivers/video/fbdev/core/fbmem.c b/drivers/video/fbdev/core/fbmem.c
index bd0be854b116..9b456163db89 100644
--- a/drivers/video/fbdev/core/fbmem.c
+++ b/drivers/video/fbdev/core/fbmem.c
@@ -768,7 +768,7 @@ fb_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
int c, cnt = 0, err = 0;
unsigned long total_size;
- if (!info || ! info->screen_base)
+ if (!info)
return -ENODEV;
if (info->state != FBINFO_STATE_RUNNING)
@@ -777,6 +777,9 @@ fb_read(struct file *file, char __user *buf, size_t count, loff_t *ppos)
if (info->fbops->fb_read)
return info->fbops->fb_read(info, buf, count, ppos);
+ if (!info->screen_base)
+ return -ENODEV;
+
total_size = info->screen_size;
if (total_size == 0)
@@ -833,7 +836,7 @@ fb_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos)
int c, cnt = 0, err = 0;
unsigned long total_size;
- if (!info || !info->screen_base)
+ if (!info)
return -ENODEV;
if (info->state != FBINFO_STATE_RUNNING)
@@ -842,6 +845,9 @@ fb_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos)
if (info->fbops->fb_write)
return info->fbops->fb_write(info, buf, count, ppos);
+ if (!info->screen_base)
+ return -ENODEV;
+
total_size = info->screen_size;
if (total_size == 0)
@@ -1074,6 +1080,19 @@ fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var)
}
EXPORT_SYMBOL(fb_set_var);
+int fb_set_var_from_user(struct fb_info *info, struct fb_var_screeninfo *var)
+{
+ int ret = fbcon_modechange_possible(info, var);
+
+ if (!ret)
+ ret = fb_set_var(info, var);
+ if (!ret)
+ fbcon_update_vcs(info, var->activate & FB_ACTIVATE_ALL);
+
+ return ret;
+}
+EXPORT_SYMBOL(fb_set_var_from_user);
+
int
fb_blank(struct fb_info *info, int blank)
{
@@ -1122,11 +1141,7 @@ static long do_fb_ioctl(struct fb_info *info, unsigned int cmd,
var.activate &= ~FB_ACTIVATE_KD_TEXT;
console_lock();
lock_fb_info(info);
- ret = fbcon_modechange_possible(info, &var);
- if (!ret)
- ret = fb_set_var(info, &var);
- if (!ret)
- fbcon_update_vcs(info, var.activate & FB_ACTIVATE_ALL);
+ ret = fb_set_var_from_user(info, &var);
unlock_fb_info(info);
console_unlock();
if (!ret && copy_to_user(argp, &var, sizeof(var)))
diff --git a/drivers/video/fbdev/core/fbsysfs.c b/drivers/video/fbdev/core/fbsysfs.c
index 4d7f63892dcc..5ced9e5510c5 100644
--- a/drivers/video/fbdev/core/fbsysfs.c
+++ b/drivers/video/fbdev/core/fbsysfs.c
@@ -100,9 +100,7 @@ static int activate(struct fb_info *fb_info, struct fb_var_screeninfo *var)
var->activate |= FB_ACTIVATE_FORCE;
console_lock();
lock_fb_info(fb_info);
- err = fb_set_var(fb_info, var);
- if (!err)
- fbcon_update_vcs(fb_info, var->activate & FB_ACTIVATE_ALL);
+ err = fb_set_var_from_user(fb_info, var);
unlock_fb_info(fb_info);
console_unlock();
if (err)
@@ -194,8 +192,15 @@ static ssize_t store_modes(struct device *device,
if (fb_new_modelist(fb_info)) {
fb_destroy_modelist(&fb_info->modelist);
list_splice(&old_list, &fb_info->modelist);
- } else
+ } else {
+ /*
+ * fb_display[i].mode and fb_info->mode both point into the old
+ * list. Clear them before it is freed.
+ */
+ fbcon_delete_modelist(&old_list);
+ fb_info->mode = NULL;
fb_destroy_modelist(&old_list);
+ }
unlock_fb_info(fb_info);
console_unlock();
diff --git a/drivers/video/fbdev/hecubafb.c b/drivers/video/fbdev/hecubafb.c
index 962a155a5b3d..98e224a9641c 100644
--- a/drivers/video/fbdev/hecubafb.c
+++ b/drivers/video/fbdev/hecubafb.c
@@ -163,8 +163,8 @@ static ssize_t hecubafb_write(struct fb_info *info, const char __user *buf,
int err = 0;
unsigned long total_size;
- if (info->state != FBINFO_STATE_RUNNING)
- return -EPERM;
+ if (!info->screen_buffer)
+ return -ENODEV;
total_size = info->fix.smem_len;
diff --git a/drivers/video/fbdev/metronomefb.c b/drivers/video/fbdev/metronomefb.c
index dcfbcb7523ad..eba185b58f0c 100644
--- a/drivers/video/fbdev/metronomefb.c
+++ b/drivers/video/fbdev/metronomefb.c
@@ -523,8 +523,8 @@ static ssize_t metronomefb_write(struct fb_info *info, const char __user *buf,
int err = 0;
unsigned long total_size;
- if (info->state != FBINFO_STATE_RUNNING)
- return -EPERM;
+ if (!info->screen_buffer)
+ return -ENODEV;
total_size = info->fix.smem_len;
diff --git a/drivers/video/fbdev/omap2/omapfb/omapfb-main.c b/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
index 5ccddcfce722..7e6e177ad2af 100644
--- a/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
+++ b/drivers/video/fbdev/omap2/omapfb/omapfb-main.c
@@ -1097,7 +1097,11 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
rg = omapfb_get_mem_region(ofbi->region);
- start = omapfb_get_region_paddr(ofbi);
+ if (ofbi->rotation_type == OMAP_DSS_ROT_VRFB)
+ start = rg->vrfb.paddr[0];
+ else
+ start = rg->paddr;
+
len = fix->smem_len;
DBG("user mmap region start %lx, len %d, off %lx\n", start, len,
@@ -1107,6 +1111,8 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
vma->vm_ops = &mmap_user_ops;
vma->vm_private_data = rg;
+ atomic_inc(&rg->map_count);
+
r = vm_iomap_memory(vma, start, len);
if (r)
goto error;
@@ -1119,7 +1125,8 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma)
return 0;
error:
- omapfb_put_mem_region(ofbi->region);
+ atomic_dec(&rg->map_count);
+ omapfb_put_mem_region(rg);
return r;
}
diff --git a/drivers/video/fbdev/ps3fb.c b/drivers/video/fbdev/ps3fb.c
index 2fe08b67eda7..ae93a23bfce9 100644
--- a/drivers/video/fbdev/ps3fb.c
+++ b/drivers/video/fbdev/ps3fb.c
@@ -29,7 +29,6 @@
#include <linux/freezer.h>
#include <linux/uaccess.h>
#include <linux/fb.h>
-#include <linux/fbcon.h>
#include <linux/init.h>
#include <asm/cell-regs.h>
@@ -828,9 +827,7 @@ static int ps3fb_ioctl(struct fb_info *info, unsigned int cmd,
/* Force, in case only special bits changed */
var.activate |= FB_ACTIVATE_FORCE;
par->new_mode_id = val;
- retval = fb_set_var(info, &var);
- if (!retval)
- fbcon_update_vcs(info, var.activate & FB_ACTIVATE_ALL);
+ retval = fb_set_var_from_user(info, &var);
console_unlock();
}
break;
diff --git a/drivers/video/fbdev/pvr2fb.c b/drivers/video/fbdev/pvr2fb.c
index 6e14884b4f8d..6c03dd356d20 100644
--- a/drivers/video/fbdev/pvr2fb.c
+++ b/drivers/video/fbdev/pvr2fb.c
@@ -647,6 +647,9 @@ static ssize_t pvr2fb_write(struct fb_info *info, const char __user *buf,
struct page **pages;
int ret, i;
+ if (!info->screen_base)
+ return -ENODEV;
+
nr_pages = (count + PAGE_SIZE - 1) >> PAGE_SHIFT;
pages = kmalloc_array(nr_pages, sizeof(struct page *), GFP_KERNEL);
diff --git a/drivers/video/fbdev/sh_mobile_lcdcfb.c b/drivers/video/fbdev/sh_mobile_lcdcfb.c
index 444c3ca9d4d4..6ce0fc3e706f 100644
--- a/drivers/video/fbdev/sh_mobile_lcdcfb.c
+++ b/drivers/video/fbdev/sh_mobile_lcdcfb.c
@@ -15,7 +15,6 @@
#include <linux/ctype.h>
#include <linux/dma-mapping.h>
#include <linux/delay.h>
-#include <linux/fbcon.h>
#include <linux/init.h>
#include <linux/interrupt.h>
#include <linux/ioctl.h>
@@ -1770,11 +1769,9 @@ static void sh_mobile_fb_reconfig(struct fb_info *info)
var.height = ch->display.height;
var.activate = FB_ACTIVATE_NOW;
- if (fb_set_var(info, &var) < 0)
+ if (fb_set_var_from_user(info, &var) < 0)
/* Couldn't reconfigure, hopefully, can continue as before */
return;
-
- fbcon_update_vcs(info, true);
}
/*
diff --git a/drivers/video/fbdev/sm712fb.c b/drivers/video/fbdev/sm712fb.c
index 3baf33635e65..88f898580a68 100644
--- a/drivers/video/fbdev/sm712fb.c
+++ b/drivers/video/fbdev/sm712fb.c
@@ -1028,12 +1028,9 @@ static ssize_t smtcfb_read(struct fb_info *info, char __user *buf,
int c, i, cnt = 0, err = 0;
unsigned long total_size;
- if (!info || !info->screen_base)
+ if (!info->screen_base)
return -ENODEV;
- if (info->state != FBINFO_STATE_RUNNING)
- return -EPERM;
-
total_size = info->screen_size;
if (total_size == 0)
@@ -1094,12 +1091,9 @@ static ssize_t smtcfb_write(struct fb_info *info, const char __user *buf,
int c, i, cnt = 0, err = 0;
unsigned long total_size;
- if (!info || !info->screen_base)
+ if (!info->screen_base)
return -ENODEV;
- if (info->state != FBINFO_STATE_RUNNING)
- return -EPERM;
-
total_size = info->screen_size;
if (total_size == 0)
diff --git a/drivers/video/fbdev/ssd1307fb.c b/drivers/video/fbdev/ssd1307fb.c
index b9fefa844574..61f84abb5e0c 100644
--- a/drivers/video/fbdev/ssd1307fb.c
+++ b/drivers/video/fbdev/ssd1307fb.c
@@ -14,6 +14,7 @@
#include <linux/module.h>
#include <linux/property.h>
#include <linux/pwm.h>
+#include <linux/spinlock.h>
#include <linux/uaccess.h>
#include <linux/regulator/consumer.h>
@@ -72,6 +73,13 @@ struct ssd1307fb_par {
struct i2c_client *client;
u32 height;
struct fb_info *info;
+ /* Pending damage, with exclusive x2/y2, protected by damage_lock. */
+ spinlock_t damage_lock;
+ bool damage_pending;
+ u32 damage_x1;
+ u32 damage_x2;
+ u32 damage_y1;
+ u32 damage_y2;
u8 lookup_table[4];
u32 page_offset;
u32 col_offset;
@@ -292,40 +300,6 @@ static int ssd1307fb_update_display(struct ssd1307fb_par *par)
return ssd1307fb_update_rect(par, 0, 0, par->width, par->height);
}
-static ssize_t ssd1307fb_write(struct fb_info *info, const char __user *buf,
- size_t count, loff_t *ppos)
-{
- struct ssd1307fb_par *par = info->par;
- unsigned long total_size;
- unsigned long p = *ppos;
- void *dst;
- int ret;
-
- total_size = info->fix.smem_len;
-
- if (p > total_size)
- return -EINVAL;
-
- if (count + p > total_size)
- count = total_size - p;
-
- if (!count)
- return -EINVAL;
-
- dst = info->screen_buffer + p;
-
- if (copy_from_user(dst, buf, count))
- return -EFAULT;
-
- ret = ssd1307fb_update_display(par);
- if (ret < 0)
- return ret;
-
- *ppos += count;
-
- return count;
-}
-
static int ssd1307fb_blank(int blank_mode, struct fb_info *info)
{
struct ssd1307fb_par *par = info->par;
@@ -336,44 +310,87 @@ static int ssd1307fb_blank(int blank_mode, struct fb_info *info)
return ssd1307fb_write_cmd(par->client, SSD1307FB_DISPLAY_ON);
}
-static void ssd1307fb_fillrect(struct fb_info *info, const struct fb_fillrect *rect)
+static void ssd1307fb_schedule_damage(struct fb_info *info, u32 x, u32 y,
+ u32 width, u32 height)
{
struct ssd1307fb_par *par = info->par;
- sys_fillrect(info, rect);
- ssd1307fb_update_rect(par, rect->dx, rect->dy, rect->width,
- rect->height);
+ unsigned long flags;
+ u32 x2, y2;
+
+ if (!width || !height || x >= par->width || y >= par->height)
+ return;
+
+ x2 = x + min(width, par->width - x);
+ y2 = y + min(height, par->height - y);
+
+ spin_lock_irqsave(&par->damage_lock, flags);
+ if (par->damage_pending) {
+ par->damage_x1 = min(par->damage_x1, x);
+ par->damage_y1 = min(par->damage_y1, y);
+ par->damage_x2 = max(par->damage_x2, x2);
+ par->damage_y2 = max(par->damage_y2, y2);
+ } else {
+ par->damage_x1 = x;
+ par->damage_y1 = y;
+ par->damage_x2 = x2;
+ par->damage_y2 = y2;
+ par->damage_pending = true;
+ }
+ spin_unlock_irqrestore(&par->damage_lock, flags);
+
+ /* Advance an already-pending mmap update as well. */
+ mod_delayed_work(system_wq, &info->deferred_work, 0);
}
-static void ssd1307fb_copyarea(struct fb_info *info, const struct fb_copyarea *area)
+static void ssd1307fb_defio_damage_range(struct fb_info *info, off_t off, size_t len)
{
struct ssd1307fb_par *par = info->par;
- sys_copyarea(info, area);
- ssd1307fb_update_rect(par, area->dx, area->dy, area->width,
- area->height);
+
+ ssd1307fb_schedule_damage(info, 0, 0, par->width, par->height);
}
-static void ssd1307fb_imageblit(struct fb_info *info, const struct fb_image *image)
+static void ssd1307fb_defio_damage_area(struct fb_info *info, u32 x, u32 y,
+ u32 width, u32 height)
{
- struct ssd1307fb_par *par = info->par;
- sys_imageblit(info, image);
- ssd1307fb_update_rect(par, image->dx, image->dy, image->width,
- image->height);
+ ssd1307fb_schedule_damage(info, x, y, width, height);
}
+FB_GEN_DEFAULT_DEFERRED_SYS_OPS(ssd1307fb,
+ ssd1307fb_defio_damage_range,
+ ssd1307fb_defio_damage_area)
+
static const struct fb_ops ssd1307fb_ops = {
.owner = THIS_MODULE,
- .fb_read = fb_sys_read,
- .fb_write = ssd1307fb_write,
+ FB_DEFAULT_DEFERRED_OPS(ssd1307fb),
.fb_blank = ssd1307fb_blank,
- .fb_fillrect = ssd1307fb_fillrect,
- .fb_copyarea = ssd1307fb_copyarea,
- .fb_imageblit = ssd1307fb_imageblit,
- .fb_mmap = fb_deferred_io_mmap,
};
static void ssd1307fb_deferred_io(struct fb_info *info, struct list_head *pagereflist)
{
- ssd1307fb_update_display(info->par);
+ struct ssd1307fb_par *par = info->par;
+ unsigned long flags;
+ u32 x, y, width, height;
+
+ spin_lock_irqsave(&par->damage_lock, flags);
+ if (!list_empty(pagereflist)) {
+ x = 0;
+ y = 0;
+ width = par->width;
+ height = par->height;
+ par->damage_pending = false;
+ } else if (par->damage_pending) {
+ x = par->damage_x1;
+ y = par->damage_y1;
+ width = par->damage_x2 - par->damage_x1;
+ height = par->damage_y2 - par->damage_y1;
+ par->damage_pending = false;
+ } else {
+ spin_unlock_irqrestore(&par->damage_lock, flags);
+ return;
+ }
+ spin_unlock_irqrestore(&par->damage_lock, flags);
+
+ ssd1307fb_update_rect(par, x, y, width, height);
}
static int ssd1307fb_init(struct ssd1307fb_par *par)
@@ -653,6 +670,7 @@ static int ssd1307fb_probe(struct i2c_client *client)
par = info->par;
par->info = info;
par->client = client;
+ spin_lock_init(&par->damage_lock);
par->device_info = device_get_match_data(dev);
diff --git a/drivers/virt/acrn/irqfd.c b/drivers/virt/acrn/irqfd.c
index d4ad211dce7a..e6280a100572 100644
--- a/drivers/virt/acrn/irqfd.c
+++ b/drivers/virt/acrn/irqfd.c
@@ -18,6 +18,9 @@
static LIST_HEAD(acrn_irqfd_clients);
+/* Cleanup work has been queued; set via test_and_set_bit(). */
+#define HSM_IRQFD_FLAG_SHUTDOWN 0
+
/**
* struct hsm_irqfd - Properties of HSM irqfd
* @vm: Associated VM pointer
@@ -27,6 +30,7 @@ static LIST_HEAD(acrn_irqfd_clients);
* @list: Entry within &acrn_vm.irqfds of irqfds of a VM
* @pt: Structure for select/poll on the associated eventfd
* @msi: MSI data
+ * @flags: Internal lifecycle flags (HSM_IRQFD_FLAG_*)
*/
struct hsm_irqfd {
struct acrn_vm *vm;
@@ -36,6 +40,7 @@ struct hsm_irqfd {
struct list_head list;
poll_table pt;
struct acrn_msi_entry msi;
+ unsigned long flags;
};
static void acrn_irqfd_inject(struct hsm_irqfd *irqfd)
@@ -46,30 +51,29 @@ static void acrn_irqfd_inject(struct hsm_irqfd *irqfd)
irqfd->msi.msi_data);
}
-static void hsm_irqfd_shutdown(struct hsm_irqfd *irqfd)
+/* Queue the cleanup work at most once. Safe from atomic context. */
+static void hsm_irqfd_queue_shutdown(struct hsm_irqfd *irqfd)
{
- u64 cnt;
-
- lockdep_assert_held(&irqfd->vm->irqfds_lock);
-
- /* remove from wait queue */
- list_del_init(&irqfd->list);
- eventfd_ctx_remove_wait_queue(irqfd->eventfd, &irqfd->wait, &cnt);
- eventfd_ctx_put(irqfd->eventfd);
- kfree(irqfd);
+ if (!test_and_set_bit(HSM_IRQFD_FLAG_SHUTDOWN, &irqfd->flags))
+ queue_work(irqfd->vm->irqfd_wq, &irqfd->shutdown);
}
+/* Sole owner of @irqfd: unhook waitqueue, drop eventfd ref, free. */
static void hsm_irqfd_shutdown_work(struct work_struct *work)
{
- struct hsm_irqfd *irqfd;
- struct acrn_vm *vm;
+ struct hsm_irqfd *irqfd = container_of(work, struct hsm_irqfd,
+ shutdown);
+ struct acrn_vm *vm = irqfd->vm;
+ u64 cnt;
- irqfd = container_of(work, struct hsm_irqfd, shutdown);
- vm = irqfd->vm;
mutex_lock(&vm->irqfds_lock);
if (!list_empty(&irqfd->list))
- hsm_irqfd_shutdown(irqfd);
+ list_del_init(&irqfd->list);
mutex_unlock(&vm->irqfds_lock);
+
+ eventfd_ctx_remove_wait_queue(irqfd->eventfd, &irqfd->wait, &cnt);
+ eventfd_ctx_put(irqfd->eventfd);
+ kfree(irqfd);
}
/* Called with wqh->lock held and interrupts disabled */
@@ -78,17 +82,16 @@ static int hsm_irqfd_wakeup(wait_queue_entry_t *wait, unsigned int mode,
{
unsigned long poll_bits = (unsigned long)key;
struct hsm_irqfd *irqfd;
- struct acrn_vm *vm;
irqfd = container_of(wait, struct hsm_irqfd, wait);
- vm = irqfd->vm;
+
if (poll_bits & POLLIN)
/* An event has been signaled, inject an interrupt */
acrn_irqfd_inject(irqfd);
if (poll_bits & POLLHUP)
- /* Do shutdown work in thread to hold wqh->lock */
- queue_work(vm->irqfd_wq, &irqfd->shutdown);
+ /* Defer teardown to the cleanup work; can't sleep here. */
+ hsm_irqfd_queue_shutdown(irqfd);
return 0;
}
@@ -145,6 +148,12 @@ static int acrn_irqfd_assign(struct acrn_vm *vm, struct acrn_irqfd *args)
init_waitqueue_func_entry(&irqfd->wait, hsm_irqfd_wakeup);
init_poll_funcptr(&irqfd->pt, hsm_irqfd_poll_func);
+ /*
+ * Hold irqfds_lock across waitqueue install and list_add so the
+ * irqfd is not visible to deassign/deinit before its waitqueue
+ * entry is in place, and any racing EPOLLHUP cleanup work blocks
+ * on irqfds_lock until publication completes.
+ */
mutex_lock(&vm->irqfds_lock);
list_for_each_entry(tmp, &vm->irqfds, list) {
if (irqfd->eventfd != tmp->eventfd)
@@ -153,14 +162,12 @@ static int acrn_irqfd_assign(struct acrn_vm *vm, struct acrn_irqfd *args)
mutex_unlock(&vm->irqfds_lock);
goto fail;
}
- list_add_tail(&irqfd->list, &vm->irqfds);
- mutex_unlock(&vm->irqfds_lock);
- /* Check the pending event in this stage */
events = vfs_poll(f.file, &irqfd->pt);
-
+ list_add_tail(&irqfd->list, &vm->irqfds);
if (events & EPOLLIN)
acrn_irqfd_inject(irqfd);
+ mutex_unlock(&vm->irqfds_lock);
fdput(f);
return 0;
@@ -187,13 +194,17 @@ static int acrn_irqfd_deassign(struct acrn_vm *vm,
mutex_lock(&vm->irqfds_lock);
list_for_each_entry_safe(irqfd, tmp, &vm->irqfds, list) {
if (irqfd->eventfd == eventfd) {
- hsm_irqfd_shutdown(irqfd);
+ list_del_init(&irqfd->list);
+ hsm_irqfd_queue_shutdown(irqfd);
break;
}
}
mutex_unlock(&vm->irqfds_lock);
eventfd_ctx_put(eventfd);
+ /* Wait for cleanup work to finish so the eventfd is fully detached. */
+ flush_workqueue(vm->irqfd_wq);
+
return 0;
}
@@ -226,9 +237,15 @@ void acrn_irqfd_deinit(struct acrn_vm *vm)
struct hsm_irqfd *irqfd, *next;
dev_dbg(acrn_dev.this_device, "VM %u irqfd deinit.\n", vm->vmid);
- destroy_workqueue(vm->irqfd_wq);
+
mutex_lock(&vm->irqfds_lock);
- list_for_each_entry_safe(irqfd, next, &vm->irqfds, list)
- hsm_irqfd_shutdown(irqfd);
+ list_for_each_entry_safe(irqfd, next, &vm->irqfds, list) {
+ list_del_init(&irqfd->list);
+ hsm_irqfd_queue_shutdown(irqfd);
+ }
mutex_unlock(&vm->irqfds_lock);
+
+ /* Drain all cleanup work before tearing the workqueue down. */
+ flush_workqueue(vm->irqfd_wq);
+ destroy_workqueue(vm->irqfd_wq);
}
diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c
index 241b94f62e56..2cd7cd177795 100644
--- a/drivers/virt/nitro_enclaves/ne_misc_dev.c
+++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c
@@ -1710,6 +1710,7 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 __user *slot_ui
return enclave_fd;
put_file:
+ enclave_file->private_data = NULL;
fput(enclave_file);
put_fd:
put_unused_fd(enclave_fd);
diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c
index 1ef094427f29..0d3481250fb9 100644
--- a/drivers/virtio/virtio.c
+++ b/drivers/virtio/virtio.c
@@ -340,6 +340,34 @@ static void virtio_dev_remove(struct device *_d)
of_node_put(dev->dev.of_node);
}
+static void virtio_dev_shutdown(struct device *_d)
+{
+ struct virtio_device *dev = dev_to_virtio(_d);
+ struct virtio_driver *drv = drv_to_virtio(dev->dev.driver);
+
+ /*
+ * Stop accesses to or from the device.
+ * We only need to do it if there's a driver - no accesses otherwise.
+ */
+ if (!drv)
+ return;
+
+ /*
+ * Some devices get wedged if you kick them after they are
+ * reset. Mark all vqs as broken to make sure we don't.
+ */
+ virtio_break_device(dev);
+ /*
+ * Guarantee that any callback will see vq->broken as true.
+ */
+ virtio_synchronize_cbs(dev);
+ /*
+ * As IOMMUs are reset on shutdown, this will block device access to memory.
+ * Some devices get wedged if this happens, so reset to make sure it does not.
+ */
+ dev->config->reset(dev);
+}
+
static struct bus_type virtio_bus = {
.name = "virtio",
.match = virtio_dev_match,
@@ -347,6 +375,7 @@ static struct bus_type virtio_bus = {
.uevent = virtio_uevent,
.probe = virtio_dev_probe,
.remove = virtio_dev_remove,
+ .shutdown = virtio_dev_shutdown,
};
int register_virtio_driver(struct virtio_driver *driver)
diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c
index 7651940aafd2..6395b8fba005 100644
--- a/drivers/virtio/virtio_mmio.c
+++ b/drivers/virtio/virtio_mmio.c
@@ -64,7 +64,6 @@
#include <linux/platform_device.h>
#include <linux/pm.h>
#include <linux/slab.h>
-#include <linux/spinlock.h>
#include <linux/virtio.h>
#include <linux/virtio_config.h>
#include <uapi/linux/virtio_mmio.h>
@@ -88,21 +87,10 @@ struct virtio_mmio_device {
void __iomem *base;
unsigned long version;
- /* a list of queues so we can dispatch IRQs */
- spinlock_t lock;
- struct list_head virtqueues;
+ /* True if enable_irq_wake() succeeded for the shared IRQ. */
+ bool wake_irq_enabled;
};
-struct virtio_mmio_vq_info {
- /* the actual virtqueue */
- struct virtqueue *vq;
-
- /* the list node for the virtqueues list */
- struct list_head node;
-};
-
-
-
/* Configuration interface */
static u64 vm_get_features(struct virtio_device *vdev)
@@ -289,9 +277,8 @@ static bool vm_notify(struct virtqueue *vq)
static irqreturn_t vm_interrupt(int irq, void *opaque)
{
struct virtio_mmio_device *vm_dev = opaque;
- struct virtio_mmio_vq_info *info;
+ struct virtqueue *vq;
unsigned long status;
- unsigned long flags;
irqreturn_t ret = IRQ_NONE;
/* Read and acknowledge interrupts */
@@ -304,10 +291,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque)
}
if (likely(status & VIRTIO_MMIO_INT_VRING)) {
- spin_lock_irqsave(&vm_dev->lock, flags);
- list_for_each_entry(info, &vm_dev->virtqueues, node)
- ret |= vring_interrupt(irq, info->vq);
- spin_unlock_irqrestore(&vm_dev->lock, flags);
+ virtio_device_for_each_vq(&vm_dev->vdev, vq)
+ ret |= vring_interrupt(irq, vq);
}
return ret;
@@ -318,14 +303,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque)
static void vm_del_vq(struct virtqueue *vq)
{
struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vq->vdev);
- struct virtio_mmio_vq_info *info = vq->priv;
- unsigned long flags;
unsigned int index = vq->index;
- spin_lock_irqsave(&vm_dev->lock, flags);
- list_del(&info->node);
- spin_unlock_irqrestore(&vm_dev->lock, flags);
-
/* Select and deactivate the queue */
writel(index, vm_dev->base + VIRTIO_MMIO_QUEUE_SEL);
if (vm_dev->version == 1) {
@@ -336,19 +315,23 @@ static void vm_del_vq(struct virtqueue *vq)
}
vring_del_virtqueue(vq);
-
- kfree(info);
}
static void vm_del_vqs(struct virtio_device *vdev)
{
struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev);
struct virtqueue *vq, *n;
+ int irq = platform_get_irq(vm_dev->pdev, 0);
list_for_each_entry_safe(vq, n, &vdev->vqs, list)
vm_del_vq(vq);
- free_irq(platform_get_irq(vm_dev->pdev, 0), vm_dev);
+ if (vm_dev->wake_irq_enabled) {
+ disable_irq_wake(irq);
+ vm_dev->wake_irq_enabled = false;
+ }
+
+ free_irq(irq, vm_dev);
}
static void vm_synchronize_cbs(struct virtio_device *vdev)
@@ -363,9 +346,7 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in
const char *name, bool ctx)
{
struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev);
- struct virtio_mmio_vq_info *info;
struct virtqueue *vq;
- unsigned long flags;
unsigned int num;
int err;
@@ -382,13 +363,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in
goto error_available;
}
- /* Allocate and fill out our active queue description */
- info = kmalloc(sizeof(*info), GFP_KERNEL);
- if (!info) {
- err = -ENOMEM;
- goto error_kmalloc;
- }
-
num = readl(vm_dev->base + VIRTIO_MMIO_QUEUE_NUM_MAX);
if (num == 0) {
err = -ENOENT;
@@ -446,13 +420,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in
writel(1, vm_dev->base + VIRTIO_MMIO_QUEUE_READY);
}
- vq->priv = info;
- info->vq = vq;
-
- spin_lock_irqsave(&vm_dev->lock, flags);
- list_add(&info->node, &vm_dev->virtqueues);
- spin_unlock_irqrestore(&vm_dev->lock, flags);
-
return vq;
error_bad_pfn:
@@ -464,8 +431,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in
writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_READY);
WARN_ON(readl(vm_dev->base + VIRTIO_MMIO_QUEUE_READY));
}
- kfree(info);
-error_kmalloc:
error_available:
return ERR_PTR(err);
}
@@ -489,8 +454,9 @@ static int vm_find_vqs(struct virtio_device *vdev, unsigned int nvqs,
if (err)
return err;
- if (of_property_read_bool(vm_dev->pdev->dev.of_node, "wakeup-source"))
- enable_irq_wake(irq);
+ if (of_property_read_bool(vm_dev->pdev->dev.of_node, "wakeup-source") &&
+ !enable_irq_wake(irq))
+ vm_dev->wake_irq_enabled = true;
for (i = 0; i < nvqs; ++i) {
if (!names[i]) {
@@ -610,8 +576,6 @@ static int virtio_mmio_probe(struct platform_device *pdev)
vm_dev->vdev.dev.release = virtio_mmio_release_dev;
vm_dev->vdev.config = &virtio_mmio_config_ops;
vm_dev->pdev = pdev;
- INIT_LIST_HEAD(&vm_dev->virtqueues);
- spin_lock_init(&vm_dev->lock);
vm_dev->base = devm_platform_ioremap_resource(pdev, 0);
if (IS_ERR(vm_dev->base)) {
diff --git a/drivers/watchdog/at91sam9_wdt.c b/drivers/watchdog/at91sam9_wdt.c
index fed7be246442..4e5171f4d006 100644
--- a/drivers/watchdog/at91sam9_wdt.c
+++ b/drivers/watchdog/at91sam9_wdt.c
@@ -242,7 +242,7 @@ static int at91_wdt_init(struct platform_device *pdev, struct at91wdt *wdt)
return 0;
out_stop_timer:
- del_timer(&wdt->timer);
+ timer_shutdown_sync(&wdt->timer);
return err;
}
@@ -387,7 +387,7 @@ static int __exit at91wdt_remove(struct platform_device *pdev)
watchdog_unregister_device(&wdt->wdd);
pr_warn("I quit now, hardware will probably reboot!\n");
- del_timer(&wdt->timer);
+ timer_shutdown_sync(&wdt->timer);
clk_disable_unprepare(wdt->sclk);
return 0;
diff --git a/drivers/watchdog/cpu5wdt.c b/drivers/watchdog/cpu5wdt.c
index 688b112e712b..9f279c0e13a6 100644
--- a/drivers/watchdog/cpu5wdt.c
+++ b/drivers/watchdog/cpu5wdt.c
@@ -252,7 +252,7 @@ static void cpu5wdt_exit(void)
if (cpu5wdt_device.queue) {
cpu5wdt_device.queue = 0;
wait_for_completion(&cpu5wdt_device.stop);
- del_timer(&cpu5wdt_device.timer);
+ timer_shutdown_sync(&cpu5wdt_device.timer);
}
misc_deregister(&cpu5wdt_misc);
diff --git a/drivers/watchdog/rtd119x_wdt.c b/drivers/watchdog/rtd119x_wdt.c
index 834b94ff3f90..e963be4df8c4 100644
--- a/drivers/watchdog/rtd119x_wdt.c
+++ b/drivers/watchdog/rtd119x_wdt.c
@@ -94,16 +94,11 @@ static const struct of_device_id rtd119x_wdt_dt_ids[] = {
{ }
};
-static void rtd119x_clk_disable_unprepare(void *data)
-{
- clk_disable_unprepare(data);
-}
-
static int rtd119x_wdt_probe(struct platform_device *pdev)
{
struct device *dev = &pdev->dev;
struct rtd119x_watchdog_device *data;
- int ret;
+ unsigned long rate;
data = devm_kzalloc(dev, sizeof(*data), GFP_KERNEL);
if (!data)
@@ -113,22 +108,18 @@ static int rtd119x_wdt_probe(struct platform_device *pdev)
if (IS_ERR(data->base))
return PTR_ERR(data->base);
- data->clk = devm_clk_get(dev, NULL);
+ data->clk = devm_clk_get_enabled(dev, NULL);
if (IS_ERR(data->clk))
return PTR_ERR(data->clk);
- ret = clk_prepare_enable(data->clk);
- if (ret)
- return ret;
- ret = devm_add_action_or_reset(dev, rtd119x_clk_disable_unprepare,
- data->clk);
- if (ret)
- return ret;
+ rate = clk_get_rate(data->clk);
+ if (!rate)
+ return -EINVAL;
data->wdt_dev.info = &rtd119x_wdt_info;
data->wdt_dev.ops = &rtd119x_wdt_ops;
data->wdt_dev.timeout = 120;
- data->wdt_dev.max_timeout = 0xffffffff / clk_get_rate(data->clk);
+ data->wdt_dev.max_timeout = 0xffffffff / rate;
data->wdt_dev.min_timeout = 1;
data->wdt_dev.parent = dev;
diff --git a/drivers/xen/events/events_2l.c b/drivers/xen/events/events_2l.c
index b8f2f971c2f0..e3585330cf98 100644
--- a/drivers/xen/events/events_2l.c
+++ b/drivers/xen/events/events_2l.c
@@ -171,11 +171,11 @@ static void evtchn_2l_handle_events(unsigned cpu, struct evtchn_loop_ctrl *ctrl)
int i;
struct shared_info *s = HYPERVISOR_shared_info;
struct vcpu_info *vcpu_info = __this_cpu_read(xen_vcpu);
+ evtchn_port_t evtchn;
/* Timer interrupt has highest priority. */
- irq = irq_from_virq(cpu, VIRQ_TIMER);
+ irq = irq_evtchn_from_virq(cpu, VIRQ_TIMER, &evtchn);
if (irq != -1) {
- evtchn_port_t evtchn = evtchn_from_irq(irq);
word_idx = evtchn / BITS_PER_LONG;
bit_idx = evtchn % BITS_PER_LONG;
if (active_evtchns(cpu, s, word_idx) & (1ULL << bit_idx))
@@ -328,9 +328,9 @@ irqreturn_t xen_debug_interrupt(int irq, void *dev_id)
for (i = 0; i < EVTCHN_2L_NR_CHANNELS; i++) {
if (sync_test_bit(i, BM(sh->evtchn_pending))) {
int word_idx = i / BITS_PER_EVTCHN_WORD;
- printk(" %d: event %d -> irq %d%s%s%s\n",
+ printk(" %d: event %d -> irq %u%s%s%s\n",
cpu_from_evtchn(i), i,
- get_evtchn_to_irq(i),
+ irq_from_evtchn(i),
sync_test_bit(word_idx, BM(&v->evtchn_pending_sel))
? "" : " l2-clear",
!sync_test_bit(i, BM(sh->evtchn_mask))
diff --git a/drivers/xen/events/events_base.c b/drivers/xen/events/events_base.c
index 9052a4873bca..8fd9d173a9a9 100644
--- a/drivers/xen/events/events_base.c
+++ b/drivers/xen/events/events_base.c
@@ -163,6 +163,8 @@ static DEFINE_PER_CPU(int [NR_VIRQS], virq_to_irq) = {[0 ... NR_VIRQS-1] = -1};
/* IRQ <-> IPI mapping */
static DEFINE_PER_CPU(int [XEN_NR_IPIS], ipi_to_irq) = {[0 ... XEN_NR_IPIS-1] = -1};
+/* Cache for IPI event channels - needed for hot cpu unplug (avoid RCU usage). */
+static DEFINE_PER_CPU(evtchn_port_t [XEN_NR_IPIS], ipi_to_evtchn) = {[0 ... XEN_NR_IPIS-1] = 0};
/* Event channel distribution data */
static atomic_t channels_on_cpu[NR_CPUS];
@@ -245,15 +247,6 @@ static int set_evtchn_to_irq(evtchn_port_t evtchn, unsigned int irq)
return 0;
}
-int get_evtchn_to_irq(evtchn_port_t evtchn)
-{
- if (evtchn >= xen_evtchn_max_channels())
- return -1;
- if (evtchn_to_irq[EVTCHN_ROW(evtchn)] == NULL)
- return -1;
- return READ_ONCE(evtchn_to_irq[EVTCHN_ROW(evtchn)][EVTCHN_COL(evtchn)]);
-}
-
/* Get info for IRQ */
static struct irq_info *info_for_irq(unsigned irq)
{
@@ -271,6 +264,19 @@ static void set_info_for_irq(unsigned int irq, struct irq_info *info)
irq_set_chip_data(irq, info);
}
+static struct irq_info *evtchn_to_info(evtchn_port_t evtchn)
+{
+ int irq;
+
+ if (evtchn >= xen_evtchn_max_channels())
+ return NULL;
+ if (evtchn_to_irq[EVTCHN_ROW(evtchn)] == NULL)
+ return NULL;
+ irq = READ_ONCE(evtchn_to_irq[EVTCHN_ROW(evtchn)][EVTCHN_COL(evtchn)]);
+
+ return (irq < 0) ? NULL : info_for_irq(irq);
+}
+
/* Per CPU channel accounting */
static void channels_on_cpu_dec(struct irq_info *info)
{
@@ -297,6 +303,13 @@ static void channels_on_cpu_inc(struct irq_info *info)
info->is_accounted = 1;
}
+static void xen_irq_free_desc(unsigned int irq)
+{
+ /* Legacy IRQ descriptors are managed by the arch. */
+ if (irq >= nr_legacy_irqs())
+ irq_free_desc(irq);
+}
+
static void delayed_free_irq(struct work_struct *work)
{
struct irq_info *info = container_of(to_rcu_work(work), struct irq_info,
@@ -308,14 +321,11 @@ static void delayed_free_irq(struct work_struct *work)
kfree(info);
- /* Legacy IRQ descriptors are managed by the arch. */
- if (irq >= nr_legacy_irqs())
- irq_free_desc(irq);
+ xen_irq_free_desc(irq);
}
/* Constructors for packed IRQ information. */
static int xen_irq_info_common_setup(struct irq_info *info,
- unsigned irq,
enum xen_irq_type type,
evtchn_port_t evtchn,
unsigned short cpu)
@@ -325,29 +335,27 @@ static int xen_irq_info_common_setup(struct irq_info *info,
BUG_ON(info->type != IRQT_UNBOUND && info->type != type);
info->type = type;
- info->irq = irq;
info->evtchn = evtchn;
info->cpu = cpu;
info->mask_reason = EVT_MASK_REASON_EXPLICIT;
raw_spin_lock_init(&info->lock);
- ret = set_evtchn_to_irq(evtchn, irq);
+ ret = set_evtchn_to_irq(evtchn, info->irq);
if (ret < 0)
return ret;
- irq_clear_status_flags(irq, IRQ_NOREQUEST|IRQ_NOAUTOEN);
+ irq_clear_status_flags(info->irq, IRQ_NOREQUEST | IRQ_NOAUTOEN);
return xen_evtchn_port_setup(evtchn);
}
-static int xen_irq_info_evtchn_setup(unsigned irq,
+static int xen_irq_info_evtchn_setup(struct irq_info *info,
evtchn_port_t evtchn,
struct xenbus_device *dev)
{
- struct irq_info *info = info_for_irq(irq);
int ret;
- ret = xen_irq_info_common_setup(info, irq, IRQT_EVTCHN, evtchn, 0);
+ ret = xen_irq_info_common_setup(info, IRQT_EVTCHN, evtchn, 0);
info->u.interdomain = dev;
if (dev)
atomic_inc(&dev->event_channels);
@@ -355,49 +363,37 @@ static int xen_irq_info_evtchn_setup(unsigned irq,
return ret;
}
-static int xen_irq_info_ipi_setup(unsigned cpu,
- unsigned irq,
- evtchn_port_t evtchn,
- enum ipi_vector ipi)
+static int xen_irq_info_ipi_setup(struct irq_info *info, unsigned int cpu,
+ evtchn_port_t evtchn, enum ipi_vector ipi)
{
- struct irq_info *info = info_for_irq(irq);
-
info->u.ipi = ipi;
- per_cpu(ipi_to_irq, cpu)[ipi] = irq;
+ per_cpu(ipi_to_irq, cpu)[ipi] = info->irq;
+ per_cpu(ipi_to_evtchn, cpu)[ipi] = evtchn;
- return xen_irq_info_common_setup(info, irq, IRQT_IPI, evtchn, 0);
+ return xen_irq_info_common_setup(info, IRQT_IPI, evtchn, 0);
}
-static int xen_irq_info_virq_setup(unsigned cpu,
- unsigned irq,
- evtchn_port_t evtchn,
- unsigned virq)
+static int xen_irq_info_virq_setup(struct irq_info *info, unsigned int cpu,
+ evtchn_port_t evtchn, unsigned int virq)
{
- struct irq_info *info = info_for_irq(irq);
-
info->u.virq = virq;
- per_cpu(virq_to_irq, cpu)[virq] = irq;
+ per_cpu(virq_to_irq, cpu)[virq] = info->irq;
- return xen_irq_info_common_setup(info, irq, IRQT_VIRQ, evtchn, 0);
+ return xen_irq_info_common_setup(info, IRQT_VIRQ, evtchn, 0);
}
-static int xen_irq_info_pirq_setup(unsigned irq,
- evtchn_port_t evtchn,
- unsigned pirq,
- unsigned gsi,
- uint16_t domid,
- unsigned char flags)
+static int xen_irq_info_pirq_setup(struct irq_info *info, evtchn_port_t evtchn,
+ unsigned int pirq, unsigned int gsi,
+ uint16_t domid, unsigned char flags)
{
- struct irq_info *info = info_for_irq(irq);
-
info->u.pirq.pirq = pirq;
info->u.pirq.gsi = gsi;
info->u.pirq.domid = domid;
info->u.pirq.flags = flags;
- return xen_irq_info_common_setup(info, irq, IRQT_PIRQ, evtchn, 0);
+ return xen_irq_info_common_setup(info, IRQT_PIRQ, evtchn, 0);
}
static void xen_irq_info_cleanup(struct irq_info *info)
@@ -411,7 +407,7 @@ static void xen_irq_info_cleanup(struct irq_info *info)
/*
* Accessors for packed IRQ information.
*/
-evtchn_port_t evtchn_from_irq(unsigned irq)
+static evtchn_port_t evtchn_from_irq(unsigned int irq)
{
const struct irq_info *info = NULL;
@@ -425,29 +421,32 @@ evtchn_port_t evtchn_from_irq(unsigned irq)
unsigned int irq_from_evtchn(evtchn_port_t evtchn)
{
- return get_evtchn_to_irq(evtchn);
+ struct irq_info *info = evtchn_to_info(evtchn);
+
+ return info ? info->irq : -1;
}
EXPORT_SYMBOL_GPL(irq_from_evtchn);
-int irq_from_virq(unsigned int cpu, unsigned int virq)
+int irq_evtchn_from_virq(unsigned int cpu, unsigned int virq,
+ evtchn_port_t *evtchn)
{
- return per_cpu(virq_to_irq, cpu)[virq];
+ int irq = per_cpu(virq_to_irq, cpu)[virq];
+
+ *evtchn = evtchn_from_irq(irq);
+
+ return irq;
}
-static enum ipi_vector ipi_from_irq(unsigned irq)
+static enum ipi_vector ipi_from_irq(struct irq_info *info)
{
- struct irq_info *info = info_for_irq(irq);
-
BUG_ON(info == NULL);
BUG_ON(info->type != IRQT_IPI);
return info->u.ipi;
}
-static unsigned virq_from_irq(unsigned irq)
+static unsigned int virq_from_irq(struct irq_info *info)
{
- struct irq_info *info = info_for_irq(irq);
-
BUG_ON(info == NULL);
BUG_ON(info->type != IRQT_VIRQ);
@@ -464,25 +463,11 @@ static unsigned pirq_from_irq(unsigned irq)
return info->u.pirq.pirq;
}
-static enum xen_irq_type type_from_irq(unsigned irq)
-{
- return info_for_irq(irq)->type;
-}
-
-static unsigned cpu_from_irq(unsigned irq)
-{
- return info_for_irq(irq)->cpu;
-}
-
unsigned int cpu_from_evtchn(evtchn_port_t evtchn)
{
- int irq = get_evtchn_to_irq(evtchn);
- unsigned ret = 0;
-
- if (irq != -1)
- ret = cpu_from_irq(irq);
+ struct irq_info *info = evtchn_to_info(evtchn);
- return ret;
+ return info ? info->cpu : 0;
}
static void do_mask(struct irq_info *info, u8 reason)
@@ -528,22 +513,17 @@ static bool pirq_needs_eoi_flag(unsigned irq)
return info->u.pirq.flags & PIRQ_NEEDS_EOI;
}
-static void bind_evtchn_to_cpu(evtchn_port_t evtchn, unsigned int cpu,
+static void bind_evtchn_to_cpu(struct irq_info *info, unsigned int cpu,
bool force_affinity)
{
- int irq = get_evtchn_to_irq(evtchn);
- struct irq_info *info = info_for_irq(irq);
-
- BUG_ON(irq == -1);
-
if (IS_ENABLED(CONFIG_SMP) && force_affinity) {
- struct irq_data *data = irq_get_irq_data(irq);
+ struct irq_data *data = irq_get_irq_data(info->irq);
irq_data_update_affinity(data, cpumask_of(cpu));
irq_data_update_effective_affinity(data, cpumask_of(cpu));
}
- xen_evtchn_port_bind_to_cpu(evtchn, cpu, info->cpu);
+ xen_evtchn_port_bind_to_cpu(info->evtchn, cpu, info->cpu);
channels_on_cpu_dec(info);
info->cpu = cpu;
@@ -733,50 +713,49 @@ void xen_irq_lateeoi(unsigned int irq, unsigned int eoi_flags)
}
EXPORT_SYMBOL_GPL(xen_irq_lateeoi);
-static void xen_irq_init(unsigned irq)
+static struct irq_info *xen_irq_init(unsigned int irq)
{
struct irq_info *info;
info = kzalloc(sizeof(*info), GFP_KERNEL);
- if (info == NULL)
- panic("Unable to allocate metadata for IRQ%d\n", irq);
+ if (info) {
+ info->irq = irq;
+ info->type = IRQT_UNBOUND;
+ info->refcnt = -1;
+ INIT_RCU_WORK(&info->rwork, delayed_free_irq);
- info->type = IRQT_UNBOUND;
- info->refcnt = -1;
- INIT_RCU_WORK(&info->rwork, delayed_free_irq);
+ set_info_for_irq(irq, info);
+ /*
+ * Interrupt affinity setting can be immediate. No point
+ * in delaying it until an interrupt is handled.
+ */
+ irq_set_status_flags(irq, IRQ_MOVE_PCNTXT);
- set_info_for_irq(irq, info);
- /*
- * Interrupt affinity setting can be immediate. No point
- * in delaying it until an interrupt is handled.
- */
- irq_set_status_flags(irq, IRQ_MOVE_PCNTXT);
+ INIT_LIST_HEAD(&info->eoi_list);
+ list_add_tail(&info->list, &xen_irq_list_head);
+ }
- INIT_LIST_HEAD(&info->eoi_list);
- list_add_tail(&info->list, &xen_irq_list_head);
+ return info;
}
-static int __must_check xen_allocate_irqs_dynamic(int nvec)
+static struct irq_info *xen_allocate_irq_dynamic(void)
{
- int i, irq = irq_alloc_descs(-1, 0, nvec, -1);
+ int irq = irq_alloc_desc_from(0, -1);
+ struct irq_info *info = NULL;
if (irq >= 0) {
- for (i = 0; i < nvec; i++)
- xen_irq_init(irq + i);
+ info = xen_irq_init(irq);
+ if (!info)
+ xen_irq_free_desc(irq);
}
- return irq;
-}
-
-static inline int __must_check xen_allocate_irq_dynamic(void)
-{
-
- return xen_allocate_irqs_dynamic(1);
+ return info;
}
-static int __must_check xen_allocate_irq_gsi(unsigned gsi)
+static struct irq_info *xen_allocate_irq_gsi(unsigned int gsi)
{
int irq;
+ struct irq_info *info;
/*
* A PV guest has no concept of a GSI (since it has no ACPI
@@ -793,15 +772,15 @@ static int __must_check xen_allocate_irq_gsi(unsigned gsi)
else
irq = irq_alloc_desc_at(gsi, -1);
- xen_irq_init(irq);
+ info = xen_irq_init(irq);
+ if (!info)
+ xen_irq_free_desc(irq);
- return irq;
+ return info;
}
-static void xen_free_irq(unsigned irq)
+static void xen_free_irq(struct irq_info *info)
{
- struct irq_info *info = info_for_irq(irq);
-
if (WARN_ON(!info))
return;
@@ -901,7 +880,7 @@ static unsigned int __startup_pirq(unsigned int irq)
goto err;
info->evtchn = evtchn;
- bind_evtchn_to_cpu(evtchn, 0, false);
+ bind_evtchn_to_cpu(info, 0, false);
rc = xen_evtchn_port_setup(evtchn);
if (rc)
@@ -967,10 +946,14 @@ int xen_irq_from_gsi(unsigned gsi)
}
EXPORT_SYMBOL_GPL(xen_irq_from_gsi);
-static void __unbind_from_irq(unsigned int irq)
+static void __unbind_from_irq(struct irq_info *info, unsigned int irq)
{
- evtchn_port_t evtchn = evtchn_from_irq(irq);
- struct irq_info *info = info_for_irq(irq);
+ evtchn_port_t evtchn;
+
+ if (!info) {
+ xen_irq_free_desc(irq);
+ return;
+ }
if (info->refcnt > 0) {
info->refcnt--;
@@ -978,16 +961,19 @@ static void __unbind_from_irq(unsigned int irq)
return;
}
+ evtchn = info->evtchn;
+
if (VALID_EVTCHN(evtchn)) {
- unsigned int cpu = cpu_from_irq(irq);
+ unsigned int cpu = info->cpu;
struct xenbus_device *dev;
- switch (type_from_irq(irq)) {
+ switch (info->type) {
case IRQT_VIRQ:
- per_cpu(virq_to_irq, cpu)[virq_from_irq(irq)] = -1;
+ per_cpu(virq_to_irq, cpu)[virq_from_irq(info)] = -1;
break;
case IRQT_IPI:
- per_cpu(ipi_to_irq, cpu)[ipi_from_irq(irq)] = -1;
+ per_cpu(ipi_to_irq, cpu)[ipi_from_irq(info)] = -1;
+ per_cpu(ipi_to_evtchn, cpu)[ipi_from_irq(info)] = 0;
break;
case IRQT_EVTCHN:
dev = info->u.interdomain;
@@ -1002,7 +988,7 @@ static void __unbind_from_irq(unsigned int irq)
xen_evtchn_close(evtchn);
}
- xen_free_irq(irq);
+ xen_free_irq(info);
}
/*
@@ -1018,24 +1004,24 @@ static void __unbind_from_irq(unsigned int irq)
int xen_bind_pirq_gsi_to_irq(unsigned gsi,
unsigned pirq, int shareable, char *name)
{
- int irq;
+ struct irq_info *info;
struct physdev_irq irq_op;
int ret;
mutex_lock(&irq_mapping_update_lock);
- irq = xen_irq_from_gsi(gsi);
- if (irq != -1) {
+ ret = xen_irq_from_gsi(gsi);
+ if (ret != -1) {
pr_info("%s: returning irq %d for gsi %u\n",
- __func__, irq, gsi);
+ __func__, ret, gsi);
goto out;
}
- irq = xen_allocate_irq_gsi(gsi);
- if (irq < 0)
+ info = xen_allocate_irq_gsi(gsi);
+ if (!info)
goto out;
- irq_op.irq = irq;
+ irq_op.irq = info->irq;
irq_op.vector = 0;
/* Only the privileged domain can do this. For non-priv, the pcifront
@@ -1043,20 +1029,19 @@ int xen_bind_pirq_gsi_to_irq(unsigned gsi,
* this in the priv domain. */
if (xen_initial_domain() &&
HYPERVISOR_physdev_op(PHYSDEVOP_alloc_irq_vector, &irq_op)) {
- xen_free_irq(irq);
- irq = -ENOSPC;
+ xen_free_irq(info);
+ ret = -ENOSPC;
goto out;
}
- ret = xen_irq_info_pirq_setup(irq, 0, pirq, gsi, DOMID_SELF,
+ ret = xen_irq_info_pirq_setup(info, 0, pirq, gsi, DOMID_SELF,
shareable ? PIRQ_SHAREABLE : 0);
if (ret < 0) {
- __unbind_from_irq(irq);
- irq = ret;
+ __unbind_from_irq(info, info->irq);
goto out;
}
- pirq_query_unmask(irq);
+ pirq_query_unmask(info->irq);
/* We try to use the handler with the appropriate semantic for the
* type of interrupt: if the interrupt is an edge triggered
* interrupt we use handle_edge_irq.
@@ -1073,16 +1058,18 @@ int xen_bind_pirq_gsi_to_irq(unsigned gsi,
* is the right choice either way.
*/
if (shareable)
- irq_set_chip_and_handler_name(irq, &xen_pirq_chip,
+ irq_set_chip_and_handler_name(info->irq, &xen_pirq_chip,
handle_fasteoi_irq, name);
else
- irq_set_chip_and_handler_name(irq, &xen_pirq_chip,
+ irq_set_chip_and_handler_name(info->irq, &xen_pirq_chip,
handle_edge_irq, name);
+ ret = info->irq;
+
out:
mutex_unlock(&irq_mapping_update_lock);
- return irq;
+ return ret;
}
#ifdef CONFIG_PCI_MSI
@@ -1104,17 +1091,24 @@ int xen_bind_pirq_msi_to_irq(struct pci_dev *dev, struct msi_desc *msidesc,
int pirq, int nvec, const char *name, domid_t domid)
{
int i, irq, ret;
+ struct irq_info *info;
mutex_lock(&irq_mapping_update_lock);
- irq = xen_allocate_irqs_dynamic(nvec);
+ irq = irq_alloc_descs(-1, 0, nvec, -1);
if (irq < 0)
goto out;
for (i = 0; i < nvec; i++) {
+ info = xen_irq_init(irq + i);
+ if (!info) {
+ ret = -ENOMEM;
+ goto error_irq;
+ }
+
irq_set_chip_and_handler_name(irq + i, &xen_pirq_chip, handle_edge_irq, name);
- ret = xen_irq_info_pirq_setup(irq + i, 0, pirq + i, 0, domid,
+ ret = xen_irq_info_pirq_setup(info, 0, pirq + i, 0, domid,
i == 0 ? 0 : PIRQ_MSI_GROUP);
if (ret < 0)
goto error_irq;
@@ -1126,9 +1120,12 @@ int xen_bind_pirq_msi_to_irq(struct pci_dev *dev, struct msi_desc *msidesc,
out:
mutex_unlock(&irq_mapping_update_lock);
return irq;
+
error_irq:
- while (nvec--)
- __unbind_from_irq(irq + nvec);
+ while (nvec--) {
+ info = info_for_irq(irq + nvec);
+ __unbind_from_irq(info, irq + nvec);
+ }
mutex_unlock(&irq_mapping_update_lock);
return ret;
}
@@ -1164,7 +1161,7 @@ int xen_destroy_irq(int irq)
}
}
- xen_free_irq(irq);
+ xen_free_irq(info);
out:
mutex_unlock(&irq_mapping_update_lock);
@@ -1203,28 +1200,27 @@ EXPORT_SYMBOL_GPL(xen_pirq_from_irq);
static int bind_evtchn_to_irq_chip(evtchn_port_t evtchn, struct irq_chip *chip,
struct xenbus_device *dev)
{
- int irq;
- int ret;
+ int ret = -ENOMEM;
+ struct irq_info *info;
if (evtchn >= xen_evtchn_max_channels())
return -ENOMEM;
mutex_lock(&irq_mapping_update_lock);
- irq = get_evtchn_to_irq(evtchn);
+ info = evtchn_to_info(evtchn);
- if (irq == -1) {
- irq = xen_allocate_irq_dynamic();
- if (irq < 0)
+ if (!info) {
+ info = xen_allocate_irq_dynamic();
+ if (!info)
goto out;
- irq_set_chip_and_handler_name(irq, chip,
+ irq_set_chip_and_handler_name(info->irq, chip,
handle_edge_irq, "event");
- ret = xen_irq_info_evtchn_setup(irq, evtchn, dev);
+ ret = xen_irq_info_evtchn_setup(info, evtchn, dev);
if (ret < 0) {
- __unbind_from_irq(irq);
- irq = ret;
+ __unbind_from_irq(info, info->irq);
goto out;
}
/*
@@ -1234,16 +1230,17 @@ static int bind_evtchn_to_irq_chip(evtchn_port_t evtchn, struct irq_chip *chip,
* affinity setting is not invoked on them so nothing would
* bind the channel.
*/
- bind_evtchn_to_cpu(evtchn, 0, false);
+ bind_evtchn_to_cpu(info, 0, false);
} else {
- struct irq_info *info = info_for_irq(irq);
- WARN_ON(info == NULL || info->type != IRQT_EVTCHN);
+ WARN_ON(info->type != IRQT_EVTCHN);
}
+ ret = info->irq;
+
out:
mutex_unlock(&irq_mapping_update_lock);
- return irq;
+ return ret;
}
int bind_evtchn_to_irq(evtchn_port_t evtchn)
@@ -1262,18 +1259,19 @@ static int bind_ipi_to_irq(unsigned int ipi, unsigned int cpu)
{
struct evtchn_bind_ipi bind_ipi;
evtchn_port_t evtchn;
- int ret, irq;
+ struct irq_info *info;
+ int ret;
mutex_lock(&irq_mapping_update_lock);
- irq = per_cpu(ipi_to_irq, cpu)[ipi];
+ ret = per_cpu(ipi_to_irq, cpu)[ipi];
- if (irq == -1) {
- irq = xen_allocate_irq_dynamic();
- if (irq < 0)
+ if (ret == -1) {
+ info = xen_allocate_irq_dynamic();
+ if (!info)
goto out;
- irq_set_chip_and_handler_name(irq, &xen_percpu_chip,
+ irq_set_chip_and_handler_name(info->irq, &xen_percpu_chip,
handle_percpu_irq, "ipi");
bind_ipi.vcpu = xen_vcpu_nr(cpu);
@@ -1282,25 +1280,25 @@ static int bind_ipi_to_irq(unsigned int ipi, unsigned int cpu)
BUG();
evtchn = bind_ipi.port;
- ret = xen_irq_info_ipi_setup(cpu, irq, evtchn, ipi);
+ ret = xen_irq_info_ipi_setup(info, cpu, evtchn, ipi);
if (ret < 0) {
- __unbind_from_irq(irq);
- irq = ret;
+ __unbind_from_irq(info, info->irq);
goto out;
}
/*
* Force the affinity mask to the target CPU so proc shows
* the correct target.
*/
- bind_evtchn_to_cpu(evtchn, cpu, true);
+ bind_evtchn_to_cpu(info, cpu, true);
+ ret = info->irq;
} else {
- struct irq_info *info = info_for_irq(irq);
+ info = info_for_irq(ret);
WARN_ON(info == NULL || info->type != IRQT_IPI);
}
out:
mutex_unlock(&irq_mapping_update_lock);
- return irq;
+ return ret;
}
static int bind_interdomain_evtchn_to_irq_chip(struct xenbus_device *dev,
@@ -1328,10 +1326,12 @@ int bind_interdomain_evtchn_to_irq_lateeoi(struct xenbus_device *dev,
}
EXPORT_SYMBOL_GPL(bind_interdomain_evtchn_to_irq_lateeoi);
-static int find_virq(unsigned int virq, unsigned int cpu, evtchn_port_t *evtchn)
+static int find_virq(unsigned int virq, unsigned int cpu, evtchn_port_t *evtchn,
+ bool percpu)
{
struct evtchn_status status;
evtchn_port_t port;
+ bool exists = false;
memset(&status, 0, sizeof(status));
for (port = 0; port < xen_evtchn_max_channels(); port++) {
@@ -1344,12 +1344,16 @@ static int find_virq(unsigned int virq, unsigned int cpu, evtchn_port_t *evtchn)
continue;
if (status.status != EVTCHNSTAT_virq)
continue;
- if (status.u.virq == virq && status.vcpu == xen_vcpu_nr(cpu)) {
+ if (status.u.virq != virq)
+ continue;
+ if (status.vcpu == xen_vcpu_nr(cpu)) {
*evtchn = port;
return 0;
+ } else if (!percpu) {
+ exists = true;
}
}
- return -ENOENT;
+ return exists ? -EEXIST : -ENOENT;
}
/**
@@ -1369,22 +1373,23 @@ int bind_virq_to_irq(unsigned int virq, unsigned int cpu, bool percpu)
{
struct evtchn_bind_virq bind_virq;
evtchn_port_t evtchn = 0;
- int irq, ret;
+ struct irq_info *info;
+ int ret;
mutex_lock(&irq_mapping_update_lock);
- irq = per_cpu(virq_to_irq, cpu)[virq];
+ ret = per_cpu(virq_to_irq, cpu)[virq];
- if (irq == -1) {
- irq = xen_allocate_irq_dynamic();
- if (irq < 0)
+ if (ret == -1) {
+ info = xen_allocate_irq_dynamic();
+ if (!info)
goto out;
if (percpu)
- irq_set_chip_and_handler_name(irq, &xen_percpu_chip,
+ irq_set_chip_and_handler_name(info->irq, &xen_percpu_chip,
handle_percpu_irq, "virq");
else
- irq_set_chip_and_handler_name(irq, &xen_dynamic_chip,
+ irq_set_chip_and_handler_name(info->irq, &xen_dynamic_chip,
handle_edge_irq, "virq");
bind_virq.virq = virq;
@@ -1395,14 +1400,16 @@ int bind_virq_to_irq(unsigned int virq, unsigned int cpu, bool percpu)
evtchn = bind_virq.port;
else {
if (ret == -EEXIST)
- ret = find_virq(virq, cpu, &evtchn);
- BUG_ON(ret < 0);
+ ret = find_virq(virq, cpu, &evtchn, percpu);
+ if (ret) {
+ __unbind_from_irq(info, info->irq);
+ goto out;
+ }
}
- ret = xen_irq_info_virq_setup(cpu, irq, evtchn, virq);
+ ret = xen_irq_info_virq_setup(info, cpu, evtchn, virq);
if (ret < 0) {
- __unbind_from_irq(irq);
- irq = ret;
+ __unbind_from_irq(info, info->irq);
goto out;
}
@@ -1410,22 +1417,26 @@ int bind_virq_to_irq(unsigned int virq, unsigned int cpu, bool percpu)
* Force the affinity mask for percpu interrupts so proc
* shows the correct target.
*/
- bind_evtchn_to_cpu(evtchn, cpu, percpu);
+ bind_evtchn_to_cpu(info, cpu, percpu);
+ ret = info->irq;
} else {
- struct irq_info *info = info_for_irq(irq);
+ info = info_for_irq(ret);
WARN_ON(info == NULL || info->type != IRQT_VIRQ);
}
out:
mutex_unlock(&irq_mapping_update_lock);
- return irq;
+ return ret;
}
static void unbind_from_irq(unsigned int irq)
{
+ struct irq_info *info;
+
mutex_lock(&irq_mapping_update_lock);
- __unbind_from_irq(irq);
+ info = info_for_irq(irq);
+ __unbind_from_irq(info, irq);
mutex_unlock(&irq_mapping_update_lock);
}
@@ -1576,13 +1587,7 @@ EXPORT_SYMBOL_GPL(xen_set_irq_priority);
int evtchn_make_refcounted(evtchn_port_t evtchn)
{
- int irq = get_evtchn_to_irq(evtchn);
- struct irq_info *info;
-
- if (irq == -1)
- return -ENOENT;
-
- info = info_for_irq(irq);
+ struct irq_info *info = evtchn_to_info(evtchn);
if (!info)
return -ENOENT;
@@ -1597,7 +1602,6 @@ EXPORT_SYMBOL_GPL(evtchn_make_refcounted);
int evtchn_get(evtchn_port_t evtchn)
{
- int irq;
struct irq_info *info;
int err = -ENOENT;
@@ -1606,11 +1610,7 @@ int evtchn_get(evtchn_port_t evtchn)
mutex_lock(&irq_mapping_update_lock);
- irq = get_evtchn_to_irq(evtchn);
- if (irq == -1)
- goto done;
-
- info = info_for_irq(irq);
+ info = evtchn_to_info(evtchn);
if (!info)
goto done;
@@ -1630,16 +1630,17 @@ EXPORT_SYMBOL_GPL(evtchn_get);
void evtchn_put(evtchn_port_t evtchn)
{
- int irq = get_evtchn_to_irq(evtchn);
- if (WARN_ON(irq == -1))
+ struct irq_info *info = evtchn_to_info(evtchn);
+
+ if (WARN_ON(!info))
return;
- unbind_from_irq(irq);
+ unbind_from_irq(info->irq);
}
EXPORT_SYMBOL_GPL(evtchn_put);
void xen_send_IPI_one(unsigned int cpu, enum ipi_vector vector)
{
- int irq;
+ evtchn_port_t evtchn;
#ifdef CONFIG_X86
if (unlikely(vector == XEN_NMI_VECTOR)) {
@@ -1650,9 +1651,9 @@ void xen_send_IPI_one(unsigned int cpu, enum ipi_vector vector)
return;
}
#endif
- irq = per_cpu(ipi_to_irq, cpu)[vector];
- BUG_ON(irq < 0);
- notify_remote_via_irq(irq);
+ evtchn = per_cpu(ipi_to_evtchn, cpu)[vector];
+ BUG_ON(evtchn == 0);
+ notify_remote_via_evtchn(evtchn);
}
struct evtchn_loop_ctrl {
@@ -1663,12 +1664,10 @@ struct evtchn_loop_ctrl {
void handle_irq_for_port(evtchn_port_t port, struct evtchn_loop_ctrl *ctrl)
{
- int irq;
- struct irq_info *info;
+ struct irq_info *info = evtchn_to_info(port);
struct xenbus_device *dev;
- irq = get_evtchn_to_irq(port);
- if (irq == -1)
+ if (!info)
return;
/*
@@ -1693,7 +1692,6 @@ void handle_irq_for_port(evtchn_port_t port, struct evtchn_loop_ctrl *ctrl)
}
}
- info = info_for_irq(irq);
if (xchg_acquire(&info->is_active, 1))
return;
@@ -1707,7 +1705,7 @@ void handle_irq_for_port(evtchn_port_t port, struct evtchn_loop_ctrl *ctrl)
info->eoi_time = get_jiffies_64() + event_eoi_delay;
}
- generic_handle_irq(irq);
+ generic_handle_irq(info->irq);
}
int xen_evtchn_do_upcall(void)
@@ -1765,16 +1763,17 @@ void rebind_evtchn_irq(evtchn_port_t evtchn, int irq)
mutex_lock(&irq_mapping_update_lock);
/* After resume the irq<->evtchn mappings are all cleared out */
- BUG_ON(get_evtchn_to_irq(evtchn) != -1);
+ BUG_ON(evtchn_to_info(evtchn));
/* Expect irq to have been bound before,
so there should be a proper type */
BUG_ON(info->type == IRQT_UNBOUND);
- (void)xen_irq_info_evtchn_setup(irq, evtchn, NULL);
+ info->irq = irq;
+ (void)xen_irq_info_evtchn_setup(info, evtchn, NULL);
mutex_unlock(&irq_mapping_update_lock);
- bind_evtchn_to_cpu(evtchn, info->cpu, false);
+ bind_evtchn_to_cpu(info, info->cpu, false);
/* Unmask the event channel. */
enable_irq(irq);
@@ -1810,7 +1809,7 @@ static int xen_rebind_evtchn_to_cpu(struct irq_info *info, unsigned int tcpu)
if (HYPERVISOR_event_channel_op(EVTCHNOP_bind_vcpu, &bind_vcpu) >= 0) {
int old_cpu = info->cpu;
- bind_evtchn_to_cpu(evtchn, tcpu, false);
+ bind_evtchn_to_cpu(info, tcpu, false);
if (info->type == IRQT_VIRQ) {
int virq = info->u.virq;
@@ -1970,7 +1969,7 @@ static void restore_pirqs(void)
if (rc) {
pr_warn("xen map irq failed gsi=%d irq=%d pirq=%d rc=%d\n",
gsi, irq, pirq, rc);
- xen_free_irq(irq);
+ xen_free_irq(info);
continue;
}
@@ -1984,13 +1983,15 @@ static void restore_cpu_virqs(unsigned int cpu)
{
struct evtchn_bind_virq bind_virq;
evtchn_port_t evtchn;
+ struct irq_info *info;
int virq, irq;
for (virq = 0; virq < NR_VIRQS; virq++) {
if ((irq = per_cpu(virq_to_irq, cpu)[virq]) == -1)
continue;
+ info = info_for_irq(irq);
- BUG_ON(virq_from_irq(irq) != virq);
+ BUG_ON(virq_from_irq(info) != virq);
/* Get a new binding from Xen. */
bind_virq.virq = virq;
@@ -2001,9 +2002,9 @@ static void restore_cpu_virqs(unsigned int cpu)
evtchn = bind_virq.port;
/* Record the new mapping. */
- (void)xen_irq_info_virq_setup(cpu, irq, evtchn, virq);
+ xen_irq_info_virq_setup(info, cpu, evtchn, virq);
/* The affinity mask is still valid */
- bind_evtchn_to_cpu(evtchn, cpu, false);
+ bind_evtchn_to_cpu(info, cpu, false);
}
}
@@ -2011,13 +2012,15 @@ static void restore_cpu_ipis(unsigned int cpu)
{
struct evtchn_bind_ipi bind_ipi;
evtchn_port_t evtchn;
+ struct irq_info *info;
int ipi, irq;
for (ipi = 0; ipi < XEN_NR_IPIS; ipi++) {
if ((irq = per_cpu(ipi_to_irq, cpu)[ipi]) == -1)
continue;
+ info = info_for_irq(irq);
- BUG_ON(ipi_from_irq(irq) != ipi);
+ BUG_ON(ipi_from_irq(info) != ipi);
/* Get a new binding from Xen. */
bind_ipi.vcpu = xen_vcpu_nr(cpu);
@@ -2027,9 +2030,9 @@ static void restore_cpu_ipis(unsigned int cpu)
evtchn = bind_ipi.port;
/* Record the new mapping. */
- (void)xen_irq_info_ipi_setup(cpu, irq, evtchn, ipi);
+ xen_irq_info_ipi_setup(info, cpu, evtchn, ipi);
/* The affinity mask is still valid */
- bind_evtchn_to_cpu(evtchn, cpu, false);
+ bind_evtchn_to_cpu(info, cpu, false);
}
}
diff --git a/drivers/xen/events/events_internal.h b/drivers/xen/events/events_internal.h
index 4d3398eff9cd..19ae31695edc 100644
--- a/drivers/xen/events/events_internal.h
+++ b/drivers/xen/events/events_internal.h
@@ -33,7 +33,6 @@ struct evtchn_ops {
extern const struct evtchn_ops *evtchn_ops;
-int get_evtchn_to_irq(evtchn_port_t evtchn);
void handle_irq_for_port(evtchn_port_t port, struct evtchn_loop_ctrl *ctrl);
unsigned int cpu_from_evtchn(evtchn_port_t evtchn);
diff --git a/drivers/xen/gntdev-dmabuf.c b/drivers/xen/gntdev-dmabuf.c
index 335451309566..7df26330af42 100644
--- a/drivers/xen/gntdev-dmabuf.c
+++ b/drivers/xen/gntdev-dmabuf.c
@@ -357,8 +357,12 @@ struct gntdev_dmabuf_export_args {
static int dmabuf_exp_from_pages(struct gntdev_dmabuf_export_args *args)
{
DEFINE_DMA_BUF_EXPORT_INFO(exp_info);
- struct gntdev_dmabuf *gntdev_dmabuf;
- int ret;
+ struct gntdev_dmabuf *gntdev_dmabuf __free(kfree) = NULL;
+ CLASS(get_unused_fd, ret)(O_CLOEXEC);
+ int fd;
+
+ if (ret < 0)
+ return ret;
gntdev_dmabuf = kzalloc(sizeof(*gntdev_dmabuf), GFP_KERNEL);
if (!gntdev_dmabuf)
@@ -383,32 +387,23 @@ static int dmabuf_exp_from_pages(struct gntdev_dmabuf_export_args *args)
exp_info.priv = gntdev_dmabuf;
gntdev_dmabuf->dmabuf = dma_buf_export(&exp_info);
- if (IS_ERR(gntdev_dmabuf->dmabuf)) {
- ret = PTR_ERR(gntdev_dmabuf->dmabuf);
- gntdev_dmabuf->dmabuf = NULL;
- goto fail;
- }
-
- ret = dma_buf_fd(gntdev_dmabuf->dmabuf, O_CLOEXEC);
- if (ret < 0)
- goto fail;
+ if (IS_ERR(gntdev_dmabuf->dmabuf))
+ return PTR_ERR(gntdev_dmabuf->dmabuf);
gntdev_dmabuf->fd = ret;
args->fd = ret;
pr_debug("Exporting DMA buffer with fd %d\n", ret);
+ get_file(gntdev_dmabuf->priv->filp);
mutex_lock(&args->dmabuf_priv->lock);
list_add(&gntdev_dmabuf->next, &args->dmabuf_priv->exp_list);
mutex_unlock(&args->dmabuf_priv->lock);
- get_file(gntdev_dmabuf->priv->filp);
- return 0;
-fail:
- if (gntdev_dmabuf->dmabuf)
- dma_buf_put(gntdev_dmabuf->dmabuf);
- kfree(gntdev_dmabuf);
- return ret;
+ fd = ret;
+ ret = -EBADF;
+ fd_install(fd, no_free_ptr(gntdev_dmabuf)->dmabuf->file);
+ return 0;
}
static struct gntdev_grant_map *
diff --git a/drivers/xen/pvcalls-front.c b/drivers/xen/pvcalls-front.c
index d5d589bda243..5e1701ca8f1a 100644
--- a/drivers/xen/pvcalls-front.c
+++ b/drivers/xen/pvcalls-front.c
@@ -32,6 +32,7 @@ struct pvcalls_bedata {
struct xen_pvcalls_front_ring ring;
grant_ref_t ref;
int irq;
+ bool disabled;
struct list_head socket_mappings;
spinlock_t socket_lock;
@@ -131,6 +132,20 @@ static inline int get_request(struct pvcalls_bedata *bedata, int *req_id)
return 0;
}
+/*
+ * Wait for the backend's response to req_id, or for the frontend to be
+ * disabled because the backend violated the wire protocol. Returns 0 once
+ * the response has arrived, or -EIO if the frontend was disabled.
+ */
+static int pvcalls_front_wait_rsp(struct pvcalls_bedata *bedata, u32 req_id)
+{
+ wait_event(bedata->inflight_req,
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled));
+
+ return READ_ONCE(bedata->disabled) ? -EIO : 0;
+}
+
static bool pvcalls_front_write_todo(struct sock_mapping *map)
{
struct pvcalls_data_intf *intf = map->active.ring;
@@ -168,7 +183,8 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
struct pvcalls_bedata *bedata;
struct xen_pvcalls_response *rsp;
uint8_t *src, *dst;
- int req_id = 0, more = 0, done = 0;
+ u32 req_id = 0;
+ int more = 0, done = 0;
if (dev == NULL)
return IRQ_HANDLED;
@@ -179,12 +195,31 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
pvcalls_exit();
return IRQ_HANDLED;
}
+ if (READ_ONCE(bedata->disabled)) {
+ pvcalls_exit();
+ return IRQ_HANDLED;
+ }
again:
while (RING_HAS_UNCONSUMED_RESPONSES(&bedata->ring)) {
rsp = RING_GET_RESPONSE(&bedata->ring, bedata->ring.rsp_cons);
req_id = rsp->req_id;
+ if (req_id >= PVCALLS_NR_RSP_PER_RING) {
+ /*
+ * The backend supplied a req_id that would index
+ * bedata->rsp[] out of bounds: a protocol violation
+ * from a malicious or buggy backend. Log once, stop
+ * trusting this backend and disable the frontend rather
+ * than silently dropping the response and continuing.
+ */
+ pr_err_once("pvcalls: backend sent out-of-range req_id %u, disabling frontend\n",
+ req_id);
+ WRITE_ONCE(bedata->disabled, true);
+ bedata->ring.rsp_cons++;
+ done = 1;
+ break;
+ }
if (rsp->cmd == PVCALLS_POLL) {
struct sock_mapping *map = (struct sock_mapping *)(uintptr_t)
rsp->u.poll.id;
@@ -217,7 +252,7 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id)
}
RING_FINAL_CHECK_FOR_RESPONSES(&bedata->ring, more);
- if (more)
+ if (more && !READ_ONCE(bedata->disabled))
goto again;
if (done)
wake_up(&bedata->inflight_req);
@@ -322,8 +357,11 @@ int pvcalls_front_socket(struct socket *sock)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit();
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -469,8 +507,11 @@ int pvcalls_front_connect(struct socket *sock, struct sockaddr *addr,
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -700,8 +741,11 @@ int pvcalls_front_bind(struct socket *sock, struct sockaddr *addr, int addr_len)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -749,8 +793,11 @@ int pvcalls_front_listen(struct socket *sock, int backlog)
if (notify)
notify_remote_via_irq(bedata->irq);
- wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ ret = pvcalls_front_wait_rsp(bedata, req_id);
+ if (ret) {
+ pvcalls_exit_sock(sock);
+ return ret;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -806,6 +853,14 @@ int pvcalls_front_accept(struct socket *sock, struct socket *newsock, int flags)
}
}
+ if (READ_ONCE(bedata->disabled)) {
+ clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
+ (void *)&map->passive.flags);
+ wake_up(&map->passive.inflight_accept_req);
+ pvcalls_exit_sock(sock);
+ return -EIO;
+ }
+
map2 = kzalloc(sizeof(*map2), GFP_KERNEL);
if (map2 == NULL) {
clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
@@ -867,10 +922,18 @@ int pvcalls_front_accept(struct socket *sock, struct socket *newsock, int flags)
}
if (wait_event_interruptible(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id)) {
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled))) {
pvcalls_exit_sock(sock);
return -EINTR;
}
+ if (READ_ONCE(bedata->disabled)) {
+ clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT,
+ (void *)&map->passive.flags);
+ wake_up(&map->passive.inflight_accept_req);
+ pvcalls_exit_sock(sock);
+ return -EIO;
+ }
/* read req_id, then the content */
smp_rmb();
@@ -1039,7 +1102,8 @@ int pvcalls_front_release(struct socket *sock)
notify_remote_via_irq(bedata->irq);
wait_event(bedata->inflight_req,
- READ_ONCE(bedata->rsp[req_id].req_id) == req_id);
+ READ_ONCE(bedata->rsp[req_id].req_id) == req_id ||
+ READ_ONCE(bedata->disabled));
if (map->active_socket) {
/*
diff --git a/fs/9p/vfs_inode_dotl.c b/fs/9p/vfs_inode_dotl.c
index e15ad46833e0..7c261b354379 100644
--- a/fs/9p/vfs_inode_dotl.c
+++ b/fs/9p/vfs_inode_dotl.c
@@ -424,8 +424,8 @@ static int v9fs_vfs_mkdir_dotl(struct user_namespace *mnt_userns,
err);
goto error;
}
- v9fs_fid_add(dentry, &fid);
v9fs_set_create_acl(inode, fid, dacl, pacl);
+ v9fs_fid_add(dentry, &fid);
d_instantiate(dentry, inode);
err = 0;
} else {
diff --git a/fs/Kconfig b/fs/Kconfig
index 7104e3eb38eb..da3fd893efa4 100644
--- a/fs/Kconfig
+++ b/fs/Kconfig
@@ -150,7 +150,6 @@ menu "DOS/FAT/EXFAT/NT Filesystems"
source "fs/fat/Kconfig"
source "fs/exfat/Kconfig"
-source "fs/ntfs/Kconfig"
source "fs/ntfs3/Kconfig"
endmenu
diff --git a/fs/Makefile b/fs/Makefile
index 9b7bcf6e54cd..59c6aab0d52a 100644
--- a/fs/Makefile
+++ b/fs/Makefile
@@ -96,7 +96,6 @@ obj-y += unicode/
obj-$(CONFIG_SYSV_FS) += sysv/
obj-$(CONFIG_SMBFS) += smb/
obj-$(CONFIG_HPFS_FS) += hpfs/
-obj-$(CONFIG_NTFS_FS) += ntfs/
obj-$(CONFIG_NTFS3_FS) += ntfs3/
obj-$(CONFIG_UFS_FS) += ufs/
obj-$(CONFIG_EFS_FS) += efs/
diff --git a/fs/afs/inode.c b/fs/afs/inode.c
index 5921dd3687e3..524e5a106337 100644
--- a/fs/afs/inode.c
+++ b/fs/afs/inode.c
@@ -809,6 +809,7 @@ void afs_evict_inode(struct inode *inode)
ASSERTCMP(inode->i_ino, ==, vnode->fid.vnode);
+ flush_delayed_work(&vnode->lock_work);
truncate_inode_pages_final(&inode->i_data);
afs_set_cache_aux(vnode, &aux);
diff --git a/fs/afs/internal.h b/fs/afs/internal.h
index fd4310272ccc..325be1d29a71 100644
--- a/fs/afs/internal.h
+++ b/fs/afs/internal.h
@@ -99,6 +99,7 @@ struct afs_call {
wait_queue_head_t waitq; /* processes awaiting completion */
struct work_struct async_work; /* async I/O processor */
struct work_struct work; /* actual work processor */
+ struct work_struct free_work; /* Deferred free processor */
struct rxrpc_call *rxcall; /* RxRPC call handle */
struct key *key; /* security for this call */
struct afs_net *net; /* The network namespace */
@@ -1283,6 +1284,7 @@ extern int __net_init afs_open_socket(struct afs_net *);
extern void __net_exit afs_close_socket(struct afs_net *);
extern void afs_charge_preallocation(struct work_struct *);
extern void afs_put_call(struct afs_call *);
+void afs_deferred_put_call(struct afs_call *call);
extern void afs_make_call(struct afs_addr_cursor *, struct afs_call *, gfp_t);
extern long afs_wait_for_call_to_complete(struct afs_call *, struct afs_addr_cursor *);
extern struct afs_call *afs_alloc_flat_call(struct afs_net *,
diff --git a/fs/afs/rxrpc.c b/fs/afs/rxrpc.c
index a0715e5dc38d..0658523bd786 100644
--- a/fs/afs/rxrpc.c
+++ b/fs/afs/rxrpc.c
@@ -16,6 +16,7 @@
struct workqueue_struct *afs_async_calls;
+static void afs_deferred_free_worker(struct work_struct *work);
static void afs_wake_up_call_waiter(struct sock *, struct rxrpc_call *, unsigned long);
static void afs_wake_up_async_call(struct sock *, struct rxrpc_call *, unsigned long);
static void afs_process_async_call(struct work_struct *);
@@ -153,6 +154,7 @@ static struct afs_call *afs_alloc_call(struct afs_net *net,
call->debug_id = atomic_inc_return(&rxrpc_debug_id);
refcount_set(&call->ref, 1);
INIT_WORK(&call->async_work, afs_process_async_call);
+ INIT_WORK(&call->free_work, afs_deferred_free_worker);
init_waitqueue_head(&call->waitq);
spin_lock_init(&call->state_lock);
call->iter = &call->def_iter;
@@ -163,6 +165,34 @@ static struct afs_call *afs_alloc_call(struct afs_net *net,
return call;
}
+static void afs_free_call(struct afs_call *call)
+{
+ struct afs_net *net = call->net;
+ int o;
+
+ ASSERT(!work_pending(&call->async_work));
+
+ if (call->rxcall) {
+ rxrpc_kernel_end_call(net->socket, call->rxcall);
+ call->rxcall = NULL;
+ }
+ if (call->type->destructor)
+ call->type->destructor(call);
+
+ afs_unuse_server_notime(call->net, call->server, afs_server_trace_put_call);
+ afs_put_addrlist(call->alist);
+ kfree(call->request);
+
+ o = atomic_read(&net->nr_outstanding_calls);
+ trace_afs_call(call->debug_id, afs_call_trace_free, 0, o,
+ __builtin_return_address(0));
+ kfree(call);
+
+ o = atomic_dec_return(&net->nr_outstanding_calls);
+ if (o == 0)
+ wake_up_var(&net->nr_outstanding_calls);
+}
+
/*
* Dispose of a reference on a call.
*/
@@ -177,30 +207,34 @@ void afs_put_call(struct afs_call *call)
o = atomic_read(&net->nr_outstanding_calls);
trace_afs_call(debug_id, afs_call_trace_put, r - 1, o,
__builtin_return_address(0));
+ if (zero)
+ afs_free_call(call);
+}
- if (zero) {
- ASSERT(!work_pending(&call->async_work));
- ASSERT(call->type->name != NULL);
-
- if (call->rxcall) {
- rxrpc_kernel_end_call(net->socket, call->rxcall);
- call->rxcall = NULL;
- }
- if (call->type->destructor)
- call->type->destructor(call);
+static void afs_deferred_free_worker(struct work_struct *work)
+{
+ struct afs_call *call = container_of(work, struct afs_call, free_work);
- afs_unuse_server_notime(call->net, call->server, afs_server_trace_put_call);
- afs_put_addrlist(call->alist);
- kfree(call->request);
+ afs_free_call(call);
+}
- trace_afs_call(call->debug_id, afs_call_trace_free, 0, o,
- __builtin_return_address(0));
- kfree(call);
+/*
+ * Dispose of a reference on a call, deferring the cleanup to a workqueue
+ * to avoid lock recursion.
+ */
+void afs_deferred_put_call(struct afs_call *call)
+{
+ struct afs_net *net = call->net;
+ unsigned int debug_id = call->debug_id;
+ bool zero;
+ int r, o;
- o = atomic_dec_return(&net->nr_outstanding_calls);
- if (o == 0)
- wake_up_var(&net->nr_outstanding_calls);
- }
+ zero = __refcount_dec_and_test(&call->ref, &r);
+ o = atomic_read(&net->nr_outstanding_calls);
+ trace_afs_call(debug_id, afs_call_trace_put, r - 1, o,
+ __builtin_return_address(0));
+ if (zero)
+ schedule_work(&call->free_work);
}
static struct afs_call *afs_get_call(struct afs_call *call,
@@ -667,7 +701,8 @@ static void afs_wake_up_call_waiter(struct sock *sk, struct rxrpc_call *rxcall,
}
/*
- * wake up an asynchronous call
+ * Wake up an asynchronous call. The caller is holding the call notify
+ * spinlock around this, so we can't call afs_put_call().
*/
static void afs_wake_up_async_call(struct sock *sk, struct rxrpc_call *rxcall,
unsigned long call_user_ID)
@@ -684,7 +719,7 @@ static void afs_wake_up_async_call(struct sock *sk, struct rxrpc_call *rxcall,
__builtin_return_address(0));
if (!queue_work(afs_async_calls, &call->async_work))
- afs_put_call(call);
+ afs_deferred_put_call(call);
}
}
diff --git a/fs/afs/super.c b/fs/afs/super.c
index 1dd6c7c9713b..3fa0c38d7a7c 100644
--- a/fs/afs/super.c
+++ b/fs/afs/super.c
@@ -670,7 +670,6 @@ static void afs_i_init_once(void *_vnode)
INIT_LIST_HEAD(&vnode->wb_keys);
INIT_LIST_HEAD(&vnode->pending_locks);
INIT_LIST_HEAD(&vnode->granted_locks);
- INIT_DELAYED_WORK(&vnode->lock_work, afs_lock_work);
INIT_LIST_HEAD(&vnode->cb_mmap_link);
seqlock_init(&vnode->cb_lock);
}
@@ -702,6 +701,7 @@ static struct inode *afs_alloc_inode(struct super_block *sb)
init_rwsem(&vnode->rmdir_lock);
INIT_WORK(&vnode->cb_work, afs_invalidate_mmap_work);
+ INIT_DELAYED_WORK(&vnode->lock_work, afs_lock_work);
_leave(" = %p", &vnode->netfs.inode);
return &vnode->netfs.inode;
diff --git a/fs/autofs/waitq.c b/fs/autofs/waitq.c
index efdc76732fae..49e8111aaf26 100644
--- a/fs/autofs/waitq.c
+++ b/fs/autofs/waitq.c
@@ -12,15 +12,13 @@
*/
static autofs_wqt_t autofs_next_wait_queue = 1;
-void autofs_catatonic_mode(struct autofs_sb_info *sbi)
+static void autofs_catatonic_mode_locked(struct autofs_sb_info *sbi)
{
struct autofs_wait_queue *wq, *nwq;
- mutex_lock(&sbi->wq_mutex);
- if (sbi->flags & AUTOFS_SBI_CATATONIC) {
- mutex_unlock(&sbi->wq_mutex);
+ lockdep_assert_held(&sbi->wq_mutex);
+ if (sbi->flags & AUTOFS_SBI_CATATONIC)
return;
- }
pr_debug("entering catatonic mode\n");
@@ -40,6 +38,21 @@ void autofs_catatonic_mode(struct autofs_sb_info *sbi)
fput(sbi->pipe); /* Close the pipe */
sbi->pipe = NULL;
sbi->pipefd = -1;
+}
+
+void autofs_catatonic_mode(struct autofs_sb_info *sbi)
+{
+ mutex_lock(&sbi->wq_mutex);
+ autofs_catatonic_mode_locked(sbi);
+ mutex_unlock(&sbi->wq_mutex);
+}
+
+static void autofs_catatonic_mode_for_pipe(struct autofs_sb_info *sbi,
+ struct file *pipe)
+{
+ mutex_lock(&sbi->wq_mutex);
+ if (sbi->pipe == pipe)
+ autofs_catatonic_mode_locked(sbi);
mutex_unlock(&sbi->wq_mutex);
}
@@ -170,7 +183,7 @@ static void autofs_notify_daemon(struct autofs_sb_info *sbi,
autofs_wait_release(sbi, wq->wait_queue_token, ret);
break;
default:
- autofs_catatonic_mode(sbi);
+ autofs_catatonic_mode_for_pipe(sbi, pipe);
break;
}
fput(pipe);
diff --git a/fs/btrfs/acl.c b/fs/btrfs/acl.c
index 548d6a5477b4..038dc7315be9 100644
--- a/fs/btrfs/acl.c
+++ b/fs/btrfs/acl.c
@@ -116,6 +116,9 @@ int btrfs_set_acl(struct user_namespace *mnt_userns, struct inode *inode,
int ret;
umode_t old_mode = inode->i_mode;
+ if (btrfs_root_readonly(BTRFS_I(inode)->root))
+ return -EROFS;
+
if (type == ACL_TYPE_ACCESS && acl) {
ret = posix_acl_update_mode(mnt_userns, inode,
&inode->i_mode, &acl);
diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c
index 1f9fbec887c0..a8f096b46b7b 100644
--- a/fs/btrfs/block-group.c
+++ b/fs/btrfs/block-group.c
@@ -2586,8 +2586,11 @@ void btrfs_create_pending_block_groups(struct btrfs_trans_handle *trans)
/* Already aborted the transaction if it failed. */
next:
btrfs_delayed_refs_rsv_release(fs_info, 1);
+
+ spin_lock(&fs_info->unused_bgs_lock);
list_del_init(&block_group->bg_list);
clear_bit(BLOCK_GROUP_FLAG_NEW, &block_group->runtime_flags);
+ spin_unlock(&fs_info->unused_bgs_lock);
}
btrfs_trans_release_chunk_metadata(trans);
}
diff --git a/fs/btrfs/block-rsv.c b/fs/btrfs/block-rsv.c
index a47c8b450996..a3fe0ef899a2 100644
--- a/fs/btrfs/block-rsv.c
+++ b/fs/btrfs/block-rsv.c
@@ -352,10 +352,25 @@ int btrfs_cond_migrate_bytes(struct btrfs_fs_info *fs_info,
void btrfs_update_global_block_rsv(struct btrfs_fs_info *fs_info)
{
struct btrfs_block_rsv *block_rsv = &fs_info->global_block_rsv;
- struct btrfs_space_info *sinfo = block_rsv->space_info;
+ struct btrfs_space_info *sinfo;
struct btrfs_root *root, *tmp;
- u64 num_bytes = btrfs_root_used(&fs_info->tree_root->root_item);
unsigned int min_items = 1;
+ u64 num_bytes;
+
+ /*
+ * A full read-only mount (rescue options) cannot start transactions,
+ * so the global reserve is never consumed. Mark it as full and skip
+ * the accounting.
+ */
+ if (btrfs_is_full_ro(fs_info)) {
+ spin_lock(&block_rsv->lock);
+ block_rsv->full = true;
+ spin_unlock(&block_rsv->lock);
+ return;
+ }
+
+ sinfo = block_rsv->space_info;
+ num_bytes = btrfs_root_used(&fs_info->tree_root->root_item);
/*
* The global block rsv is based on the size of the extent tree, the
diff --git a/fs/btrfs/ctree.c b/fs/btrfs/ctree.c
index 36f54f0f21d8..9e611f45b4b3 100644
--- a/fs/btrfs/ctree.c
+++ b/fs/btrfs/ctree.c
@@ -502,10 +502,16 @@ int btrfs_force_cow_block(struct btrfs_trans_handle *trans,
atomic_inc(&cow->refs);
rcu_assign_pointer(root->node, cow);
- btrfs_free_tree_block(trans, btrfs_root_id(root), buf,
- parent_start, last_ref);
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(root), buf,
+ parent_start, last_ref);
free_extent_buffer(buf);
add_root_to_dirty_list(root);
+ if (ret < 0) {
+ btrfs_tree_unlock(cow);
+ free_extent_buffer(cow);
+ btrfs_abort_transaction(trans, ret);
+ return ret;
+ }
} else {
WARN_ON(trans->transid != btrfs_header_generation(parent));
btrfs_tree_mod_log_insert_key(parent, parent_slot,
@@ -524,8 +530,14 @@ int btrfs_force_cow_block(struct btrfs_trans_handle *trans,
return ret;
}
}
- btrfs_free_tree_block(trans, btrfs_root_id(root), buf,
- parent_start, last_ref);
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(root), buf,
+ parent_start, last_ref);
+ if (ret < 0) {
+ btrfs_tree_unlock(cow);
+ free_extent_buffer(cow);
+ btrfs_abort_transaction(trans, ret);
+ return ret;
+ }
}
trace_btrfs_cow_block(root, buf, cow);
@@ -999,9 +1011,13 @@ static noinline int balance_level(struct btrfs_trans_handle *trans,
free_extent_buffer(mid);
root_sub_used(root, mid->len);
- btrfs_free_tree_block(trans, btrfs_root_id(root), mid, 0, 1);
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(root), mid, 0, 1);
/* once for the root ptr */
free_extent_buffer_stale(mid);
+ if (ret < 0) {
+ btrfs_abort_transaction(trans, ret);
+ goto enospc;
+ }
return 0;
}
if (btrfs_header_nritems(mid) >
@@ -1058,10 +1074,14 @@ static noinline int balance_level(struct btrfs_trans_handle *trans,
btrfs_tree_unlock(right);
del_ptr(root, path, level + 1, pslot + 1);
root_sub_used(root, right->len);
- btrfs_free_tree_block(trans, btrfs_root_id(root), right,
- 0, 1);
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(root),
+ right, 0, 1);
free_extent_buffer_stale(right);
right = NULL;
+ if (ret < 0) {
+ btrfs_abort_transaction(trans, ret);
+ goto enospc;
+ }
} else {
struct btrfs_disk_key right_key;
btrfs_node_key(right, &right_key, 0);
@@ -1107,9 +1127,15 @@ static noinline int balance_level(struct btrfs_trans_handle *trans,
btrfs_tree_unlock(mid);
del_ptr(root, path, level + 1, pslot);
root_sub_used(root, mid->len);
- btrfs_free_tree_block(trans, btrfs_root_id(root), mid, 0, 1);
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(root), mid, 0, 1);
free_extent_buffer_stale(mid);
mid = NULL;
+ if (ret < 0) {
+ path->locks[level] = 0;
+ path->nodes[level] = NULL;
+ btrfs_abort_transaction(trans, ret);
+ goto enospc;
+ }
} else {
/* update the parent key to reflect our changes */
struct btrfs_disk_key mid_key;
@@ -4314,11 +4340,13 @@ static void del_ptr(struct btrfs_root *root, struct btrfs_path *path,
* The path must have already been setup for deleting the leaf, including
* all the proper balancing. path->nodes[1] must be locked.
*/
-static noinline void btrfs_del_leaf(struct btrfs_trans_handle *trans,
- struct btrfs_root *root,
- struct btrfs_path *path,
- struct extent_buffer *leaf)
+static noinline int btrfs_del_leaf(struct btrfs_trans_handle *trans,
+ struct btrfs_root *root,
+ struct btrfs_path *path,
+ struct extent_buffer *leaf)
{
+ int ret;
+
WARN_ON(btrfs_header_generation(leaf) != trans->transid);
del_ptr(root, path, 1, path->slots[1]);
@@ -4331,8 +4359,12 @@ static noinline void btrfs_del_leaf(struct btrfs_trans_handle *trans,
root_sub_used(root, leaf->len);
atomic_inc(&leaf->refs);
- btrfs_free_tree_block(trans, btrfs_root_id(root), leaf, 0, 1);
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(root), leaf, 0, 1);
free_extent_buffer_stale(leaf);
+ if (ret < 0)
+ btrfs_abort_transaction(trans, ret);
+
+ return ret;
}
/*
* delete the item at the leaf level in path. If that empties
@@ -4387,7 +4419,9 @@ int btrfs_del_items(struct btrfs_trans_handle *trans, struct btrfs_root *root,
btrfs_set_header_level(leaf, 0);
} else {
btrfs_clean_tree_block(leaf);
- btrfs_del_leaf(trans, root, path, leaf);
+ ret = btrfs_del_leaf(trans, root, path, leaf);
+ if (ret < 0)
+ return ret;
}
} else {
int used = leaf_space_used(leaf, 0, nritems);
@@ -4448,9 +4482,8 @@ int btrfs_del_items(struct btrfs_trans_handle *trans, struct btrfs_root *root,
if (btrfs_header_nritems(leaf) == 0) {
path->slots[1] = slot;
- btrfs_del_leaf(trans, root, path, leaf);
+ ret = btrfs_del_leaf(trans, root, path, leaf);
free_extent_buffer(leaf);
- ret = 0;
} else {
/* if we're still in the path, make sure
* we're dirty. Otherwise, one of the
diff --git a/fs/btrfs/ctree.h b/fs/btrfs/ctree.h
index 16795e91aee2..c1cfc03fefbe 100644
--- a/fs/btrfs/ctree.h
+++ b/fs/btrfs/ctree.h
@@ -1626,6 +1626,12 @@ enum {
BTRFS_MOUNT_IGNOREDATACSUMS = (1UL << 30),
};
+/* These mount options require a full read-only fs, no new transaction is allowed. */
+#define BTRFS_MOUNT_FULL_RO_MASK \
+ (BTRFS_MOUNT_NOLOGREPLAY | \
+ BTRFS_MOUNT_IGNOREBADROOTS | \
+ BTRFS_MOUNT_IGNOREDATACSUMS)
+
#define BTRFS_DEFAULT_COMMIT_INTERVAL (30)
#define BTRFS_DEFAULT_MAX_INLINE (2048)
@@ -1635,6 +1641,15 @@ enum {
#define btrfs_test_opt(fs_info, opt) ((fs_info)->mount_opt & \
BTRFS_MOUNT_##opt)
+static inline bool btrfs_is_full_ro(const struct btrfs_fs_info *fs_info)
+{
+ if (!sb_rdonly(fs_info->sb))
+ return false;
+ if (unlikely(fs_info->mount_opt & BTRFS_MOUNT_FULL_RO_MASK))
+ return true;
+ return false;
+}
+
#define btrfs_set_and_info(fs_info, opt, fmt, args...) \
do { \
if (!btrfs_test_opt(fs_info, opt)) \
@@ -2888,10 +2903,10 @@ struct extent_buffer *btrfs_alloc_tree_block(struct btrfs_trans_handle *trans,
int level, u64 hint,
u64 empty_size,
enum btrfs_lock_nesting nest);
-void btrfs_free_tree_block(struct btrfs_trans_handle *trans,
- u64 root_id,
- struct extent_buffer *buf,
- u64 parent, int last_ref);
+int btrfs_free_tree_block(struct btrfs_trans_handle *trans,
+ u64 root_id,
+ struct extent_buffer *buf,
+ u64 parent, int last_ref);
int btrfs_alloc_reserved_file_extent(struct btrfs_trans_handle *trans,
struct btrfs_root *root, u64 owner,
u64 offset, u64 ram_bytes,
diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c
index 9c856a73d533..f9c43aadc78a 100644
--- a/fs/btrfs/dev-replace.c
+++ b/fs/btrfs/dev-replace.c
@@ -520,6 +520,7 @@ static int mark_block_group_to_copy(struct btrfs_fs_info *fs_info,
path->reada = READA_FORWARD;
path->search_commit_root = 1;
path->skip_locking = 1;
+ path->need_commit_sem = 1;
key.objectid = src_dev->devid;
key.type = BTRFS_DEV_EXTENT_KEY;
diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c
index 203ff9bbad43..a71d773d0c3b 100644
--- a/fs/btrfs/disk-io.c
+++ b/fs/btrfs/disk-io.c
@@ -3798,6 +3798,10 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device
}
}
+ /* If the fs has any rescue options, no transaction is allowed. */
+ if (btrfs_is_full_ro(fs_info))
+ set_bit(BTRFS_FS_STATE_ERROR, &fs_info->fs_state);
+
fs_info->fs_root = btrfs_get_fs_root(fs_info, BTRFS_FS_TREE_OBJECTID, true);
if (IS_ERR(fs_info->fs_root)) {
err = PTR_ERR(fs_info->fs_root);
@@ -5121,6 +5125,7 @@ static int btrfs_destroy_marked_extents(struct btrfs_fs_info *fs_info,
free_extent_buffer_stale(eb);
}
}
+ extent_io_tree_release(dirty_pages);
return ret;
}
diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c
index 9a3c304d76b4..fb3306ff616b 100644
--- a/fs/btrfs/extent-tree.c
+++ b/fs/btrfs/extent-tree.c
@@ -1707,6 +1707,7 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans,
struct btrfs_delayed_extent_op *extent_op,
int insert_reserved)
{
+ struct btrfs_fs_info *fs_info = trans->fs_info;
int ret = 0;
if (TRANS_ABORTED(trans)) {
@@ -1716,19 +1717,22 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans,
}
if (node->type == BTRFS_TREE_BLOCK_REF_KEY ||
- node->type == BTRFS_SHARED_BLOCK_REF_KEY)
+ node->type == BTRFS_SHARED_BLOCK_REF_KEY) {
ret = run_delayed_tree_ref(trans, node, extent_op,
insert_reserved);
- else if (node->type == BTRFS_EXTENT_DATA_REF_KEY ||
- node->type == BTRFS_SHARED_DATA_REF_KEY)
+ } else if (node->type == BTRFS_EXTENT_DATA_REF_KEY ||
+ node->type == BTRFS_SHARED_DATA_REF_KEY) {
ret = run_delayed_data_ref(trans, node, extent_op,
insert_reserved);
- else
- BUG();
+ } else {
+ ret = -EUCLEAN;
+ btrfs_err(fs_info, "unexpected delayed ref node type: %u", node->type);
+ }
+
if (ret && insert_reserved)
btrfs_pin_extent(trans, node->bytenr, node->num_bytes, 1);
if (ret < 0)
- btrfs_err(trans->fs_info,
+ btrfs_err(fs_info,
"failed to run delayed ref for logical %llu num_bytes %llu type %u action %u ref_mod %d: %d",
node->bytenr, node->num_bytes, node->type,
node->action, node->ref_mod, ret);
@@ -2834,7 +2838,15 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans)
block_group->length,
&trimmed);
+ /*
+ * Not strictly necessary to lock, as the block_group should be
+ * read-only from btrfs_delete_unused_bgs().
+ */
+ ASSERT(block_group->ro);
+ spin_lock(&fs_info->unused_bgs_lock);
list_del_init(&block_group->bg_list);
+ spin_unlock(&fs_info->unused_bgs_lock);
+
btrfs_unfreeze_block_group(block_group);
btrfs_put_block_group(block_group);
@@ -3280,10 +3292,10 @@ static noinline int check_ref_cleanup(struct btrfs_trans_handle *trans,
return 0;
}
-void btrfs_free_tree_block(struct btrfs_trans_handle *trans,
- u64 root_id,
- struct extent_buffer *buf,
- u64 parent, int last_ref)
+int btrfs_free_tree_block(struct btrfs_trans_handle *trans,
+ u64 root_id,
+ struct extent_buffer *buf,
+ u64 parent, int last_ref)
{
struct btrfs_fs_info *fs_info = trans->fs_info;
struct btrfs_ref generic_ref = { 0 };
@@ -3297,7 +3309,8 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans,
if (root_id != BTRFS_TREE_LOG_OBJECTID) {
btrfs_ref_tree_mod(fs_info, &generic_ref);
ret = btrfs_add_delayed_tree_ref(trans, &generic_ref, NULL);
- BUG_ON(ret); /* -ENOMEM */
+ if (ret < 0)
+ return ret;
}
if (last_ref && btrfs_header_generation(buf) == trans->transid) {
@@ -3361,6 +3374,7 @@ void btrfs_free_tree_block(struct btrfs_trans_handle *trans,
*/
clear_bit(EXTENT_BUFFER_CORRUPT, &buf->bflags);
}
+ return 0;
}
/* Can return -ENOMEM */
@@ -5533,7 +5547,7 @@ static noinline int walk_up_proc(struct btrfs_trans_handle *trans,
struct walk_control *wc)
{
struct btrfs_fs_info *fs_info = root->fs_info;
- int ret;
+ int ret = 0;
int level = wc->level;
struct extent_buffer *eb = path->nodes[level];
u64 parent = 0;
@@ -5628,12 +5642,14 @@ static noinline int walk_up_proc(struct btrfs_trans_handle *trans,
goto owner_mismatch;
}
- btrfs_free_tree_block(trans, btrfs_root_id(root), eb, parent,
- wc->refs[level] == 1);
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(root), eb, parent,
+ wc->refs[level] == 1);
+ if (ret < 0)
+ btrfs_abort_transaction(trans, ret);
out:
wc->refs[level] = 0;
wc->flags[level] = 0;
- return 0;
+ return ret;
owner_mismatch:
btrfs_err_rl(fs_info, "unexpected tree owner, have %llu expect %llu",
diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c
index aeea8930b973..c2ad6cafb33a 100644
--- a/fs/btrfs/extent_io.c
+++ b/fs/btrfs/extent_io.c
@@ -557,9 +557,15 @@ static int repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 start,
struct btrfs_io_context *bioc = NULL;
int ret = 0;
- ASSERT(!(fs_info->sb->s_flags & SB_RDONLY));
BUG_ON(!mirror_num);
+ /*
+ * The fs either mounted RO or hit critical errors, no need
+ * to continue repairing.
+ */
+ if (unlikely(sb_rdonly(fs_info->sb)))
+ return 0;
+
if (btrfs_repair_one_zone(fs_info, logical))
return 0;
@@ -3496,6 +3502,7 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo,
struct fiemap_cache *cache,
u64 offset, u64 phys, u64 len, u32 flags)
{
+ u64 cache_end;
int ret = 0;
/* Set at the end of extent_fiemap(). */
@@ -3505,15 +3512,102 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo,
goto assign;
/*
- * Sanity check, extent_fiemap() should have ensured that new
- * fiemap extent won't overlap with cached one.
- * Not recoverable.
+ * When iterating the extents of the inode, at extent_fiemap(), we may
+ * find an extent that starts at an offset behind the end offset of the
+ * previous extent we processed. This happens if fiemap is called
+ * without FIEMAP_FLAG_SYNC and there are ordered extents completing
+ * while we call btrfs_next_leaf() (through fiemap_next_leaf_item()).
*
- * NOTE: Physical address can overlap, due to compression
+ * For example we are in leaf X processing its last item, which is the
+ * file extent item for file range [512K, 1M[, and after
+ * btrfs_next_leaf() releases the path, there's an ordered extent that
+ * completes for the file range [768K, 2M[, and that results in trimming
+ * the file extent item so that it now corresponds to the file range
+ * [512K, 768K[ and a new file extent item is inserted for the file
+ * range [768K, 2M[, which may end up as the last item of leaf X or as
+ * the first item of the next leaf - in either case btrfs_next_leaf()
+ * will leave us with a path pointing to the new extent item, for the
+ * file range [768K, 2M[, since that's the first key that follows the
+ * last one we processed. So in order not to report overlapping extents
+ * to user space, we trim the length of the previously cached extent and
+ * emit it.
+ *
+ * Upon calling btrfs_next_leaf() we may also find an extent with an
+ * offset smaller than or equals to cache->offset, and this happens
+ * when we had a hole or prealloc extent with several delalloc ranges in
+ * it, but after btrfs_next_leaf() released the path, delalloc was
+ * flushed and the resulting ordered extents were completed, so we can
+ * now have found a file extent item for an offset that is smaller than
+ * or equals to what we have in cache->offset. We deal with this as
+ * described below.
*/
- if (cache->offset + cache->len > offset) {
- WARN_ON(1);
- return -EINVAL;
+ cache_end = cache->offset + cache->len;
+ if (cache_end > offset) {
+ if (offset == cache->offset) {
+ /*
+ * We cached a dealloc range (found in the io tree) for
+ * a hole or prealloc extent and we have now found a
+ * file extent item for the same offset. What we have
+ * now is more recent and up to date, so discard what
+ * we had in the cache and use what we have just found.
+ */
+ goto assign;
+ } else if (offset > cache->offset) {
+ /*
+ * The extent range we previously found ends after the
+ * offset of the file extent item we found and that
+ * offset falls somewhere in the middle of that previous
+ * extent range. So adjust the range we previously found
+ * to end at the offset of the file extent item we have
+ * just found, since this extent is more up to date.
+ * Emit that adjusted range and cache the file extent
+ * item we have just found. This corresponds to the case
+ * where a previously found file extent item was split
+ * due to an ordered extent completing.
+ */
+ cache->len = offset - cache->offset;
+ goto emit;
+ } else {
+ const u64 range_end = offset + len;
+
+ /*
+ * The offset of the file extent item we have just found
+ * is behind the cached offset. This means we were
+ * processing a hole or prealloc extent for which we
+ * have found delalloc ranges (in the io tree), so what
+ * we have in the cache is the last delalloc range we
+ * found while the file extent item we found can be
+ * either for a whole delalloc range we previously
+ * emmitted or only a part of that range.
+ *
+ * We have two cases here:
+ *
+ * 1) The file extent item's range ends at or behind the
+ * cached extent's end. In this case just ignore the
+ * current file extent item because we don't want to
+ * overlap with previous ranges that may have been
+ * emmitted already;
+ *
+ * 2) The file extent item starts behind the currently
+ * cached extent but its end offset goes beyond the
+ * end offset of the cached extent. We don't want to
+ * overlap with a previous range that may have been
+ * emmitted already, so we emit the currently cached
+ * extent and then partially store the current file
+ * extent item's range in the cache, for the subrange
+ * going the cached extent's end to the end of the
+ * file extent item.
+ */
+ if (range_end <= cache_end)
+ return 0;
+
+ if (!(flags & (FIEMAP_EXTENT_ENCODED | FIEMAP_EXTENT_DELALLOC)))
+ phys += cache_end - offset;
+
+ offset = cache_end;
+ len = range_end - cache_end;
+ goto emit;
+ }
}
/*
@@ -3533,6 +3627,7 @@ static int emit_fiemap_extent(struct fiemap_extent_info *fieinfo,
return 0;
}
+emit:
/* Not mergeable, need to submit cached one */
ret = fiemap_fill_next_extent(fieinfo, cache->offset, cache->phys,
cache->len, cache->flags);
@@ -3706,15 +3801,33 @@ static int fiemap_process_hole(struct btrfs_inode *inode,
* it beyond i_size.
*/
while (cur_offset < end && cur_offset < i_size) {
+ struct extent_state *cached_state = NULL;
u64 delalloc_start;
u64 delalloc_end;
u64 prealloc_start;
+ u64 lockstart;
+ u64 lockend;
u64 prealloc_len = 0;
bool delalloc;
+ lockstart = round_down(cur_offset, inode->root->fs_info->sectorsize);
+ lockend = round_up(end, inode->root->fs_info->sectorsize);
+
+ /*
+ * We are only locking for the delalloc range because that's the
+ * only thing that can change here. With fiemap we have a lock
+ * on the inode, so no buffered or direct writes can happen.
+ *
+ * However mmaps and normal page writeback will cause this to
+ * change arbitrarily. We have to lock the extent lock here to
+ * make sure that nobody messes with the tree while we're doing
+ * btrfs_find_delalloc_in_range.
+ */
+ lock_extent(&inode->io_tree, lockstart, lockend, &cached_state);
delalloc = btrfs_find_delalloc_in_range(inode, cur_offset, end,
&delalloc_start,
&delalloc_end);
+ unlock_extent(&inode->io_tree, lockstart, lockend, &cached_state);
if (!delalloc)
break;
@@ -3884,7 +3997,6 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
u64 start, u64 len)
{
const u64 ino = btrfs_ino(inode);
- struct extent_state *cached_state = NULL;
struct btrfs_path *path;
struct btrfs_root *root = inode->root;
struct fiemap_cache cache = { 0 };
@@ -3893,8 +4005,9 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
struct ulist *tmp_ulist;
u64 last_extent_end;
u64 prev_extent_end;
- u64 lockstart;
- u64 lockend;
+ u64 range_start;
+ u64 range_end;
+ const u64 sectorsize = inode->root->fs_info->sectorsize;
bool stopped = false;
int ret;
@@ -3907,12 +4020,11 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
goto out;
}
- lockstart = round_down(start, root->fs_info->sectorsize);
- lockend = round_up(start + len, root->fs_info->sectorsize);
- prev_extent_end = lockstart;
+ range_start = round_down(start, sectorsize);
+ range_end = round_up(start + len, sectorsize);
+ prev_extent_end = range_start;
btrfs_inode_lock(&inode->vfs_inode, BTRFS_ILOCK_SHARED);
- lock_extent(&inode->io_tree, lockstart, lockend, &cached_state);
ret = fiemap_find_last_extent_offset(inode, path, &last_extent_end);
if (ret < 0)
@@ -3920,7 +4032,7 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
btrfs_release_path(path);
path->reada = READA_FORWARD;
- ret = fiemap_search_slot(inode, path, lockstart);
+ ret = fiemap_search_slot(inode, path, range_start);
if (ret < 0) {
goto out_unlock;
} else if (ret > 0) {
@@ -3932,7 +4044,7 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
goto check_eof_delalloc;
}
- while (prev_extent_end < lockend) {
+ while (prev_extent_end < range_end) {
struct extent_buffer *leaf = path->nodes[0];
struct btrfs_file_extent_item *ei;
struct btrfs_key key;
@@ -3955,17 +4067,17 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
* The first iteration can leave us at an extent item that ends
* before our range's start. Move to the next item.
*/
- if (extent_end <= lockstart)
+ if (extent_end <= range_start)
goto next_item;
/* We have in implicit hole (NO_HOLES feature enabled). */
if (prev_extent_end < key.offset) {
- const u64 range_end = min(key.offset, lockend) - 1;
+ const u64 hole_end = min(key.offset, range_end) - 1;
ret = fiemap_process_hole(inode, fieinfo, &cache,
backref_cache, 0, 0, 0,
roots, tmp_ulist,
- prev_extent_end, range_end);
+ prev_extent_end, hole_end);
if (ret < 0) {
goto out_unlock;
} else if (ret > 0) {
@@ -3975,7 +4087,7 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
}
/* We've reached the end of the fiemap range, stop. */
- if (key.offset >= lockend) {
+ if (key.offset >= range_end) {
stopped = true;
break;
}
@@ -4070,28 +4182,40 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
btrfs_free_path(path);
path = NULL;
- if (!stopped && prev_extent_end < lockend) {
+ if (!stopped && prev_extent_end < range_end) {
ret = fiemap_process_hole(inode, fieinfo, &cache, backref_cache,
0, 0, 0, roots, tmp_ulist,
- prev_extent_end, lockend - 1);
+ prev_extent_end, range_end - 1);
if (ret < 0)
goto out_unlock;
- prev_extent_end = lockend;
+ prev_extent_end = range_end;
}
if (cache.cached && cache.offset + cache.len >= last_extent_end) {
const u64 i_size = i_size_read(&inode->vfs_inode);
if (prev_extent_end < i_size) {
+ struct extent_state *cached_state = NULL;
u64 delalloc_start;
u64 delalloc_end;
+ u64 lockstart;
+ u64 lockend;
bool delalloc;
+ lockstart = round_down(prev_extent_end, sectorsize);
+ lockend = round_up(i_size, sectorsize);
+
+ /*
+ * See the comment in fiemap_process_hole as to why
+ * we're doing the locking here.
+ */
+ lock_extent(&inode->io_tree, lockstart, lockend, &cached_state);
delalloc = btrfs_find_delalloc_in_range(inode,
prev_extent_end,
i_size - 1,
&delalloc_start,
&delalloc_end);
+ unlock_extent(&inode->io_tree, lockstart, lockend, &cached_state);
if (!delalloc)
cache.flags |= FIEMAP_EXTENT_LAST;
} else {
@@ -4102,7 +4226,6 @@ int extent_fiemap(struct btrfs_inode *inode, struct fiemap_extent_info *fieinfo,
ret = emit_last_fiemap_cache(fieinfo, &cache);
out_unlock:
- unlock_extent(&inode->io_tree, lockstart, lockend, &cached_state);
btrfs_inode_unlock(&inode->vfs_inode, BTRFS_ILOCK_SHARED);
out:
kfree(backref_cache);
diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index 0970ec776ff8..d34407035b6a 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -949,69 +949,63 @@ static noinline int prepare_pages(struct inode *inode, struct page **pages,
/*
* This function locks the extent and properly waits for data=ordered extents
- * to finish before allowing the pages to be modified if need.
+ * to finish before allowing the pages to be modified.
*
* The return value:
- * 1 - the extent is locked
- * 0 - the extent is not locked, and everything is OK
+ * 0 - the extent is locked
* -EAGAIN - need re-prepare the pages
* the other < 0 number - Something wrong happens
*/
static noinline int
-lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct page **pages,
- size_t num_pages, loff_t pos,
- size_t write_bytes,
- u64 *lockstart, u64 *lockend, bool nowait,
- struct extent_state **cached_state)
+lock_and_cleanup_extent(struct btrfs_inode *inode, struct page **pages,
+ size_t num_pages, loff_t pos,
+ size_t write_bytes,
+ u64 *lockstart, u64 *lockend, bool nowait,
+ struct extent_state **cached_state)
{
struct btrfs_fs_info *fs_info = inode->root->fs_info;
+ struct btrfs_ordered_extent *ordered;
u64 start_pos;
u64 last_pos;
int i;
- int ret = 0;
start_pos = round_down(pos, fs_info->sectorsize);
last_pos = round_up(pos + write_bytes, fs_info->sectorsize) - 1;
- if (start_pos < inode->vfs_inode.i_size) {
- struct btrfs_ordered_extent *ordered;
-
- if (nowait) {
- if (!try_lock_extent(&inode->io_tree, start_pos, last_pos)) {
- for (i = 0; i < num_pages; i++) {
- unlock_page(pages[i]);
- put_page(pages[i]);
- pages[i] = NULL;
- }
-
- return -EAGAIN;
- }
- } else {
- lock_extent(&inode->io_tree, start_pos, last_pos, cached_state);
- }
-
- ordered = btrfs_lookup_ordered_range(inode, start_pos,
- last_pos - start_pos + 1);
- if (ordered &&
- ordered->file_offset + ordered->num_bytes > start_pos &&
- ordered->file_offset <= last_pos) {
- unlock_extent(&inode->io_tree, start_pos, last_pos,
- cached_state);
+ if (nowait) {
+ if (!try_lock_extent(&inode->io_tree, start_pos, last_pos)) {
for (i = 0; i < num_pages; i++) {
unlock_page(pages[i]);
put_page(pages[i]);
+ pages[i] = NULL;
}
- btrfs_start_ordered_extent(ordered, 1);
- btrfs_put_ordered_extent(ordered);
+
return -EAGAIN;
}
- if (ordered)
- btrfs_put_ordered_extent(ordered);
+ } else {
+ lock_extent(&inode->io_tree, start_pos, last_pos, cached_state);
+ }
- *lockstart = start_pos;
- *lockend = last_pos;
- ret = 1;
+ ordered = btrfs_lookup_ordered_range(inode, start_pos,
+ last_pos - start_pos + 1);
+ if (ordered &&
+ ordered->file_offset + ordered->num_bytes > start_pos &&
+ ordered->file_offset <= last_pos) {
+ unlock_extent(&inode->io_tree, start_pos, last_pos,
+ cached_state);
+ for (i = 0; i < num_pages; i++) {
+ unlock_page(pages[i]);
+ put_page(pages[i]);
+ }
+ btrfs_start_ordered_extent(ordered, 1);
+ btrfs_put_ordered_extent(ordered);
+ return -EAGAIN;
}
+ if (ordered)
+ btrfs_put_ordered_extent(ordered);
+
+ *lockstart = start_pos;
+ *lockend = last_pos;
/*
* We should be called after prepare_pages() which should have locked
@@ -1020,7 +1014,7 @@ lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct page **pages,
for (i = 0; i < num_pages; i++)
WARN_ON(!PageLocked(pages[i]));
- return ret;
+ return 0;
}
/*
@@ -1214,7 +1208,6 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb,
size_t copied;
size_t dirty_sectors;
size_t num_sectors;
- int extents_locked;
/*
* Fault pages before locking them in prepare_pages
@@ -1298,17 +1291,15 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb,
break;
}
- extents_locked = lock_and_cleanup_extent_if_need(
- BTRFS_I(inode), pages,
- num_pages, pos, write_bytes, &lockstart,
- &lockend, nowait, &cached_state);
- if (extents_locked < 0) {
- if (!nowait && extents_locked == -EAGAIN)
+ ret = lock_and_cleanup_extent(BTRFS_I(inode), pages,
+ num_pages, pos, write_bytes, &lockstart,
+ &lockend, nowait, &cached_state);
+ if (ret < 0) {
+ if (!nowait && ret == -EAGAIN)
goto again;
btrfs_delalloc_release_extents(BTRFS_I(inode),
reserve_bytes);
- ret = extents_locked;
break;
}
@@ -1361,18 +1352,8 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb,
dirty_pages, pos, copied,
&cached_state, only_release_metadata);
- /*
- * If we have not locked the extent range, because the range's
- * start offset is >= i_size, we might still have a non-NULL
- * cached extent state, acquired while marking the extent range
- * as delalloc through btrfs_dirty_pages(). Therefore free any
- * possible cached extent state to avoid a memory leak.
- */
- if (extents_locked)
- unlock_extent(&BTRFS_I(inode)->io_tree, lockstart,
- lockend, &cached_state);
- else
- free_extent_state(cached_state);
+ unlock_extent(&BTRFS_I(inode)->io_tree, lockstart, lockend,
+ &cached_state);
btrfs_delalloc_release_extents(BTRFS_I(inode), reserve_bytes);
if (ret) {
diff --git a/fs/btrfs/free-space-tree.c b/fs/btrfs/free-space-tree.c
index 9e966e8876f5..8c6b21531cf5 100644
--- a/fs/btrfs/free-space-tree.c
+++ b/fs/btrfs/free-space-tree.c
@@ -1292,10 +1292,14 @@ int btrfs_delete_free_space_tree(struct btrfs_fs_info *fs_info)
btrfs_tree_lock(free_space_root->node);
btrfs_clean_tree_block(free_space_root->node);
btrfs_tree_unlock(free_space_root->node);
- btrfs_free_tree_block(trans, btrfs_root_id(free_space_root),
- free_space_root->node, 0, 1);
-
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(free_space_root),
+ free_space_root->node, 0, 1);
btrfs_put_root(free_space_root);
+ if (ret < 0) {
+ btrfs_abort_transaction(trans, ret);
+ btrfs_end_transaction(trans);
+ return ret;
+ }
return btrfs_commit_transaction(trans);
diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c
index e53b42a047e8..976aad24ae0a 100644
--- a/fs/btrfs/inode.c
+++ b/fs/btrfs/inode.c
@@ -1265,6 +1265,17 @@ static noinline int cow_file_range(struct btrfs_inode *inode,
alloc_hint = get_extent_allocation_hint(inode, start, num_bytes);
+ /*
+ * We're not doing compressed IO, don't unlock the first page (which
+ * the caller expects to stay locked), don't clear any dirty bits and
+ * don't set any writeback bits.
+ *
+ * Do set the Ordered (Private2) bit so we know this page was properly
+ * setup for writepage.
+ */
+ page_ops = unlock ? PAGE_UNLOCK : 0;
+ page_ops |= PAGE_SET_ORDERED;
+
/*
* Relocation relies on the relocated extents to have exactly the same
* size as the original extents. Normally writeback for relocation data
@@ -1335,21 +1346,6 @@ static noinline int cow_file_range(struct btrfs_inode *inode,
btrfs_dec_block_group_reservations(fs_info, ins.objectid);
- /*
- * We're not doing compressed IO, don't unlock the first page
- * (which the caller expects to stay locked), don't clear any
- * dirty bits and don't set any writeback bits
- *
- * Do set the Ordered (Private2) bit so we know this page was
- * properly setup for writepage.
- */
- page_ops = unlock ? PAGE_UNLOCK : 0;
- page_ops |= PAGE_SET_ORDERED;
-
- extent_clear_unlock_delalloc(inode, start, start + ram_size - 1,
- locked_page,
- EXTENT_LOCKED | EXTENT_DELALLOC,
- page_ops);
if (num_bytes < cur_alloc_size)
num_bytes = 0;
else
@@ -1366,6 +1362,8 @@ static noinline int cow_file_range(struct btrfs_inode *inode,
if (ret)
goto out_unlock;
}
+ extent_clear_unlock_delalloc(inode, orig_start, end, locked_page,
+ EXTENT_LOCKED | EXTENT_DELALLOC, page_ops);
out:
return ret;
@@ -1380,10 +1378,15 @@ static noinline int cow_file_range(struct btrfs_inode *inode,
* caller to write out the successfully allocated region and retry.
*/
if (done_offset && ret == -EAGAIN) {
- if (orig_start < start)
+ if (orig_start < start) {
+ extent_clear_unlock_delalloc(inode, orig_start, start - 1,
+ locked_page,
+ EXTENT_LOCKED | EXTENT_DELALLOC,
+ page_ops);
*done_offset = start - 1;
- else
+ } else {
*done_offset = start;
+ }
return ret;
} else if (ret == -EAGAIN) {
/* Convert to -ENOSPC since the caller cannot retry. */
@@ -1399,29 +1402,31 @@ static noinline int cow_file_range(struct btrfs_inode *inode,
* We process each region below.
*/
- clear_bits = EXTENT_LOCKED | EXTENT_DELALLOC | EXTENT_DELALLOC_NEW |
- EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV;
- page_ops = PAGE_UNLOCK | PAGE_START_WRITEBACK | PAGE_END_WRITEBACK;
-
/*
* For the range (1). We have already instantiated the ordered extents
* for this region. They are cleaned up by
* btrfs_cleanup_ordered_extents() in e.g,
- * btrfs_run_delalloc_range(). EXTENT_LOCKED | EXTENT_DELALLOC are
- * already cleared in the above loop. And, EXTENT_DELALLOC_NEW |
- * EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV are handled by the cleanup
- * function.
+ * btrfs_run_delalloc_range().
+ * EXTENT_DELALLOC_NEW | EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV
+ * are also handled by the cleanup function.
*
- * However, in case of unlock == 0, we still need to unlock the pages
- * (except @locked_page) to ensure all the pages are unlocked.
+ * So here we only clear EXTENT_LOCKED and EXTENT_DELALLOC flag, and
+ * finish the writeback of the involved folios, which will be never submitted.
*/
- if (!unlock && orig_start < start) {
+ if (orig_start < start) {
+ clear_bits = EXTENT_LOCKED | EXTENT_DELALLOC;
+ page_ops = PAGE_UNLOCK | PAGE_START_WRITEBACK | PAGE_END_WRITEBACK;
+
if (!locked_page)
mapping_set_error(inode->vfs_inode.i_mapping, ret);
extent_clear_unlock_delalloc(inode, orig_start, start - 1,
- locked_page, 0, page_ops);
+ locked_page, clear_bits, page_ops);
}
+ clear_bits = EXTENT_LOCKED | EXTENT_DELALLOC | EXTENT_DELALLOC_NEW |
+ EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV;
+ page_ops = PAGE_UNLOCK | PAGE_START_WRITEBACK | PAGE_END_WRITEBACK;
+
/*
* For the range (2). If we reserved an extent for our delalloc range
* (or a subrange) and failed to create the respective ordered extent,
@@ -7690,17 +7695,24 @@ static int btrfs_get_blocks_direct_write(struct extent_map **map,
len, orig_block_len,
ram_bytes, type);
btrfs_dec_nocow_writers(bg);
- if (type == BTRFS_ORDERED_PREALLOC) {
- free_extent_map(em);
- *map = em2;
- em = em2;
- }
-
if (IS_ERR(em2)) {
ret = PTR_ERR(em2);
+ free_extent_map(em);
+ *map = NULL;
goto out;
}
+ /*
+ * True NOCOW writes don't need to create a new extent map,
+ * while PREALLOC writes must replace the existing one.
+ */
+ if (em2) {
+ ASSERT(type == BTRFS_ORDERED_PREALLOC);
+ free_extent_map(em);
+ *map = em2;
+ em = em2;
+ }
+
dio_data->nocow_done = true;
} else {
/* Our caller expects us to free the input extent map. */
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index e278da90888e..46534e5d40bc 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -696,6 +696,8 @@ static noinline int create_subvol(struct user_namespace *mnt_userns,
ret = btrfs_insert_root(trans, fs_info->tree_root, &key,
root_item);
if (ret) {
+ int ret2;
+
/*
* Since we don't abort the transaction in this case, free the
* tree block so that we don't leak space and leave the
@@ -706,7 +708,9 @@ static noinline int create_subvol(struct user_namespace *mnt_userns,
btrfs_tree_lock(leaf);
btrfs_clean_tree_block(leaf);
btrfs_tree_unlock(leaf);
- btrfs_free_tree_block(trans, objectid, leaf, 0, 1);
+ ret2 = btrfs_free_tree_block(trans, objectid, leaf, 0, 1);
+ if (ret2 < 0)
+ btrfs_abort_transaction(trans, ret2);
free_extent_buffer(leaf);
goto out;
}
@@ -1626,26 +1630,21 @@ static_assert(IS_ALIGNED(CLUSTER_SIZE, PAGE_SIZE));
*
* - Extent bits are locked
*/
-static int defrag_one_locked_target(struct btrfs_inode *inode,
- struct defrag_target_range *target,
- struct page **pages, int nr_pages,
- struct extent_state **cached_state)
+static void defrag_one_locked_target(struct btrfs_inode *inode,
+ struct defrag_target_range *target,
+ struct page **pages, int nr_pages,
+ struct extent_state **cached_state)
{
struct btrfs_fs_info *fs_info = inode->root->fs_info;
- struct extent_changeset *data_reserved = NULL;
const u64 start = target->start;
const u64 len = target->len;
unsigned long last_index = (start + len - 1) >> PAGE_SHIFT;
unsigned long start_index = start >> PAGE_SHIFT;
unsigned long first_index = page_index(pages[0]);
- int ret = 0;
int i;
ASSERT(last_index - first_index + 1 <= nr_pages);
- ret = btrfs_delalloc_reserve_space(inode, &data_reserved, start, len);
- if (ret < 0)
- return ret;
clear_extent_bit(&inode->io_tree, start, start + len - 1,
EXTENT_DELALLOC | EXTENT_DO_ACCOUNTING |
EXTENT_DEFRAG, cached_state);
@@ -1656,10 +1655,6 @@ static int defrag_one_locked_target(struct btrfs_inode *inode,
ClearPageChecked(pages[i]);
btrfs_page_clamp_set_dirty(fs_info, pages[i], start, len);
}
- btrfs_delalloc_release_extents(inode, len);
- extent_changeset_free(data_reserved);
-
- return ret;
}
static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
@@ -1675,6 +1670,8 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
u64 last_index = (start + len - 1) >> PAGE_SHIFT;
u64 start_index = start >> PAGE_SHIFT;
unsigned int nr_pages = last_index - start_index + 1;
+ struct extent_changeset *data_reserved = NULL;
+ u64 last_defrag_end = start;
int ret = 0;
int i;
@@ -1685,6 +1682,22 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
if (!pages)
return -ENOMEM;
+ /*
+ * Reserve delalloc space before locking the range and before locking
+ * and dirtying any pages - otherwise we could deadlock, for example
+ * after defrag of one range we dirty pages and keep them locked when
+ * we move to the next range, so reserving delalloc space right before
+ * each range could trigger flushing of delalloc and deadlock on the
+ * extent lock or trigger a transaction commit with flushoncommit, which
+ * can either deadlock on the lock of a page made dirty in the previous
+ * range or the extent lock.
+ */
+ ret = btrfs_delalloc_reserve_space(inode, &data_reserved, start, len);
+ if (ret < 0) {
+ kfree(pages);
+ return ret;
+ }
+
/* Prepare all pages */
for (i = 0; i < nr_pages; i++) {
pages[i] = defrag_prepare_one_page(inode, start_index + i);
@@ -1715,10 +1728,11 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
goto unlock_extent;
list_for_each_entry(entry, &target_list, list) {
- ret = defrag_one_locked_target(inode, entry, pages, nr_pages,
- &cached_state);
- if (ret < 0)
- break;
+ defrag_one_locked_target(inode, entry, pages, nr_pages, &cached_state);
+ if (entry->start > last_defrag_end)
+ btrfs_delalloc_release_space(inode, data_reserved, last_defrag_end,
+ entry->start - last_defrag_end, true);
+ last_defrag_end = entry->start + entry->len;
}
list_for_each_entry_safe(entry, tmp, &target_list, list) {
@@ -1737,6 +1751,12 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len,
}
}
kfree(pages);
+ btrfs_delalloc_release_extents(inode, len);
+ if (last_defrag_end < start + len)
+ btrfs_delalloc_release_space(inode, data_reserved, last_defrag_end,
+ start + len - last_defrag_end, true);
+ extent_changeset_free(data_reserved);
+
return ret;
}
@@ -4663,7 +4683,8 @@ static long btrfs_ioctl_qgroup_assign(struct file *file, void __user *arg)
goto drop_write;
}
- trans = btrfs_join_transaction(root);
+ /* 2 BTRFS_QGROUP_RELATION_KEY items. */
+ trans = btrfs_start_transaction(root, 2);
if (IS_ERR(trans)) {
ret = PTR_ERR(trans);
goto out;
@@ -4725,7 +4746,11 @@ static long btrfs_ioctl_qgroup_create(struct file *file, void __user *arg)
goto out;
}
- trans = btrfs_join_transaction(root);
+ /*
+ * 1 BTRFS_QGROUP_INFO_KEY item.
+ * 1 BTRFS_QGROUP_LIMIT_KEY item.
+ */
+ trans = btrfs_start_transaction(root, 2);
if (IS_ERR(trans)) {
ret = PTR_ERR(trans);
goto out;
@@ -4771,7 +4796,8 @@ static long btrfs_ioctl_qgroup_limit(struct file *file, void __user *arg)
goto drop_write;
}
- trans = btrfs_join_transaction(root);
+ /* 1 BTRFS_QGROUP_LIMIT_KEY item. */
+ trans = btrfs_start_transaction(root, 1);
if (IS_ERR(trans)) {
ret = PTR_ERR(trans);
goto out;
diff --git a/fs/btrfs/lzo.c b/fs/btrfs/lzo.c
index 89bc5f825e0a..6e6dd4449697 100644
--- a/fs/btrfs/lzo.c
+++ b/fs/btrfs/lzo.c
@@ -15,6 +15,7 @@
#include <linux/refcount.h>
#include "compression.h"
#include "ctree.h"
+#include "btrfs_inode.h"
#define LZO_LEN 4
@@ -390,6 +391,17 @@ int lzo_decompress_bio(struct list_head *ws, struct compressed_bio *cb)
goto out;
}
+ /* The segment must not extend beyond the compressed input. */
+ if (unlikely(cur_in + seg_len > cb->compressed_len)) {
+ struct btrfs_inode *inode = BTRFS_I(cb->inode);
+
+ btrfs_err(fs_info,
+ "lzo segment overflows compressed input, root %llu inode %llu offset %llu cur_in %u len %u compressed len %u",
+ inode->root->root_key.objectid, btrfs_ino(inode),
+ cb->start, cur_in, seg_len, cb->compressed_len);
+ return -EUCLEAN;
+ }
+
/* Copy the compressed segment payload into workspace */
copy_compressed_segment(cb, workspace->cbuf, seg_len, &cur_in);
@@ -437,7 +449,7 @@ int lzo_decompress(struct list_head *ws, unsigned char *data_in,
char *kaddr;
unsigned long bytes;
- if (srclen < LZO_LEN || srclen > max_segment_len + LZO_LEN * 2)
+ if (srclen <= LZO_LEN * 2 || srclen > max_segment_len + LZO_LEN * 2)
return -EUCLEAN;
in_len = read_compress_length(data_in);
diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c
index e80d73ad0810..ff3cced9eff0 100644
--- a/fs/btrfs/qgroup.c
+++ b/fs/btrfs/qgroup.c
@@ -1325,9 +1325,11 @@ int btrfs_quota_disable(struct btrfs_fs_info *fs_info)
btrfs_tree_lock(quota_root->node);
btrfs_clean_tree_block(quota_root->node);
btrfs_tree_unlock(quota_root->node);
- btrfs_free_tree_block(trans, btrfs_root_id(quota_root),
- quota_root->node, 0, 1);
+ ret = btrfs_free_tree_block(trans, btrfs_root_id(quota_root),
+ quota_root->node, 0, 1);
+ if (ret < 0)
+ btrfs_abort_transaction(trans, ret);
out:
btrfs_put_root(quota_root);
@@ -3876,12 +3878,13 @@ static int qgroup_free_reserved_data(struct btrfs_inode *inode,
struct ulist_node *unode;
struct ulist_iterator uiter;
struct extent_changeset changeset;
+ const u32 sectorsize = root->fs_info->sectorsize;
+ const u64 aligned_start = round_down(start, sectorsize);
+ const u64 aligned_len = round_up(start + len, sectorsize) - aligned_start;
u64 freed = 0;
int ret;
extent_changeset_init(&changeset);
- len = round_up(start + len, root->fs_info->sectorsize);
- start = round_down(start, root->fs_info->sectorsize);
ULIST_ITER_INIT(&uiter);
while ((unode = ulist_next(&reserved->range_changed, &uiter))) {
@@ -3893,12 +3896,15 @@ static int qgroup_free_reserved_data(struct btrfs_inode *inode,
extent_changeset_release(&changeset);
- /* Only free range in range [start, start + len) */
- if (range_start >= start + len ||
- range_start + range_len <= start)
+ /*
+ * Only free the range within
+ * [aligned_start, aligned_start + aligned_len).
+ */
+ if (range_start >= aligned_start + aligned_len ||
+ range_start + range_len <= aligned_start)
continue;
- free_start = max(range_start, start);
- free_len = min(start + len, range_start + range_len) -
+ free_start = max(range_start, aligned_start);
+ free_len = min(aligned_start + aligned_len, range_start + range_len) -
free_start;
/*
* TODO: To also modify reserved->ranges_reserved to reflect
diff --git a/fs/btrfs/reflink.c b/fs/btrfs/reflink.c
index be33fd5a2176..00abc300782e 100644
--- a/fs/btrfs/reflink.c
+++ b/fs/btrfs/reflink.c
@@ -175,10 +175,12 @@ static int clone_copy_inline_extent(struct inode *dst,
struct btrfs_drop_extents_args drop_args = { 0 };
int ret;
struct btrfs_key key;
+ bool copied_inline_to_page = false;
if (new_key->offset > 0) {
ret = copy_inline_to_page(BTRFS_I(dst), new_key->offset,
inline_data, size, datal, comp_type);
+ copied_inline_to_page = (ret == 0);
goto out;
}
@@ -278,6 +280,60 @@ static int clone_copy_inline_extent(struct inode *dst,
ret = btrfs_inode_set_file_extent_range(BTRFS_I(dst), 0, aligned_end);
out:
if (!ret && !trans) {
+ if (copied_inline_to_page &&
+ new_key->offset + datal > i_size_read(dst)) {
+ /*
+ * If we copied the inline extent data to a page/folio
+ * beyond the i_size of the destination inode, then we
+ * need to increase the i_size before we start a
+ * transaction to update the inode item. This is to
+ * prevent a deadlock when the flushoncommit mount
+ * option is used, which happens like this:
+ *
+ * 1) Task A clones an inline extent from inode X to an
+ * offset of inode Y that is beyond Y's current
+ * i_size. This means we copied the inline extent's
+ * data to a folio of inode Y that is beyond its EOF,
+ * using the call above to copy_inline_to_page();
+ *
+ * 2) Task B starts a transaction commit and calls
+ * btrfs_start_delalloc_flush() to flush delalloc;
+ *
+ * 3) The delalloc flushing sees the new dirty folio of
+ * inode Y and when it attempts to flush it, it ends
+ * up at extent_writepage() and sees that the offset
+ * of the folio is beyond the i_size of inode Y, so
+ * it attempts to invalidate the folio by calling
+ * folio_invalidate(), which ends up at btrfs' folio
+ * invalidate callback - btrfs_invalidate_folio().
+ * There it tries to lock the folio's range in inode
+ * Y's extent io tree, but it blocks since it's
+ * currently locked by task A - during reflink we
+ * lock the inodes and the source and destination
+ * ranges after flushing all delalloc and waiting for
+ * ordered extent completion - after that we don't
+ * expect to have dirty folios in the ranges, the
+ * exception is if we have to copy an inline extent's
+ * data (because the destination offset is not zero);
+ *
+ * 4) Task A then does the 'goto out' below and attempts
+ * to start a transaction to update the inode item,
+ * and then it's blocked since the current
+ * transaction is in the TRANS_STATE_COMMIT_START
+ * state. Therefore task A has to wait for the
+ * current transaction to become unblocked (its
+ * state >= TRANS_STATE_UNBLOCKED).
+ *
+ * This leads to a deadlock - the task committing the
+ * transaction waiting for the delalloc flushing which
+ * is blocked during folio invalidation on the inode's
+ * extent lock and the reflink task waiting for the
+ * current transaction to be unblocked so that it can
+ * start a new one to update the inode item (while
+ * holding the extent lock).
+ */
+ i_size_write(dst, new_key->offset + datal);
+ }
/*
* No transaction here means we copied the inline extent into a
* page of the destination inode.
@@ -312,6 +368,8 @@ static int clone_copy_inline_extent(struct inode *dst,
ret = copy_inline_to_page(BTRFS_I(dst), new_key->offset,
inline_data, size, datal, comp_type);
+ copied_inline_to_page = (ret == 0);
+
goto out;
}
diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c
index 84b153eb9d6b..6b81a41c4a53 100644
--- a/fs/btrfs/relocation.c
+++ b/fs/btrfs/relocation.c
@@ -4228,6 +4228,24 @@ static noinline_for_stack int mark_garbage_root(struct btrfs_root *root)
return ret;
}
+static void release_recovered_fs_roots(struct list_head *roots, bool drop_reloc_refs)
+{
+ struct btrfs_root *root;
+ struct btrfs_root *next;
+
+ list_for_each_entry_safe(root, next, roots, reloc_dirty_list) {
+ list_del_init(&root->reloc_dirty_list);
+ if (drop_reloc_refs) {
+ struct btrfs_root *reloc_root = root->reloc_root;
+
+ ASSERT(reloc_root);
+ root->reloc_root = NULL;
+ btrfs_put_root(reloc_root);
+ }
+ btrfs_put_root(root);
+ }
+}
+
/*
* recover relocation interrupted by system crash.
*
@@ -4237,6 +4255,7 @@ static noinline_for_stack int mark_garbage_root(struct btrfs_root *root)
int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
{
LIST_HEAD(reloc_roots);
+ LIST_HEAD(recovered_roots);
struct btrfs_key key;
struct btrfs_root *fs_root;
struct btrfs_root *reloc_root;
@@ -4244,8 +4263,8 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
struct extent_buffer *leaf;
struct reloc_control *rc = NULL;
struct btrfs_trans_handle *trans;
- int ret;
- int err = 0;
+ int ret2;
+ int ret = 0;
path = btrfs_alloc_path();
if (!path)
@@ -4259,15 +4278,14 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
while (1) {
ret = btrfs_search_slot(NULL, fs_info->tree_root, &key,
path, 0, 0);
- if (ret < 0) {
- err = ret;
+ if (ret < 0)
goto out;
- }
if (ret > 0) {
if (path->slots[0] == 0)
break;
path->slots[0]--;
}
+ ret = 0;
leaf = path->nodes[0];
btrfs_item_key_to_cpu(leaf, &key, path->slots[0]);
btrfs_release_path(path);
@@ -4278,7 +4296,7 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
reloc_root = btrfs_read_tree_root(fs_info->tree_root, &key);
if (IS_ERR(reloc_root)) {
- err = PTR_ERR(reloc_root);
+ ret = PTR_ERR(reloc_root);
goto out;
}
@@ -4290,15 +4308,12 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
reloc_root->root_key.offset, false);
if (IS_ERR(fs_root)) {
ret = PTR_ERR(fs_root);
- if (ret != -ENOENT) {
- err = ret;
+ if (ret != -ENOENT)
goto out;
- }
ret = mark_garbage_root(reloc_root);
- if (ret < 0) {
- err = ret;
+ if (ret < 0)
goto out;
- }
+ ret = 0;
} else {
btrfs_put_root(fs_root);
}
@@ -4316,15 +4331,13 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
rc = alloc_reloc_control(fs_info);
if (!rc) {
- err = -ENOMEM;
+ ret = -ENOMEM;
goto out;
}
ret = reloc_chunk_start(fs_info);
- if (ret < 0) {
- err = ret;
+ if (ret < 0)
goto out_end;
- }
rc->extent_root = btrfs_extent_root(fs_info, 0);
@@ -4332,7 +4345,7 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
trans = btrfs_join_transaction(rc->extent_root);
if (IS_ERR(trans)) {
- err = PTR_ERR(trans);
+ ret = PTR_ERR(trans);
goto out_unset;
}
@@ -4352,27 +4365,29 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
fs_root = btrfs_get_fs_root(fs_info, reloc_root->root_key.offset,
false);
if (IS_ERR(fs_root)) {
- err = PTR_ERR(fs_root);
+ ret = PTR_ERR(fs_root);
list_add_tail(&reloc_root->root_list, &reloc_roots);
btrfs_end_transaction(trans);
- goto out_unset;
+ goto out_drop_reloc_refs;
}
- err = __add_reloc_root(reloc_root);
- ASSERT(err != -EEXIST);
- if (err) {
+ ret = __add_reloc_root(reloc_root);
+ ASSERT(ret != -EEXIST);
+ if (ret) {
list_add_tail(&reloc_root->root_list, &reloc_roots);
btrfs_put_root(fs_root);
btrfs_end_transaction(trans);
- goto out_unset;
+ goto out_drop_reloc_refs;
}
+ ASSERT(list_empty(&fs_root->reloc_dirty_list));
fs_root->reloc_root = btrfs_grab_root(reloc_root);
- btrfs_put_root(fs_root);
+ list_add_tail(&fs_root->reloc_dirty_list, &recovered_roots);
}
- err = btrfs_commit_transaction(trans);
- if (err)
- goto out_unset;
+ ret = btrfs_commit_transaction(trans);
+ if (ret)
+ goto out_drop_reloc_refs;
+ release_recovered_fs_roots(&recovered_roots, false);
merge_reloc_roots(rc);
@@ -4380,14 +4395,16 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
trans = btrfs_join_transaction(rc->extent_root);
if (IS_ERR(trans)) {
- err = PTR_ERR(trans);
+ ret = PTR_ERR(trans);
goto out_clean;
}
- err = btrfs_commit_transaction(trans);
+ ret = btrfs_commit_transaction(trans);
out_clean:
- ret = clean_dirty_subvols(rc);
- if (ret < 0 && !err)
- err = ret;
+ ret2 = clean_dirty_subvols(rc);
+ if (ret2 < 0 && !ret)
+ ret = ret2;
+out_drop_reloc_refs:
+ release_recovered_fs_roots(&recovered_roots, true);
out_unset:
unset_reloc_control(rc);
reloc_chunk_end(fs_info);
@@ -4398,14 +4415,14 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info)
btrfs_free_path(path);
- if (err == 0) {
+ if (ret == 0) {
/* cleanup orphan inode in data relocation tree */
fs_root = btrfs_grab_root(fs_info->data_reloc_root);
ASSERT(fs_root);
- err = btrfs_orphan_cleanup(fs_root);
+ ret = btrfs_orphan_cleanup(fs_root);
btrfs_put_root(fs_root);
}
- return err;
+ return ret;
}
/*
diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c
index 8cae31edc68d..d06b26e1dbea 100644
--- a/fs/btrfs/scrub.c
+++ b/fs/btrfs/scrub.c
@@ -851,8 +851,10 @@ static int scrub_print_warning_inode(u64 inum, u64 offset, u64 root,
}
ret = paths_from_inode(inum, ipath);
- if (ret < 0)
+ if (ret < 0) {
+ btrfs_put_root(local_root);
goto err;
+ }
/*
* we deliberately ignore the bit ipath might have been too small to
@@ -3630,6 +3632,11 @@ static noinline_for_stack int scrub_stripe(struct scrub_ctx *sctx,
root = btrfs_extent_root(fs_info, bg->start);
csum_root = btrfs_csum_root(fs_info, bg->start);
+ if (unlikely(!root || !csum_root)) {
+ btrfs_err(fs_info, "no valid extent or csum root for scrub");
+ btrfs_free_path(path);
+ return -EUCLEAN;
+ }
/*
* collect all data csums for the stripe to avoid seeking during
diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c
index af7b5aae842a..245dc18e8947 100644
--- a/fs/btrfs/transaction.c
+++ b/fs/btrfs/transaction.c
@@ -148,7 +148,13 @@ void btrfs_put_transaction(struct btrfs_transaction *transaction)
cache = list_first_entry(&transaction->deleted_bgs,
struct btrfs_block_group,
bg_list);
+ /*
+ * Not strictly necessary to lock, as no other task will be using a
+ * block_group on the deleted_bgs list during a transaction abort.
+ */
+ spin_lock(&transaction->fs_info->unused_bgs_lock);
list_del_init(&cache->bg_list);
+ spin_unlock(&transaction->fs_info->unused_bgs_lock);
btrfs_unfreeze_block_group(cache);
btrfs_put_block_group(cache);
}
@@ -1225,14 +1231,13 @@ static int btrfs_write_and_wait_transaction(struct btrfs_trans_handle *trans)
blk_finish_plug(&plug);
ret2 = btrfs_wait_extents(fs_info, dirty_pages);
- extent_io_tree_release(&trans->transaction->dirty_pages);
-
if (ret)
return ret;
- else if (ret2)
+ if (ret2)
return ret2;
- else
- return 0;
+
+ extent_io_tree_release(&trans->transaction->dirty_pages);
+ return 0;
}
/*
@@ -2104,7 +2109,13 @@ static void btrfs_cleanup_pending_block_groups(struct btrfs_trans_handle *trans)
list_for_each_entry_safe(block_group, tmp, &trans->new_bgs, bg_list) {
btrfs_delayed_refs_rsv_release(fs_info, 1);
+ /*
+ * Not strictly necessary to lock, as no other task will be using a
+ * block_group on the new_bgs list during a transaction abort.
+ */
+ spin_lock(&fs_info->unused_bgs_lock);
list_del_init(&block_group->bg_list);
+ spin_unlock(&fs_info->unused_bgs_lock);
}
}
diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c
index ceb2a9556fe7..c7697a77e770 100644
--- a/fs/btrfs/tree-log.c
+++ b/fs/btrfs/tree-log.c
@@ -4200,24 +4200,34 @@ static void fill_inode_item(struct btrfs_trans_handle *trans,
u64 logged_isize)
{
struct btrfs_map_token token;
+ u64 gen = BTRFS_I(inode)->generation;
u64 flags;
btrfs_init_map_token(&token, leaf);
if (log_inode_only) {
- /* set the generation to zero so the recover code
- * can tell the difference between an logging
- * just to say 'this inode exists' and a logging
- * to say 'update this inode with these values'
+ /*
+ * Set the generation to zero so the recover code can tell the
+ * difference between a logging just to say 'this inode exists'
+ * and a logging to say 'update this inode with these values'.
+ * But only if the inode was not already logged before.
+ * We access ->logged_trans directly since it was already set
+ * up in the call chain by btrfs_log_inode(), and data_race()
+ * to avoid false alerts from KCSAN and since it was set already
+ * and one can set it to 0 since that only happens on eviction
+ * and we are holding a ref on the inode.
*/
- btrfs_set_token_inode_generation(&token, item, 0);
+ ASSERT(data_race(BTRFS_I(inode)->logged_trans) > 0);
+ if (data_race(BTRFS_I(inode)->logged_trans) < trans->transid)
+ gen = 0;
+
btrfs_set_token_inode_size(&token, item, logged_isize);
} else {
- btrfs_set_token_inode_generation(&token, item,
- BTRFS_I(inode)->generation);
btrfs_set_token_inode_size(&token, item, inode->i_size);
}
+ btrfs_set_token_inode_generation(&token, item, gen);
+
btrfs_set_token_inode_uid(&token, item, i_uid_read(inode));
btrfs_set_token_inode_gid(&token, item, i_gid_read(inode));
btrfs_set_token_inode_mode(&token, item, inode->i_mode);
@@ -5033,42 +5043,63 @@ static int btrfs_log_changed_extents(struct btrfs_trans_handle *trans,
return 0;
}
-static int logged_inode_size(struct btrfs_root *log, struct btrfs_inode *inode,
- struct btrfs_path *path, u64 *size_ret)
+static int get_inode_size_to_log(struct btrfs_trans_handle *trans,
+ struct btrfs_inode *inode,
+ struct btrfs_path *path, u64 *size_ret)
{
struct btrfs_key key;
+ struct btrfs_inode_item *item;
int ret;
key.objectid = btrfs_ino(inode);
key.type = BTRFS_INODE_ITEM_KEY;
key.offset = 0;
- ret = btrfs_search_slot(NULL, log, &key, path, 0, 0);
- if (ret < 0) {
- return ret;
- } else if (ret > 0) {
- *size_ret = 0;
- } else {
- struct btrfs_inode_item *item;
+ /*
+ * Our caller called inode_logged(), so logged_trans is up to date.
+ * Use data_race() to silence any warning from KCSAN. Once logged_trans
+ * is set, it can only be reset to 0 after inode eviction.
+ */
+ if (data_race(inode->logged_trans) == trans->transid) {
+ ret = btrfs_search_slot(NULL, inode->root->log_root, &key, path, 0, 0);
+ } else if (inode->generation < trans->transid) {
+ path->search_commit_root = true;
+ path->skip_locking = true;
+ ret = btrfs_search_slot(NULL, inode->root, &key, path, 0, 0);
+ path->search_commit_root = false;
+ path->skip_locking = false;
- item = btrfs_item_ptr(path->nodes[0], path->slots[0],
- struct btrfs_inode_item);
- *size_ret = btrfs_inode_size(path->nodes[0], item);
- /*
- * If the in-memory inode's i_size is smaller then the inode
- * size stored in the btree, return the inode's i_size, so
- * that we get a correct inode size after replaying the log
- * when before a power failure we had a shrinking truncate
- * followed by addition of a new name (rename / new hard link).
- * Otherwise return the inode size from the btree, to avoid
- * data loss when replaying a log due to previously doing a
- * write that expands the inode's size and logging a new name
- * immediately after.
- */
- if (*size_ret > inode->vfs_inode.i_size)
- *size_ret = inode->vfs_inode.i_size;
+ } else {
+ *size_ret = 0;
+ return 0;
}
+ /*
+ * If the inode was logged before or is from a past transaction, then
+ * its inode item must exist in the log root or in the commit root.
+ */
+ ASSERT(ret <= 0);
+ if (WARN_ON_ONCE(ret > 0))
+ ret = -ENOENT;
+
+ if (ret < 0)
+ return ret;
+
+ item = btrfs_item_ptr(path->nodes[0], path->slots[0],
+ struct btrfs_inode_item);
+ *size_ret = btrfs_inode_size(path->nodes[0], item);
+ /*
+ * If the in-memory inode's i_size is smaller then the inode size stored
+ * in the btree, return the inode's i_size, so that we get a correct
+ * inode size after replaying the log when before a power failure we had
+ * a shrinking truncate followed by addition of a new name (rename / new
+ * hard link). Otherwise return the inode size from the btree, to avoid
+ * data loss when replaying a log due to previously doing a write that
+ * expands the inode's size and logging a new name immediately after.
+ */
+ if (*size_ret > inode->vfs_inode.i_size)
+ *size_ret = inode->vfs_inode.i_size;
+
btrfs_release_path(path);
return 0;
}
@@ -5793,6 +5824,7 @@ static int log_conflicting_inodes(struct btrfs_trans_handle *trans,
struct btrfs_log_ctx *ctx)
{
struct btrfs_fs_info *fs_info = root->fs_info;
+ const bool orig_log_new_dentries = ctx->log_new_dentries;
int ret = 0;
/*
@@ -5854,8 +5886,12 @@ static int log_conflicting_inodes(struct btrfs_trans_handle *trans,
* dir index key range logged for the directory. So we
* must make sure the deletion is recorded.
*/
+ ctx->log_new_dentries = false;
ret = btrfs_log_inode(trans, BTRFS_I(inode),
LOG_INODE_ALL, ctx);
+ if (!ret && ctx->log_new_dentries)
+ ret = log_new_dir_dentries(trans, BTRFS_I(inode), ctx);
+
btrfs_add_delayed_iput(inode);
if (ret)
break;
@@ -5890,6 +5926,7 @@ static int log_conflicting_inodes(struct btrfs_trans_handle *trans,
break;
}
+ ctx->log_new_dentries = orig_log_new_dentries;
ctx->logging_conflict_inodes = false;
if (ret)
free_conflicting_inodes(ctx);
@@ -6567,7 +6604,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans,
ret = drop_inode_items(trans, log, path, inode,
BTRFS_XATTR_ITEM_KEY);
} else {
- if (inode_only == LOG_INODE_EXISTS && ctx->logged_before) {
+ if (inode_only == LOG_INODE_EXISTS) {
/*
* Make sure the new inode item we write to the log has
* the same isize as the current one (if it exists).
@@ -6581,7 +6618,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans,
* (zeroes), as if an expanding truncate happened,
* instead of getting a file of 4Kb only.
*/
- ret = logged_inode_size(log, inode, path, &logged_isize);
+ ret = get_inode_size_to_log(trans, inode, path, &logged_isize);
if (ret)
goto out_unlock;
}
diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c
index f44a3fc14971..a2f3b57a070a 100644
--- a/fs/btrfs/volumes.c
+++ b/fs/btrfs/volumes.c
@@ -1471,30 +1471,158 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, fmode_t flags,
}
/*
- * Try to find a chunk that intersects [start, start + len] range and when one
- * such is found, record the end of it in *start
+ * Find the first pending extent intersecting a range.
+ *
+ * @device: the device to search
+ * @start: start of the range to check
+ * @len: length of the range to check
+ * @pending_start: output pointer for the start of the found pending extent
+ * @pending_end: output pointer for the end of the found pending extent (inclusive)
+ *
+ * Search for a pending chunk allocation that intersects the half-open range
+ * [start, start + len).
+ *
+ * Return: true if a pending extent was found, false otherwise.
+ * If the return value is true, store the first pending extent in
+ * [*pending_start, *pending_end]. Otherwise, the two output variables
+ * may still be modified, to something outside the range and should not
+ * be used.
*/
-static bool contains_pending_extent(struct btrfs_device *device, u64 *start,
- u64 len)
+static bool first_pending_extent(struct btrfs_device *device, u64 start, u64 len,
+ u64 *pending_start, u64 *pending_end)
{
- u64 physical_start, physical_end;
-
lockdep_assert_held(&device->fs_info->chunk_mutex);
- if (!find_first_extent_bit(&device->alloc_state, *start,
- &physical_start, &physical_end,
+ if (!find_first_extent_bit(&device->alloc_state, start,
+ pending_start, pending_end,
CHUNK_ALLOCATED, NULL)) {
- if (in_range(physical_start, *start, len) ||
- in_range(*start, physical_start,
- physical_end + 1 - physical_start)) {
- *start = physical_end + 1;
+ if (in_range(*pending_start, start, len) ||
+ in_range(start, *pending_start, *pending_end + 1 - *pending_start)) {
return true;
}
}
return false;
}
+/*
+ * Find the first real hole accounting for pending extents.
+ *
+ * @device: the device containing the candidate hole
+ * @start: input/output pointer for the hole start position
+ * @len: input/output pointer for the hole length
+ * @min_hole_size: the size of hole we are looking for
+ *
+ * Given a potential hole specified by [*start, *start + *len), check for pending
+ * chunk allocations within that range. If pending extents are found, the hole is
+ * adjusted to represent the first true free space that is large enough when
+ * accounting for pending chunks.
+ *
+ * Note that this function must handle various cases involving non consecutive
+ * pending extents.
+ *
+ * Returns: true if a suitable hole was found and false otherwise.
+ * If the return value is true, then *start and *len are set to represent the hole.
+ * If the return value is false, then *start is set to the largest hole we
+ * found and *len is set to its length.
+ * If there are no holes at all, then *start is set to the end of the range and
+ * *len is set to 0.
+ */
+static bool find_hole_in_pending_extents(struct btrfs_device *device, u64 *start,
+ u64 *len, u64 min_hole_size)
+{
+ u64 pending_start, pending_end;
+ u64 end;
+ u64 max_hole_start = 0;
+ u64 max_hole_len = 0;
+
+ lockdep_assert_held(&device->fs_info->chunk_mutex);
+
+ if (*len == 0)
+ return false;
+
+ end = *start + *len - 1;
+
+ /*
+ * Loop until we either see a large enough hole or check every pending
+ * extent overlapping the candidate hole.
+ * At every hole that we observe, record it if it is the new max.
+ * At the end of the iteration, set the output variables to the max hole.
+ */
+ while (true) {
+ if (first_pending_extent(device, *start, *len, &pending_start, &pending_end)) {
+ /*
+ * Case 1: the pending extent overlaps the start of
+ * candidate hole. That means the true hole is after the
+ * pending extent, but we need to find the next pending
+ * extent to properly size the hole. In the next loop,
+ * we will reduce to case 2 or 3.
+ * e.g.,
+ *
+ * |----pending A----| real hole |----pending B----|
+ * | candidate hole |
+ * *start end
+ */
+ if (pending_start <= *start) {
+ *start = pending_end + 1;
+ goto next;
+ }
+ /*
+ * Case 2: The pending extent starts after *start (and overlaps
+ * [*start, end), so the first hole just goes up to the start
+ * of the pending extent.
+ * e.g.,
+ *
+ * | real hole |----pending A----|
+ * | candidate hole |
+ * *start end
+ */
+ *len = pending_start - *start;
+ if (*len > max_hole_len) {
+ max_hole_start = *start;
+ max_hole_len = *len;
+ }
+ if (*len >= min_hole_size)
+ break;
+ /*
+ * If the hole wasn't big enough, then we advance past
+ * the pending extent and keep looking.
+ */
+ *start = pending_end + 1;
+ goto next;
+ } else {
+ /*
+ * Case 3: There is no pending extent overlapping the
+ * range [*start, *start + *len - 1], so the only remaining
+ * hole is the remaining range.
+ * e.g.,
+ *
+ * | candidate hole |
+ * | real hole |
+ * *start end
+ */
+
+ if (*len > max_hole_len) {
+ max_hole_start = *start;
+ max_hole_len = *len;
+ }
+ break;
+ }
+next:
+ if (*start > end)
+ break;
+ *len = end - *start + 1;
+ }
+ if (max_hole_len) {
+ *start = max_hole_start;
+ *len = max_hole_len;
+ } else {
+ *start = end + 1;
+ *len = 0;
+ }
+ return max_hole_len >= min_hole_size;
+}
+
static u64 dev_extent_search_start(struct btrfs_device *device, u64 start)
{
switch (device->fs_devices->chunk_alloc_policy) {
@@ -1556,58 +1684,57 @@ static bool dev_extent_hole_check_zoned(struct btrfs_device *device,
return changed;
}
-/**
- * dev_extent_hole_check - check if specified hole is suitable for allocation
- * @device: the device which we have the hole
- * @hole_start: starting position of the hole
- * @hole_size: the size of the hole
- * @num_bytes: the size of the free space that we need
+/*
+ * Validate and adjust a hole for chunk allocation
+ *
+ * @device: the device containing the candidate hole
+ * @hole_start: input/output pointer for the hole start position
+ * @hole_size: input/output pointer for the hole size
+ * @num_bytes: minimum allocation size required
*
- * This function may modify @hole_start and @hole_size to reflect the suitable
- * position for allocation. Returns 1 if hole position is updated, 0 otherwise.
+ * Check if the specified hole is suitable for allocation and adjust it if
+ * necessary. The hole may be modified to skip over pending chunk allocations
+ * and to satisfy stricter zoned requirements on zoned filesystems.
+ *
+ * For regular (non-zoned) allocation, if the hole after adjustment is smaller
+ * than @num_bytes, the search continues past additional pending extents until
+ * either a sufficiently large hole is found or no more pending extents exist.
+ *
+ * Return: true if a suitable hole was found and false otherwise.
+ * If the return value is true, then *hole_start and *hole_size are set to
+ * represent the hole we found.
+ * If the return value is false, then *hole_start is set to the largest
+ * hole we found and *hole_size is set to its length.
+ * If there are no holes at all, then *hole_start is set to the end of the range
+ * and *hole_size is set to 0.
*/
static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start,
u64 *hole_size, u64 num_bytes)
{
- bool changed = false;
- u64 hole_end = *hole_start + *hole_size;
+ bool found = false;
+ const u64 hole_end = *hole_start + *hole_size - 1;
- for (;;) {
- /*
- * Check before we set max_hole_start, otherwise we could end up
- * sending back this offset anyway.
- */
- if (contains_pending_extent(device, hole_start, *hole_size)) {
- if (hole_end >= *hole_start)
- *hole_size = hole_end - *hole_start;
- else
- *hole_size = 0;
- changed = true;
- }
+ ASSERT(*hole_size > 0);
- switch (device->fs_devices->chunk_alloc_policy) {
- case BTRFS_CHUNK_ALLOC_REGULAR:
- /* No extra check */
- break;
- case BTRFS_CHUNK_ALLOC_ZONED:
- if (dev_extent_hole_check_zoned(device, hole_start,
- hole_size, num_bytes)) {
- changed = true;
- /*
- * The changed hole can contain pending extent.
- * Loop again to check that.
- */
- continue;
- }
- break;
- default:
- BUG();
- }
+again:
+ *hole_size = hole_end - *hole_start + 1;
+ found = find_hole_in_pending_extents(device, hole_start, hole_size, num_bytes);
+ if (!found)
+ return found;
+ ASSERT(*hole_size >= num_bytes);
+ switch (device->fs_devices->chunk_alloc_policy) {
+ case BTRFS_CHUNK_ALLOC_REGULAR:
+ return found;
+ case BTRFS_CHUNK_ALLOC_ZONED:
+ if (dev_extent_hole_check_zoned(device, hole_start, hole_size, num_bytes))
+ goto again;
break;
+ default:
+ BUG();
}
- return changed;
+ return found;
}
/*
@@ -1667,7 +1794,6 @@ static int find_free_dev_extent_start(struct btrfs_device *device,
max_hole_start = search_start;
max_hole_size = 0;
-again:
if (search_start >= search_end ||
test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state)) {
ret = -ENOSPC;
@@ -1754,11 +1880,7 @@ static int find_free_dev_extent_start(struct btrfs_device *device,
*/
if (search_end > search_start) {
hole_size = search_end - search_start;
- if (dev_extent_hole_check(device, &search_start, &hole_size,
- num_bytes)) {
- btrfs_release_path(path);
- goto again;
- }
+ dev_extent_hole_check(device, &search_start, &hole_size, num_bytes);
if (hole_size > max_hole_size) {
max_hole_start = search_start;
@@ -4939,6 +5061,7 @@ int btrfs_shrink_device(struct btrfs_device *device, u64 new_size)
u64 old_size = btrfs_device_get_total_bytes(device);
u64 diff;
u64 start;
+ u64 pending_start, pending_end;
new_size = round_down(new_size, fs_info->sectorsize);
start = new_size;
@@ -4972,7 +5095,7 @@ int btrfs_shrink_device(struct btrfs_device *device, u64 new_size)
* in-memory chunks are synced to disk so that the loop below sees them
* and relocates them accordingly.
*/
- if (contains_pending_extent(device, &start, diff)) {
+ if (first_pending_extent(device, start, diff, &pending_start, &pending_end)) {
mutex_unlock(&fs_info->chunk_mutex);
ret = btrfs_commit_transaction(trans);
if (ret)
diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c
index 67a9b735899b..6b189276d766 100644
--- a/fs/ceph/dir.c
+++ b/fs/ceph/dir.c
@@ -1564,11 +1564,11 @@ static int __dir_lease_check(struct dentry *dentry, void *arg)
if (ret > 0) {
if (time_before(jiffies, di->time + lwc->dir_lease_ttl))
return STOP;
+ if (!lwc->expire_dir_lease)
+ return KEEP;
/* Move dentry to tail of dir lease list if we don't want
* to delete it. So dentries in the list are checked in a
* round robin manner */
- if (!lwc->expire_dir_lease)
- return TOUCH;
if (dentry->d_lockref.count > 0 ||
(di->flags & CEPH_DENTRY_REFERENCED))
return TOUCH;
@@ -1595,7 +1595,7 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc)
lwc.dir_lease = false;
lwc.nr_to_scan = CEPH_CAPS_PER_RELEASE * 2;
freed = __dentry_leases_walk(mdsc, &lwc, __dentry_lease_check);
- if (!lwc.nr_to_scan) /* more invalid leases */
+ if (freed > 0 && !lwc.nr_to_scan) /* more invalid leases */
return -EAGAIN;
if (lwc.nr_to_scan < CEPH_CAPS_PER_RELEASE)
@@ -1605,6 +1605,10 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc)
lwc.expire_dir_lease = freed < count;
lwc.dir_lease_ttl = mdsc->fsc->mount_options->caps_wanted_delay_max * HZ;
freed +=__dentry_leases_walk(mdsc, &lwc, __dir_lease_check);
+ if (freed == 0 && count == 0)
+ /* no progress possible currently, retry futile */
+ return 0;
+
if (!lwc.nr_to_scan) /* more to check */
return -EAGAIN;
diff --git a/fs/ceph/export.c b/fs/ceph/export.c
index 23b945320c55..c2d3ca5417e9 100644
--- a/fs/ceph/export.c
+++ b/fs/ceph/export.c
@@ -433,6 +433,16 @@ static struct dentry *ceph_fh_to_parent(struct super_block *sb,
return dentry;
}
+static int ceph_export_copy_name(char *name, const char *src, u32 len)
+{
+ if (len > NAME_MAX)
+ return -ENAMETOOLONG;
+
+ memcpy(name, src, len);
+ name[len] = '\0';
+ return 0;
+}
+
static int __get_snap_name(struct dentry *parent, char *name,
struct dentry *child)
{
@@ -498,9 +508,8 @@ static int __get_snap_name(struct dentry *parent, char *name,
BUG_ON(!rde->inode.in);
if (ceph_snap(inode) ==
le64_to_cpu(rde->inode.in->snapid)) {
- memcpy(name, rde->name, rde->name_len);
- name[rde->name_len] = '\0';
- err = 0;
+ err = ceph_export_copy_name(name, rde->name,
+ rde->name_len);
goto out;
}
}
@@ -562,8 +571,12 @@ static int ceph_get_name(struct dentry *parent, char *name,
if (!err) {
struct ceph_mds_reply_info_parsed *rinfo = &req->r_reply_info;
- memcpy(name, rinfo->dname, rinfo->dname_len);
- name[rinfo->dname_len] = 0;
+
+ err = ceph_export_copy_name(name, rinfo->dname,
+ rinfo->dname_len);
+ }
+
+ if (!err) {
dout("get_name %p ino %llx.%llx name %s\n",
child, ceph_vinop(inode), name);
} else {
diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c
index 939f9e870de0..0f28aeae054f 100644
--- a/fs/ceph/mds_client.c
+++ b/fs/ceph/mds_client.c
@@ -430,9 +430,33 @@ static int parse_reply_info_filelock(void **p, void *end,
#define DELEGATED_INO_AVAILABLE xa_mk_value(1)
+static int ceph_insert_deleg_ino(struct ceph_mds_session *s, u64 ino)
+{
+ int err;
+
+ /*
+ * Cap how many delegated inodes a single session may hold. This is
+ * the only place that grows the count, so atomic_add_unless() bounds
+ * it at exactly CEPH_MAX_DELEG_INOS; s_num_deleg_inos can never exceed
+ * that.
+ */
+ if (!atomic_add_unless(&s->s_num_deleg_inos, 1, CEPH_MAX_DELEG_INOS)) {
+ pr_warn_ratelimited("ceph: MDS session already holds %d delegated inodes\n",
+ CEPH_MAX_DELEG_INOS);
+ return -EOVERFLOW;
+ }
+
+ err = xa_insert(&s->s_delegated_inos, ino, DELEGATED_INO_AVAILABLE,
+ GFP_KERNEL);
+ if (err)
+ atomic_dec(&s->s_num_deleg_inos);
+ return err;
+}
+
static int ceph_parse_deleg_inos(void **p, void *end,
struct ceph_mds_session *s)
{
+ u64 msg_deleg_inos = 0;
u32 sets;
ceph_decode_32_safe(p, end, sets, bad);
@@ -449,16 +473,33 @@ static int ceph_parse_deleg_inos(void **p, void *end,
start, len);
continue;
}
+
+ /*
+ * Bound the number of inodes one reply may delegate.
+ * ceph_insert_deleg_ino() separately caps the per-session
+ * population, so this only has to stop one reply from spinning
+ * the insert loop under an attacker-controlled len.
+ */
+ if (len > (u64)CEPH_MAX_DELEG_INOS ||
+ msg_deleg_inos > (u64)CEPH_MAX_DELEG_INOS - len) {
+ pr_warn_ratelimited("ceph: MDS reply delegates too many inodes (have %llu, +%llu, max %d)\n",
+ msg_deleg_inos, len, CEPH_MAX_DELEG_INOS);
+ return -EIO;
+ }
+ msg_deleg_inos += len;
+
while (len--) {
- int err = xa_insert(&s->s_delegated_inos, start++,
- DELEGATED_INO_AVAILABLE,
- GFP_KERNEL);
+ int err = ceph_insert_deleg_ino(s, start++);
+
if (!err) {
dout("added delegated inode 0x%llx\n",
start - 1);
} else if (err == -EBUSY) {
pr_warn("MDS delegated inode 0x%llx more than once.\n",
start - 1);
+ } else if (err == -EOVERFLOW) {
+ /* ceph_insert_deleg_ino() already warned. */
+ return -EIO;
} else {
return err;
}
@@ -476,16 +517,17 @@ u64 ceph_get_deleg_ino(struct ceph_mds_session *s)
xa_for_each(&s->s_delegated_inos, ino, val) {
val = xa_erase(&s->s_delegated_inos, ino);
- if (val == DELEGATED_INO_AVAILABLE)
+ if (val == DELEGATED_INO_AVAILABLE) {
+ atomic_dec(&s->s_num_deleg_inos);
return ino;
+ }
}
return 0;
}
int ceph_restore_deleg_ino(struct ceph_mds_session *s, u64 ino)
{
- return xa_insert(&s->s_delegated_inos, ino, DELEGATED_INO_AVAILABLE,
- GFP_KERNEL);
+ return ceph_insert_deleg_ino(s, ino);
}
#else /* BITS_PER_LONG == 64 */
/*
@@ -852,6 +894,7 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc,
INIT_LIST_HEAD(&s->s_waiting);
INIT_LIST_HEAD(&s->s_unsafe);
xa_init(&s->s_delegated_inos);
+ atomic_set(&s->s_num_deleg_inos, 0);
INIT_LIST_HEAD(&s->s_cap_releases);
INIT_WORK(&s->s_cap_release_work, ceph_cap_release_work);
@@ -2453,6 +2496,7 @@ char *ceph_mdsc_build_path(struct ceph_mds_client *mdsc, struct dentry *dentry,
* cannot ever succeed. Creating paths that long is
* possible with Ceph, but Linux cannot use them.
*/
+ __putname(path);
return ERR_PTR(-ENAMETOOLONG);
}
@@ -4219,6 +4263,7 @@ static void send_mds_reconnect(struct ceph_mds_client *mdsc,
goto fail_nomsg;
xa_destroy(&session->s_delegated_inos);
+ atomic_set(&session->s_num_deleg_inos, 0);
mutex_lock(&session->s_mutex);
session->s_state = CEPH_MDS_SESSION_RECONNECTING;
@@ -4426,9 +4471,11 @@ static void check_new_map(struct ceph_mds_client *mdsc,
ceph_mdsmap_get_addr(newmap, i),
sizeof(struct ceph_entity_addr))) {
/* just close it */
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
mutex_lock(&s->s_mutex);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
ceph_con_close(&s->s_con);
mutex_unlock(&s->s_mutex);
s->s_state = CEPH_MDS_SESSION_RESTARTING;
@@ -4441,10 +4488,12 @@ static void check_new_map(struct ceph_mds_client *mdsc,
*/
if (s->s_state == CEPH_MDS_SESSION_RESTARTING &&
newstate >= CEPH_MDS_STATE_RECONNECT) {
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
clear_bit(i, targets);
send_mds_reconnect(mdsc, s);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
}
/*
@@ -4456,9 +4505,11 @@ static void check_new_map(struct ceph_mds_client *mdsc,
oldstate != CEPH_MDS_STATE_STARTING)
pr_info("mds%d recovery completed\n", s->s_mds);
kick_requests(mdsc, i);
+ ceph_get_mds_session(s);
mutex_unlock(&mdsc->mutex);
mutex_lock(&s->s_mutex);
mutex_lock(&mdsc->mutex);
+ ceph_put_mds_session(s);
ceph_kick_flushing_caps(mdsc, s);
mutex_unlock(&s->s_mutex);
wake_up_session_caps(s, RECONNECT);
diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h
index e7b0194724d7..428f4fd6cec2 100644
--- a/fs/ceph/mds_client.h
+++ b/fs/ceph/mds_client.h
@@ -219,6 +219,7 @@ struct ceph_mds_session {
struct list_head s_waiting; /* waiting requests */
struct list_head s_unsafe; /* unsafe requests */
struct xarray s_delegated_inos;
+ atomic_t s_num_deleg_inos;
};
/*
diff --git a/fs/ceph/super.h b/fs/ceph/super.h
index 62a691065292..d2b49620f30c 100644
--- a/fs/ceph/super.h
+++ b/fs/ceph/super.h
@@ -572,6 +572,15 @@ static inline int ceph_ino_compare(struct inode *inode, void *data)
#define CEPH_MDS_INO_LOG_OFFSET (2 * CEPH_MAX_MDS)
#define CEPH_INO_SYSTEM_BASE ((6*CEPH_MAX_MDS) + (CEPH_MAX_MDS * CEPH_NUM_STRAY))
+/*
+ * Upper bound on the number of delegated inodes a single MDS session may
+ * hold. The MDS normally hands out a small preallocation window (the
+ * userspace mds_client_prealloc_inos option defaults to 1000) and refills
+ * it as the client consumes entries. This leaves generous headroom while
+ * bounding the CPU and memory a malformed delegation interval can consume.
+ */
+#define CEPH_MAX_DELEG_INOS 8192
+
static inline bool ceph_vino_is_reserved(const struct ceph_vino vino)
{
if (vino.ino >= CEPH_INO_SYSTEM_BASE ||
diff --git a/fs/ceph/xattr.c b/fs/ceph/xattr.c
index e0f517db6455..9c5445eb671e 100644
--- a/fs/ceph/xattr.c
+++ b/fs/ceph/xattr.c
@@ -1208,6 +1208,22 @@ int __ceph_setxattr(struct inode *inode, const char *name,
ceph_cap_string(issued));
__build_xattrs(inode);
+ /*
+ * __build_xattrs() may have released and reacquired i_ceph_lock,
+ * during which handle_cap_grant() could have replaced i_xattrs.blob
+ * with a newer MDS-provided blob and bumped i_xattrs.version. If that
+ * caused __build_xattrs() to rebuild the rb-tree from the new blob,
+ * count/names_size/vals_size may now be larger than when
+ * required_blob_size was computed above. Recompute it here so the
+ * prealloc_blob size check below reflects the current tree state.
+ */
+ required_blob_size = __get_required_blob_size(ci, name_len, val_len);
+ if (required_blob_size > mdsc->mdsmap->m_max_xattr_size) {
+ dout("sync (size too large): %d > %llu\n",
+ required_blob_size, mdsc->mdsmap->m_max_xattr_size);
+ goto do_sync;
+ }
+
if (!ci->i_xattrs.prealloc_blob ||
required_blob_size > ci->i_xattrs.prealloc_blob->alloc_len) {
struct ceph_buffer *blob;
diff --git a/fs/coredump.c b/fs/coredump.c
index b0d782367fcc..f92376a45cd4 100644
--- a/fs/coredump.c
+++ b/fs/coredump.c
@@ -407,7 +407,9 @@ static int zap_threads(struct task_struct *tsk,
int nr = -EAGAIN;
spin_lock_irq(&tsk->sighand->siglock);
- if (!(signal->flags & SIGNAL_GROUP_EXIT) && !signal->group_exec_task) {
+ /* A freeze requested before the dump would be lost with TIF_SIGPENDING. */
+ if (!(signal->flags & SIGNAL_GROUP_EXIT) && !signal->group_exec_task &&
+ !freezing(tsk) && !(tsk->jobctl & JOBCTL_TRAP_FREEZE)) {
signal->core_state = core_state;
nr = zap_process(tsk, exit_code);
clear_tsk_thread_flag(tsk, TIF_SIGPENDING);
@@ -460,6 +462,8 @@ static void coredump_finish(bool core_dumped)
current->signal->core_state = NULL;
spin_unlock_irq(¤t->sighand->siglock);
+ /* A released thread may exit and be freed before it is woken. */
+ guard(rcu)();
while ((curr = next) != NULL) {
next = curr->next;
task = curr->task;
@@ -468,6 +472,7 @@ static void coredump_finish(bool core_dumped)
* ->task == NULL before we read ->next.
*/
smp_mb();
+ /* Any wakeup now lets the thread exit, rcu keeps it alive. */
curr->task = NULL;
wake_up_process(task);
}
@@ -475,13 +480,8 @@ static void coredump_finish(bool core_dumped)
static bool dump_interrupted(void)
{
- /*
- * SIGKILL or freezing() interrupt the coredumping. Perhaps we
- * can do try_to_freeze() and check __fatal_signal_pending(),
- * but then we need to teach dump_write() to restart and clear
- * TIF_SIGPENDING.
- */
- return fatal_signal_pending(current) || freezing(current);
+ /* Only SIGKILL and the freezers set it after zap_threads(). */
+ return task_sigpending(current);
}
static void wait_for_dump_helpers(struct file *file)
diff --git a/fs/dcache.c b/fs/dcache.c
index 3208ff5bfca8..205552e31b20 100644
--- a/fs/dcache.c
+++ b/fs/dcache.c
@@ -1502,6 +1502,8 @@ EXPORT_SYMBOL(path_has_submounts);
*
* Only one of d_invalidate() and d_set_mounted() must succeed. For
* this reason take rename_lock and d_lock on dentry and ancestors.
+ * Likewise for dont_mount() which marks a dentry that is being removed
+ * under d_lock.
*/
int d_set_mounted(struct dentry *dentry)
{
@@ -1518,7 +1520,7 @@ int d_set_mounted(struct dentry *dentry)
spin_unlock(&p->d_lock);
}
spin_lock(&dentry->d_lock);
- if (!d_unlinked(dentry)) {
+ if (!d_unlinked(dentry) && !cant_mount(dentry)) {
ret = -EBUSY;
if (!d_mountpoint(dentry)) {
dentry->d_flags |= DCACHE_MOUNTED;
diff --git a/fs/dlm/config.c b/fs/dlm/config.c
index ac8b62106ce0..20b60709eccf 100644
--- a/fs/dlm/config.c
+++ b/fs/dlm/config.c
@@ -183,7 +183,7 @@ static int dlm_check_protocol_and_dlm_running(unsigned int x)
return -EINVAL;
}
- if (dlm_allow_conn)
+ if (dlm_lowcomms_is_running())
return -EBUSY;
return 0;
@@ -194,7 +194,7 @@ static int dlm_check_zero_and_dlm_running(unsigned int x)
if (!x)
return -EINVAL;
- if (dlm_allow_conn)
+ if (dlm_lowcomms_is_running())
return -EBUSY;
return 0;
diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c
index d5aeda3399f9..8c2d4bc29017 100644
--- a/fs/dlm/lock.c
+++ b/fs/dlm/lock.c
@@ -456,6 +456,8 @@ int dlm_search_rsb_tree(struct rb_root *tree, const void *name, int len,
struct dlm_rsb *r;
int rc;
+ if (len > DLM_RESNAME_MAXLEN)
+ return -EINVAL;
while (node) {
r = rb_entry(node, struct dlm_rsb, res_hashnode);
rc = rsb_cmp(r, name, len);
diff --git a/fs/dlm/lowcomms.c b/fs/dlm/lowcomms.c
index 51a641822d6c..4ad492d035dd 100644
--- a/fs/dlm/lowcomms.c
+++ b/fs/dlm/lowcomms.c
@@ -65,7 +65,6 @@
/* Number of messages to send before rescheduling */
#define MAX_SEND_MSG_COUNT 25
-#define DLM_SHUTDOWN_WAIT_TIMEOUT msecs_to_jiffies(10000)
struct connection {
struct socket *sock; /* NULL if not connected */
@@ -79,14 +78,11 @@ struct connection {
#define CF_CLOSE 6
#define CF_APP_LIMITED 7
#define CF_CLOSING 8
-#define CF_SHUTDOWN 9
-#define CF_CONNECTED 10
-#define CF_RECONNECT 11
-#define CF_DELAY_CONNECT 12
-#define CF_EOF 13
+#define CF_CONNECTED 9
+#define CF_RECONNECT 10
+#define CF_DELAY_CONNECT 11
struct list_head writequeue; /* List of outgoing writequeue_entries */
spinlock_t writequeue_lock;
- atomic_t writequeue_cnt;
int retries;
#define MAX_CONNECT_RETRIES 3
struct hlist_node list;
@@ -94,7 +90,6 @@ struct connection {
struct connection *sendcon;
struct work_struct rwork; /* Receive workqueue */
struct work_struct swork; /* Send workqueue */
- wait_queue_head_t shutdown_wait; /* wait for graceful shutdown */
unsigned char *rx_buf;
int rx_buflen;
int rx_leftover;
@@ -157,10 +152,6 @@ struct dlm_proto_ops {
int (*listen_validate)(void);
void (*listen_sockopts)(struct socket *sock);
int (*listen_bind)(struct socket *sock);
- /* What to do to shutdown */
- void (*shutdown_action)(struct connection *con);
- /* What to do to eof check */
- bool (*eof_condition)(struct connection *con);
};
static struct listen_sock_callbacks {
@@ -176,7 +167,6 @@ static DEFINE_SPINLOCK(dlm_node_addrs_spin);
static struct listen_connection listen_con;
static struct sockaddr_storage dlm_local_addr[DLM_MAX_ADDR_COUNT];
static int dlm_local_count;
-int dlm_allow_conn;
/* Work queues */
static struct workqueue_struct *recv_workqueue;
@@ -191,6 +181,11 @@ static const struct dlm_proto_ops *dlm_proto_ops;
static void process_recv_sockets(struct work_struct *work);
static void process_send_sockets(struct work_struct *work);
+bool dlm_lowcomms_is_running(void)
+{
+ return !!listen_con.sock;
+}
+
static void writequeue_entry_ctor(void *data)
{
struct writequeue_entry *entry = data;
@@ -240,11 +235,6 @@ static struct connection *__find_con(int nodeid, int r)
return NULL;
}
-static bool tcp_eof_condition(struct connection *con)
-{
- return atomic_read(&con->writequeue_cnt);
-}
-
static int dlm_con_init(struct connection *con, int nodeid)
{
con->rx_buflen = dlm_config.ci_buffer_size;
@@ -256,10 +246,8 @@ static int dlm_con_init(struct connection *con, int nodeid)
mutex_init(&con->sock_mutex);
INIT_LIST_HEAD(&con->writequeue);
spin_lock_init(&con->writequeue_lock);
- atomic_set(&con->writequeue_cnt, 0);
INIT_WORK(&con->swork, process_send_sockets);
INIT_WORK(&con->rwork, process_recv_sockets);
- init_waitqueue_head(&con->shutdown_wait);
return 0;
}
@@ -514,9 +502,6 @@ static void lowcomms_data_ready(struct sock *sk)
static void lowcomms_listen_data_ready(struct sock *sk)
{
- if (!dlm_allow_conn)
- return;
-
queue_work(recv_workqueue, &listen_con.rwork);
}
@@ -665,17 +650,6 @@ static void lowcomms_error_report(struct sock *sk)
orig_report(sk);
}
-/* Note: sk_callback_lock must be locked before calling this function. */
-static void save_listen_callbacks(struct socket *sock)
-{
- struct sock *sk = sock->sk;
-
- listen_sock.sk_data_ready = sk->sk_data_ready;
- listen_sock.sk_state_change = sk->sk_state_change;
- listen_sock.sk_write_space = sk->sk_write_space;
- listen_sock.sk_error_report = sk->sk_error_report;
-}
-
static void restore_callbacks(struct socket *sock)
{
struct sock *sk = sock->sk;
@@ -689,21 +663,6 @@ static void restore_callbacks(struct socket *sock)
release_sock(sk);
}
-static void add_listen_sock(struct socket *sock, struct listen_connection *con)
-{
- struct sock *sk = sock->sk;
-
- lock_sock(sk);
- save_listen_callbacks(sock);
- con->sock = sock;
-
- sk->sk_user_data = con;
- sk->sk_allocation = GFP_NOFS;
- /* Install a data_ready callback */
- sk->sk_data_ready = lowcomms_listen_data_ready;
- release_sock(sk);
-}
-
/* Make a socket active */
static void add_sock(struct socket *sock, struct connection *con)
{
@@ -718,6 +677,7 @@ static void add_sock(struct socket *sock, struct connection *con)
sk->sk_write_space = lowcomms_write_space;
sk->sk_state_change = lowcomms_state_change;
sk->sk_allocation = GFP_NOFS;
+ sk->sk_use_task_frag = false;
sk->sk_error_report = lowcomms_error_report;
release_sock(sk);
}
@@ -773,7 +733,6 @@ static void free_entry(struct writequeue_entry *e)
}
list_del(&e->list);
- atomic_dec(&e->con->writequeue_cnt);
kref_put(&e->ref, dlm_page_release);
}
@@ -836,56 +795,10 @@ static void close_connection(struct connection *con, bool and_other,
clear_bit(CF_CONNECTED, &con->flags);
clear_bit(CF_DELAY_CONNECT, &con->flags);
clear_bit(CF_RECONNECT, &con->flags);
- clear_bit(CF_EOF, &con->flags);
mutex_unlock(&con->sock_mutex);
clear_bit(CF_CLOSING, &con->flags);
}
-static void shutdown_connection(struct connection *con)
-{
- int ret;
-
- flush_work(&con->swork);
-
- mutex_lock(&con->sock_mutex);
- /* nothing to shutdown */
- if (!con->sock) {
- mutex_unlock(&con->sock_mutex);
- return;
- }
-
- set_bit(CF_SHUTDOWN, &con->flags);
- ret = kernel_sock_shutdown(con->sock, SHUT_WR);
- mutex_unlock(&con->sock_mutex);
- if (ret) {
- log_print("Connection %p failed to shutdown: %d will force close",
- con, ret);
- goto force_close;
- } else {
- ret = wait_event_timeout(con->shutdown_wait,
- !test_bit(CF_SHUTDOWN, &con->flags),
- DLM_SHUTDOWN_WAIT_TIMEOUT);
- if (ret == 0) {
- log_print("Connection %p shutdown timed out, will force close",
- con);
- goto force_close;
- }
- }
-
- return;
-
-force_close:
- clear_bit(CF_SHUTDOWN, &con->flags);
- close_connection(con, false, true, true);
-}
-
-static void dlm_tcp_shutdown(struct connection *con)
-{
- if (con->othercon)
- shutdown_connection(con->othercon);
- shutdown_connection(con);
-}
-
static int con_realloc_receive_buf(struct connection *con, int newlen)
{
unsigned char *newbuf;
@@ -977,19 +890,8 @@ static int receive_from_sock(struct connection *con)
log_print("connection %p got EOF from %d",
con, con->nodeid);
- if (dlm_proto_ops->eof_condition &&
- dlm_proto_ops->eof_condition(con)) {
- set_bit(CF_EOF, &con->flags);
- mutex_unlock(&con->sock_mutex);
- } else {
- mutex_unlock(&con->sock_mutex);
- close_connection(con, false, true, false);
-
- /* handling for tcp shutdown */
- clear_bit(CF_SHUTDOWN, &con->flags);
- wake_up(&con->shutdown_wait);
- }
-
+ mutex_unlock(&con->sock_mutex);
+ close_connection(con, false, true, false);
/* signal to breaking receive worker */
ret = -1;
} else {
@@ -1252,7 +1154,6 @@ static struct writequeue_entry *new_wq_entry(struct connection *con, int len,
kref_get(&e->ref);
*ppc = page_address(e->page);
e->end += len;
- atomic_inc(&con->writequeue_cnt);
if (cb)
cb(data);
@@ -1453,20 +1354,6 @@ static void send_to_sock(struct connection *con)
}
spin_unlock(&con->writequeue_lock);
- /* close if we got EOF */
- if (test_and_clear_bit(CF_EOF, &con->flags)) {
- mutex_unlock(&con->sock_mutex);
- close_connection(con, false, false, true);
-
- /* handling for tcp shutdown */
- clear_bit(CF_SHUTDOWN, &con->flags);
- wake_up(&con->shutdown_wait);
- } else {
- mutex_unlock(&con->sock_mutex);
- }
-
- return;
-
out:
mutex_unlock(&con->sock_mutex);
return;
@@ -1666,30 +1553,35 @@ static int work_start(void)
return 0;
}
-static void shutdown_conn(struct connection *con)
+void dlm_lowcomms_shutdown(void)
{
- if (dlm_proto_ops->shutdown_action)
- dlm_proto_ops->shutdown_action(con);
+ /* stop lowcomms_listen_data_ready calls */
+ lock_sock(listen_con.sock->sk);
+ listen_con.sock->sk->sk_data_ready = listen_sock.sk_data_ready;
+ release_sock(listen_con.sock->sk);
+
+ cancel_work_sync(&listen_con.rwork);
+ dlm_close_sock(&listen_con.sock);
}
-void dlm_lowcomms_shutdown(void)
+void dlm_lowcomms_shutdown_node(int nodeid, bool force)
{
+ struct connection *con;
int idx;
- /* Set all the flags to prevent any
- * socket activity.
- */
- dlm_allow_conn = 0;
-
- if (recv_workqueue)
- flush_workqueue(recv_workqueue);
- if (send_workqueue)
- flush_workqueue(send_workqueue);
-
- dlm_close_sock(&listen_con.sock);
-
idx = srcu_read_lock(&connections_srcu);
- foreach_conn(shutdown_conn);
+ con = nodeid2con(nodeid, 0);
+ if (WARN_ON_ONCE(!con)) {
+ srcu_read_unlock(&connections_srcu, idx);
+ return;
+ }
+
+ flush_work(&con->swork);
+ WARN_ON_ONCE(!force && !list_empty(&con->writequeue));
+ clean_one_writequeue(con);
+ if (con->othercon)
+ clean_one_writequeue(con->othercon);
+ close_connection(con, true, true, true);
srcu_read_unlock(&connections_srcu, idx);
}
@@ -1805,8 +1697,18 @@ static int dlm_listen_for_all(void)
if (result < 0)
goto out;
- save_listen_callbacks(sock);
- add_listen_sock(sock, &listen_con);
+ lock_sock(sock->sk);
+ listen_sock.sk_data_ready = sock->sk->sk_data_ready;
+ listen_sock.sk_write_space = sock->sk->sk_write_space;
+ listen_sock.sk_error_report = sock->sk->sk_error_report;
+ listen_sock.sk_state_change = sock->sk->sk_state_change;
+
+ listen_con.sock = sock;
+
+ sock->sk->sk_allocation = GFP_NOFS;
+ sock->sk->sk_use_task_frag = false;
+ sock->sk->sk_data_ready = lowcomms_listen_data_ready;
+ release_sock(sock->sk);
INIT_WORK(&listen_con.rwork, process_listen_recv_socket);
result = sock->ops->listen(sock, 5);
@@ -1901,8 +1803,6 @@ static const struct dlm_proto_ops dlm_tcp_ops = {
.listen_validate = dlm_tcp_listen_validate,
.listen_sockopts = dlm_tcp_listen_sockopts,
.listen_bind = dlm_tcp_listen_bind,
- .shutdown_action = dlm_tcp_shutdown,
- .eof_condition = tcp_eof_condition,
};
static int dlm_sctp_bind(struct socket *sock)
@@ -1982,8 +1882,6 @@ int dlm_lowcomms_start(void)
if (error)
goto fail;
- dlm_allow_conn = 1;
-
/* Start listening */
switch (dlm_config.ci_protocol) {
case DLM_PROTO_TCP:
@@ -2008,7 +1906,6 @@ int dlm_lowcomms_start(void)
fail_listen:
dlm_proto_ops = NULL;
fail_proto_ops:
- dlm_allow_conn = 0;
work_stop();
fail:
return error;
diff --git a/fs/dlm/lowcomms.h b/fs/dlm/lowcomms.h
index bbce7a18416d..3e8dca66183b 100644
--- a/fs/dlm/lowcomms.h
+++ b/fs/dlm/lowcomms.h
@@ -29,11 +29,12 @@ static inline int nodeid_hash(int nodeid)
return nodeid & (CONN_HASH_SIZE-1);
}
-/* switch to check if dlm is running */
-extern int dlm_allow_conn;
+/* check if dlm is running */
+bool dlm_lowcomms_is_running(void);
int dlm_lowcomms_start(void);
void dlm_lowcomms_shutdown(void);
+void dlm_lowcomms_shutdown_node(int nodeid, bool force);
void dlm_lowcomms_stop(void);
void dlm_lowcomms_init(void);
void dlm_lowcomms_exit(void);
diff --git a/fs/dlm/midcomms.c b/fs/dlm/midcomms.c
index 71e426da48c3..553cdb378037 100644
--- a/fs/dlm/midcomms.c
+++ b/fs/dlm/midcomms.c
@@ -1040,10 +1040,10 @@ static void midcomms_new_msg_cb(void *data)
atomic_inc(&mh->node->send_queue_cnt);
spin_lock(&mh->node->send_queue_lock);
+ /* need to be locked with list_add_tail_rcu() because list is ordered */
+ mh->seq = mh->node->seq_send++;
list_add_tail_rcu(&mh->list, &mh->node->send_queue);
spin_unlock(&mh->node->send_queue_lock);
-
- mh->seq = mh->node->seq_send++;
}
static struct dlm_msg *dlm_midcomms_get_msg_3_2(struct dlm_mhandle *mh, int nodeid,
@@ -1416,11 +1416,13 @@ static void midcomms_shutdown(struct midcomms_node *node)
pr_debug("active shutdown timed out for node %d with state %s\n",
node->nodeid, dlm_state_str(node->state));
midcomms_node_reset(node);
+ dlm_lowcomms_shutdown_node(node->nodeid, true);
return;
}
pr_debug("active shutdown done for node %d with state %s\n",
node->nodeid, dlm_state_str(node->state));
+ dlm_lowcomms_shutdown_node(node->nodeid, false);
}
void dlm_midcomms_shutdown(void)
@@ -1428,6 +1430,8 @@ void dlm_midcomms_shutdown(void)
struct midcomms_node *node;
int i, idx;
+ dlm_lowcomms_shutdown();
+
mutex_lock(&close_lock);
idx = srcu_read_lock(&nodes_srcu);
for (i = 0; i < CONN_HASH_SIZE; i++) {
@@ -1445,8 +1449,6 @@ void dlm_midcomms_shutdown(void)
}
srcu_read_unlock(&nodes_srcu, idx);
mutex_unlock(&close_lock);
-
- dlm_lowcomms_shutdown();
}
int dlm_midcomms_close(int nodeid)
diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c
index b1ac88a820f2..c6e6ceae48a3 100644
--- a/fs/erofs/decompressor_lzma.c
+++ b/fs/erofs/decompressor_lzma.c
@@ -6,6 +6,7 @@
struct z_erofs_lzma {
struct z_erofs_lzma *next;
struct xz_dec_microlzma *state;
+ unsigned int dict_size;
struct xz_buf buf;
u8 bounce[PAGE_SIZE];
};
@@ -132,11 +133,19 @@ int z_erofs_load_lzma_config(struct super_block *sb,
err = 0;
/* 2. walk each isolated stream and grow max dict_size if needed */
for (strm = head; strm; strm = strm->next) {
+ struct xz_dec_microlzma *state;
+
+ if (strm->dict_size >= dict_size)
+ continue;
+ state = xz_dec_microlzma_alloc(XZ_PREALLOC, dict_size);
+ if (!state) {
+ err = -ENOMEM;
+ break;
+ }
if (strm->state)
xz_dec_microlzma_end(strm->state);
- strm->state = xz_dec_microlzma_alloc(XZ_PREALLOC, dict_size);
- if (!strm->state)
- err = -ENOMEM;
+ strm->state = state;
+ strm->dict_size = dict_size;
}
/* 3. push back all to the global list and update max dict_size */
@@ -146,7 +155,8 @@ int z_erofs_load_lzma_config(struct super_block *sb,
spin_unlock(&z_erofs_lzma_lock);
wake_up_all(&z_erofs_lzma_wq);
- z_erofs_lzma_max_dictsize = dict_size;
+ if (!err)
+ z_erofs_lzma_max_dictsize = dict_size;
mutex_unlock(&lzma_resize_mutex);
return err;
}
diff --git a/fs/erofs/xattr.c b/fs/erofs/xattr.c
index 7c09ad1beb83..05e5733d7859 100644
--- a/fs/erofs/xattr.c
+++ b/fs/erofs/xattr.c
@@ -35,6 +35,7 @@ static int init_inode_xattrs(struct inode *inode)
return 0;
}
+ it.buf = __EROFS_BUF_INITIALIZER;
if (wait_on_bit_lock(&vi->flags, EROFS_I_BL_XATTR_BIT, TASK_KILLABLE))
return -ERESTARTSYS;
@@ -67,7 +68,6 @@ static int init_inode_xattrs(struct inode *inode)
goto out_unlock;
}
- it.buf = __EROFS_BUF_INITIALIZER;
it.blkaddr = erofs_blknr(sb, erofs_iloc(inode) + vi->inode_isize);
it.ofs = erofs_blkoff(sb, erofs_iloc(inode) + vi->inode_isize);
@@ -83,7 +83,6 @@ static int init_inode_xattrs(struct inode *inode)
vi->xattr_shared_xattrs = kmalloc_array(vi->xattr_shared_count,
sizeof(uint), GFP_KERNEL);
if (!vi->xattr_shared_xattrs) {
- erofs_put_metabuf(&it.buf);
ret = -ENOMEM;
goto out_unlock;
}
@@ -110,13 +109,13 @@ static int init_inode_xattrs(struct inode *inode)
le32_to_cpu(*(__le32 *)(it.kaddr + it.ofs));
it.ofs += sizeof(__le32);
}
- erofs_put_metabuf(&it.buf);
/* paired with smp_mb() at the beginning of the function. */
smp_mb();
set_bit(EROFS_I_EA_INITED_BIT, &vi->flags);
out_unlock:
+ erofs_put_metabuf(&it.buf);
clear_and_wake_up_bit(EROFS_I_BL_XATTR_BIT, &vi->flags);
return ret;
}
diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c
index 27cfc9aa3072..7dbfdfab6e35 100644
--- a/fs/erofs/zdata.c
+++ b/fs/erofs/zdata.c
@@ -689,6 +689,7 @@ static int z_erofs_collector_begin(struct z_erofs_decompress_frontend *fe)
{
struct erofs_map_blocks *map = &fe->map;
struct erofs_workgroup *grp = NULL;
+ void *ptr = NULL;
int ret;
DBG_BUGON(fe->pcl);
@@ -704,6 +705,19 @@ static int z_erofs_collector_begin(struct z_erofs_decompress_frontend *fe)
return -EFSCORRUPTED;
}
+ if (map->m_flags & EROFS_MAP_META) {
+ ptr = erofs_read_metabuf(&map->buf, fe->inode->i_sb,
+ erofs_blknr(fe->inode->i_sb, map->m_pa),
+ EROFS_NO_KMAP);
+ if (IS_ERR(ptr)) {
+ ret = PTR_ERR(ptr);
+ erofs_err(fe->inode->i_sb,
+ "failed to get inline page, err %d", ret);
+ return ret;
+ }
+ ptr = map->buf.page;
+ }
+
if (grp) {
fe->pcl = container_of(grp, struct z_erofs_pcluster, obj);
ret = -EEXIST;
@@ -719,6 +733,15 @@ static int z_erofs_collector_begin(struct z_erofs_decompress_frontend *fe)
}
z_erofs_bvec_iter_begin(&fe->biter, &fe->pcl->bvset,
Z_EROFS_INLINE_BVECS, fe->pcl->vcnt);
+ if (z_erofs_is_inline_pcluster(fe->pcl)) {
+ get_page(ptr);
+ WRITE_ONCE(fe->pcl->compressed_bvecs[0].page, ptr);
+ fe->pcl->pageofs_in = map->m_pa & ~PAGE_MASK;
+ fe->mode = Z_EROFS_PCLUSTER_FOLLOWED_NOINPLACE;
+ } else {
+ /* bind cache first when cached decompression is preferred */
+ z_erofs_bind_cache(fe);
+ }
/* since file-backed online pages are traversed in reverse order */
fe->icur = z_erofs_pclusterpages(fe->pcl);
return 0;
@@ -840,27 +863,6 @@ static int z_erofs_do_read_page(struct z_erofs_decompress_frontend *fe,
if (err)
goto out;
- if (z_erofs_is_inline_pcluster(fe->pcl)) {
- void *mp;
-
- mp = erofs_read_metabuf(&fe->map.buf, inode->i_sb,
- erofs_blknr(inode->i_sb, map->m_pa),
- EROFS_NO_KMAP);
- if (IS_ERR(mp)) {
- err = PTR_ERR(mp);
- erofs_err(inode->i_sb,
- "failed to get inline page, err %d", err);
- goto out;
- }
- get_page(fe->map.buf.page);
- WRITE_ONCE(fe->pcl->compressed_bvecs[0].page,
- fe->map.buf.page);
- fe->pcl->pageofs_in = map->m_pa & ~PAGE_MASK;
- fe->mode = Z_EROFS_PCLUSTER_FOLLOWED_NOINPLACE;
- } else {
- /* bind cache first when cached decompression is preferred */
- z_erofs_bind_cache(fe);
- }
hitted:
/*
* Ensure the current partial page belongs to this submit chain rather
@@ -1281,10 +1283,10 @@ static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io,
if (atomic_add_return(bios, &io->pending_bios))
return;
if (z_erofs_in_atomic()) {
- queue_work(z_erofs_workqueue, &io->u.work);
/* enable sync decompression for readahead */
if (sbi->opt.sync_decompress == EROFS_SYNC_DECOMPRESS_AUTO)
sbi->opt.sync_decompress = EROFS_SYNC_DECOMPRESS_FORCE_ON;
+ queue_work(z_erofs_workqueue, &io->u.work);
return;
}
gfp_flag = memalloc_noio_save();
diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c
index d2d7fe826091..9159461be610 100644
--- a/fs/erofs/zmap.c
+++ b/fs/erofs/zmap.c
@@ -30,7 +30,7 @@ struct z_erofs_maprecorder {
struct erofs_map_blocks *map;
void *kaddr;
- unsigned long lcn;
+ u64 lcn;
/* compression extent information gathered */
u8 type, headtype;
u16 clusterofs;
@@ -41,7 +41,7 @@ struct z_erofs_maprecorder {
};
static int legacy_load_cluster_from_disk(struct z_erofs_maprecorder *m,
- unsigned long lcn)
+ u64 lcn)
{
struct inode *const inode = m->inode;
struct erofs_inode *const vi = EROFS_I(inode);
@@ -245,7 +245,7 @@ static int unpack_compacted_index(struct z_erofs_maprecorder *m,
}
static int compacted_load_cluster_from_disk(struct z_erofs_maprecorder *m,
- unsigned long lcn, bool lookahead)
+ u64 lcn, bool lookahead)
{
struct inode *const inode = m->inode;
struct erofs_inode *const vi = EROFS_I(inode);
@@ -296,7 +296,7 @@ static int compacted_load_cluster_from_disk(struct z_erofs_maprecorder *m,
}
static int z_erofs_load_cluster_from_disk(struct z_erofs_maprecorder *m,
- unsigned int lcn, bool lookahead)
+ u64 lcn, bool lookahead)
{
const unsigned int datamode = EROFS_I(m->inode)->datalayout;
@@ -316,7 +316,7 @@ static int z_erofs_extent_lookback(struct z_erofs_maprecorder *m,
const unsigned int lclusterbits = vi->z_logical_clusterbits;
while (m->lcn >= lookback_distance) {
- unsigned long lcn = m->lcn - lookback_distance;
+ u64 lcn = m->lcn - lookback_distance;
int err;
/* load extent head logical cluster if needed */
@@ -343,7 +343,7 @@ static int z_erofs_extent_lookback(struct z_erofs_maprecorder *m,
return 0;
default:
erofs_err(m->inode->i_sb,
- "unknown type %u @ lcn %lu of nid %llu",
+ "unknown type %u @ lcn %llu of nid %llu",
m->type, lcn, vi->nid);
DBG_BUGON(1);
return -EOPNOTSUPP;
@@ -363,7 +363,7 @@ static int z_erofs_get_extent_compressedlen(struct z_erofs_maprecorder *m,
struct erofs_inode *const vi = EROFS_I(m->inode);
struct erofs_map_blocks *const map = m->map;
const unsigned int lclusterbits = vi->z_logical_clusterbits;
- unsigned long lcn;
+ u64 lcn;
int err;
DBG_BUGON(m->type != Z_EROFS_LCLUSTER_TYPE_PLAIN &&
@@ -416,7 +416,7 @@ static int z_erofs_get_extent_compressedlen(struct z_erofs_maprecorder *m,
fallthrough;
default:
erofs_err(m->inode->i_sb,
- "cannot found CBLKCNT @ lcn %lu of nid %llu",
+ "cannot found CBLKCNT @ lcn %llu of nid %llu",
lcn, vi->nid);
DBG_BUGON(1);
return -EFSCORRUPTED;
@@ -426,7 +426,7 @@ static int z_erofs_get_extent_compressedlen(struct z_erofs_maprecorder *m,
return 0;
err_bonus_cblkcnt:
erofs_err(m->inode->i_sb,
- "bogus CBLKCNT @ lcn %lu of nid %llu",
+ "bogus CBLKCNT @ lcn %llu of nid %llu",
lcn, vi->nid);
DBG_BUGON(1);
return -EFSCORRUPTED;
@@ -577,7 +577,8 @@ static int z_erofs_do_map_blocks(struct inode *inode,
err = -EFSCORRUPTED;
goto unmap_out;
}
- afmt = vi->z_advise & Z_EROFS_ADVISE_INTERLACED_PCLUSTER ?
+ afmt = (vi->z_advise & Z_EROFS_ADVISE_INTERLACED_PCLUSTER) &&
+ !(map->m_flags & EROFS_MAP_META) ?
Z_EROFS_COMPRESSION_INTERLACED :
Z_EROFS_COMPRESSION_SHIFTED;
} else {
diff --git a/fs/exec.c b/fs/exec.c
index 6ff6367793b9..7391ec00a1cc 100644
--- a/fs/exec.c
+++ b/fs/exec.c
@@ -1268,6 +1268,21 @@ int begin_new_exec(struct linux_binprm * bprm)
if (retval)
goto out;
+ /*
+ * This must be done here to ensure that POSIX CPU timers which were
+ * armed on the current task are dequeued from me::posix_cputimers.
+ * Otherwise in case of a TID switch the deletion of the related POSIX
+ * timer would not remove an enqueued timer because the TID lookup
+ * of the old TID fails.
+ */
+#ifdef CONFIG_POSIX_TIMERS
+ spin_lock_irq(&me->sighand->siglock);
+ posix_cpu_timers_exit(me);
+ spin_unlock_irq(&me->sighand->siglock);
+ exit_itimers(me);
+ flush_itimer_signals();
+#endif
+
/*
* Cancel any io_uring activity across execve
*/
@@ -1316,14 +1331,6 @@ int begin_new_exec(struct linux_binprm * bprm)
bprm->mm = NULL;
-#ifdef CONFIG_POSIX_TIMERS
- spin_lock_irq(&me->sighand->siglock);
- posix_cpu_timers_exit(me);
- spin_unlock_irq(&me->sighand->siglock);
- exit_itimers(me);
- flush_itimer_signals();
-#endif
-
/*
* Make the signal table private.
*/
diff --git a/fs/exfat/dir.c b/fs/exfat/dir.c
index dc12e463be1a..d8c789d76807 100644
--- a/fs/exfat/dir.c
+++ b/fs/exfat/dir.c
@@ -931,7 +931,7 @@ enum {
*/
int exfat_find_dir_entry(struct super_block *sb, struct exfat_inode_info *ei,
struct exfat_chain *p_dir, struct exfat_uni_name *p_uniname,
- int num_entries, unsigned int type, struct exfat_hint *hint_opt)
+ struct exfat_hint *hint_opt)
{
int i, rewind = 0, dentry = 0, end_eidx = 0, num_ext = 0, len;
int order, step, name_len = 0;
@@ -942,6 +942,11 @@ int exfat_find_dir_entry(struct super_block *sb, struct exfat_inode_info *ei,
struct exfat_hint *hint_stat = &ei->hint_stat;
struct exfat_hint_femp candi_empty;
struct exfat_sb_info *sbi = EXFAT_SB(sb);
+ int num_entries = exfat_calc_num_entries(p_uniname);
+ unsigned int clu_count = 0;
+
+ if (num_entries < 0)
+ return num_entries;
dentries_per_clu = sbi->dentries_per_clu;
@@ -1010,10 +1015,8 @@ int exfat_find_dir_entry(struct super_block *sb, struct exfat_inode_info *ei,
step = DIRENT_STEP_FILE;
hint_opt->clu = clu.dir;
hint_opt->eidx = i;
- if (type == TYPE_ALL || type == entry_type) {
- num_ext = ep->dentry.file.num_ext;
- step = DIRENT_STEP_STRM;
- }
+ num_ext = ep->dentry.file.num_ext;
+ step = DIRENT_STEP_STRM;
brelse(bh);
continue;
}
@@ -1098,6 +1101,10 @@ int exfat_find_dir_entry(struct super_block *sb, struct exfat_inode_info *ei,
} else {
if (exfat_get_next_cluster(sb, &clu.dir))
return -EIO;
+
+ /* break if the cluster chain includes a loop */
+ if (unlikely(++clu_count > EXFAT_DATA_CLUSTER_COUNT(sbi)))
+ goto not_found;
}
}
@@ -1176,6 +1183,7 @@ int exfat_count_dir_entries(struct super_block *sb, struct exfat_chain *p_dir)
int i, count = 0;
int dentries_per_clu;
unsigned int entry_type;
+ unsigned int clu_count = 0;
struct exfat_chain clu;
struct exfat_dentry *ep;
struct exfat_sb_info *sbi = EXFAT_SB(sb);
@@ -1208,6 +1216,12 @@ int exfat_count_dir_entries(struct super_block *sb, struct exfat_chain *p_dir)
} else {
if (exfat_get_next_cluster(sb, &(clu.dir)))
return -EIO;
+
+ if (unlikely(++clu_count > sbi->used_clusters)) {
+ exfat_fs_error(sb, "FAT or bitmap is corrupted");
+ return -EIO;
+ }
+
}
}
diff --git a/fs/exfat/exfat_fs.h b/fs/exfat/exfat_fs.h
index 5a1251207ab2..eae52c5b20b2 100644
--- a/fs/exfat/exfat_fs.h
+++ b/fs/exfat/exfat_fs.h
@@ -62,7 +62,6 @@ enum {
#define TYPE_PADDING 0x0402
#define TYPE_ACLTAB 0x0403
#define TYPE_BENIGN_SEC 0x0800
-#define TYPE_ALL 0x0FFF
#define MAX_CHARSET_SIZE 6 /* max size of multi-byte character */
#define MAX_NAME_LENGTH 255 /* max len of file name excluding NULL */
@@ -467,7 +466,7 @@ void exfat_update_dir_chksum_with_entry_set(struct exfat_entry_set_cache *es);
int exfat_calc_num_entries(struct exfat_uni_name *p_uniname);
int exfat_find_dir_entry(struct super_block *sb, struct exfat_inode_info *ei,
struct exfat_chain *p_dir, struct exfat_uni_name *p_uniname,
- int num_entries, unsigned int type, struct exfat_hint *hint_opt);
+ struct exfat_hint *hint_opt);
int exfat_alloc_new_dir(struct inode *inode, struct exfat_chain *clu);
struct exfat_dentry *exfat_get_dentry(struct super_block *sb,
struct exfat_chain *p_dir, int entry, struct buffer_head **bh);
diff --git a/fs/exfat/fatent.c b/fs/exfat/fatent.c
index e7e160d02256..bacd68bc7b8a 100644
--- a/fs/exfat/fatent.c
+++ b/fs/exfat/fatent.c
@@ -470,5 +470,15 @@ int exfat_count_num_clusters(struct super_block *sb,
}
*ret_count = count;
+
+ /*
+ * since exfat_count_used_clusters() is not called, sbi->used_clusters
+ * cannot be used here.
+ */
+ if (unlikely(i == sbi->num_clusters && clu != EXFAT_EOF_CLUSTER)) {
+ exfat_fs_error(sb, "The cluster chain has a loop");
+ return -EIO;
+ }
+
return 0;
}
diff --git a/fs/exfat/namei.c b/fs/exfat/namei.c
index 51743ea61733..605c9d617d7c 100644
--- a/fs/exfat/namei.c
+++ b/fs/exfat/namei.c
@@ -600,7 +600,7 @@ static int exfat_create(struct user_namespace *mnt_userns, struct inode *dir,
static int exfat_find(struct inode *dir, struct qstr *qname,
struct exfat_dir_entry *info)
{
- int ret, dentry, num_entries, count;
+ int ret, dentry, count;
struct exfat_chain cdir;
struct exfat_uni_name uni_name;
struct super_block *sb = dir->i_sb;
@@ -619,10 +619,6 @@ static int exfat_find(struct inode *dir, struct qstr *qname,
if (ret)
return ret;
- num_entries = exfat_calc_num_entries(&uni_name);
- if (num_entries < 0)
- return num_entries;
-
/* check the validation of hint_stat and initialize it if required */
if (ei->version != (inode_peek_iversion_raw(dir) & 0xffffffff)) {
ei->hint_stat.clu = cdir.dir;
@@ -632,9 +628,7 @@ static int exfat_find(struct inode *dir, struct qstr *qname,
}
/* search the file name for directories */
- dentry = exfat_find_dir_entry(sb, ei, &cdir, &uni_name,
- num_entries, TYPE_ALL, &hint_opt);
-
+ dentry = exfat_find_dir_entry(sb, ei, &cdir, &uni_name, &hint_opt);
if (dentry < 0)
return dentry; /* -error value */
@@ -656,14 +650,19 @@ static int exfat_find(struct inode *dir, struct qstr *qname,
info->type = exfat_get_entry_type(ep);
info->attr = le16_to_cpu(ep->dentry.file.attr);
info->size = le64_to_cpu(ep2->dentry.stream.valid_size);
+
+ info->start_clu = le32_to_cpu(ep2->dentry.stream.start_clu);
+ if (!is_valid_cluster(sbi, info->start_clu) && info->size) {
+ exfat_warn(sb, "start_clu is invalid cluster(0x%x)",
+ info->start_clu);
+ info->size = 0;
+ }
+
if (info->size == 0) {
info->flags = ALLOC_NO_FAT_CHAIN;
info->start_clu = EXFAT_EOF_CLUSTER;
- } else {
+ } else
info->flags = ep2->dentry.stream.flags;
- info->start_clu =
- le32_to_cpu(ep2->dentry.stream.start_clu);
- }
exfat_get_entry_time(sbi, &info->crtime,
ep->dentry.file.create_tz,
@@ -893,6 +892,7 @@ static int exfat_check_dir_empty(struct super_block *sb,
{
int i, dentries_per_clu;
unsigned int type;
+ unsigned int clu_count = 0;
struct exfat_chain clu;
struct exfat_dentry *ep;
struct exfat_sb_info *sbi = EXFAT_SB(sb);
@@ -929,6 +929,10 @@ static int exfat_check_dir_empty(struct super_block *sb,
} else {
if (exfat_get_next_cluster(sb, &(clu.dir)))
return -EIO;
+
+ /* break if the cluster chain includes a loop */
+ if (unlikely(++clu_count > EXFAT_DATA_CLUSTER_COUNT(sbi)))
+ break;
}
}
diff --git a/fs/exfat/nls.c b/fs/exfat/nls.c
index 0e05c6b5a286..4bfe807716de 100644
--- a/fs/exfat/nls.c
+++ b/fs/exfat/nls.c
@@ -778,15 +778,23 @@ int exfat_create_upcase_table(struct super_block *sb)
tbl_clu = le32_to_cpu(ep->dentry.upcase.start_clu);
tbl_size = le64_to_cpu(ep->dentry.upcase.size);
-
- sector = exfat_cluster_to_sector(sbi, tbl_clu);
- num_sectors = ((tbl_size - 1) >> blksize_bits) + 1;
- ret = exfat_load_upcase_table(sb, sector, num_sectors,
- le32_to_cpu(ep->dentry.upcase.checksum));
-
+ if (tbl_size) {
+ sector = exfat_cluster_to_sector(sbi, tbl_clu);
+ num_sectors = ((tbl_size - 1) >> blksize_bits) + 1;
+ ret = exfat_load_upcase_table(sb, sector, num_sectors,
+ le32_to_cpu(ep->dentry.upcase.checksum));
+ } else {
+ exfat_fs_error(sb,
+ "bad upcase table size (0 bytes). Please run fsck");
+ ret = -EINVAL;
+ }
brelse(bh);
- if (ret && ret != -EIO)
+
+ if (ret && ret != -EIO) {
+ /* free memory from exfat_load_upcase_table call */
+ exfat_free_upcase_table(sbi);
goto load_default;
+ }
/* load successfully */
return ret;
@@ -796,6 +804,8 @@ int exfat_create_upcase_table(struct super_block *sb)
return -EIO;
}
+ exfat_fs_error(sb, "no upcase table entry. Please run fsck");
+
load_default:
/* load default upcase table */
return exfat_load_default_upcase_table(sb);
diff --git a/fs/exfat/super.c b/fs/exfat/super.c
index 52b852a4121b..e377fbc6e847 100644
--- a/fs/exfat/super.c
+++ b/fs/exfat/super.c
@@ -336,13 +336,12 @@ static void exfat_hash_init(struct super_block *sb)
INIT_HLIST_HEAD(&sbi->inode_hashtable[i]);
}
-static int exfat_read_root(struct inode *inode)
+static int exfat_read_root(struct inode *inode, struct exfat_chain *root_clu)
{
struct super_block *sb = inode->i_sb;
struct exfat_sb_info *sbi = EXFAT_SB(sb);
struct exfat_inode_info *ei = EXFAT_I(inode);
- struct exfat_chain cdir;
- int num_subdirs, num_clu = 0;
+ int num_subdirs;
exfat_chain_set(&ei->dir, sbi->root_dir, 0, ALLOC_FAT_CHAIN);
ei->entry = -1;
@@ -355,12 +354,9 @@ static int exfat_read_root(struct inode *inode)
ei->hint_stat.clu = sbi->root_dir;
ei->hint_femp.eidx = EXFAT_HINT_NONE;
- exfat_chain_set(&cdir, sbi->root_dir, 0, ALLOC_FAT_CHAIN);
- if (exfat_count_num_clusters(sb, &cdir, &num_clu))
- return -EIO;
- i_size_write(inode, num_clu << sbi->cluster_size_bits);
+ i_size_write(inode, EXFAT_CLU_TO_B(root_clu->size, sbi));
- num_subdirs = exfat_count_dir_entries(sb, &cdir);
+ num_subdirs = exfat_count_dir_entries(sb, root_clu);
if (num_subdirs < 0)
return -EIO;
set_nlink(inode, num_subdirs + EXFAT_MIN_SUBDIR);
@@ -580,7 +576,8 @@ static int exfat_verify_boot_region(struct super_block *sb)
}
/* mount the file system volume */
-static int __exfat_fill_super(struct super_block *sb)
+static int __exfat_fill_super(struct super_block *sb,
+ struct exfat_chain *root_clu)
{
int ret;
struct exfat_sb_info *sbi = EXFAT_SB(sb);
@@ -597,6 +594,18 @@ static int __exfat_fill_super(struct super_block *sb)
goto free_bh;
}
+ /*
+ * Call exfat_count_num_cluster() before searching for up-case and
+ * bitmap directory entries to avoid infinite loop if they are missing
+ * and the cluster chain includes a loop.
+ */
+ exfat_chain_set(root_clu, sbi->root_dir, 0, ALLOC_FAT_CHAIN);
+ ret = exfat_count_num_clusters(sb, root_clu, &root_clu->size);
+ if (ret) {
+ exfat_err(sb, "failed to count the number of clusters in root");
+ goto free_bh;
+ }
+
ret = exfat_create_upcase_table(sb);
if (ret) {
exfat_err(sb, "failed to load upcase table");
@@ -631,6 +640,7 @@ static int exfat_fill_super(struct super_block *sb, struct fs_context *fc)
struct exfat_sb_info *sbi = sb->s_fs_info;
struct exfat_mount_options *opts = &sbi->options;
struct inode *root_inode;
+ struct exfat_chain root_clu;
int err;
if (opts->allow_utime == (unsigned short)-1)
@@ -649,7 +659,7 @@ static int exfat_fill_super(struct super_block *sb, struct fs_context *fc)
sb->s_time_min = EXFAT_MIN_TIMESTAMP_SECS;
sb->s_time_max = EXFAT_MAX_TIMESTAMP_SECS;
- err = __exfat_fill_super(sb);
+ err = __exfat_fill_super(sb, &root_clu);
if (err) {
exfat_err(sb, "failed to recognize exfat type");
goto check_nls_io;
@@ -684,7 +694,7 @@ static int exfat_fill_super(struct super_block *sb, struct fs_context *fc)
root_inode->i_ino = EXFAT_ROOT_INO;
inode_set_iversion(root_inode, 1);
- err = exfat_read_root(root_inode);
+ err = exfat_read_root(root_inode, &root_clu);
if (err) {
exfat_err(sb, "failed to initialize root inode");
goto put_inode;
diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c
index e5362c376a7a..bae9d5897696 100644
--- a/fs/ext2/inode.c
+++ b/fs/ext2/inode.c
@@ -1289,11 +1289,10 @@ static int ext2_setsize(struct inode *inode, loff_t newsize)
filemap_invalidate_unlock(inode->i_mapping);
inode->i_mtime = inode->i_ctime = current_time(inode);
+ mark_inode_dirty(inode);
if (inode_needs_sync(inode)) {
sync_mapping_buffers(inode->i_mapping);
sync_inode_metadata(inode, 1);
- } else {
- mark_inode_dirty(inode);
}
return 0;
diff --git a/fs/ext2/xattr.c b/fs/ext2/xattr.c
index 2f89b1073307..fd06fb19528f 100644
--- a/fs/ext2/xattr.c
+++ b/fs/ext2/xattr.c
@@ -776,6 +776,7 @@ ext2_xattr_set2(struct inode *inode, struct buffer_head *old_bh,
/* Update the inode. */
EXT2_I(inode)->i_file_acl = new_bh ? new_bh->b_blocknr : 0;
inode->i_ctime = current_time(inode);
+ mark_inode_dirty(inode);
if (IS_SYNC(inode)) {
error = sync_inode_metadata(inode, 1);
/* In case sync failed due to ENOSPC the inode was actually
@@ -788,8 +789,7 @@ ext2_xattr_set2(struct inode *inode, struct buffer_head *old_bh,
}
goto cleanup;
}
- } else
- mark_inode_dirty(inode);
+ }
error = 0;
if (old_bh && old_bh != new_bh) {
diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h
index b1a4fcfdb548..ae786abecc9a 100644
--- a/fs/ext4/ext4.h
+++ b/fs/ext4/ext4.h
@@ -1688,6 +1688,8 @@ struct ext4_sb_info {
struct list_head s_es_list; /* List of inodes with reclaimable extents */
long s_es_nr_inode;
struct ext4_es_stats s_es_stats;
+ /* ext4_get_link() reads the sbi in rcu pathwalk */
+ struct rcu_head s_rcu;
struct mb_cache *s_ea_block_cache;
struct mb_cache *s_ea_inode_cache;
spinlock_t s_es_lock ____cacheline_aligned_in_smp;
@@ -1778,6 +1780,30 @@ static inline struct ext4_inode_info *EXT4_I(struct inode *inode)
return container_of(inode, struct ext4_inode_info, vfs_inode);
}
+static inline int ext4_writepages_down_read(struct super_block *sb)
+{
+ percpu_down_read(&EXT4_SB(sb)->s_writepages_rwsem);
+ return memalloc_nofs_save();
+}
+
+static inline void ext4_writepages_up_read(struct super_block *sb, int ctx)
+{
+ memalloc_nofs_restore(ctx);
+ percpu_up_read(&EXT4_SB(sb)->s_writepages_rwsem);
+}
+
+static inline int ext4_writepages_down_write(struct super_block *sb)
+{
+ percpu_down_write(&EXT4_SB(sb)->s_writepages_rwsem);
+ return memalloc_nofs_save();
+}
+
+static inline void ext4_writepages_up_write(struct super_block *sb, int ctx)
+{
+ memalloc_nofs_restore(ctx);
+ percpu_up_write(&EXT4_SB(sb)->s_writepages_rwsem);
+}
+
static inline int ext4_valid_inum(struct super_block *sb, unsigned long ino)
{
return ino == EXT4_ROOT_INO ||
@@ -2984,7 +3010,8 @@ typedef enum {
EXT4_IGET_SPECIAL = 0x0001, /* OK to iget a system inode */
EXT4_IGET_HANDLE = 0x0002, /* Inode # is from a handle */
EXT4_IGET_BAD = 0x0004, /* Allow to iget a bad inode */
- EXT4_IGET_EA_INODE = 0x0008 /* Inode should contain an EA value */
+ EXT4_IGET_EA_INODE = 0x0008, /* Inode should contain an EA value */
+ EXT4_IGET_NOWAIT = 0x0010 /* Non-blocking lookup (skip if freeing) */
} ext4_iget_flags;
extern struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c
index 6d95dab53847..ed62b5247358 100644
--- a/fs/ext4/extents.c
+++ b/fs/ext4/extents.c
@@ -3268,7 +3268,8 @@ static int ext4_split_extent_at(handle_t *handle,
if (IS_ERR(path)) {
EXT4_ERROR_INODE(inode, "Failed split extent on %u, err %ld",
split, PTR_ERR(path));
- return PTR_ERR(path);
+ err = PTR_ERR(path);
+ goto out_err;
}
depth = ext_depth(inode);
ex = path[depth].p_ext;
@@ -3323,9 +3324,12 @@ static int ext4_split_extent_at(handle_t *handle,
* and err is a non-zero error code.
*/
ext4_ext_dirty(handle, inode, path + path->p_depth);
- return err;
out:
ext4_ext_show_leaf(inode, *ppath);
+out_err:
+ if (err)
+ /* Remove all remaining potentially stale extents. */
+ ext4_es_remove_extent(inode, ee_block, ee_len);
return err;
}
@@ -3726,11 +3730,15 @@ static int ext4_split_convert_extents(handle_t *handle,
/* Convert to unwritten */
if (flags & EXT4_GET_BLOCKS_CONVERT_UNWRITTEN) {
split_flag |= EXT4_EXT_DATA_VALID1;
- /* Convert to initialized */
- } else if (flags & EXT4_GET_BLOCKS_CONVERT) {
+ /* Split the existing unwritten extent */
+ } else if (flags & (EXT4_GET_BLOCKS_UNWRIT_EXT |
+ EXT4_GET_BLOCKS_CONVERT)) {
split_flag |= ee_block + ee_len <= eof_block ?
EXT4_EXT_MAY_ZEROOUT : 0;
- split_flag |= (EXT4_EXT_MARK_UNWRIT2 | EXT4_EXT_DATA_VALID2);
+ split_flag |= EXT4_EXT_MARK_UNWRIT2;
+ /* Convert to initialized */
+ if (flags & EXT4_GET_BLOCKS_CONVERT)
+ split_flag |= EXT4_EXT_DATA_VALID2;
}
flags |= EXT4_GET_BLOCKS_PRE_IO;
return ext4_split_extent(handle, inode, ppath, map, split_flag, flags);
@@ -3895,7 +3903,7 @@ ext4_ext_handle_unwritten_extents(handle_t *handle, struct inode *inode,
/* get_block() before submitting IO, split the extent */
if (flags & EXT4_GET_BLOCKS_PRE_IO) {
ret = ext4_split_convert_extents(handle, inode, map, ppath,
- flags | EXT4_GET_BLOCKS_CONVERT);
+ flags);
if (ret < 0) {
err = ret;
goto out2;
diff --git a/fs/ext4/extents_status.c b/fs/ext4/extents_status.c
index 592229027af7..3744d49ef285 100644
--- a/fs/ext4/extents_status.c
+++ b/fs/ext4/extents_status.c
@@ -1751,12 +1751,17 @@ int ext4_es_register_shrinker(struct ext4_sb_info *sbi)
}
void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi)
+{
+ unregister_shrinker(&sbi->s_es_shrinker);
+}
+
+/* ext4_get_link() bumps the hit and miss counters in rcu pathwalk */
+void ext4_es_destroy_stats(struct ext4_sb_info *sbi)
{
percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_hits);
percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_misses);
percpu_counter_destroy(&sbi->s_es_stats.es_stats_all_cnt);
percpu_counter_destroy(&sbi->s_es_stats.es_stats_shk_cnt);
- unregister_shrinker(&sbi->s_es_shrinker);
}
/*
diff --git a/fs/ext4/extents_status.h b/fs/ext4/extents_status.h
index 481ec4381bee..8a03771269b6 100644
--- a/fs/ext4/extents_status.h
+++ b/fs/ext4/extents_status.h
@@ -241,6 +241,7 @@ static inline void ext4_es_store_pblock_status(struct extent_status *es,
extern int ext4_es_register_shrinker(struct ext4_sb_info *sbi);
extern void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi);
+extern void ext4_es_destroy_stats(struct ext4_sb_info *sbi);
extern int ext4_seq_es_shrinker_info_show(struct seq_file *seq, void *v);
diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c
index ef801e13ca46..2739c9a04c8f 100644
--- a/fs/ext4/inode.c
+++ b/fs/ext4/inode.c
@@ -1451,7 +1451,8 @@ static int ext4_journalled_write_end(struct file *file,
BUG_ON(!ext4_handle_valid(handle));
- if (ext4_has_inline_data(inode))
+ if (ext4_has_inline_data(inode) &&
+ ext4_test_inode_state(inode, EXT4_STATE_MAY_INLINE_DATA))
return ext4_write_inline_data_end(inode, pos, len, copied, page);
if (unlikely(copied < len) && !PageUptodate(page)) {
@@ -2710,11 +2711,12 @@ static int ext4_writepages(struct address_space *mapping,
struct ext4_sb_info *sbi = EXT4_SB(mapping->host->i_sb);
struct blk_plug plug;
bool give_up_on_write = false;
+ int alloc_ctx;
if (unlikely(ext4_forced_shutdown(EXT4_SB(inode->i_sb))))
return -EIO;
- percpu_down_read(&sbi->s_writepages_rwsem);
+ alloc_ctx = ext4_writepages_down_read(inode->i_sb);
trace_ext4_writepages(inode, wbc);
/*
@@ -2924,7 +2926,7 @@ static int ext4_writepages(struct address_space *mapping,
out_writepages:
trace_ext4_writepages_result(inode, wbc, ret,
nr_to_write - wbc->nr_to_write);
- percpu_up_read(&sbi->s_writepages_rwsem);
+ ext4_writepages_up_read(inode->i_sb, alloc_ctx);
return ret;
}
@@ -2935,17 +2937,18 @@ static int ext4_dax_writepages(struct address_space *mapping,
long nr_to_write = wbc->nr_to_write;
struct inode *inode = mapping->host;
struct ext4_sb_info *sbi = EXT4_SB(mapping->host->i_sb);
+ int alloc_ctx;
if (unlikely(ext4_forced_shutdown(EXT4_SB(inode->i_sb))))
return -EIO;
- percpu_down_read(&sbi->s_writepages_rwsem);
+ alloc_ctx = ext4_writepages_down_read(inode->i_sb);
trace_ext4_writepages(inode, wbc);
ret = dax_writeback_mapping_range(mapping, sbi->s_daxdev, wbc);
trace_ext4_writepages_result(inode, wbc, ret,
nr_to_write - wbc->nr_to_write);
- percpu_up_read(&sbi->s_writepages_rwsem);
+ ext4_writepages_up_read(inode->i_sb, alloc_ctx);
return ret;
}
@@ -4834,6 +4837,21 @@ static int check_igot_inode(struct inode *inode, ext4_iget_flags flags,
return -EFSCORRUPTED;
}
+static int ext4_iget_match(struct inode *inode, unsigned long ino,
+ void *data)
+{
+ if (inode->i_ino != ino)
+ return 0;
+ spin_lock(&inode->i_lock);
+ if (inode->i_state & (I_FREEING | I_WILL_FREE | I_CREATING)) {
+ spin_unlock(&inode->i_lock);
+ return -1;
+ }
+ __iget(inode);
+ spin_unlock(&inode->i_lock);
+ return 1;
+}
+
struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
ext4_iget_flags flags, const char *function,
unsigned int line)
@@ -4867,9 +4885,24 @@ struct inode *__ext4_iget(struct super_block *sb, unsigned long ino,
return ERR_PTR(-EFSCORRUPTED);
}
- inode = iget_locked(sb, ino);
- if (!inode)
- return ERR_PTR(-ENOMEM);
+ if (flags & EXT4_IGET_NOWAIT) {
+ inode = find_inode_nowait(sb, ino, ext4_iget_match, NULL);
+ if (!inode)
+ return ERR_PTR(-ENOENT);
+
+ if (inode->i_state & I_NEW)
+ wait_on_inode(inode);
+
+ if (unlikely(inode_unhashed(inode))) {
+ iput(inode);
+ return ERR_PTR(-ENOENT);
+ }
+ } else {
+ inode = iget_locked(sb, ino);
+ if (!inode)
+ return ERR_PTR(-ENOMEM);
+ }
+
if (!(inode->i_state & I_NEW)) {
ret = check_igot_inode(inode, flags, function, line);
if (ret) {
@@ -6172,7 +6205,7 @@ int ext4_change_inode_journal_flag(struct inode *inode, int val)
journal_t *journal;
handle_t *handle;
int err;
- struct ext4_sb_info *sbi = EXT4_SB(inode->i_sb);
+ int alloc_ctx;
/*
* We have to be very careful here: changing a data block's
@@ -6210,7 +6243,7 @@ int ext4_change_inode_journal_flag(struct inode *inode, int val)
}
}
- percpu_down_write(&sbi->s_writepages_rwsem);
+ alloc_ctx = ext4_writepages_down_write(inode->i_sb);
jbd2_journal_lock_updates(journal);
/*
@@ -6227,7 +6260,7 @@ int ext4_change_inode_journal_flag(struct inode *inode, int val)
err = jbd2_journal_flush(journal, 0);
if (err < 0) {
jbd2_journal_unlock_updates(journal);
- percpu_up_write(&sbi->s_writepages_rwsem);
+ ext4_writepages_up_write(inode->i_sb, alloc_ctx);
return err;
}
ext4_clear_inode_flag(inode, EXT4_INODE_JOURNAL_DATA);
@@ -6235,7 +6268,7 @@ int ext4_change_inode_journal_flag(struct inode *inode, int val)
ext4_set_aops(inode);
jbd2_journal_unlock_updates(journal);
- percpu_up_write(&sbi->s_writepages_rwsem);
+ ext4_writepages_up_write(inode->i_sb, alloc_ctx);
if (val)
filemap_invalidate_unlock(inode->i_mapping);
diff --git a/fs/ext4/ioctl.c b/fs/ext4/ioctl.c
index a6d5a4e37a28..8cee51df4bbd 100644
--- a/fs/ext4/ioctl.c
+++ b/fs/ext4/ioctl.c
@@ -1348,6 +1348,11 @@ static long __ext4_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
goto mext_out;
}
+ if (file_inode(filp)->i_sb != file_inode(donor.file)->i_sb) {
+ err = -EXDEV;
+ goto mext_out;
+ }
+
if (ext4_has_feature_bigalloc(sb)) {
ext4_msg(sb, KERN_ERR,
"Online defrag not supported with bigalloc");
diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c
index df6fa0a56855..e77d56c533c0 100644
--- a/fs/ext4/mballoc.c
+++ b/fs/ext4/mballoc.c
@@ -1351,7 +1351,7 @@ static int ext4_mb_init_cache(struct page *page, char *incore, gfp_t gfp)
* Lock the buddy and bitmap pages. This make sure other parallel init_group
* on the same buddy page doesn't happen whild holding the buddy page lock.
* Return locked buddy and bitmap pages on e4b struct. If buddy and bitmap
- * are on the same page e4b->bd_buddy_page is NULL and return value is 0.
+ * are on the same page e4b->bd_buddy_folio is NULL and return value is 0.
*/
static int ext4_mb_get_buddy_page_lock(struct super_block *sb,
ext4_group_t group, struct ext4_buddy *e4b, gfp_t gfp)
@@ -1359,10 +1359,10 @@ static int ext4_mb_get_buddy_page_lock(struct super_block *sb,
struct inode *inode = EXT4_SB(sb)->s_buddy_cache;
int block, pnum, poff;
int blocks_per_page;
- struct page *page;
+ struct folio *folio;
- e4b->bd_buddy_page = NULL;
- e4b->bd_bitmap_page = NULL;
+ e4b->bd_buddy_folio = NULL;
+ e4b->bd_bitmap_folio = NULL;
blocks_per_page = PAGE_SIZE / sb->s_blocksize;
/*
@@ -1373,37 +1373,38 @@ static int ext4_mb_get_buddy_page_lock(struct super_block *sb,
block = group * 2;
pnum = block / blocks_per_page;
poff = block % blocks_per_page;
- page = find_or_create_page(inode->i_mapping, pnum, gfp);
- if (!page)
- return -ENOMEM;
- BUG_ON(page->mapping != inode->i_mapping);
- e4b->bd_bitmap_page = page;
- e4b->bd_bitmap = page_address(page) + (poff * sb->s_blocksize);
+ folio = __filemap_get_folio(inode->i_mapping, pnum,
+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
+ if (IS_ERR(folio))
+ return PTR_ERR(folio);
+ BUG_ON(folio->mapping != inode->i_mapping);
+ e4b->bd_bitmap_folio = folio;
+ e4b->bd_bitmap = folio_address(folio) + (poff * sb->s_blocksize);
if (blocks_per_page >= 2) {
/* buddy and bitmap are on the same page */
return 0;
}
- block++;
- pnum = block / blocks_per_page;
- page = find_or_create_page(inode->i_mapping, pnum, gfp);
- if (!page)
- return -ENOMEM;
- BUG_ON(page->mapping != inode->i_mapping);
- e4b->bd_buddy_page = page;
+ /* blocks_per_page == 1, hence we need another page for the buddy */
+ folio = __filemap_get_folio(inode->i_mapping, block + 1,
+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
+ if (IS_ERR(folio))
+ return PTR_ERR(folio);
+ BUG_ON(folio->mapping != inode->i_mapping);
+ e4b->bd_buddy_folio = folio;
return 0;
}
static void ext4_mb_put_buddy_page_lock(struct ext4_buddy *e4b)
{
- if (e4b->bd_bitmap_page) {
- unlock_page(e4b->bd_bitmap_page);
- put_page(e4b->bd_bitmap_page);
+ if (e4b->bd_bitmap_folio) {
+ folio_unlock(e4b->bd_bitmap_folio);
+ folio_put(e4b->bd_bitmap_folio);
}
- if (e4b->bd_buddy_page) {
- unlock_page(e4b->bd_buddy_page);
- put_page(e4b->bd_buddy_page);
+ if (e4b->bd_buddy_folio) {
+ folio_unlock(e4b->bd_buddy_folio);
+ folio_put(e4b->bd_buddy_folio);
}
}
@@ -1418,7 +1419,7 @@ int ext4_mb_init_group(struct super_block *sb, ext4_group_t group, gfp_t gfp)
struct ext4_group_info *this_grp;
struct ext4_buddy e4b;
- struct page *page;
+ struct folio *folio;
int ret = 0;
might_sleep();
@@ -1445,16 +1446,16 @@ int ext4_mb_init_group(struct super_block *sb, ext4_group_t group, gfp_t gfp)
goto err;
}
- page = e4b.bd_bitmap_page;
- ret = ext4_mb_init_cache(page, NULL, gfp);
+ folio = e4b.bd_bitmap_folio;
+ ret = ext4_mb_init_cache(&folio->page, NULL, gfp);
if (ret)
goto err;
- if (!PageUptodate(page)) {
+ if (!folio_test_uptodate(folio)) {
ret = -EIO;
goto err;
}
- if (e4b.bd_buddy_page == NULL) {
+ if (e4b.bd_buddy_folio == NULL) {
/*
* If both the bitmap and buddy are in
* the same page we don't need to force
@@ -1464,11 +1465,11 @@ int ext4_mb_init_group(struct super_block *sb, ext4_group_t group, gfp_t gfp)
goto err;
}
/* init buddy cache */
- page = e4b.bd_buddy_page;
- ret = ext4_mb_init_cache(page, e4b.bd_bitmap, gfp);
+ folio = e4b.bd_buddy_folio;
+ ret = ext4_mb_init_cache(&folio->page, e4b.bd_bitmap, gfp);
if (ret)
goto err;
- if (!PageUptodate(page)) {
+ if (!folio_test_uptodate(folio)) {
ret = -EIO;
goto err;
}
@@ -1490,7 +1491,7 @@ ext4_mb_load_buddy_gfp(struct super_block *sb, ext4_group_t group,
int block;
int pnum;
int poff;
- struct page *page;
+ struct folio *folio;
int ret;
struct ext4_group_info *grp;
struct ext4_sb_info *sbi = EXT4_SB(sb);
@@ -1508,8 +1509,8 @@ ext4_mb_load_buddy_gfp(struct super_block *sb, ext4_group_t group,
e4b->bd_info = grp;
e4b->bd_sb = sb;
e4b->bd_group = group;
- e4b->bd_buddy_page = NULL;
- e4b->bd_bitmap_page = NULL;
+ e4b->bd_buddy_folio = NULL;
+ e4b->bd_bitmap_folio = NULL;
if (unlikely(EXT4_MB_GRP_NEED_INIT(grp))) {
/*
@@ -1530,92 +1531,105 @@ ext4_mb_load_buddy_gfp(struct super_block *sb, ext4_group_t group,
pnum = block / blocks_per_page;
poff = block % blocks_per_page;
- /* we could use find_or_create_page(), but it locks page
- * what we'd like to avoid in fast path ... */
- page = find_get_page_flags(inode->i_mapping, pnum, FGP_ACCESSED);
- if (page == NULL || !PageUptodate(page)) {
- if (page)
- /*
- * drop the page reference and try
- * to get the page with lock. If we
- * are not uptodate that implies
- * somebody just created the page but
- * is yet to initialize the same. So
- * wait for it to initialize.
- */
- put_page(page);
- page = find_or_create_page(inode->i_mapping, pnum, gfp);
- if (page) {
- BUG_ON(page->mapping != inode->i_mapping);
- if (!PageUptodate(page)) {
- ret = ext4_mb_init_cache(page, NULL, gfp);
+ /* Avoid locking the folio in the fast path ... */
+ folio = __filemap_get_folio(inode->i_mapping, pnum, FGP_ACCESSED, 0);
+ if (IS_ERR(folio) || !folio_test_uptodate(folio) || folio_test_locked(folio)) {
+ /*
+ * folio_test_locked is employed to detect ongoing folio
+ * migrations, since concurrent migrations can lead to
+ * bitmap inconsistency. And if we are not uptodate that
+ * implies somebody just created the folio but is yet to
+ * initialize it. We can drop the folio reference and
+ * try to get the folio with lock in both cases to avoid
+ * concurrency.
+ */
+ if (!IS_ERR(folio))
+ folio_put(folio);
+ folio = __filemap_get_folio(inode->i_mapping, pnum,
+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
+ if (!IS_ERR(folio)) {
+ if (WARN_RATELIMIT(folio->mapping != inode->i_mapping,
+ "ext4: bitmap's mapping != inode->i_mapping\n")) {
+ /* should never happen */
+ folio_unlock(folio);
+ ret = -EINVAL;
+ goto err;
+ }
+ if (!folio_test_uptodate(folio)) {
+ ret = ext4_mb_init_cache(&folio->page, NULL, gfp);
if (ret) {
- unlock_page(page);
+ folio_unlock(folio);
goto err;
}
- mb_cmp_bitmaps(e4b, page_address(page) +
+ mb_cmp_bitmaps(e4b, folio_address(folio) +
(poff * sb->s_blocksize));
}
- unlock_page(page);
+ folio_unlock(folio);
}
}
- if (page == NULL) {
- ret = -ENOMEM;
+ if (IS_ERR(folio)) {
+ ret = PTR_ERR(folio);
goto err;
}
- if (!PageUptodate(page)) {
+ if (!folio_test_uptodate(folio)) {
ret = -EIO;
goto err;
}
- /* Pages marked accessed already */
- e4b->bd_bitmap_page = page;
- e4b->bd_bitmap = page_address(page) + (poff * sb->s_blocksize);
+ /* Folios marked accessed already */
+ e4b->bd_bitmap_folio = folio;
+ e4b->bd_bitmap = folio_address(folio) + (poff * sb->s_blocksize);
block++;
pnum = block / blocks_per_page;
poff = block % blocks_per_page;
- page = find_get_page_flags(inode->i_mapping, pnum, FGP_ACCESSED);
- if (page == NULL || !PageUptodate(page)) {
- if (page)
- put_page(page);
- page = find_or_create_page(inode->i_mapping, pnum, gfp);
- if (page) {
- BUG_ON(page->mapping != inode->i_mapping);
- if (!PageUptodate(page)) {
- ret = ext4_mb_init_cache(page, e4b->bd_bitmap,
+ folio = __filemap_get_folio(inode->i_mapping, pnum, FGP_ACCESSED, 0);
+ if (IS_ERR(folio) || !folio_test_uptodate(folio) || folio_test_locked(folio)) {
+ if (!IS_ERR(folio))
+ folio_put(folio);
+ folio = __filemap_get_folio(inode->i_mapping, pnum,
+ FGP_LOCK | FGP_ACCESSED | FGP_CREAT, gfp);
+ if (!IS_ERR(folio)) {
+ if (WARN_RATELIMIT(folio->mapping != inode->i_mapping,
+ "ext4: buddy bitmap's mapping != inode->i_mapping\n")) {
+ /* should never happen */
+ folio_unlock(folio);
+ ret = -EINVAL;
+ goto err;
+ }
+ if (!folio_test_uptodate(folio)) {
+ ret = ext4_mb_init_cache(&folio->page, e4b->bd_bitmap,
gfp);
if (ret) {
- unlock_page(page);
+ folio_unlock(folio);
goto err;
}
}
- unlock_page(page);
+ folio_unlock(folio);
}
}
- if (page == NULL) {
- ret = -ENOMEM;
+ if (IS_ERR(folio)) {
+ ret = PTR_ERR(folio);
goto err;
}
- if (!PageUptodate(page)) {
+ if (!folio_test_uptodate(folio)) {
ret = -EIO;
goto err;
}
- /* Pages marked accessed already */
- e4b->bd_buddy_page = page;
- e4b->bd_buddy = page_address(page) + (poff * sb->s_blocksize);
+ /* Folios marked accessed already */
+ e4b->bd_buddy_folio = folio;
+ e4b->bd_buddy = folio_address(folio) + (poff * sb->s_blocksize);
return 0;
err:
- if (page)
- put_page(page);
- if (e4b->bd_bitmap_page)
- put_page(e4b->bd_bitmap_page);
- if (e4b->bd_buddy_page)
- put_page(e4b->bd_buddy_page);
+ if (!IS_ERR_OR_NULL(folio))
+ folio_put(folio);
+ if (e4b->bd_bitmap_folio)
+ folio_put(e4b->bd_bitmap_folio);
+
e4b->bd_buddy = NULL;
e4b->bd_bitmap = NULL;
return ret;
@@ -1629,10 +1643,10 @@ static int ext4_mb_load_buddy(struct super_block *sb, ext4_group_t group,
static void ext4_mb_unload_buddy(struct ext4_buddy *e4b)
{
- if (e4b->bd_bitmap_page)
- put_page(e4b->bd_bitmap_page);
- if (e4b->bd_buddy_page)
- put_page(e4b->bd_buddy_page);
+ if (e4b->bd_bitmap_folio)
+ folio_put(e4b->bd_bitmap_folio);
+ if (e4b->bd_buddy_folio)
+ folio_put(e4b->bd_buddy_folio);
}
@@ -2055,9 +2069,9 @@ static void ext4_mb_use_best_found(struct ext4_allocation_context *ac,
* double allocate blocks. The reference is dropped
* in ext4_mb_release_context
*/
- ac->ac_bitmap_page = e4b->bd_bitmap_page;
+ ac->ac_bitmap_page = &e4b->bd_bitmap_folio->page;
get_page(ac->ac_bitmap_page);
- ac->ac_buddy_page = e4b->bd_buddy_page;
+ ac->ac_buddy_page = &e4b->bd_buddy_folio->page;
get_page(ac->ac_buddy_page);
/* store last allocated for subsequent stream allocation */
if (ac->ac_flags & EXT4_MB_STREAM_ALLOC) {
@@ -2306,7 +2320,9 @@ void ext4_mb_simple_scan_group(struct ext4_allocation_context *ac,
continue;
buddy = mb_find_buddy(e4b, i, &max);
- BUG_ON(buddy == NULL);
+ if (WARN_RATELIMIT(buddy == NULL,
+ "ext4: mb_simple_scan_group: mb_find_buddy failed, (%d)\n", i))
+ continue;
k = mb_find_next_zero_bit(buddy, max, 0);
if (k >= max) {
@@ -3714,8 +3730,8 @@ static void ext4_free_data_in_buddy(struct super_block *sb,
/* No more items in the per group rb tree
* balance refcounts from ext4_mb_free_metadata()
*/
- put_page(e4b.bd_buddy_page);
- put_page(e4b.bd_bitmap_page);
+ folio_put(e4b.bd_buddy_folio);
+ folio_put(e4b.bd_bitmap_folio);
}
ext4_unlock_group(sb, entry->efd_group);
ext4_mb_unload_buddy(&e4b);
@@ -4317,15 +4333,14 @@ static void ext4_discard_allocated_blocks(struct ext4_allocation_context *ac)
if (ac->ac_f_ex.fe_len == 0)
return;
err = ext4_mb_load_buddy(ac->ac_sb, ac->ac_f_ex.fe_group, &e4b);
- if (err) {
+ if (WARN_RATELIMIT(err,
+ "ext4: mb_load_buddy failed (%d)", err))
/*
* This should never happen since we pin the
* pages in the ext4_allocation_context so
* ext4_mb_load_buddy() should never fail.
*/
- WARN(1, "mb_load_buddy failed (%d)", err);
return;
- }
ext4_lock_group(ac->ac_sb, ac->ac_f_ex.fe_group);
mb_free_blocks(ac->ac_inode, &e4b, ac->ac_f_ex.fe_start,
ac->ac_f_ex.fe_len);
@@ -5878,8 +5893,8 @@ ext4_mb_free_metadata(handle_t *handle, struct ext4_buddy *e4b,
struct rb_node *parent = NULL, *new_node;
BUG_ON(!ext4_handle_valid(handle));
- BUG_ON(e4b->bd_bitmap_page == NULL);
- BUG_ON(e4b->bd_buddy_page == NULL);
+ BUG_ON(e4b->bd_bitmap_folio == NULL);
+ BUG_ON(e4b->bd_buddy_folio == NULL);
new_node = &new_entry->efd_node;
cluster = new_entry->efd_start_cluster;
@@ -5890,8 +5905,8 @@ ext4_mb_free_metadata(handle_t *handle, struct ext4_buddy *e4b,
* otherwise we'll refresh it from
* on-disk bitmap and lose not-yet-available
* blocks */
- get_page(e4b->bd_buddy_page);
- get_page(e4b->bd_bitmap_page);
+ folio_get(e4b->bd_buddy_folio);
+ folio_get(e4b->bd_bitmap_folio);
}
while (*n) {
parent = *n;
diff --git a/fs/ext4/mballoc.h b/fs/ext4/mballoc.h
index 2d95fcab941f..fe4dbbbbe872 100644
--- a/fs/ext4/mballoc.h
+++ b/fs/ext4/mballoc.h
@@ -201,9 +201,9 @@ struct ext4_allocation_context {
#define AC_STATUS_BREAK 3
struct ext4_buddy {
- struct page *bd_buddy_page;
+ struct folio *bd_buddy_folio;
void *bd_buddy;
- struct page *bd_bitmap_page;
+ struct folio *bd_bitmap_folio;
void *bd_bitmap;
struct ext4_group_info *bd_info;
struct super_block *bd_sb;
diff --git a/fs/ext4/migrate.c b/fs/ext4/migrate.c
index 0be0467ae6dd..917e7d9b563e 100644
--- a/fs/ext4/migrate.c
+++ b/fs/ext4/migrate.c
@@ -408,7 +408,6 @@ static int free_ext_block(handle_t *handle, struct inode *inode)
int ext4_ext_migrate(struct inode *inode)
{
- struct ext4_sb_info *sbi = EXT4_SB(inode->i_sb);
handle_t *handle;
int retval = 0, i;
__le32 *i_data;
@@ -418,6 +417,7 @@ int ext4_ext_migrate(struct inode *inode)
unsigned long max_entries;
__u32 goal, tmp_csum_seed;
uid_t owner[2];
+ int alloc_ctx;
/*
* If the filesystem does not support extents, or the inode
@@ -434,7 +434,7 @@ int ext4_ext_migrate(struct inode *inode)
*/
return retval;
- percpu_down_write(&sbi->s_writepages_rwsem);
+ alloc_ctx = ext4_writepages_down_write(inode->i_sb);
/*
* Worst case we can touch the allocation bitmaps and a block
@@ -457,6 +457,7 @@ int ext4_ext_migrate(struct inode *inode)
if (IS_ERR(tmp_inode)) {
retval = PTR_ERR(tmp_inode);
ext4_journal_stop(handle);
+ tmp_inode = NULL;
goto out_unlock;
}
/*
@@ -584,9 +585,9 @@ int ext4_ext_migrate(struct inode *inode)
ext4_journal_stop(handle);
out_tmp_inode:
unlock_new_inode(tmp_inode);
- iput(tmp_inode);
out_unlock:
- percpu_up_write(&sbi->s_writepages_rwsem);
+ ext4_writepages_up_write(inode->i_sb, alloc_ctx);
+ iput(tmp_inode);
return retval;
}
@@ -605,6 +606,7 @@ int ext4_ind_migrate(struct inode *inode)
ext4_fsblk_t blk;
handle_t *handle;
int ret, ret2 = 0;
+ int alloc_ctx;
if (!ext4_has_feature_extents(inode->i_sb) ||
(!ext4_test_inode_flag(inode, EXT4_INODE_EXTENTS)))
@@ -621,7 +623,7 @@ int ext4_ind_migrate(struct inode *inode)
if (test_opt(inode->i_sb, DELALLOC))
ext4_alloc_da_blocks(inode);
- percpu_down_write(&sbi->s_writepages_rwsem);
+ alloc_ctx = ext4_writepages_down_write(inode->i_sb);
handle = ext4_journal_start(inode, EXT4_HT_MIGRATE, 1);
if (IS_ERR(handle)) {
@@ -665,6 +667,6 @@ int ext4_ind_migrate(struct inode *inode)
up_write(&EXT4_I(inode)->i_data_sem);
ext4_journal_stop(handle);
out_unlock:
- percpu_up_write(&sbi->s_writepages_rwsem);
+ ext4_writepages_up_write(inode->i_sb, alloc_ctx);
return ret;
}
diff --git a/fs/ext4/super.c b/fs/ext4/super.c
index f2a569868e41..fa6d9260030a 100644
--- a/fs/ext4/super.c
+++ b/fs/ext4/super.c
@@ -1167,12 +1167,12 @@ static void dump_orphan_list(struct super_block *sb, struct ext4_sb_info *sbi)
#ifdef CONFIG_QUOTA
static int ext4_quota_off(struct super_block *sb, int type);
-static inline void ext4_quota_off_umount(struct super_block *sb)
+static inline void ext4_quotas_off(struct super_block *sb, int type)
{
- int type;
+ BUG_ON(type > EXT4_MAXQUOTAS);
/* Use our quota_off function to clear inode flags etc. */
- for (type = 0; type < EXT4_MAXQUOTAS; type++)
+ for (type--; type >= 0; type--)
ext4_quota_off(sb, type);
}
@@ -1188,7 +1188,7 @@ static inline char *get_qf_name(struct super_block *sb,
lockdep_is_held(&sb->s_umount));
}
#else
-static inline void ext4_quota_off_umount(struct super_block *sb)
+static inline void ext4_quotas_off(struct super_block *sb, int type)
{
}
#endif
@@ -1260,6 +1260,14 @@ static void ext4_flex_groups_free(struct ext4_sb_info *sbi)
}
}
+static void ext4_put_sbi_rcu(struct rcu_head *head)
+{
+ struct ext4_sb_info *sbi = container_of(head, struct ext4_sb_info, s_rcu);
+
+ ext4_es_destroy_stats(sbi);
+ kfree(sbi);
+}
+
static void ext4_put_super(struct super_block *sb)
{
struct ext4_sb_info *sbi = EXT4_SB(sb);
@@ -1283,7 +1291,7 @@ static void ext4_put_super(struct super_block *sb)
ext4_msg(sb, KERN_INFO, "unmounting filesystem.");
ext4_unregister_li_request(sb);
- ext4_quota_off_umount(sb);
+ ext4_quotas_off(sb, EXT4_MAXQUOTAS);
flush_work(&sbi->s_error_work);
destroy_workqueue(sbi->rsv_conversion_wq);
@@ -1344,7 +1352,6 @@ static void ext4_put_super(struct super_block *sb)
ext4_stop_mmpd(sbi);
brelse(sbi->s_sbh);
- sb->s_fs_info = NULL;
/*
* Now that we are completely done shutting down the
* superblock, we need to actually destroy the kobject.
@@ -1359,7 +1366,8 @@ static void ext4_put_super(struct super_block *sb)
#if IS_ENABLED(CONFIG_UNICODE)
utf8_unload(sb->s_encoding);
#endif
- kfree(sbi);
+ /* ext4_get_link() reads the sbi and the counters in rcu pathwalk */
+ call_rcu(&sbi->s_rcu, ext4_put_sbi_rcu);
}
static struct kmem_cache *ext4_inode_cachep;
@@ -5547,19 +5555,15 @@ static int __ext4_fill_super(struct fs_context *fc, struct super_block *sb)
if (err)
goto failed_mount6;
- err = ext4_register_sysfs(sb);
- if (err)
- goto failed_mount7;
-
err = ext4_init_orphan_info(sb);
if (err)
- goto failed_mount8;
+ goto failed_mount7;
#ifdef CONFIG_QUOTA
/* Enable quota usage during mount. */
if (ext4_has_feature_quota(sb) && !sb_rdonly(sb)) {
err = ext4_enable_quotas(sb);
if (err)
- goto failed_mount9;
+ goto failed_mount8;
}
#endif /* CONFIG_QUOTA */
@@ -5585,7 +5589,7 @@ static int __ext4_fill_super(struct fs_context *fc, struct super_block *sb)
ext4_msg(sb, KERN_INFO, "recovery complete");
err = ext4_mark_recovery_complete(sb, es);
if (err)
- goto failed_mount10;
+ goto failed_mount9;
}
if (test_opt(sb, DISCARD) && !bdev_max_discard_sectors(sb->s_bdev))
@@ -5602,15 +5606,17 @@ static int __ext4_fill_super(struct fs_context *fc, struct super_block *sb)
atomic_set(&sbi->s_warning_count, 0);
atomic_set(&sbi->s_msg_count, 0);
+ /* Register sysfs after all initializations are complete. */
+ err = ext4_register_sysfs(sb);
+ if (err)
+ goto failed_mount9;
+
return 0;
-failed_mount10:
- ext4_quota_off_umount(sb);
-failed_mount9: __maybe_unused
+failed_mount9:
+ ext4_quotas_off(sb, EXT4_MAXQUOTAS);
+failed_mount8: __maybe_unused
ext4_release_orphan_info(sb);
-failed_mount8:
- ext4_unregister_sysfs(sb);
- kobject_put(&sbi->s_kobj);
failed_mount7:
ext4_unregister_li_request(sb);
failed_mount6:
@@ -5642,6 +5648,7 @@ failed_mount9: __maybe_unused
}
failed_mount3a:
ext4_es_unregister_shrinker(sbi);
+ ext4_es_destroy_stats(sbi);
failed_mount3:
/* flush s_error_work before sbi destroy */
flush_work(&sbi->s_error_work);
@@ -7091,20 +7098,8 @@ int ext4_enable_quotas(struct super_block *sb)
"(type=%d, err=%d, ino=%lu). "
"Please run e2fsck to fix.", type,
err, qf_inums[type]);
- for (type--; type >= 0; type--) {
- struct inode *inode;
-
- inode = sb_dqopt(sb)->files[type];
- if (inode)
- inode = igrab(inode);
- dquot_quota_off(sb, type);
- if (inode) {
- lockdep_set_quota_inode(inode,
- I_DATA_SEM_NORMAL);
- iput(inode);
- }
- }
+ ext4_quotas_off(sb, type);
return err;
}
}
diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c
index a944028502f2..e66275a5d83f 100644
--- a/fs/ext4/xattr.c
+++ b/fs/ext4/xattr.c
@@ -1497,7 +1497,7 @@ ext4_xattr_inode_cache_find(struct inode *inode, const void *value,
while (ce) {
ea_inode = ext4_iget(inode->i_sb, ce->e_value,
- EXT4_IGET_EA_INODE);
+ EXT4_IGET_EA_INODE | EXT4_IGET_NOWAIT);
if (IS_ERR(ea_inode))
goto next_entry;
ext4_xattr_inode_set_class(ea_inode);
diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c
index 49000b1a7a65..3d3b0e313af8 100644
--- a/fs/f2fs/checkpoint.c
+++ b/fs/f2fs/checkpoint.c
@@ -102,7 +102,7 @@ static struct page *__get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index,
}
if (unlikely(!PageUptodate(page))) {
- f2fs_handle_page_eio(sbi, page->index, META);
+ f2fs_handle_page_eio(sbi, page_folio(page), META);
f2fs_put_page(page, 1);
return ERR_PTR(-EIO);
}
@@ -1609,8 +1609,9 @@ static int do_checkpoint(struct f2fs_sb_info *sbi, struct cp_control *cpc)
*/
if (f2fs_sb_has_encrypt(sbi) || f2fs_sb_has_verity(sbi) ||
f2fs_sb_has_compression(sbi))
- invalidate_mapping_pages(META_MAPPING(sbi),
- MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - 1);
+ f2fs_bug_on(sbi,
+ invalidate_inode_pages2_range(META_MAPPING(sbi),
+ MAIN_BLKADDR(sbi), MAX_BLKADDR(sbi) - 1));
f2fs_release_ino_entry(sbi, false);
diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c
index 5d0c41abb405..37a684c682c7 100644
--- a/fs/f2fs/compress.c
+++ b/fs/f2fs/compress.c
@@ -1123,6 +1123,7 @@ static int prepare_compress_overwrite(struct compress_ctx *cc,
f2fs_compress_ctx_add_page(cc, page);
if (!PageUptodate(page)) {
+ f2fs_handle_page_eio(sbi, page_folio(page), DATA);
release_and_retry:
f2fs_put_rpages(cc);
f2fs_unlock_rpages(cc, i + 1);
@@ -1214,13 +1215,14 @@ int f2fs_truncate_partial_cluster(struct inode *inode, u64 from, bool lock)
int i;
for (i = cluster_size - 1; i >= 0; i--) {
- loff_t start = rpages[i]->index << PAGE_SHIFT;
+ struct folio *folio = page_folio(rpages[i]);
+ loff_t start = (loff_t)folio->index << PAGE_SHIFT;
if (from <= start) {
- zero_user_segment(rpages[i], 0, PAGE_SIZE);
+ folio_zero_segment(folio, 0, folio_size(folio));
} else {
- zero_user_segment(rpages[i], from - start,
- PAGE_SIZE);
+ folio_zero_segment(folio, from - start,
+ folio_size(folio));
break;
}
}
diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c
index 94c46b1b04c5..6384ac931bb7 100644
--- a/fs/f2fs/data.c
+++ b/fs/f2fs/data.c
@@ -1458,11 +1458,9 @@ static int __allocate_data_block(struct dnode_of_data *dn, int seg_type)
old_blkaddr = dn->data_blkaddr;
f2fs_allocate_data_block(sbi, NULL, old_blkaddr, &dn->data_blkaddr,
&sum, seg_type, NULL);
- if (GET_SEGNO(sbi, old_blkaddr) != NULL_SEGNO) {
- invalidate_mapping_pages(META_MAPPING(sbi),
- old_blkaddr, old_blkaddr);
- f2fs_invalidate_compress_page(sbi, old_blkaddr);
- }
+ if (GET_SEGNO(sbi, old_blkaddr) != NULL_SEGNO)
+ f2fs_invalidate_internal_cache(sbi, old_blkaddr);
+
f2fs_update_data_blkaddr(dn, dn->data_blkaddr);
return 0;
}
@@ -2243,6 +2241,12 @@ int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret,
int i;
int ret = 0;
+ if (unlikely(f2fs_cp_error(sbi))) {
+ ret = -EIO;
+ from_dnode = false;
+ goto out_put_dnode;
+ }
+
f2fs_bug_on(sbi, f2fs_cluster_is_empty(cc));
last_block_in_file = bytes_to_blks(inode,
diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c
index dd6e8a10193b..711b4262a0c7 100644
--- a/fs/f2fs/debug.c
+++ b/fs/f2fs/debug.c
@@ -280,10 +280,6 @@ static void update_mem_info(struct f2fs_sb_info *sbi)
get_cache:
si->cache_mem = 0;
- /* build gc */
- if (sbi->gc_thread)
- si->cache_mem += sizeof(struct f2fs_gc_kthread);
-
/* build merge flush thread */
if (SM_I(sbi)->fcc_info)
si->cache_mem += sizeof(struct flush_cmd_control);
diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c
index 29ac1e77b5d7..cadc32959a23 100644
--- a/fs/f2fs/dir.c
+++ b/fs/f2fs/dir.c
@@ -374,6 +374,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir,
de = find_in_block(dir, dentry_page, fname, &max_slots, use_hash);
if (IS_ERR(de)) {
+ f2fs_put_page(dentry_page, 0);
*res_page = ERR_CAST(de);
de = NULL;
break;
diff --git a/fs/f2fs/extent_cache.c b/fs/f2fs/extent_cache.c
index a5c63c7da299..b4c14efc83cd 100644
--- a/fs/f2fs/extent_cache.c
+++ b/fs/f2fs/extent_cache.c
@@ -15,26 +15,24 @@
#include "node.h"
#include <trace/events/f2fs.h>
-bool sanity_check_extent_cache(struct inode *inode)
+bool sanity_check_extent_cache(struct inode *inode, struct page *ipage)
{
struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
- struct f2fs_inode_info *fi = F2FS_I(inode);
- struct extent_info *ei;
+ struct f2fs_extent *i_ext = &F2FS_INODE(ipage)->i_ext;
+ struct extent_info ei;
- if (!fi->extent_tree[EX_READ])
- return true;
+ get_read_extent_info(&ei, i_ext);
- ei = &fi->extent_tree[EX_READ]->largest;
+ if (!ei.len)
+ return true;
- if (ei->len &&
- (!f2fs_is_valid_blkaddr(sbi, ei->blk,
- DATA_GENERIC_ENHANCE) ||
- !f2fs_is_valid_blkaddr(sbi, ei->blk + ei->len - 1,
- DATA_GENERIC_ENHANCE))) {
+ if (!f2fs_is_valid_blkaddr(sbi, ei.blk, DATA_GENERIC_ENHANCE) ||
+ !f2fs_is_valid_blkaddr(sbi, ei.blk + ei.len - 1,
+ DATA_GENERIC_ENHANCE)) {
set_sbi_flag(sbi, SBI_NEED_FSCK);
f2fs_warn(sbi, "%s: inode (ino=%lx) extent info [%u, %u, %u] is incorrect, run fsck to fix",
__func__, inode->i_ino,
- ei->blk, ei->fofs, ei->len);
+ ei.blk, ei.fofs, ei.len);
return false;
}
return true;
@@ -58,24 +56,11 @@ static void __set_extent_info(struct extent_info *ei,
}
}
-static bool __may_read_extent_tree(struct inode *inode)
-{
- struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
-
- if (!test_opt(sbi, READ_EXTENT_CACHE))
- return false;
- if (is_inode_flag_set(inode, FI_NO_EXTENT))
- return false;
- if (is_inode_flag_set(inode, FI_COMPRESSED_FILE) &&
- !f2fs_sb_has_readonly(sbi))
- return false;
- return S_ISREG(inode->i_mode);
-}
-
static bool __init_may_extent_tree(struct inode *inode, enum extent_type type)
{
if (type == EX_READ)
- return __may_read_extent_tree(inode);
+ return test_opt(F2FS_I_SB(inode), READ_EXTENT_CACHE) &&
+ S_ISREG(inode->i_mode);
return false;
}
@@ -88,7 +73,18 @@ static bool __may_extent_tree(struct inode *inode, enum extent_type type)
if (list_empty(&F2FS_I_SB(inode)->s_list))
return false;
- return __init_may_extent_tree(inode, type);
+ if (!__init_may_extent_tree(inode, type))
+ return false;
+
+ if (is_inode_flag_set(inode, FI_NO_EXTENT))
+ return false;
+
+ if (type == EX_READ) {
+ if (is_inode_flag_set(inode, FI_COMPRESSED_FILE) &&
+ !f2fs_sb_has_readonly(F2FS_I_SB(inode)))
+ return false;
+ }
+ return true;
}
static void __try_update_largest_extent(struct extent_tree *et,
@@ -406,21 +402,22 @@ static struct extent_tree *__grab_extent_tree(struct inode *inode,
}
static unsigned int __free_extent_tree(struct f2fs_sb_info *sbi,
- struct extent_tree *et)
+ struct extent_tree *et, unsigned int nr_shrink)
{
struct rb_node *node, *next;
struct extent_node *en;
- unsigned int count = atomic_read(&et->node_cnt);
+ unsigned int count;
node = rb_first_cached(&et->root);
- while (node) {
+
+ for (count = 0; node && count < nr_shrink; count++) {
next = rb_next(node);
en = rb_entry(node, struct extent_node, rb_node);
__release_extent_node(sbi, et, en);
node = next;
}
- return count - atomic_read(&et->node_cnt);
+ return count;
}
static void __drop_largest_extent(struct extent_tree *et,
@@ -444,24 +441,22 @@ void f2fs_init_read_extent_tree(struct inode *inode, struct page *ipage)
if (!__may_extent_tree(inode, EX_READ)) {
/* drop largest read extent */
- if (i_ext && i_ext->len) {
+ if (i_ext->len) {
f2fs_wait_on_page_writeback(ipage, NODE, true, true);
i_ext->len = 0;
set_page_dirty(ipage);
}
- goto out;
+ set_inode_flag(inode, FI_NO_EXTENT);
+ return;
}
et = __grab_extent_tree(inode, EX_READ);
- if (!i_ext || !i_ext->len)
- goto out;
-
get_read_extent_info(&ei, i_ext);
write_lock(&et->lock);
- if (atomic_read(&et->node_cnt))
- goto unlock_out;
+ if (atomic_read(&et->node_cnt) || !ei.len)
+ goto skip;
en = __attach_extent_node(sbi, et, &ei, NULL,
&et->root.rb_root.rb_node, true);
@@ -473,11 +468,13 @@ void f2fs_init_read_extent_tree(struct inode *inode, struct page *ipage)
list_add_tail(&en->list, &eti->extent_list);
spin_unlock(&eti->extent_lock);
}
-unlock_out:
+skip:
+ /* Let's drop, if checkpoint got corrupted. */
+ if (f2fs_cp_error(sbi)) {
+ et->largest.len = 0;
+ et->largest_updated = true;
+ }
write_unlock(&et->lock);
-out:
- if (!F2FS_I(inode)->extent_tree[EX_READ])
- set_inode_flag(inode, FI_NO_EXTENT);
}
void f2fs_init_extent_tree(struct inode *inode)
@@ -614,6 +611,30 @@ static struct extent_node *__insert_extent_tree(struct f2fs_sb_info *sbi,
return en;
}
+static unsigned int __destroy_extent_node(struct inode *inode,
+ enum extent_type type)
+{
+ struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
+ struct extent_tree *et = F2FS_I(inode)->extent_tree[type];
+ unsigned int nr_shrink = READ_EXTENT_CACHE_SHRINK_NUMBER;
+ unsigned int node_cnt = 0;
+
+ if (!et || !atomic_read(&et->node_cnt))
+ return 0;
+
+ while (atomic_read(&et->node_cnt)) {
+ write_lock(&et->lock);
+ if (!is_inode_flag_set(inode, FI_NO_EXTENT))
+ set_inode_flag(inode, FI_NO_EXTENT);
+ node_cnt += __free_extent_tree(sbi, et, nr_shrink);
+ write_unlock(&et->lock);
+ }
+
+ f2fs_bug_on(sbi, atomic_read(&et->node_cnt));
+
+ return node_cnt;
+}
+
static void __update_extent_tree_range(struct inode *inode,
struct extent_info *tei, enum extent_type type)
{
@@ -636,12 +657,12 @@ static void __update_extent_tree_range(struct inode *inode,
tei->blk, 0);
write_lock(&et->lock);
- if (type == EX_READ) {
- if (is_inode_flag_set(inode, FI_NO_EXTENT)) {
- write_unlock(&et->lock);
- return;
- }
+ if (is_inode_flag_set(inode, FI_NO_EXTENT)) {
+ write_unlock(&et->lock);
+ return;
+ }
+ if (type == EX_READ) {
prev = et->largest;
dei.len = 0;
@@ -743,9 +764,6 @@ static void __update_extent_tree_range(struct inode *inode,
}
}
- if (is_inode_flag_set(inode, FI_NO_EXTENT))
- __free_extent_tree(sbi, et);
-
if (et->largest_updated) {
et->largest_updated = false;
updated = true;
@@ -753,6 +771,9 @@ static void __update_extent_tree_range(struct inode *inode,
write_unlock(&et->lock);
+ if (is_inode_flag_set(inode, FI_NO_EXTENT))
+ __destroy_extent_node(inode, EX_READ);
+
if (updated)
f2fs_mark_inode_dirty_sync(inode, true);
}
@@ -838,10 +859,14 @@ static unsigned int __shrink_extent_tree(struct f2fs_sb_info *sbi, int nr_shrink
list_for_each_entry_safe(et, next, &eti->zombie_list, list) {
if (atomic_read(&et->node_cnt)) {
write_lock(&et->lock);
- node_cnt += __free_extent_tree(sbi, et);
+ node_cnt += __free_extent_tree(sbi, et,
+ nr_shrink - node_cnt - tree_cnt);
write_unlock(&et->lock);
}
- f2fs_bug_on(sbi, atomic_read(&et->node_cnt));
+
+ if (atomic_read(&et->node_cnt))
+ goto unlock_out;
+
list_del_init(&et->list);
radix_tree_delete(&eti->extent_tree_root, et->ino);
kmem_cache_free(extent_tree_slab, et);
@@ -932,23 +957,6 @@ unsigned int f2fs_shrink_read_extent_tree(struct f2fs_sb_info *sbi, int nr_shrin
return __shrink_extent_tree(sbi, nr_shrink, EX_READ);
}
-static unsigned int __destroy_extent_node(struct inode *inode,
- enum extent_type type)
-{
- struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
- struct extent_tree *et = F2FS_I(inode)->extent_tree[type];
- unsigned int node_cnt = 0;
-
- if (!et || !atomic_read(&et->node_cnt))
- return 0;
-
- write_lock(&et->lock);
- node_cnt = __free_extent_tree(sbi, et);
- write_unlock(&et->lock);
-
- return node_cnt;
-}
-
void f2fs_destroy_extent_node(struct inode *inode)
{
__destroy_extent_node(inode, EX_READ);
@@ -956,7 +964,6 @@ void f2fs_destroy_extent_node(struct inode *inode)
static void __drop_extent_tree(struct inode *inode, enum extent_type type)
{
- struct f2fs_sb_info *sbi = F2FS_I_SB(inode);
struct extent_tree *et = F2FS_I(inode)->extent_tree[type];
bool updated = false;
@@ -964,7 +971,6 @@ static void __drop_extent_tree(struct inode *inode, enum extent_type type)
return;
write_lock(&et->lock);
- __free_extent_tree(sbi, et);
if (type == EX_READ) {
set_inode_flag(inode, FI_NO_EXTENT);
if (et->largest.len) {
@@ -973,6 +979,9 @@ static void __drop_extent_tree(struct inode *inode, enum extent_type type)
}
}
write_unlock(&et->lock);
+
+ __destroy_extent_node(inode, type);
+
if (updated)
f2fs_mark_inode_dirty_sync(inode, true);
}
diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h
index cbf4032b5b7e..5b63b92a1027 100644
--- a/fs/f2fs/f2fs.h
+++ b/fs/f2fs/f2fs.h
@@ -1599,6 +1599,26 @@ struct decompress_io_ctx {
#define MAX_COMPRESS_LOG_SIZE 8
#define MAX_COMPRESS_WINDOW_SIZE(log_size) ((PAGE_SIZE) << (log_size))
+struct f2fs_gc_kthread {
+ struct task_struct *f2fs_gc_task;
+ wait_queue_head_t gc_wait_queue_head;
+
+ /* for gc sleep time */
+ unsigned int urgent_sleep_time;
+ unsigned int min_sleep_time;
+ unsigned int max_sleep_time;
+ unsigned int no_gc_sleep_time;
+
+ /* for changing gc mode */
+ unsigned int gc_wake;
+
+ /* for GC_MERGE mount option */
+ wait_queue_head_t fggc_wq; /*
+ * caller of f2fs_balance_fs()
+ * will wait on this wait queue.
+ */
+};
+
struct f2fs_sb_info {
struct super_block *sb; /* pointer to VFS super block */
struct proc_dir_entry *s_proc; /* proc entry */
@@ -1715,7 +1735,7 @@ struct f2fs_sb_info {
* semaphore for GC, avoid
* race between GC and GC or CP
*/
- struct f2fs_gc_kthread *gc_thread; /* GC thread */
+ struct f2fs_gc_kthread gc_thread; /* GC thread */
struct atgc_management am; /* atgc management */
unsigned int cur_victim_sec; /* current victim section num */
unsigned int gc_mode; /* current GC state */
@@ -3446,7 +3466,6 @@ static inline bool __is_valid_data_blkaddr(block_t blkaddr)
* file.c
*/
int f2fs_sync_file(struct file *file, loff_t start, loff_t end, int datasync);
-void f2fs_truncate_data_blocks(struct dnode_of_data *dn);
int f2fs_do_truncate_blocks(struct inode *inode, u64 from, bool lock);
int f2fs_truncate_blocks(struct inode *inode, u64 from, bool lock);
int f2fs_truncate(struct inode *inode);
@@ -4145,7 +4164,7 @@ void f2fs_leave_shrinker(struct f2fs_sb_info *sbi);
/*
* extent_cache.c
*/
-bool sanity_check_extent_cache(struct inode *inode);
+bool sanity_check_extent_cache(struct inode *inode, struct page *ipage);
struct rb_entry *f2fs_lookup_rb_tree(struct rb_root_cached *root,
struct rb_entry *cached_re, unsigned int ofs);
struct rb_node **f2fs_lookup_rb_tree_for_insert(struct f2fs_sb_info *sbi,
@@ -4562,9 +4581,11 @@ static inline void f2fs_io_schedule_timeout(long timeout)
io_schedule_timeout(timeout);
}
-static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi, pgoff_t ofs,
- enum page_type type)
+static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi,
+ struct folio *folio, enum page_type type)
{
+ pgoff_t ofs = folio->index;
+
if (unlikely(f2fs_cp_error(sbi)))
return;
@@ -4577,6 +4598,44 @@ static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi, pgoff_t ofs,
}
}
+static inline bool f2fs_is_readonly(struct f2fs_sb_info *sbi)
+{
+ return f2fs_sb_has_readonly(sbi) || f2fs_readonly(sbi->sb);
+}
+
+static inline void f2fs_truncate_meta_inode_pages(struct f2fs_sb_info *sbi,
+ block_t blkaddr, unsigned int cnt)
+{
+ bool need_submit = false;
+ int i = 0;
+
+ do {
+ struct page *page;
+
+ page = find_get_page(META_MAPPING(sbi), blkaddr + i);
+ if (page) {
+ if (PageWriteback(page))
+ need_submit = true;
+ f2fs_put_page(page, 0);
+ }
+ } while (++i < cnt && !need_submit);
+
+ if (need_submit)
+ f2fs_submit_merged_write_cond(sbi, sbi->meta_inode,
+ NULL, 0, DATA);
+
+ truncate_inode_pages_range(META_MAPPING(sbi),
+ F2FS_BLK_TO_BYTES((loff_t)blkaddr),
+ F2FS_BLK_END_BYTES((loff_t)(blkaddr + cnt - 1)));
+}
+
+static inline void f2fs_invalidate_internal_cache(struct f2fs_sb_info *sbi,
+ block_t blkaddr)
+{
+ f2fs_truncate_meta_inode_pages(sbi, blkaddr, 1);
+ f2fs_invalidate_compress_page(sbi, blkaddr);
+}
+
#define EFSBADCRC EBADMSG /* Bad CRC detected */
#define EFSCORRUPTED EUCLEAN /* Filesystem is corrupted */
diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c
index 36c3103035c3..4dd995b0c472 100644
--- a/fs/f2fs/file.c
+++ b/fs/f2fs/file.c
@@ -666,11 +666,6 @@ void f2fs_truncate_data_blocks_range(struct dnode_of_data *dn, int count)
dn->ofs_in_node, nr_free);
}
-void f2fs_truncate_data_blocks(struct dnode_of_data *dn)
-{
- f2fs_truncate_data_blocks_range(dn, ADDRS_PER_BLOCK(dn->inode));
-}
-
static int truncate_partial_data_page(struct inode *inode, u64 from,
bool cache_only)
{
@@ -833,8 +828,16 @@ int f2fs_truncate(struct inode *inode)
/* we should check inline_data size */
if (!f2fs_may_inline_data(inode)) {
err = f2fs_convert_inline_inode(inode);
- if (err)
+ if (err) {
+ /*
+ * Always truncate page #0 to avoid page cache
+ * leak in evict() path.
+ */
+ truncate_inode_pages_range(inode->i_mapping,
+ F2FS_BLK_TO_BYTES(0),
+ F2FS_BLK_END_BYTES(0));
return err;
+ }
}
err = f2fs_truncate_blocks(inode, i_size_read(inode), true);
@@ -2929,8 +2932,9 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
struct inode *src = file_inode(file_in);
struct inode *dst = file_inode(file_out);
struct f2fs_sb_info *sbi = F2FS_I_SB(src);
- size_t olen = len, dst_max_i_size = 0;
- size_t dst_osize;
+ size_t olen = len;
+ loff_t dst_max_i_size = 0;
+ loff_t dst_osize, dst_end;
int ret;
if (file_in->f_path.mnt != file_out->f_path.mnt ||
@@ -2987,8 +2991,16 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in,
}
dst_osize = dst->i_size;
- if (pos_out + olen > dst->i_size)
- dst_max_i_size = pos_out + olen;
+ if (olen > LLONG_MAX - pos_out)
+ goto out_unlock;
+ dst_end = pos_out + olen;
+ if (dst_end > dst->i_size) {
+ ret = inode_newsize_ok(dst, dst_end);
+ if (ret)
+ goto out_unlock;
+ ret = -EINVAL;
+ dst_max_i_size = dst_end;
+ }
/* verify the end result is block aligned */
if (!IS_ALIGNED(pos_in, F2FS_BLKSIZE) ||
diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c
index 687b2ce82c85..851f55aa37f4 100644
--- a/fs/f2fs/gc.c
+++ b/fs/f2fs/gc.c
@@ -31,9 +31,9 @@ static unsigned int count_bits(const unsigned long *addr,
static int gc_thread_func(void *data)
{
struct f2fs_sb_info *sbi = data;
- struct f2fs_gc_kthread *gc_th = sbi->gc_thread;
- wait_queue_head_t *wq = &sbi->gc_thread->gc_wait_queue_head;
- wait_queue_head_t *fggc_wq = &sbi->gc_thread->fggc_wq;
+ struct f2fs_gc_kthread *gc_th = &sbi->gc_thread;
+ wait_queue_head_t *wq = &sbi->gc_thread.gc_wait_queue_head;
+ wait_queue_head_t *fggc_wq = &sbi->gc_thread.fggc_wq;
unsigned int wait_ms;
struct f2fs_gc_control gc_control = {
.victim_segno = NULL_SEGNO,
@@ -169,16 +169,10 @@ static int gc_thread_func(void *data)
int f2fs_start_gc_thread(struct f2fs_sb_info *sbi)
{
- struct f2fs_gc_kthread *gc_th;
+ struct f2fs_gc_kthread *gc_th = &sbi->gc_thread;
dev_t dev = sbi->sb->s_bdev->bd_dev;
int err = 0;
- gc_th = f2fs_kmalloc(sbi, sizeof(struct f2fs_gc_kthread), GFP_KERNEL);
- if (!gc_th) {
- err = -ENOMEM;
- goto out;
- }
-
gc_th->urgent_sleep_time = DEF_GC_THREAD_URGENT_SLEEP_TIME;
gc_th->min_sleep_time = DEF_GC_THREAD_MIN_SLEEP_TIME;
gc_th->max_sleep_time = DEF_GC_THREAD_MAX_SLEEP_TIME;
@@ -186,30 +180,27 @@ int f2fs_start_gc_thread(struct f2fs_sb_info *sbi)
gc_th->gc_wake = 0;
- sbi->gc_thread = gc_th;
- init_waitqueue_head(&sbi->gc_thread->gc_wait_queue_head);
- init_waitqueue_head(&sbi->gc_thread->fggc_wq);
- sbi->gc_thread->f2fs_gc_task = kthread_run(gc_thread_func, sbi,
+ init_waitqueue_head(&gc_th->gc_wait_queue_head);
+ init_waitqueue_head(&gc_th->fggc_wq);
+ gc_th->f2fs_gc_task = kthread_run(gc_thread_func, sbi,
"f2fs_gc-%u:%u", MAJOR(dev), MINOR(dev));
if (IS_ERR(gc_th->f2fs_gc_task)) {
err = PTR_ERR(gc_th->f2fs_gc_task);
- kfree(gc_th);
- sbi->gc_thread = NULL;
+ gc_th->f2fs_gc_task = NULL;
}
-out:
return err;
}
void f2fs_stop_gc_thread(struct f2fs_sb_info *sbi)
{
- struct f2fs_gc_kthread *gc_th = sbi->gc_thread;
+ struct f2fs_gc_kthread *gc_th = &sbi->gc_thread;
- if (!gc_th)
+ if (!gc_th->f2fs_gc_task)
return;
+
kthread_stop(gc_th->f2fs_gc_task);
+ gc_th->f2fs_gc_task = NULL;
wake_up_all(&gc_th->fggc_wq);
- kfree(gc_th);
- sbi->gc_thread = NULL;
}
static int select_gc_type(struct f2fs_sb_info *sbi, int gc_type)
@@ -1192,7 +1183,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index)
.in_list = 0,
.retry = 0,
};
- int err;
+ int err = 0;
page = f2fs_grab_cache_page(mapping, index, true);
if (!page)
@@ -1246,6 +1237,9 @@ static int ra_data_block(struct inode *inode, pgoff_t index)
goto put_page;
}
+ if (PageUptodate(fio.encrypted_page))
+ goto put_encrypted_page;
+
err = f2fs_submit_page_bio(&fio);
if (err)
goto put_encrypted_page;
@@ -1386,9 +1380,8 @@ static int move_data_block(struct inode *inode, block_t bidx,
memcpy(page_address(fio.encrypted_page),
page_address(mpage), PAGE_SIZE);
f2fs_put_page(mpage, 1);
- invalidate_mapping_pages(META_MAPPING(fio.sbi),
- fio.old_blkaddr, fio.old_blkaddr);
- f2fs_invalidate_compress_page(fio.sbi, fio.old_blkaddr);
+
+ f2fs_invalidate_internal_cache(fio.sbi, fio.old_blkaddr);
set_page_dirty(fio.encrypted_page);
if (clear_page_dirty_for_io(fio.encrypted_page))
@@ -1577,6 +1570,16 @@ static int gc_data_segment(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
continue;
}
+ if (f2fs_has_inline_data(inode)) {
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_err(sbi,
+ "inode %lx has both inline_data flag and "
+ "data block, nid=%u, ofs_in_node=%u",
+ inode->i_ino, dni.nid, ofs_in_node);
+ iput(inode);
+ continue;
+ }
+
err = f2fs_gc_pinned_control(inode, gc_type, segno);
if (err == -EAGAIN) {
iput(inode);
@@ -1992,8 +1995,9 @@ void f2fs_build_gc_manager(struct f2fs_sb_info *sbi)
static int free_segment_range(struct f2fs_sb_info *sbi,
unsigned int secs, bool gc_only)
{
- unsigned int segno, next_inuse, start, end;
+ unsigned int segno, secno, next_inuse, start, end, end_secno;
struct cp_control cpc = { CP_RESIZE, 0, 0, 0 };
+ unsigned int freed_secs = 0;
int gc_mode, gc_type;
int err = 0;
int type;
@@ -2002,6 +2006,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
MAIN_SECS(sbi) -= secs;
start = MAIN_SECS(sbi) * sbi->segs_per_sec;
end = MAIN_SEGS(sbi) - 1;
+ end_secno = GET_SEC_FROM_SEG(sbi, end);
mutex_lock(&DIRTY_I(sbi)->seglist_lock);
for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++)
@@ -2013,8 +2018,16 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
sbi->next_victim_seg[gc_type] = NULL_SEGNO;
mutex_unlock(&DIRTY_I(sbi)->seglist_lock);
+ spin_lock(&FREE_I(sbi)->segmap_lock);
+ for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) {
+ if (!test_bit(secno, FREE_I(sbi)->free_secmap))
+ freed_secs++;
+ }
+ FREE_I(sbi)->free_sections -= freed_secs;
+ spin_unlock(&FREE_I(sbi)->segmap_lock);
+
/* Move out cursegs from the target range */
- for (type = CURSEG_HOT_DATA; type < NR_CURSEG_PERSIST_TYPE; type++)
+ for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++)
f2fs_allocate_segment_for_resize(sbi, type, start, end);
/* do GC to move out valid blocks in the range */
@@ -2050,6 +2063,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi,
f2fs_bug_on(sbi, 1);
}
out:
+ spin_lock(&FREE_I(sbi)->segmap_lock);
+ FREE_I(sbi)->free_sections += freed_secs;
+ spin_unlock(&FREE_I(sbi)->segmap_lock);
MAIN_SECS(sbi) += secs;
return err;
}
diff --git a/fs/f2fs/gc.h b/fs/f2fs/gc.h
index 47357101b03c..0ff8138f0174 100644
--- a/fs/f2fs/gc.h
+++ b/fs/f2fs/gc.h
@@ -32,26 +32,6 @@
#define NR_GC_CHECKPOINT_SECS (3) /* data/node/dentry sections */
-struct f2fs_gc_kthread {
- struct task_struct *f2fs_gc_task;
- wait_queue_head_t gc_wait_queue_head;
-
- /* for gc sleep time */
- unsigned int urgent_sleep_time;
- unsigned int min_sleep_time;
- unsigned int max_sleep_time;
- unsigned int no_gc_sleep_time;
-
- /* for changing gc mode */
- unsigned int gc_wake;
-
- /* for GC_MERGE mount option */
- wait_queue_head_t fggc_wq; /*
- * caller of f2fs_balance_fs()
- * will wait on this wait queue.
- */
-};
-
struct gc_inode_list {
struct list_head ilist;
struct radix_tree_root iroot;
diff --git a/fs/f2fs/inode.c b/fs/f2fs/inode.c
index 1715663e26a8..e97aa6a8ac53 100644
--- a/fs/f2fs/inode.c
+++ b/fs/f2fs/inode.c
@@ -455,15 +455,15 @@ static int do_read_inode(struct inode *inode)
init_idisk_time(inode);
- /* Need all the flag bits */
- f2fs_init_read_extent_tree(inode, node_page);
-
- if (!sanity_check_extent_cache(inode)) {
+ if (!sanity_check_extent_cache(inode, node_page)) {
f2fs_put_page(node_page, 1);
f2fs_handle_error(sbi, ERROR_CORRUPTED_INODE);
return -EFSCORRUPTED;
}
+ /* Need all the flag bits */
+ f2fs_init_read_extent_tree(inode, node_page);
+
f2fs_put_page(node_page, 1);
stat_inc_inline_xattr(inode);
diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c
index 8d2e5d3910c5..233448ba3a29 100644
--- a/fs/f2fs/node.c
+++ b/fs/f2fs/node.c
@@ -417,6 +417,7 @@ bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino)
struct nat_entry *e;
bool need_update = true;
+ f2fs_down_read(&sbi->node_write);
f2fs_down_read(&nm_i->nat_tree_lock);
e = __lookup_nat_cache(nm_i, ino);
if (e && get_nat_flag(e, HAS_LAST_FSYNC) &&
@@ -424,6 +425,7 @@ bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino)
get_nat_flag(e, HAS_FSYNCED_INODE)))
need_update = false;
f2fs_up_read(&nm_i->nat_tree_lock);
+ f2fs_up_read(&sbi->node_write);
return need_update;
}
@@ -918,6 +920,16 @@ static int truncate_node(struct dnode_of_data *dn)
if (err)
return err;
+ if (ni.blk_addr != NEW_ADDR &&
+ !f2fs_is_valid_blkaddr(sbi, ni.blk_addr, DATA_GENERIC_ENHANCE)) {
+ f2fs_err(sbi,
+ "nat entry is corrupted, run fsck to fix it, ino:%u, "
+ "nid:%u, blkaddr:%u", ni.ino, ni.nid, ni.blk_addr);
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_handle_error(sbi, ERROR_INCONSISTENT_NAT);
+ return -EFSCORRUPTED;
+ }
+
/* Deallocate node address */
f2fs_invalidate_blocks(sbi, ni.blk_addr);
dec_valid_node_count(sbi, dn->inode, dn->nid == dn->inode->i_ino);
@@ -946,6 +958,7 @@ static int truncate_node(struct dnode_of_data *dn)
static int truncate_dnode(struct dnode_of_data *dn)
{
+ struct f2fs_sb_info *sbi = F2FS_I_SB(dn->inode);
struct page *page;
int err;
@@ -953,16 +966,25 @@ static int truncate_dnode(struct dnode_of_data *dn)
return 1;
/* get direct node */
- page = f2fs_get_node_page(F2FS_I_SB(dn->inode), dn->nid);
+ page = f2fs_get_node_page(sbi, dn->nid);
if (PTR_ERR(page) == -ENOENT)
return 1;
else if (IS_ERR(page))
return PTR_ERR(page);
+ if (IS_INODE(page) || ino_of_node(page) != dn->inode->i_ino) {
+ f2fs_err(sbi, "incorrect node reference, ino: %lu, nid: %u, ino_of_node: %u",
+ dn->inode->i_ino, dn->nid, ino_of_node(page));
+ set_sbi_flag(sbi, SBI_NEED_FSCK);
+ f2fs_handle_error(sbi, ERROR_INVALID_NODE_REFERENCE);
+ f2fs_put_page(page, 1);
+ return -EFSCORRUPTED;
+ }
+
/* Make dnode_of_data for parameter */
dn->node_page = page;
dn->ofs_in_node = 0;
- f2fs_truncate_data_blocks(dn);
+ f2fs_truncate_data_blocks_range(dn, ADDRS_PER_BLOCK(dn->inode));
err = truncate_node(dn);
if (err) {
f2fs_put_page(page, 1);
@@ -1488,11 +1510,11 @@ static struct page *__get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid,
f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER);
err = -EFSCORRUPTED;
out_err:
- ClearPageUptodate(page);
+ clear_node_page_dirty(page);
out_put_err:
/* ENOENT comes from read_node_page which is not an error. */
if (err != -ENOENT)
- f2fs_handle_page_eio(sbi, page->index, NODE);
+ f2fs_handle_page_eio(sbi, page_folio(page), NODE);
f2fs_put_page(page, 1);
return ERR_PTR(err);
}
@@ -1602,9 +1624,10 @@ static struct page *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino)
return last_page;
}
-static int __write_node_page(struct page *page, bool atomic, bool *submitted,
- struct writeback_control *wbc, bool do_balance,
- enum iostat_type io_type, unsigned int *seq_id)
+static int __write_node_page(struct page *page, bool atomic, bool do_fsync,
+ bool *submitted, struct writeback_control *wbc,
+ bool do_balance, enum iostat_type io_type,
+ unsigned int *seq_id)
{
struct f2fs_sb_info *sbi = F2FS_P_SB(page);
nid_t nid;
@@ -1670,13 +1693,14 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted,
goto redirty_out;
}
- if (atomic) {
- if (!test_opt(sbi, NOBARRIER) && !f2fs_sb_has_blkzoned(sbi))
- fio.op_flags |= REQ_PREFLUSH | REQ_FUA;
- if (IS_INODE(page))
- set_dentry_mark(page,
+ if (atomic && !test_opt(sbi, NOBARRIER) && !f2fs_sb_has_blkzoned(sbi))
+ fio.op_flags |= REQ_PREFLUSH | REQ_FUA;
+
+ set_dentry_mark(page, false);
+ set_fsync_mark(page, do_fsync);
+ if (IS_INODE(page) && (atomic || is_fsync_dnode(page)))
+ set_dentry_mark(page,
f2fs_need_dentry_mark(sbi, ino_of_node(page)));
- }
/* should add to global list before clearing PAGECACHE status */
if (f2fs_in_warm_node_list(sbi, page)) {
@@ -1737,7 +1761,7 @@ int f2fs_move_node_page(struct page *node_page, int gc_type)
goto out_page;
}
- if (__write_node_page(node_page, false, NULL,
+ if (__write_node_page(node_page, false, false, NULL,
&wbc, false, FS_GC_NODE_IO, NULL)) {
err = -EAGAIN;
unlock_page(node_page);
@@ -1758,7 +1782,7 @@ int f2fs_move_node_page(struct page *node_page, int gc_type)
static int f2fs_write_node_page(struct page *page,
struct writeback_control *wbc)
{
- return __write_node_page(page, false, NULL, wbc, false,
+ return __write_node_page(page, false, false, NULL, wbc, false,
FS_NODE_IO, NULL);
}
@@ -1791,6 +1815,7 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
for (i = 0; i < nr_pages; i++) {
struct page *page = pvec.pages[i];
bool submitted = false;
+ bool do_fsync = false;
if (unlikely(f2fs_cp_error(sbi))) {
f2fs_put_page(last_page, 0);
@@ -1821,19 +1846,13 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
f2fs_wait_on_page_writeback(page, NODE, true, true);
- set_fsync_mark(page, 0);
- set_dentry_mark(page, 0);
-
if (!atomic || page == last_page) {
- set_fsync_mark(page, 1);
+ do_fsync = true;
percpu_counter_inc(&sbi->rf_node_block_count);
if (IS_INODE(page)) {
if (is_inode_flag_set(inode,
FI_DIRTY_INODE))
f2fs_update_inode(inode, page);
- if (!atomic)
- set_dentry_mark(page,
- f2fs_need_dentry_mark(sbi, ino));
}
/* may be written by other thread */
if (!PageDirty(page))
@@ -1845,8 +1864,9 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
ret = __write_node_page(page, atomic &&
page == last_page,
- &submitted, wbc, true,
- FS_NODE_IO, seq_id);
+ do_fsync, &submitted,
+ wbc, true, FS_NODE_IO,
+ seq_id);
if (ret) {
unlock_page(page);
f2fs_put_page(last_page, 0);
@@ -1871,6 +1891,11 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode,
f2fs_debug(sbi, "Retry to write fsync mark: ino=%u, idx=%lx",
ino, last_page->index);
lock_page(last_page);
+ if (unlikely(last_page->mapping != NODE_MAPPING(sbi))) {
+ f2fs_put_page(last_page, 1);
+ ret = -EAGAIN;
+ goto out;
+ }
f2fs_wait_on_page_writeback(last_page, NODE, true, true);
set_page_dirty(last_page);
unlock_page(last_page);
@@ -2051,10 +2076,7 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi,
if (!clear_page_dirty_for_io(page))
goto continue_unlock;
- set_fsync_mark(page, 0);
- set_dentry_mark(page, 0);
-
- ret = __write_node_page(page, false, &submitted,
+ ret = __write_node_page(page, false, false, &submitted,
wbc, do_balance, io_type, NULL);
if (ret)
unlock_page(page);
diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c
index b33e5220aa18..45572d1d2d4e 100644
--- a/fs/f2fs/recovery.c
+++ b/fs/f2fs/recovery.c
@@ -176,7 +176,8 @@ static int recover_dentry(struct inode *inode, struct page *ipage,
struct inode *dir, *einode;
struct fsync_inode_entry *entry;
int err = 0;
- char *name;
+ const char *name;
+ int name_len;
entry = get_fsync_inode(dir_list, pino);
if (!entry) {
@@ -235,12 +236,16 @@ static int recover_dentry(struct inode *inode, struct page *ipage,
out_put:
f2fs_put_page(page, 0);
out:
- if (file_enc_name(inode))
+ if (file_enc_name(inode)) {
name = "<encrypted>";
- else
+ name_len = sizeof("<encrypted>") - 1;
+ } else {
name = raw_inode->i_name;
- f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %s, dir = %lx, err = %d",
- __func__, ino_of_node(ipage), name,
+ name_len = min_t(unsigned int, le32_to_cpu(raw_inode->i_namelen),
+ F2FS_NAME_LEN);
+ }
+ f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %lx, err = %d",
+ __func__, ino_of_node(ipage), name_len, name,
IS_ERR(dir) ? 0 : dir->i_ino, err);
return err;
}
@@ -287,7 +292,8 @@ static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri)
static int recover_inode(struct inode *inode, struct page *page)
{
struct f2fs_inode *raw = F2FS_INODE(page);
- char *name;
+ const char *name;
+ int name_len;
int err;
inode->i_mode = le16_to_cpu(raw->i_mode);
@@ -336,13 +342,18 @@ static int recover_inode(struct inode *inode, struct page *page)
f2fs_mark_inode_dirty_sync(inode, true);
- if (file_enc_name(inode))
+ if (file_enc_name(inode)) {
name = "<encrypted>";
- else
- name = F2FS_INODE(page)->i_name;
+ name_len = sizeof("<encrypted>") - 1;
+ } else {
+ name = raw->i_name;
+ name_len = min_t(unsigned int, le32_to_cpu(raw->i_namelen),
+ F2FS_NAME_LEN);
+ }
- f2fs_notice(F2FS_I_SB(inode), "recover_inode: ino = %x, name = %s, inline = %x",
- ino_of_node(page), name, raw->i_inline);
+ f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x",
+ __func__, ino_of_node(page), name_len, name,
+ raw->i_inline);
return 0;
}
diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c
index d185b209f89a..bf52ba8a32c0 100644
--- a/fs/f2fs/segment.c
+++ b/fs/f2fs/segment.c
@@ -415,15 +415,14 @@ void f2fs_balance_fs(struct f2fs_sb_info *sbi, bool need)
f2fs_submit_merged_write(sbi, DATA);
f2fs_submit_all_merged_ipu_writes(sbi);
- if (test_opt(sbi, GC_MERGE) && sbi->gc_thread &&
- sbi->gc_thread->f2fs_gc_task) {
+ if (test_opt(sbi, GC_MERGE) && sbi->gc_thread.f2fs_gc_task) {
DEFINE_WAIT(wait);
- prepare_to_wait(&sbi->gc_thread->fggc_wq, &wait,
+ prepare_to_wait(&sbi->gc_thread.fggc_wq, &wait,
TASK_UNINTERRUPTIBLE);
- wake_up(&sbi->gc_thread->gc_wait_queue_head);
+ wake_up(&sbi->gc_thread.gc_wait_queue_head);
io_schedule();
- finish_wait(&sbi->gc_thread->fggc_wq, &wait);
+ finish_wait(&sbi->gc_thread.fggc_wq, &wait);
} else {
struct f2fs_gc_control gc_control = {
.victim_segno = NULL_SEGNO,
@@ -1126,6 +1125,15 @@ static int __submit_discard_cmd(struct f2fs_sb_info *sbi,
if (is_sbi_flag_set(sbi, SBI_NEED_FSCK))
return 0;
+ /*
+ * stop issuing discard for any of below cases:
+ * 1. device is conventional zone, but it doesn't support discard.
+ * 2. device is regulare device, after snapshot it doesn't support
+ * discard.
+ */
+ if (!bdev_max_discard_sectors(bdev))
+ return -EOPNOTSUPP;
+
trace_f2fs_issue_discard(bdev, dc->start, dc->len);
lstart = dc->lstart;
@@ -2299,8 +2307,7 @@ void f2fs_invalidate_blocks(struct f2fs_sb_info *sbi, block_t addr)
if (addr == NEW_ADDR || addr == COMPRESS_ADDR)
return;
- invalidate_mapping_pages(META_MAPPING(sbi), addr, addr);
- f2fs_invalidate_compress_page(sbi, addr);
+ f2fs_invalidate_internal_cache(sbi, addr);
/* add it into sit main buffer */
down_write(&sit_i->sentry_lock);
@@ -3370,11 +3377,8 @@ static void do_write_page(struct f2fs_summary *sum, struct f2fs_io_info *fio)
reallocate:
f2fs_allocate_data_block(fio->sbi, fio->page, fio->old_blkaddr,
&fio->new_blkaddr, sum, type, fio);
- if (GET_SEGNO(fio->sbi, fio->old_blkaddr) != NULL_SEGNO) {
- invalidate_mapping_pages(META_MAPPING(fio->sbi),
- fio->old_blkaddr, fio->old_blkaddr);
- f2fs_invalidate_compress_page(fio->sbi, fio->old_blkaddr);
- }
+ if (GET_SEGNO(fio->sbi, fio->old_blkaddr) != NULL_SEGNO)
+ f2fs_invalidate_internal_cache(fio->sbi, fio->old_blkaddr);
/* writeout dirty page into bdev */
f2fs_submit_page_write(fio);
@@ -3467,8 +3471,7 @@ int f2fs_inplace_write_data(struct f2fs_io_info *fio)
}
if (fio->post_read)
- invalidate_mapping_pages(META_MAPPING(sbi),
- fio->new_blkaddr, fio->new_blkaddr);
+ f2fs_truncate_meta_inode_pages(sbi, fio->new_blkaddr, 1);
stat_inc_inplace_blocks(fio->sbi);
@@ -3569,9 +3572,7 @@ void f2fs_do_replace_block(struct f2fs_sb_info *sbi, struct f2fs_summary *sum,
update_sit_entry(sbi, new_blkaddr, 1);
}
if (GET_SEGNO(sbi, old_blkaddr) != NULL_SEGNO) {
- invalidate_mapping_pages(META_MAPPING(sbi),
- old_blkaddr, old_blkaddr);
- f2fs_invalidate_compress_page(sbi, old_blkaddr);
+ f2fs_invalidate_internal_cache(sbi, old_blkaddr);
if (!from_gc)
update_segment_mtime(sbi, old_blkaddr, 0);
update_sit_entry(sbi, old_blkaddr, -1);
@@ -3660,7 +3661,7 @@ void f2fs_wait_on_block_writeback_range(struct inode *inode, block_t blkaddr,
for (i = 0; i < len; i++)
f2fs_wait_on_block_writeback(inode, blkaddr + i);
- invalidate_mapping_pages(META_MAPPING(sbi), blkaddr, blkaddr + len - 1);
+ f2fs_truncate_meta_inode_pages(sbi, blkaddr, len);
}
static int read_compacted_summaries(struct f2fs_sb_info *sbi)
diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c
index 9d1b5699f5bf..a1c33931d175 100644
--- a/fs/f2fs/super.c
+++ b/fs/f2fs/super.c
@@ -1400,6 +1400,11 @@ static int parse_options(struct super_block *sb, char *options, bool is_remount)
return -EINVAL;
}
+ if (f2fs_is_readonly(sbi) && test_opt(sbi, FLUSH_MERGE)) {
+ f2fs_err(sbi, "FLUSH_MERGE not compatible with readonly mode");
+ return -EINVAL;
+ }
+
if (f2fs_sb_has_readonly(sbi) && !f2fs_readonly(sbi->sb)) {
f2fs_err(sbi, "Allow to mount readonly mode only");
return -EROFS;
@@ -1737,6 +1742,18 @@ static int f2fs_freeze(struct super_block *sb)
static int f2fs_unfreeze(struct super_block *sb)
{
+ struct f2fs_sb_info *sbi = F2FS_SB(sb);
+
+ /*
+ * It will update discard_max_bytes of mounted lvm device to zero
+ * after creating snapshot on this lvm device, let's drop all
+ * remained discards.
+ * We don't need to disable real-time discard because discard_max_bytes
+ * will recover after removal of snapshot.
+ */
+ if (test_opt(sbi, DISCARD) && !f2fs_hw_support_discard(sbi))
+ f2fs_issue_discard_timeout(sbi, false);
+
clear_sbi_flag(F2FS_SB(sb), SBI_IS_FREEZING);
return 0;
}
@@ -1997,8 +2014,10 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root)
seq_puts(seq, ",inline_dentry");
else
seq_puts(seq, ",noinline_dentry");
- if (!f2fs_readonly(sbi->sb) && test_opt(sbi, FLUSH_MERGE))
+ if (test_opt(sbi, FLUSH_MERGE))
seq_puts(seq, ",flush_merge");
+ else
+ seq_puts(seq, ",noflush_merge");
if (test_opt(sbi, NOBARRIER))
seq_puts(seq, ",nobarrier");
if (test_opt(sbi, FASTBOOT))
@@ -2096,9 +2115,22 @@ static int f2fs_show_options(struct seq_file *seq, struct dentry *root)
return 0;
}
-static void default_options(struct f2fs_sb_info *sbi)
+static void default_options(struct f2fs_sb_info *sbi, bool remount)
{
/* init some FS parameters */
+ if (!remount) {
+ set_opt(sbi, READ_EXTENT_CACHE);
+ clear_opt(sbi, DISABLE_CHECKPOINT);
+
+ if (f2fs_hw_support_discard(sbi) || f2fs_hw_should_discard(sbi))
+ set_opt(sbi, DISCARD);
+
+ if (f2fs_sb_has_blkzoned(sbi))
+ F2FS_OPTION(sbi).discard_unit = DISCARD_UNIT_SECTION;
+ else
+ F2FS_OPTION(sbi).discard_unit = DISCARD_UNIT_BLOCK;
+ }
+
if (f2fs_sb_has_readonly(sbi))
F2FS_OPTION(sbi).active_logs = NR_CURSEG_RO_TYPE;
else
@@ -2119,22 +2151,16 @@ static void default_options(struct f2fs_sb_info *sbi)
set_opt(sbi, INLINE_XATTR);
set_opt(sbi, INLINE_DATA);
set_opt(sbi, INLINE_DENTRY);
- set_opt(sbi, READ_EXTENT_CACHE);
set_opt(sbi, NOHEAP);
- clear_opt(sbi, DISABLE_CHECKPOINT);
set_opt(sbi, MERGE_CHECKPOINT);
F2FS_OPTION(sbi).unusable_cap = 0;
sbi->sb->s_flags |= SB_LAZYTIME;
- set_opt(sbi, FLUSH_MERGE);
- if (f2fs_hw_support_discard(sbi) || f2fs_hw_should_discard(sbi))
- set_opt(sbi, DISCARD);
- if (f2fs_sb_has_blkzoned(sbi)) {
+ if (!f2fs_is_readonly(sbi))
+ set_opt(sbi, FLUSH_MERGE);
+ if (f2fs_sb_has_blkzoned(sbi))
F2FS_OPTION(sbi).fs_mode = FS_MODE_LFS;
- F2FS_OPTION(sbi).discard_unit = DISCARD_UNIT_SECTION;
- } else {
+ else
F2FS_OPTION(sbi).fs_mode = FS_MODE_ADAPTIVE;
- F2FS_OPTION(sbi).discard_unit = DISCARD_UNIT_BLOCK;
- }
#ifdef CONFIG_F2FS_FS_XATTR
set_opt(sbi, XATTR_USER);
@@ -2311,7 +2337,7 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
clear_sbi_flag(sbi, SBI_NEED_SB_WRITE);
}
- default_options(sbi);
+ default_options(sbi, true);
/* parse mount options */
err = parse_options(sb, data, true);
@@ -2393,11 +2419,11 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data)
if ((*flags & SB_RDONLY) ||
(F2FS_OPTION(sbi).bggc_mode == BGGC_MODE_OFF &&
!test_opt(sbi, GC_MERGE))) {
- if (sbi->gc_thread) {
+ if (sbi->gc_thread.f2fs_gc_task) {
f2fs_stop_gc_thread(sbi);
need_restart_gc = true;
}
- } else if (!sbi->gc_thread) {
+ } else if (!sbi->gc_thread.f2fs_gc_task) {
err = f2fs_start_gc_thread(sbi);
if (err)
goto restore_opts;
@@ -4243,7 +4269,7 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent)
sbi->s_chksum_seed = f2fs_chksum(sbi, ~0, raw_super->uuid,
sizeof(raw_super->uuid));
- default_options(sbi);
+ default_options(sbi, false);
/* parse mount options */
options = kstrdup((const char *)data, GFP_KERNEL);
if (data && !options) {
diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c
index 503a50542ac5..52457176e95e 100644
--- a/fs/f2fs/sysfs.c
+++ b/fs/f2fs/sysfs.c
@@ -68,7 +68,7 @@ static ssize_t f2fs_sbi_show(struct f2fs_attr *a,
static unsigned char *__struct_ptr(struct f2fs_sb_info *sbi, int struct_type)
{
if (struct_type == GC_THREAD)
- return (unsigned char *)sbi->gc_thread;
+ return (unsigned char *)&sbi->gc_thread;
else if (struct_type == SM_INFO)
return (unsigned char *)SM_I(sbi);
else if (struct_type == DCC_INFO)
@@ -326,19 +326,14 @@ static ssize_t f2fs_sbi_show(struct f2fs_attr *a,
if (!strcmp(a->attr.name, "ckpt_thread_ioprio")) {
struct ckpt_req_control *cprc = &sbi->cprc_info;
- int len = 0;
int class = IOPRIO_PRIO_CLASS(cprc->ckpt_thread_ioprio);
- int data = IOPRIO_PRIO_DATA(cprc->ckpt_thread_ioprio);
+ int level = IOPRIO_PRIO_LEVEL(cprc->ckpt_thread_ioprio);
- if (class == IOPRIO_CLASS_RT)
- len += scnprintf(buf + len, PAGE_SIZE - len, "rt,");
- else if (class == IOPRIO_CLASS_BE)
- len += scnprintf(buf + len, PAGE_SIZE - len, "be,");
- else
+ if (class != IOPRIO_CLASS_RT && class != IOPRIO_CLASS_BE)
return -EINVAL;
- len += scnprintf(buf + len, PAGE_SIZE - len, "%d\n", data);
- return len;
+ return sysfs_emit(buf, "%s,%d\n",
+ class == IOPRIO_CLASS_RT ? "rt" : "be", level);
}
#ifdef CONFIG_F2FS_FS_COMPRESSION
@@ -458,7 +453,7 @@ static ssize_t __sbi_store(struct f2fs_attr *a,
const char *name = strim((char *)buf);
struct ckpt_req_control *cprc = &sbi->cprc_info;
int class;
- long data;
+ long level;
int ret;
if (!strncmp(name, "rt,", 3))
@@ -469,14 +464,14 @@ static ssize_t __sbi_store(struct f2fs_attr *a,
return -EINVAL;
name += 3;
- ret = kstrtol(name, 10, &data);
+ ret = kstrtol(name, 10, &level);
if (ret)
return ret;
- if (data >= IOPRIO_NR_LEVELS || data < 0)
+ if (level >= IOPRIO_NR_LEVELS || level < 0)
return -EINVAL;
- cprc->ckpt_thread_ioprio = IOPRIO_PRIO_VALUE(class, data);
- if (test_opt(sbi, MERGE_CHECKPOINT)) {
+ cprc->ckpt_thread_ioprio = IOPRIO_PRIO_VALUE(class, level);
+ if (cprc->f2fs_issue_ckpt) {
ret = set_task_ioprio(cprc->f2fs_issue_ckpt,
cprc->ckpt_thread_ioprio);
if (ret)
@@ -543,20 +538,20 @@ static ssize_t __sbi_store(struct f2fs_attr *a,
sbi->gc_mode = GC_NORMAL;
} else if (t == 1) {
sbi->gc_mode = GC_URGENT_HIGH;
- if (sbi->gc_thread) {
- sbi->gc_thread->gc_wake = 1;
+ if (sbi->gc_thread.f2fs_gc_task) {
+ sbi->gc_thread.gc_wake = 1;
wake_up_interruptible_all(
- &sbi->gc_thread->gc_wait_queue_head);
+ &sbi->gc_thread.gc_wait_queue_head);
wake_up_discard_thread(sbi, true);
}
} else if (t == 2) {
sbi->gc_mode = GC_URGENT_LOW;
} else if (t == 3) {
sbi->gc_mode = GC_URGENT_MID;
- if (sbi->gc_thread) {
- sbi->gc_thread->gc_wake = 1;
+ if (sbi->gc_thread.f2fs_gc_task) {
+ sbi->gc_thread.gc_wake = 1;
wake_up_interruptible_all(
- &sbi->gc_thread->gc_wait_queue_head);
+ &sbi->gc_thread.gc_wait_queue_head);
}
} else {
return -EINVAL;
@@ -708,13 +703,14 @@ static ssize_t f2fs_sbi_store(struct f2fs_attr *a,
ssize_t ret;
bool gc_entry = (!strcmp(a->attr.name, "gc_urgent") ||
a->struct_type == GC_THREAD);
+ bool thread_entry = !strcmp(a->attr.name, "ckpt_thread_ioprio");
- if (gc_entry) {
+ if (gc_entry || thread_entry) {
if (!down_read_trylock(&sbi->sb->s_umount))
return -EAGAIN;
}
ret = __sbi_store(a, sbi, buf, count);
- if (gc_entry)
+ if (gc_entry || thread_entry)
up_read(&sbi->sb->s_umount);
return ret;
diff --git a/fs/fcntl.c b/fs/fcntl.c
index 1bde178e4aa2..13ba6b47877b 100644
--- a/fs/fcntl.c
+++ b/fs/fcntl.c
@@ -290,6 +290,9 @@ static long fcntl_rw_hint(struct file *file, unsigned int cmd,
return -EFAULT;
return 0;
case F_SET_RW_HINT:
+ /* nothing is ever written to it */
+ if (IS_IMMUTABLE(inode))
+ return -EPERM;
if (copy_from_user(&hint, argp, sizeof(hint)))
return -EFAULT;
if (!rw_hint_valid(hint))
diff --git a/fs/fsopen.c b/fs/fsopen.c
index 25689eaef511..d755a4c7cd56 100644
--- a/fs/fsopen.c
+++ b/fs/fsopen.c
@@ -191,6 +191,9 @@ SYSCALL_DEFINE3(fspick, int, dfd, const char __user *, path, unsigned int, flags
ret = -EINVAL;
if (target.mnt->mnt_root != target.dentry)
goto err_path;
+ /* kernel-internal superblocks are nobody's to reconfigure */
+ if (target.dentry->d_sb->s_flags & SB_NOUSER)
+ goto err_path;
fc = fs_context_for_reconfigure(target.dentry, 0, 0);
if (IS_ERR(fc)) {
diff --git a/fs/fuse/inode.c b/fs/fuse/inode.c
index 332cc885b815..269699abdbfc 100644
--- a/fs/fuse/inode.c
+++ b/fs/fuse/inode.c
@@ -1537,6 +1537,8 @@ static int fuse_fill_super_submount(struct super_block *sb,
fuse_fill_attr_from_inode(&root_attr, parent_fi);
root = fuse_iget(sb, parent_fi->nodeid, 0, &root_attr, 0, 0);
+ if (!root)
+ return -ENOMEM;
/*
* This inode is just a duplicate, so it is not looked up and
* its nlookup should not be incremented. fuse_iget() does
diff --git a/fs/gfs2/glock.c b/fs/gfs2/glock.c
index 428c1db295fa..d0ec9487e882 100644
--- a/fs/gfs2/glock.c
+++ b/fs/gfs2/glock.c
@@ -167,19 +167,46 @@ static bool glock_blocked_by_withdraw(struct gfs2_glock *gl)
return true;
}
-void gfs2_glock_free(struct gfs2_glock *gl)
+static void __gfs2_glock_free(struct gfs2_glock *gl)
{
- struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
-
gfs2_glock_assert_withdraw(gl, atomic_read(&gl->gl_revokes) == 0);
rhashtable_remove_fast(&gl_hash_table, &gl->gl_node, ht_parms);
smp_mb();
wake_up_glock(gl);
call_rcu(&gl->gl_rcu, gfs2_glock_dealloc);
+}
+
+void gfs2_glock_free(struct gfs2_glock *gl) {
+ struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
+
+ __gfs2_glock_free(gl);
if (atomic_dec_and_test(&sdp->sd_glock_disposal))
wake_up(&sdp->sd_glock_wait);
}
+void gfs2_glock_free_later(struct gfs2_glock *gl) {
+ struct gfs2_sbd *sdp = gl->gl_name.ln_sbd;
+
+ spin_lock(&lru_lock);
+ list_add(&gl->gl_lru, &sdp->sd_dead_glocks);
+ spin_unlock(&lru_lock);
+ if (atomic_dec_and_test(&sdp->sd_glock_disposal))
+ wake_up(&sdp->sd_glock_wait);
+}
+
+static void gfs2_free_dead_glocks(struct gfs2_sbd *sdp)
+{
+ struct list_head *list = &sdp->sd_dead_glocks;
+
+ while(!list_empty(list)) {
+ struct gfs2_glock *gl;
+
+ gl = list_first_entry(list, struct gfs2_glock, gl_lru);
+ list_del_init(&gl->gl_lru);
+ __gfs2_glock_free(gl);
+ }
+}
+
/**
* gfs2_glock_hold() - increment reference count on glock
* @gl: The glock to hold
@@ -2263,6 +2290,8 @@ void gfs2_gl_hash_clear(struct gfs2_sbd *sdp)
wait_event_timeout(sdp->sd_glock_wait,
atomic_read(&sdp->sd_glock_disposal) == 0,
HZ * 600);
+ gfs2_lm_unmount(sdp);
+ gfs2_free_dead_glocks(sdp);
glock_hash_walk(dump_glock_func, sdp);
}
diff --git a/fs/gfs2/glock.h b/fs/gfs2/glock.h
index f6dd94ce7499..ca1669cf9299 100644
--- a/fs/gfs2/glock.h
+++ b/fs/gfs2/glock.h
@@ -270,6 +270,7 @@ extern void gfs2_gl_dq_holders(struct gfs2_sbd *sdp);
extern void gfs2_glock_thaw(struct gfs2_sbd *sdp);
extern void gfs2_glock_add_to_lru(struct gfs2_glock *gl);
extern void gfs2_glock_free(struct gfs2_glock *gl);
+extern void gfs2_glock_free_later(struct gfs2_glock *gl);
extern int __init gfs2_glock_init(void);
extern void gfs2_glock_exit(void);
diff --git a/fs/gfs2/incore.h b/fs/gfs2/incore.h
index 113aeb587702..7a630de1c1a0 100644
--- a/fs/gfs2/incore.h
+++ b/fs/gfs2/incore.h
@@ -837,6 +837,7 @@ struct gfs2_sbd {
struct gfs2_holder sd_freeze_gh;
atomic_t sd_freeze_state;
struct mutex sd_freeze_mutex;
+ struct list_head sd_dead_glocks;
char sd_fsname[GFS2_FSNAME_LEN + 3 * sizeof(int) + 2];
char sd_table_name[GFS2_FSNAME_LEN];
diff --git a/fs/gfs2/lock_dlm.c b/fs/gfs2/lock_dlm.c
index 4f44173e8a36..001555ccceeb 100644
--- a/fs/gfs2/lock_dlm.c
+++ b/fs/gfs2/lock_dlm.c
@@ -121,6 +121,11 @@ static void gdlm_ast(void *arg)
struct gfs2_glock *gl = arg;
unsigned ret = gl->gl_state;
+ /* If the glock is dead, we only react to a dlm_unlock() reply. */
+ if (__lockref_is_dead(&gl->gl_lockref) &&
+ gl->gl_lksb.sb_status != -DLM_EUNLOCK)
+ return;
+
gfs2_update_reply_times(gl);
BUG_ON(gl->gl_lksb.sb_flags & DLM_SBF_DEMOTED);
@@ -171,6 +176,9 @@ static void gdlm_bast(void *arg, int mode)
{
struct gfs2_glock *gl = arg;
+ if (__lockref_is_dead(&gl->gl_lockref))
+ return;
+
switch (mode) {
case DLM_LOCK_EX:
gfs2_glock_cb(gl, LM_ST_UNLOCKED);
@@ -291,6 +299,8 @@ static void gdlm_put_lock(struct gfs2_glock *gl)
struct lm_lockstruct *ls = &sdp->sd_lockstruct;
int error;
+ BUG_ON(!__lockref_is_dead(&gl->gl_lockref));
+
if (gl->gl_lksb.sb_lkid == 0) {
gfs2_glock_free(gl);
return;
@@ -305,7 +315,7 @@ static void gdlm_put_lock(struct gfs2_glock *gl)
if (test_bit(SDF_SKIP_DLM_UNLOCK, &sdp->sd_flags) &&
!gl->gl_lksb.sb_lvbptr) {
- gfs2_glock_free(gl);
+ gfs2_glock_free_later(gl);
return;
}
diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c
index 26a70b9676d5..0b457be8d99e 100644
--- a/fs/gfs2/ops_fstype.c
+++ b/fs/gfs2/ops_fstype.c
@@ -137,6 +137,7 @@ static struct gfs2_sbd *init_sbd(struct super_block *sb)
init_waitqueue_head(&sdp->sd_log_flush_wait);
atomic_set(&sdp->sd_freeze_state, SFS_UNFROZEN);
mutex_init(&sdp->sd_freeze_mutex);
+ INIT_LIST_HEAD(&sdp->sd_dead_glocks);
return sdp;
@@ -1272,12 +1273,6 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc)
if (error) {
gfs2_freeze_unlock(&sdp->sd_freeze_gh);
- if (sdp->sd_quotad_process)
- kthread_stop(sdp->sd_quotad_process);
- sdp->sd_quotad_process = NULL;
- if (sdp->sd_logd_process)
- kthread_stop(sdp->sd_logd_process);
- sdp->sd_logd_process = NULL;
fs_err(sdp, "can't make FS RW: %d\n", error);
goto fail_per_node;
}
@@ -1287,6 +1282,12 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc)
fail_per_node:
init_per_node(sdp, UNDO);
+ if (sdp->sd_quotad_process)
+ kthread_stop(sdp->sd_quotad_process);
+ sdp->sd_quotad_process = NULL;
+ if (sdp->sd_logd_process)
+ kthread_stop(sdp->sd_logd_process);
+ sdp->sd_logd_process = NULL;
fail_inodes:
init_inodes(sdp, UNDO);
fail_sb:
diff --git a/fs/gfs2/super.c b/fs/gfs2/super.c
index 14b3f66938ea..644e74c33f8d 100644
--- a/fs/gfs2/super.c
+++ b/fs/gfs2/super.c
@@ -157,8 +157,10 @@ int gfs2_make_fs_rw(struct gfs2_sbd *sdp)
gfs2_log_pointers_init(sdp, head.lh_blkno);
error = gfs2_quota_init(sdp);
- if (!error && gfs2_withdrawn(sdp))
+ if (!error && gfs2_withdrawn(sdp)) {
+ gfs2_quota_cleanup(sdp);
error = -EIO;
+ }
if (!error)
set_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags);
return error;
@@ -641,10 +643,7 @@ static void gfs2_put_super(struct super_block *sb)
gfs2_gl_hash_clear(sdp);
truncate_inode_pages_final(&sdp->sd_aspace);
gfs2_delete_debugfs_file(sdp);
- /* Unmount the locking protocol */
- gfs2_lm_unmount(sdp);
- /* At this point, we're through participating in the lockspace */
gfs2_sys_fs_del(sdp);
free_sbd(sdp);
}
diff --git a/fs/inode.c b/fs/inode.c
index 0036d5c68fd4..99e980ddbdce 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -443,6 +443,7 @@ void __iget(struct inode *inode)
{
atomic_inc(&inode->i_count);
}
+EXPORT_SYMBOL(__iget);
/*
* get additional reference to inode; caller must already hold one.
@@ -642,21 +643,18 @@ void dump_mapping(const struct address_space *mapping)
void clear_inode(struct inode *inode)
{
/*
- * We have to cycle the i_pages lock here because reclaim can be in the
- * process of removing the last page (in __filemap_remove_folio())
- * and we must not free the mapping under it.
+ * We have to cycle the i_pages lock here because reclaim
+ * can be in the process of removing the last page (in
+ * __filemap_remove_folio()) and we must not free the mapping
+ * under it. We also remove nodes which are empty; these
+ * can occur in two different ways. The first is that radix
+ * tree expansion can fail partway and the second is that THP
+ * collapse_file() can allocate some temporary nodes and not
+ * clean them up.
*/
- xa_lock_irq(&inode->i_data.i_pages);
+ xa_destroy(&inode->i_data.i_pages);
+
BUG_ON(inode->i_data.nrpages);
- /*
- * Almost always, mapping_empty(&inode->i_data) here; but there are
- * two known and long-standing ways in which nodes may get left behind
- * (when deep radix-tree node allocation failed partway; or when THP
- * collapse_file() failed). Until those two known cases are cleaned up,
- * or a cleanup function is called here, do not BUG_ON(!mapping_empty),
- * nor even WARN_ON(!mapping_empty).
- */
- xa_unlock_irq(&inode->i_data.i_pages);
BUG_ON(!list_empty(&inode->i_data.private_list));
BUG_ON(!(inode->i_state & I_FREEING));
BUG_ON(inode->i_state & I_CLEAR);
diff --git a/fs/jfs/jfs_dtree.c b/fs/jfs/jfs_dtree.c
index 93db6eec4465..ab11849cf9cc 100644
--- a/fs/jfs/jfs_dtree.c
+++ b/fs/jfs/jfs_dtree.c
@@ -2613,7 +2613,7 @@ void dtInitRoot(tid_t tid, struct inode *ip, u32 idotdot)
* fsck.jfs should really fix this, but it currently does not.
* Called from jfs_readdir when bad index is detected.
*/
-static void add_missing_indices(struct inode *inode, s64 bn)
+static int add_missing_indices(struct inode *inode, s64 bn)
{
struct ldtentry *d;
struct dt_lock *dtlck;
@@ -2622,7 +2622,7 @@ static void add_missing_indices(struct inode *inode, s64 bn)
struct lv *lv;
struct metapage *mp;
dtpage_t *p;
- int rc;
+ int rc = 0;
s8 *stbl;
tid_t tid;
struct tlock *tlck;
@@ -2647,6 +2647,16 @@ static void add_missing_indices(struct inode *inode, s64 bn)
stbl = DT_GETSTBL(p);
for (i = 0; i < p->header.nextindex; i++) {
+ if (stbl[i] < 0) {
+ jfs_err("jfs: add_missing_indices: Invalid stbl[%d] = %d for inode %ld, block = %lld",
+ i, stbl[i], (long)inode->i_ino, (long long)bn);
+ rc = -EIO;
+
+ DT_PUTPAGE(mp);
+ txAbort(tid, 0);
+ goto end;
+ }
+
d = (struct ldtentry *) &p->slot[stbl[i]];
index = le32_to_cpu(d->index);
if ((index < 2) || (index >= JFS_IP(inode)->next_index)) {
@@ -2664,6 +2674,7 @@ static void add_missing_indices(struct inode *inode, s64 bn)
(void) txCommit(tid, 1, &inode, 0);
end:
txEnd(tid);
+ return rc;
}
/*
@@ -3017,7 +3028,8 @@ int jfs_readdir(struct file *file, struct dir_context *ctx)
}
if (fix_page) {
- add_missing_indices(ip, bn);
+ if ((rc = add_missing_indices(ip, bn)))
+ goto out;
page_fixed = 1;
}
diff --git a/fs/libfs.c b/fs/libfs.c
index 63bc52c20f7e..60ec2dd90358 100644
--- a/fs/libfs.c
+++ b/fs/libfs.c
@@ -449,6 +449,31 @@ int simple_rmdir(struct inode *dir, struct dentry *dentry)
}
EXPORT_SYMBOL(simple_rmdir);
+/**
+ * simple_rename_timestamp - update the various inode timestamps for rename
+ * @old_dir: old parent directory
+ * @old_dentry: dentry that is being renamed
+ * @new_dir: new parent directory
+ * @new_dentry: target for rename
+ *
+ * POSIX mandates that the old and new parent directories have their ctime and
+ * mtime updated, and that inodes of @old_dentry and @new_dentry (if any), have
+ * their ctime updated.
+ */
+void simple_rename_timestamp(struct inode *old_dir, struct dentry *old_dentry,
+ struct inode *new_dir, struct dentry *new_dentry)
+{
+ struct inode *newino = d_inode(new_dentry);
+
+ old_dir->i_mtime = inode_set_ctime_current(old_dir);
+ if (new_dir != old_dir)
+ new_dir->i_mtime = inode_set_ctime_current(new_dir);
+ inode_set_ctime_current(d_inode(old_dentry));
+ if (newino)
+ inode_set_ctime_current(newino);
+}
+EXPORT_SYMBOL_GPL(simple_rename_timestamp);
+
int simple_rename_exchange(struct inode *old_dir, struct dentry *old_dentry,
struct inode *new_dir, struct dentry *new_dentry)
{
@@ -464,11 +489,7 @@ int simple_rename_exchange(struct inode *old_dir, struct dentry *old_dentry,
inc_nlink(old_dir);
}
}
- old_dir->i_ctime = old_dir->i_mtime =
- new_dir->i_ctime = new_dir->i_mtime =
- d_inode(old_dentry)->i_ctime =
- d_inode(new_dentry)->i_ctime = current_time(old_dir);
-
+ simple_rename_timestamp(old_dir, old_dentry, new_dir, new_dentry);
return 0;
}
EXPORT_SYMBOL_GPL(simple_rename_exchange);
@@ -477,7 +498,6 @@ int simple_rename(struct user_namespace *mnt_userns, struct inode *old_dir,
struct dentry *old_dentry, struct inode *new_dir,
struct dentry *new_dentry, unsigned int flags)
{
- struct inode *inode = d_inode(old_dentry);
int they_are_dirs = d_is_dir(old_dentry);
if (flags & ~(RENAME_NOREPLACE | RENAME_EXCHANGE))
@@ -500,9 +520,7 @@ int simple_rename(struct user_namespace *mnt_userns, struct inode *old_dir,
inc_nlink(new_dir);
}
- old_dir->i_ctime = old_dir->i_mtime = new_dir->i_ctime =
- new_dir->i_mtime = inode->i_ctime = current_time(old_dir);
-
+ simple_rename_timestamp(old_dir, old_dentry, new_dir, new_dentry);
return 0;
}
EXPORT_SYMBOL(simple_rename);
diff --git a/fs/lockd/clntproc.c b/fs/lockd/clntproc.c
index 99fffc9cb958..7f867122e388 100644
--- a/fs/lockd/clntproc.c
+++ b/fs/lockd/clntproc.c
@@ -479,8 +479,11 @@ static const struct file_lock_operations nlmclnt_lock_ops = {
static void nlmclnt_locks_init_private(struct file_lock *fl, struct nlm_host *host)
{
fl->fl_u.nfs_fl.state = 0;
+ fl->fl_ops = NULL;
fl->fl_u.nfs_fl.owner = nlmclnt_find_lockowner(host, fl->fl_owner);
INIT_LIST_HEAD(&fl->fl_u.nfs_fl.list);
+ if (!fl->fl_u.nfs_fl.owner)
+ return;
fl->fl_ops = &nlmclnt_lock_ops;
}
diff --git a/fs/lockd/svc.c b/fs/lockd/svc.c
index c33f78513f00..1905e7a6b916 100644
--- a/fs/lockd/svc.c
+++ b/fs/lockd/svc.c
@@ -49,7 +49,7 @@
static struct svc_program nlmsvc_program;
-const struct nlmsvc_binding *nlmsvc_ops;
+const struct nlmsvc_binding __rcu *nlmsvc_ops;
EXPORT_SYMBOL_GPL(nlmsvc_ops);
static DEFINE_MUTEX(nlmsvc_mutex);
@@ -113,7 +113,7 @@ static void set_grace_period(struct net *net)
static void restart_grace(void)
{
- if (nlmsvc_ops) {
+ if (rcu_access_pointer(nlmsvc_ops)) {
struct net *net = &init_net;
struct lockd_net *ln = net_generic(net, lockd_net_id);
@@ -175,7 +175,7 @@ lockd(void *vrqstp)
svc_process(rqstp);
}
flush_signals(current);
- if (nlmsvc_ops)
+ if (rcu_access_pointer(nlmsvc_ops))
nlmsvc_invalidate_all();
nlm_shutdown_hosts();
cancel_delayed_work_sync(&ln->grace_period_end);
diff --git a/fs/lockd/svc4proc.c b/fs/lockd/svc4proc.c
index 28ba7b1460aa..bb5653f4bf2e 100644
--- a/fs/lockd/svc4proc.c
+++ b/fs/lockd/svc4proc.c
@@ -29,7 +29,7 @@ nlm4svc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp,
__be32 error = 0;
/* nfsd callbacks must have been installed for this procedure */
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return nlm_lck_denied_nolocks;
if (lock->lock_start > OFFSET_MAX ||
@@ -487,7 +487,7 @@ nlm4svc_proc_granted_res(struct svc_rqst *rqstp)
{
struct nlm_res *argp = rqstp->rq_argp;
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return rpc_success;
dprintk("lockd: GRANTED_RES called\n");
diff --git a/fs/lockd/svcproc.c b/fs/lockd/svcproc.c
index 1a4459763644..bf1c22028cf0 100644
--- a/fs/lockd/svcproc.c
+++ b/fs/lockd/svcproc.c
@@ -59,7 +59,7 @@ nlmsvc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp,
__be32 error = 0;
/* nfsd callbacks must have been installed for this procedure */
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return nlm_lck_denied_nolocks;
/* Obtain host handle */
@@ -521,7 +521,7 @@ nlmsvc_proc_granted_res(struct svc_rqst *rqstp)
{
struct nlm_res *argp = rqstp->rq_argp;
- if (!nlmsvc_ops)
+ if (!rcu_access_pointer(nlmsvc_ops))
return rpc_success;
dprintk("lockd: GRANTED_RES called\n");
diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c
index 9e2a67bfd070..fa72f1aef464 100644
--- a/fs/lockd/svcsubs.c
+++ b/fs/lockd/svcsubs.c
@@ -86,12 +86,23 @@ int lock_to_openmode(struct file_lock *lock)
static __be32 nlm_do_fopen(struct svc_rqst *rqstp,
struct nlm_file *file, int mode)
{
+ const struct nlmsvc_binding *ops;
struct file **fp = &file->f_file[mode];
__be32 nfserr;
if (*fp)
return 0;
- nfserr = nlmsvc_ops->fopen(rqstp, &file->f_handle, fp, mode);
+
+ rcu_read_lock();
+ ops = rcu_dereference(nlmsvc_ops);
+ if (!ops || !try_module_get(ops->owner)) {
+ rcu_read_unlock();
+ return nlm_lck_denied_nolocks;
+ }
+ rcu_read_unlock();
+
+ nfserr = ops->fopen(rqstp, &file->f_handle, fp, mode);
+ module_put(ops->owner);
if (nfserr)
dprintk("lockd: open failed (error %d)\n", nfserr);
return nfserr;
@@ -159,6 +170,33 @@ nlm_lookup_file(struct svc_rqst *rqstp, struct nlm_file **result,
goto out_unlock;
}
+/*
+ * Release the struct file references held by a nlm_file.
+ */
+static void nlm_release_files(struct nlm_file *file)
+{
+ const struct nlmsvc_binding *ops;
+ bool have_ops;
+
+ rcu_read_lock();
+ ops = rcu_dereference(nlmsvc_ops);
+ have_ops = ops && try_module_get(ops->owner);
+ rcu_read_unlock();
+
+ if (have_ops) {
+ if (file->f_file[O_RDONLY])
+ ops->fclose(file->f_file[O_RDONLY]);
+ if (file->f_file[O_WRONLY])
+ ops->fclose(file->f_file[O_WRONLY]);
+ module_put(ops->owner);
+ } else {
+ if (file->f_file[O_RDONLY])
+ fput(file->f_file[O_RDONLY]);
+ if (file->f_file[O_WRONLY])
+ fput(file->f_file[O_WRONLY]);
+ }
+}
+
/*
* Delete a file after having released all locks, blocks and shares
*/
@@ -168,10 +206,7 @@ nlm_delete_file(struct nlm_file *file)
nlm_debug_print_file("closing file", file);
if (!hlist_unhashed(&file->f_list)) {
hlist_del(&file->f_list);
- if (file->f_file[O_RDONLY])
- nlmsvc_ops->fclose(file->f_file[O_RDONLY]);
- if (file->f_file[O_WRONLY])
- nlmsvc_ops->fclose(file->f_file[O_WRONLY]);
+ nlm_release_files(file);
kfree(file);
} else {
printk(KERN_WARNING "lockd: attempt to release unknown file!\n");
@@ -493,7 +528,7 @@ EXPORT_SYMBOL_GPL(nlmsvc_unlock_all_by_sb);
static int
nlmsvc_match_ip(void *datap, struct nlm_host *host)
{
- return rpc_cmp_addr(nlm_srcaddr(host), datap);
+ return rpc_cmp_addr(nlm_srcaddr(datap), (struct sockaddr *)host);
}
/**
diff --git a/fs/mbcache.c b/fs/mbcache.c
index 2a4b8b549e93..637e30744613 100644
--- a/fs/mbcache.c
+++ b/fs/mbcache.c
@@ -406,6 +406,7 @@ void mb_cache_destroy(struct mb_cache *cache)
{
struct mb_cache_entry *entry, *next;
+ cancel_work_sync(&cache->c_shrink_work);
unregister_shrinker(&cache->c_shrink);
/*
diff --git a/fs/namei.c b/fs/namei.c
index f4fb5bf5b81f..d3ce9e1351b5 100644
--- a/fs/namei.c
+++ b/fs/namei.c
@@ -3713,6 +3713,10 @@ static int vfs_tmpfile(struct user_namespace *mnt_userns,
int error;
int open_flag = file->f_flags;
+ /* A tmpfile is I_LINKABLE, so guard its owner like may_o_create(). */
+ if (!fsuidgid_has_mapping(dir->i_sb, mnt_userns))
+ return -EOVERFLOW;
+
/* we want directory to be writable */
error = inode_permission(mnt_userns, dir, MAY_WRITE | MAY_EXEC);
if (error)
diff --git a/fs/namespace.c b/fs/namespace.c
index ed5e9e7251e3..c07e9e857f99 100644
--- a/fs/namespace.c
+++ b/fs/namespace.c
@@ -1122,7 +1122,7 @@ static struct mount *clone_mnt(struct mount *old, struct dentry *root,
}
mnt->mnt.mnt_flags = old->mnt.mnt_flags;
- mnt->mnt.mnt_flags &= ~(MNT_WRITE_HOLD|MNT_MARKED|MNT_INTERNAL);
+ mnt->mnt.mnt_flags &= ~(MNT_WRITE_HOLD|MNT_MARKED|MNT_INTERNAL|MNT_UMOUNT);
atomic_inc(&sb->s_active);
mnt->mnt.mnt_userns = mnt_user_ns(&old->mnt);
@@ -1561,6 +1561,8 @@ static void umount_tree(struct mount *mnt, enum umount_tree_flags how)
/* Gather the mounts to umount */
for (p = mnt; p; p = next_mnt(p, mnt)) {
+ /* A mount is unmounted once. */
+ WARN_ON_ONCE(p->mnt.mnt_flags & MNT_UMOUNT);
p->mnt.mnt_flags |= MNT_UMOUNT;
list_move(&p->mnt_list, &tmp_list);
}
@@ -1735,6 +1737,9 @@ static int do_umount(struct mount *mnt, int flags)
* detach_mounts allows lazily unmounting those mounts instead of
* leaking them.
*
+ * The dentry is unhashed before the mounts go so that no lookup finds
+ * what they covered. The caller removes it for good afterwards.
+ *
* The caller may hold dentry->d_inode->i_mutex.
*/
void __detach_mounts(struct dentry *dentry)
@@ -1748,6 +1753,8 @@ void __detach_mounts(struct dentry *dentry)
if (!mp)
goto out_unlock;
+ /* the name goes first, what covered it goes second */
+ d_drop(dentry);
event++;
while (!hlist_empty(&mp->m_list)) {
mnt = hlist_entry(mp->m_list.first, struct mount, mnt_mp_list);
@@ -2212,14 +2219,30 @@ static int attach_recursive_mnt(struct mount *source_mnt,
struct mountpoint *dest_mp,
bool moving)
{
- struct user_namespace *user_ns = current->nsproxy->mnt_ns->user_ns;
HLIST_HEAD(tree_list);
struct mnt_namespace *ns = dest_mnt->mnt_ns;
+ struct user_namespace *user_ns = ns->user_ns;
struct mountpoint *smp;
struct mount *child, *p;
struct hlist_node *n;
int err;
+ /*
+ * A caller in an unprivileged mount namespaces may trigger an
+ * automount and propagate locked mounts into privileged mount
+ * namespaces. Take ownership from the target mount namespace.
+ * It's equivalent for everything but the automount case.
+ *
+ * Detached trees in anonymous mount namespaces by be handed
+ * over via SCM_RIGHTS or inherited in other ways on purpose
+ * the attaching task's mount namespace is authoritative, not
+ * the creator of the detached tree.
+ */
+ if (is_anon_ns(ns))
+ user_ns = current->nsproxy->mnt_ns->user_ns;
+ else
+ user_ns = ns->user_ns;
+
/* Preallocate a mountpoint in case the new mounts need
* to be tucked under other mounts.
*/
@@ -2428,6 +2451,9 @@ static struct mount *__do_loopback(struct path *old_path, int recurse)
if (!check_mnt(old) && old_path->dentry->d_op != &ns_dentry_operations)
return mnt;
+ if (recurse && !old->mnt_ns)
+ return ERR_PTR(-EINVAL);
+
if (!recurse && has_locked_children(old, old_path->dentry))
return mnt;
@@ -2965,7 +2991,7 @@ static int do_add_mount(struct mount *newmnt, struct mountpoint *mp,
if (!(mnt_flags & MNT_SHRINKABLE))
return -EINVAL;
/* ... and for those we'd better have mountpoint still alive */
- if (!parent->mnt_ns)
+ if (!is_mounted(&parent->mnt))
return -EINVAL;
}
@@ -3198,6 +3224,11 @@ void mark_mounts_for_expiry(struct list_head *mounts)
}
while (!list_empty(&graveyard)) {
mnt = list_first_entry(&graveyard, struct mount, mnt_expire);
+ /* an earlier umount_tree() may have moved a busy mount here */
+ if (propagate_mount_busy(mnt, 1)) {
+ list_move(&mnt->mnt_expire, mounts);
+ continue;
+ }
touch_mnt_namespace(mnt->mnt_ns);
umount_tree(mnt, UMOUNT_PROPAGATE|UMOUNT_SYNC);
}
@@ -3207,17 +3238,37 @@ void mark_mounts_for_expiry(struct list_head *mounts)
EXPORT_SYMBOL_GPL(mark_mounts_for_expiry);
+/*
+ * Unmount @mnt if it's a shrinkable mount without children that nobody uses.
+ *
+ * mount_lock must be held for write
+ */
+static bool shrink_submount(struct mount *mnt)
+{
+ if (propagate_mount_busy(mnt, 1))
+ return false;
+ touch_mnt_namespace(mnt->mnt_ns);
+ umount_tree(mnt, UMOUNT_PROPAGATE|UMOUNT_SYNC);
+ return true;
+}
+
/*
* Ripoff of 'select_parent()'
*
- * search the list of submounts for a given mountpoint, and move any
- * shrinkable submounts to the 'graveyard' list.
+ * unmount the shrinkable submounts of @parent that aren't busy, children
+ * before their parent, and say whether anything went
+ *
+ * The cursor into the children of @this_parent survives the umount of a
+ * child mount without child mounts. The mounts that get umounted together with
+ * it are located under receiving mounts of @this_parent and never under
+ * @this_parent itself. The one exception is @this_parent getting unmounted
+ * then the walk starts over.
*/
-static int select_submounts(struct mount *parent, struct list_head *graveyard)
+static bool __shrink_submounts(struct mount *parent)
{
struct mount *this_parent = parent;
struct list_head *next;
- int found = 0;
+ bool shrunk = false;
repeat:
next = this_parent->mnt_mounts.next;
@@ -3236,42 +3287,45 @@ static int select_submounts(struct mount *parent, struct list_head *graveyard)
this_parent = mnt;
goto repeat;
}
-
- if (!propagate_mount_busy(mnt, 1)) {
- list_move_tail(&mnt->mnt_expire, graveyard);
- found++;
- }
+ if (!shrink_submount(mnt))
+ continue;
+ shrunk = true;
+ if (unlikely(this_parent->mnt.mnt_flags & MNT_UMOUNT))
+ return true;
}
/*
* All done at this level ... ascend and resume the search
*/
if (this_parent != parent) {
- next = this_parent->mnt_child.next;
- this_parent = this_parent->mnt_parent;
+ struct mount *mnt = this_parent;
+
+ next = mnt->mnt_child.next;
+ this_parent = mnt->mnt_parent;
+ /* its children are gone, maybe it can go as well */
+ if (shrink_submount(mnt)) {
+ shrunk = true;
+ if (unlikely(this_parent->mnt.mnt_flags & MNT_UMOUNT))
+ return true;
+ }
goto resume;
}
- return found;
+ return shrunk;
}
/*
- * process a list of expirable mountpoints with the intent of discarding any
- * submounts of a specific parent mountpoint
+ * unmount the shrinkable submounts of @mnt that aren't busy
+ *
+ * The busy check and the umount of a mount are adjacent. An umount can
+ * still empty or move a mount in a part of the tree that was walked
+ * already, so walk again until nothing goes.
*
* mount_lock must be held for write
*/
static void shrink_submounts(struct mount *mnt)
{
- LIST_HEAD(graveyard);
- struct mount *m;
-
- /* extract submounts of 'mountpoint' from the expiration list */
- while (select_submounts(mnt, &graveyard)) {
- while (!list_empty(&graveyard)) {
- m = list_first_entry(&graveyard, struct mount,
- mnt_expire);
- touch_mnt_namespace(m->mnt_ns);
- umount_tree(m, UMOUNT_PROPAGATE|UMOUNT_SYNC);
- }
+ for (;;) {
+ if (!__shrink_submounts(mnt))
+ break;
}
}
@@ -3943,14 +3997,15 @@ SYSCALL_DEFINE2(pivot_root, const char __user *, new_root,
new_mnt = real_mount(new.mnt);
root_mnt = real_mount(root.mnt);
old_mnt = real_mount(old.mnt);
+ /* only a mounted mount has a parent that namespace_sem pins */
+ if (!check_mnt(root_mnt) || !check_mnt(new_mnt))
+ goto out4;
ex_parent = new_mnt->mnt_parent;
root_parent = root_mnt->mnt_parent;
if (IS_MNT_SHARED(old_mnt) ||
IS_MNT_SHARED(ex_parent) ||
IS_MNT_SHARED(root_parent))
goto out4;
- if (!check_mnt(root_mnt) || !check_mnt(new_mnt))
- goto out4;
if (new_mnt->mnt.mnt_flags & MNT_LOCKED)
goto out4;
error = -ENOENT;
@@ -4185,6 +4240,13 @@ static int do_mount_setattr(struct path *path, struct mount_kattr *kattr)
* propagation.
*/
namespace_lock();
+ /* invent_group_ids() walks the tree, only walk a mounted one */
+ if (!is_mounted(&mnt->mnt) ||
+ ((mnt_has_parent(mnt) || !is_anon_ns(mnt->mnt_ns)) &&
+ !check_mnt(mnt))) {
+ namespace_unlock();
+ return -EINVAL;
+ }
if (kattr->propagation == MS_SHARED) {
err = invent_group_ids(mnt, kattr->recurse);
if (err) {
@@ -4197,6 +4259,7 @@ static int do_mount_setattr(struct path *path, struct mount_kattr *kattr)
err = -EINVAL;
lock_mount_hash();
+ /* Checked under namespace_sem already if the propagation changes. */
/* Ensure that this isn't anything purely vfs internal. */
if (!is_mounted(&mnt->mnt))
goto out;
diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c
index fc6413953600..09f4d9d1960a 100644
--- a/fs/nfs/blocklayout/dev.c
+++ b/fs/nfs/blocklayout/dev.c
@@ -16,12 +16,14 @@
static void
bl_free_device(struct pnfs_block_dev *dev)
{
- if (dev->nr_children) {
+ if (dev->children) {
int i;
for (i = 0; i < dev->nr_children; i++)
bl_free_device(&dev->children[i]);
kfree(dev->children);
+ dev->children = NULL;
+ dev->nr_children = 0;
} else {
if (dev->pr_registered) {
const struct pr_ops *ops =
@@ -34,8 +36,10 @@ bl_free_device(struct pnfs_block_dev *dev)
pr_err("failed to unregister PR key.\n");
}
- if (dev->bdev)
+ if (dev->bdev) {
blkdev_put(dev->bdev, FMODE_READ | FMODE_WRITE);
+ dev->bdev = NULL;
+ }
}
}
@@ -375,6 +379,7 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d,
out_blkdev_put:
blkdev_put(d->bdev, FMODE_READ | FMODE_WRITE);
+ d->bdev = NULL;
return error;
}
@@ -410,8 +415,11 @@ bl_parse_concat(struct nfs_server *server, struct pnfs_block_dev *d,
for (i = 0; i < v->concat.volumes_count; i++) {
ret = bl_parse_deviceid(server, &d->children[i],
volumes, v->concat.volumes[i], gfp_mask);
- if (ret)
+ if (ret) {
+ bl_free_device(&d->children[i]);
+ bl_free_device(d);
return ret;
+ }
d->nr_children++;
d->children[i].start += len;
@@ -439,8 +447,11 @@ bl_parse_stripe(struct nfs_server *server, struct pnfs_block_dev *d,
for (i = 0; i < v->stripe.volumes_count; i++) {
ret = bl_parse_deviceid(server, &d->children[i],
volumes, v->stripe.volumes[i], gfp_mask);
- if (ret)
+ if (ret) {
+ bl_free_device(&d->children[i]);
+ bl_free_device(d);
return ret;
+ }
d->nr_children++;
len += d->children[i].len;
diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c
index fa519ce5c841..5b538ade2210 100644
--- a/fs/nfs/callback_xdr.c
+++ b/fs/nfs/callback_xdr.c
@@ -419,9 +419,9 @@ static __be32 decode_cb_sequence_args(struct svc_rqst *rqstp,
args->csa_nrclists = ntohl(*p++);
args->csa_rclists = NULL;
if (args->csa_nrclists) {
- args->csa_rclists = kmalloc_array(args->csa_nrclists,
- sizeof(*args->csa_rclists),
- GFP_KERNEL);
+ args->csa_rclists = kcalloc(args->csa_nrclists,
+ sizeof(*args->csa_rclists),
+ GFP_KERNEL);
if (unlikely(args->csa_rclists == NULL))
return htonl(NFS4ERR_RESOURCE);
diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c
index 17b38da17288..64a869d40774 100644
--- a/fs/nfs/delegation.c
+++ b/fs/nfs/delegation.c
@@ -145,6 +145,7 @@ int nfs4_check_delegation(struct inode *inode, fmode_t flags)
static int nfs_delegation_claim_locks(struct nfs4_state *state, const nfs4_stateid *stateid)
{
struct inode *inode = state->inode;
+ struct nfs_inode *nfsi = NFS_I(inode);
struct file_lock *fl;
struct file_lock_context *flctx = inode->i_flctx;
struct list_head *list;
@@ -154,6 +155,9 @@ static int nfs_delegation_claim_locks(struct nfs4_state *state, const nfs4_state
goto out;
list = &flctx->flc_posix;
+
+ /* Guard against reclaim and new lock/unlock calls */
+ down_write(&nfsi->rwsem);
spin_lock(&flctx->flc_lock);
restart:
list_for_each_entry(fl, list, fl_list) {
@@ -161,8 +165,10 @@ static int nfs_delegation_claim_locks(struct nfs4_state *state, const nfs4_state
continue;
spin_unlock(&flctx->flc_lock);
status = nfs4_lock_delegation_recall(fl, state, stateid);
- if (status < 0)
+ if (status < 0) {
+ up_write(&nfsi->rwsem);
goto out;
+ }
spin_lock(&flctx->flc_lock);
}
if (list == &flctx->flc_posix) {
@@ -170,6 +176,7 @@ static int nfs_delegation_claim_locks(struct nfs4_state *state, const nfs4_state
goto restart;
}
spin_unlock(&flctx->flc_lock);
+ up_write(&nfsi->rwsem);
out:
return status;
}
@@ -436,11 +443,14 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
struct nfs_inode *nfsi = NFS_I(inode);
struct nfs_delegation *delegation, *old_delegation;
struct nfs_delegation *freeme = NULL;
+ bool orphaned = false;
int status = 0;
delegation = kmalloc(sizeof(*delegation), GFP_KERNEL_ACCOUNT);
- if (delegation == NULL)
+ if (delegation == NULL) {
+ nfs4_proc_delegreturn(inode, cred, stateid, 0);
return -ENOMEM;
+ }
nfs4_stateid_copy(&delegation->stateid, stateid);
refcount_set(&delegation->refcount, 1);
delegation->type = type;
@@ -482,12 +492,16 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
goto out;
}
if (test_and_set_bit(NFS_DELEGATION_RETURNING,
- &old_delegation->flags))
+ &old_delegation->flags)) {
+ orphaned = true;
goto out;
+ }
}
freeme = nfs_detach_delegation_locked(nfsi, old_delegation, clp);
- if (freeme == NULL)
+ if (freeme == NULL) {
+ orphaned = true;
goto out;
+ }
add_new:
/*
* If we didn't revalidate the change attribute before setting
@@ -514,8 +528,11 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred,
trace_nfs4_set_delegation(inode, type);
out:
spin_unlock(&clp->cl_lock);
- if (delegation != NULL)
+ if (delegation != NULL) {
+ if (orphaned)
+ nfs_do_return_delegation(inode, delegation, 0);
__nfs_free_delegation(delegation);
+ }
if (freeme != NULL) {
nfs_do_return_delegation(inode, freeme, 0);
nfs_free_delegation(freeme);
diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c
index a5f69d1fce3f..88140175a445 100644
--- a/fs/nfs/flexfilelayout/flexfilelayout.c
+++ b/fs/nfs/flexfilelayout/flexfilelayout.c
@@ -1127,7 +1127,8 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task,
struct pnfs_layout_hdr *lo = lseg->pls_layout;
struct inode *inode = lo->plh_inode;
struct nfs4_deviceid_node *devid = FF_LAYOUT_DEVID_NODE(lseg, idx);
- struct nfs4_slot_table *tbl = &clp->cl_session->fc_slot_table;
+ struct nfs4_slot_table *tbl = nfs4_has_session(clp) ?
+ &clp->cl_session->fc_slot_table : clp->cl_slot_tbl;
switch (op_status) {
case NFS4_OK:
diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c
index 9f352f1a6eb1..321c6e86b0b0 100644
--- a/fs/nfs/nfs4proc.c
+++ b/fs/nfs/nfs4proc.c
@@ -3798,8 +3798,8 @@ int nfs4_do_close(struct nfs4_state *state, gfp_t gfp_mask, int wait)
calldata->res.seqid = calldata->arg.seqid;
calldata->res.server = server;
calldata->res.lr_ret = -NFS4ERR_NOMATCHING_LAYOUT;
- calldata->lr.roc = pnfs_roc(state->inode,
- &calldata->lr.arg, &calldata->lr.res, msg.rpc_cred);
+ calldata->lr.roc = pnfs_roc(state->inode, &calldata->lr.arg,
+ &calldata->lr.res, msg.rpc_cred, wait);
if (calldata->lr.roc) {
calldata->arg.lr_args = &calldata->lr.arg;
calldata->res.lr_res = &calldata->lr.res;
@@ -6748,7 +6748,7 @@ static int _nfs4_proc_delegreturn(struct inode *inode, const struct cred *cred,
data->inode = nfs_igrab_and_active(inode);
if (data->inode || issync) {
data->lr.roc = pnfs_roc(inode, &data->lr.arg, &data->lr.res,
- cred);
+ cred, issync);
if (data->lr.roc) {
data->args.lr_args = &data->lr.arg;
data->res.lr_res = &data->lr.res;
@@ -6961,7 +6961,6 @@ static void nfs4_locku_done(struct rpc_task *task, void *data)
switch (task->tk_status) {
case 0:
renew_lease(calldata->server, calldata->timestamp);
- locks_lock_inode_wait(calldata->lsp->ls_state->inode, &calldata->fl);
if (nfs4_update_lock_stateid(calldata->lsp,
&calldata->res.stateid))
break;
@@ -7224,11 +7223,6 @@ static void nfs4_lock_done(struct rpc_task *task, void *calldata)
case 0:
renew_lease(NFS_SERVER(d_inode(data->ctx->dentry)),
data->timestamp);
- if (data->arg.new_lock && !data->cancelled) {
- data->fl.fl_flags &= ~(FL_SLEEP | FL_ACCESS);
- if (locks_lock_inode_wait(lsp->ls_state->inode, &data->fl) < 0)
- goto out_restart;
- }
if (data->arg.new_lock_owner != 0) {
nfs_confirm_seqid(&lsp->ls_seqid, 0);
nfs4_stateid_copy(&lsp->ls_stateid, &data->res.stateid);
@@ -7337,11 +7331,10 @@ static int _nfs4_do_setlk(struct nfs4_state *state, int cmd, struct file_lock *f
msg.rpc_argp = &data->arg;
msg.rpc_resp = &data->res;
task_setup_data.callback_data = data;
- if (recovery_type > NFS_LOCK_NEW) {
- if (recovery_type == NFS_LOCK_RECLAIM)
- data->arg.reclaim = NFS_LOCK_RECLAIM;
- } else
- data->arg.new_lock = 1;
+
+ if (recovery_type == NFS_LOCK_RECLAIM)
+ data->arg.reclaim = NFS_LOCK_RECLAIM;
+
task = rpc_run_task(&task_setup_data);
if (IS_ERR(task))
return PTR_ERR(task);
@@ -7453,6 +7446,13 @@ static int _nfs4_proc_setlk(struct nfs4_state *state, int cmd, struct file_lock
up_read(&nfsi->rwsem);
mutex_unlock(&sp->so_delegreturn_mutex);
status = _nfs4_do_setlk(state, cmd, request, NFS_LOCK_NEW);
+ if (status)
+ goto out;
+
+ down_read(&nfsi->rwsem);
+ request->fl_flags &= ~(FL_SLEEP | FL_ACCESS);
+ status = locks_lock_inode_wait(state->inode, request);
+ up_read(&nfsi->rwsem);
out:
request->fl_flags = fl_flags;
return status;
@@ -7640,6 +7640,7 @@ static int nfs4_add_lease(struct file *file, long arg, struct file_lock **lease,
{
struct inode *inode = file_inode(file);
fmode_t type = arg == F_RDLCK ? FMODE_READ : FMODE_WRITE;
+ fl_owner_t owner = (*lease)->fl_owner;
int ret;
/* No delegation, no lease */
@@ -7649,7 +7650,8 @@ static int nfs4_add_lease(struct file *file, long arg, struct file_lock **lease,
if (ret || nfs4_have_delegation(inode, type))
return ret;
/* We raced with a delegation return */
- nfs4_delete_lease(file, priv);
+ dprintk("%s: raced with a delegation return\n", __func__);
+ nfs4_delete_lease(file, &owner);
return -EAGAIN;
}
diff --git a/fs/nfs/nfs4session.c b/fs/nfs/nfs4session.c
index 5db460476bf2..c59e710eb7cc 100644
--- a/fs/nfs/nfs4session.c
+++ b/fs/nfs/nfs4session.c
@@ -634,16 +634,22 @@ int nfs4_init_ds_session(struct nfs_client *clp, unsigned long lease_time)
int ret;
spin_lock(&clp->cl_lock);
- if (test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) {
- /*
- * Do not set NFS_CS_CHECK_LEASE_TIME instead set the
- * DS lease to be equal to the MDS lease.
- */
+ /*
+ * Do not set NFS_CS_CHECK_LEASE_TIME instead set the
+ * DS lease to be equal to the MDS lease.
+ *
+ * A v4.0 DS has no session, so seed the lease every time.
+ */
+ if (!session ||
+ test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) {
clp->cl_lease_time = lease_time;
clp->cl_last_renewal = jiffies;
}
spin_unlock(&clp->cl_lock);
+ if (!session)
+ return 0;
+
ret = nfs41_check_session_ready(clp);
if (ret)
return ret;
diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c
index 99a93c63708d..26d509b6ebd1 100644
--- a/fs/nfs/pnfs.c
+++ b/fs/nfs/pnfs.c
@@ -1427,10 +1427,9 @@ pnfs_commit_and_return_layout(struct inode *inode)
return ret;
}
-bool pnfs_roc(struct inode *ino,
- struct nfs4_layoutreturn_args *args,
- struct nfs4_layoutreturn_res *res,
- const struct cred *cred)
+bool pnfs_roc(struct inode *ino, struct nfs4_layoutreturn_args *args,
+ struct nfs4_layoutreturn_res *res, const struct cred *cred,
+ bool sync)
{
struct nfs_inode *nfsi = NFS_I(ino);
struct nfs_open_context *ctx;
@@ -1441,7 +1440,7 @@ bool pnfs_roc(struct inode *ino,
nfs4_stateid stateid;
enum pnfs_iomode iomode = 0;
bool layoutreturn = false, roc = false;
- bool skip_read = false;
+ bool skip_read;
if (!nfs_have_layout(ino))
return false;
@@ -1454,20 +1453,14 @@ bool pnfs_roc(struct inode *ino,
lo = NULL;
goto out_noroc;
}
- pnfs_get_layout_hdr(lo);
- if (test_bit(NFS_LAYOUT_RETURN_LOCK, &lo->plh_flags)) {
- spin_unlock(&ino->i_lock);
- rcu_read_unlock();
- wait_on_bit(&lo->plh_flags, NFS_LAYOUT_RETURN,
- TASK_UNINTERRUPTIBLE);
- pnfs_put_layout_hdr(lo);
- goto retry;
- }
/* no roc if we hold a delegation */
+ skip_read = false;
if (nfs4_check_delegation(ino, FMODE_READ)) {
- if (nfs4_check_delegation(ino, FMODE_WRITE))
+ if (nfs4_check_delegation(ino, FMODE_WRITE)) {
+ lo = NULL;
goto out_noroc;
+ }
skip_read = true;
}
@@ -1476,12 +1469,43 @@ bool pnfs_roc(struct inode *ino,
if (state == NULL)
continue;
/* Don't return layout if there is open file state */
- if (state->state & FMODE_WRITE)
+ if (state->state & FMODE_WRITE) {
+ lo = NULL;
goto out_noroc;
+ }
if (state->state & FMODE_READ)
skip_read = true;
}
+ if (skip_read) {
+ bool writes = false;
+
+ list_for_each_entry(lseg, &lo->plh_segs, pls_list) {
+ if (lseg->pls_range.iomode != IOMODE_READ) {
+ writes = true;
+ break;
+ }
+ }
+ if (!writes) {
+ lo = NULL;
+ goto out_noroc;
+ }
+ }
+
+ pnfs_get_layout_hdr(lo);
+ if (test_bit(NFS_LAYOUT_RETURN_LOCK, &lo->plh_flags)) {
+ if (!sync) {
+ pnfs_set_plh_return_info(
+ lo, skip_read ? IOMODE_RW : IOMODE_ANY, 0);
+ goto out_noroc;
+ }
+ spin_unlock(&ino->i_lock);
+ rcu_read_unlock();
+ wait_on_bit(&lo->plh_flags, NFS_LAYOUT_RETURN,
+ TASK_UNINTERRUPTIBLE);
+ pnfs_put_layout_hdr(lo);
+ goto retry;
+ }
list_for_each_entry_safe(lseg, next, &lo->plh_segs, pls_list) {
if (skip_read && lseg->pls_range.iomode == IOMODE_READ)
@@ -1521,7 +1545,7 @@ bool pnfs_roc(struct inode *ino,
out_noroc:
spin_unlock(&ino->i_lock);
rcu_read_unlock();
- pnfs_layoutcommit_inode(ino, true);
+ pnfs_layoutcommit_inode(ino, sync);
if (roc) {
struct pnfs_layoutdriver_type *ld = NFS_SERVER(ino)->pnfs_curr_ld;
if (ld->prepare_layoutreturn)
diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h
index c95f8a26ee1b..0cc6f9a295f2 100644
--- a/fs/nfs/pnfs.h
+++ b/fs/nfs/pnfs.h
@@ -296,10 +296,9 @@ int pnfs_mark_matching_lsegs_return(struct pnfs_layout_hdr *lo,
u32 seq);
int pnfs_mark_layout_stateid_invalid(struct pnfs_layout_hdr *lo,
struct list_head *lseg_list);
-bool pnfs_roc(struct inode *ino,
- struct nfs4_layoutreturn_args *args,
- struct nfs4_layoutreturn_res *res,
- const struct cred *cred);
+bool pnfs_roc(struct inode *ino, struct nfs4_layoutreturn_args *args,
+ struct nfs4_layoutreturn_res *res, const struct cred *cred,
+ bool sync);
int pnfs_roc_done(struct rpc_task *task, struct nfs4_layoutreturn_args **argpp,
struct nfs4_layoutreturn_res **respp, int *ret);
void pnfs_roc_release(struct nfs4_layoutreturn_args *args,
@@ -770,12 +769,10 @@ pnfs_layoutcommit_outstanding(struct inode *inode)
return false;
}
-
-static inline bool
-pnfs_roc(struct inode *ino,
- struct nfs4_layoutreturn_args *args,
- struct nfs4_layoutreturn_res *res,
- const struct cred *cred)
+static inline bool pnfs_roc(struct inode *ino,
+ struct nfs4_layoutreturn_args *args,
+ struct nfs4_layoutreturn_res *res,
+ const struct cred *cred, bool sync)
{
return false;
}
diff --git a/fs/nfsd/export.c b/fs/nfsd/export.c
index a9d794cfd10e..191842a5d7ee 100644
--- a/fs/nfsd/export.c
+++ b/fs/nfsd/export.c
@@ -40,15 +40,24 @@
#define EXPKEY_HASHMAX (1 << EXPKEY_HASHBITS)
#define EXPKEY_HASHMASK (EXPKEY_HASHMAX -1)
-static void expkey_put(struct kref *ref)
+static void expkey_put_work(struct work_struct *work)
{
- struct svc_expkey *key = container_of(ref, struct svc_expkey, h.ref);
+ struct svc_expkey *key =
+ container_of(to_rcu_work(work), struct svc_expkey, ek_rcu_work);
if (test_bit(CACHE_VALID, &key->h.flags) &&
!test_bit(CACHE_NEGATIVE, &key->h.flags))
path_put(&key->ek_path);
auth_domain_put(key->ek_client);
- kfree_rcu(key, ek_rcu);
+ kfree(key);
+}
+
+static void expkey_put(struct kref *ref)
+{
+ struct svc_expkey *key = container_of(ref, struct svc_expkey, h.ref);
+
+ INIT_RCU_WORK(&key->ek_rcu_work, expkey_put_work);
+ queue_rcu_work(system_wq, &key->ek_rcu_work);
}
static int expkey_upcall(struct cache_detail *cd, struct cache_head *h)
@@ -351,16 +360,26 @@ static void export_stats_destroy(struct export_stats *stats)
EXP_STATS_COUNTERS_NUM);
}
-static void svc_export_put(struct kref *ref)
+static void svc_export_put_work(struct work_struct *work)
{
- struct svc_export *exp = container_of(ref, struct svc_export, h.ref);
+ struct svc_export *exp =
+ container_of(to_rcu_work(work), struct svc_export, ex_rcu_work);
+
path_put(&exp->ex_path);
auth_domain_put(exp->ex_client);
nfsd4_fslocs_free(&exp->ex_fslocs);
export_stats_destroy(exp->ex_stats);
kfree(exp->ex_stats);
kfree(exp->ex_uuid);
- kfree_rcu(exp, ex_rcu);
+ kfree(exp);
+}
+
+static void svc_export_put(struct kref *ref)
+{
+ struct svc_export *exp = container_of(ref, struct svc_export, h.ref);
+
+ INIT_RCU_WORK(&exp->ex_rcu_work, svc_export_put_work);
+ queue_rcu_work(system_wq, &exp->ex_rcu_work);
}
static int svc_export_upcall(struct cache_detail *cd, struct cache_head *h)
diff --git a/fs/nfsd/export.h b/fs/nfsd/export.h
index f73e23bb24a1..fa545d8dcc36 100644
--- a/fs/nfsd/export.h
+++ b/fs/nfsd/export.h
@@ -75,7 +75,7 @@ struct svc_export {
u32 ex_layout_types;
struct nfsd4_deviceid_map *ex_devid_map;
struct cache_detail *cd;
- struct rcu_head ex_rcu;
+ struct rcu_work ex_rcu_work;
struct export_stats *ex_stats;
};
@@ -91,7 +91,7 @@ struct svc_expkey {
u32 ek_fsid[6];
struct path ek_path;
- struct rcu_head ek_rcu;
+ struct rcu_work ek_rcu_work;
};
#define EX_ISSYNC(exp) (!((exp)->ex_flags & NFSEXP_ASYNC))
diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c
index 46eb06e32f0c..ae8343bdc045 100644
--- a/fs/nfsd/filecache.c
+++ b/fs/nfsd/filecache.c
@@ -60,12 +60,6 @@ static DEFINE_PER_CPU(unsigned long, nfsd_file_releases);
static DEFINE_PER_CPU(unsigned long, nfsd_file_total_age);
static DEFINE_PER_CPU(unsigned long, nfsd_file_evictions);
-struct nfsd_fcache_disposal {
- struct work_struct work;
- spinlock_t lock;
- struct list_head freeme;
-};
-
static struct workqueue_struct *nfsd_filecache_wq __read_mostly;
static struct kmem_cache *nfsd_file_slab;
@@ -417,12 +411,11 @@ nfsd_file_dispose_list_delayed(struct list_head *dispose)
struct nfsd_file *nf = list_first_entry(dispose,
struct nfsd_file, nf_gc);
struct nfsd_net *nn = net_generic(nf->nf_net, nfsd_net_id);
- struct nfsd_fcache_disposal *l = nn->fcache_disposal;
- spin_lock(&l->lock);
- list_move_tail(&nf->nf_gc, &l->freeme);
- spin_unlock(&l->lock);
- queue_work(nfsd_filecache_wq, &l->work);
+ spin_lock(&nn->fcache_dispose_lock);
+ list_move_tail(&nf->nf_gc, &nn->fcache_dispose_list);
+ spin_unlock(&nn->fcache_dispose_lock);
+ queue_work(nfsd_filecache_wq, &nn->fcache_dispose_work);
}
}
@@ -651,12 +644,12 @@ static void
nfsd_file_delayed_close(struct work_struct *work)
{
LIST_HEAD(head);
- struct nfsd_fcache_disposal *l = container_of(work,
- struct nfsd_fcache_disposal, work);
+ struct nfsd_net *nn = container_of(work, struct nfsd_net,
+ fcache_dispose_work);
- spin_lock(&l->lock);
- list_splice_init(&l->freeme, &head);
- spin_unlock(&l->lock);
+ spin_lock(&nn->fcache_dispose_lock);
+ list_splice_init(&nn->fcache_dispose_list, &head);
+ spin_unlock(&nn->fcache_dispose_lock);
nfsd_file_dispose_list(&head);
}
@@ -825,44 +818,15 @@ __nfsd_file_cache_purge(struct net *net)
nfsd_file_dispose_list(&dispose);
}
-static struct nfsd_fcache_disposal *
-nfsd_alloc_fcache_disposal(void)
-{
- struct nfsd_fcache_disposal *l;
-
- l = kmalloc(sizeof(*l), GFP_KERNEL);
- if (!l)
- return NULL;
- INIT_WORK(&l->work, nfsd_file_delayed_close);
- spin_lock_init(&l->lock);
- INIT_LIST_HEAD(&l->freeme);
- return l;
-}
-
-static void
-nfsd_free_fcache_disposal(struct nfsd_fcache_disposal *l)
-{
- cancel_work_sync(&l->work);
- nfsd_file_dispose_list(&l->freeme);
- kfree(l);
-}
-
-static void
-nfsd_free_fcache_disposal_net(struct net *net)
-{
- struct nfsd_net *nn = net_generic(net, nfsd_net_id);
- struct nfsd_fcache_disposal *l = nn->fcache_disposal;
-
- nfsd_free_fcache_disposal(l);
-}
-
int
nfsd_file_cache_start_net(struct net *net)
{
struct nfsd_net *nn = net_generic(net, nfsd_net_id);
- nn->fcache_disposal = nfsd_alloc_fcache_disposal();
- return nn->fcache_disposal ? 0 : -ENOMEM;
+ spin_lock_init(&nn->fcache_dispose_lock);
+ INIT_LIST_HEAD(&nn->fcache_dispose_list);
+ INIT_WORK(&nn->fcache_dispose_work, nfsd_file_delayed_close);
+ return 0;
}
/**
@@ -881,8 +845,11 @@ nfsd_file_cache_purge(struct net *net)
void
nfsd_file_cache_shutdown_net(struct net *net)
{
+ struct nfsd_net *nn = net_generic(net, nfsd_net_id);
+
nfsd_file_cache_purge(net);
- nfsd_free_fcache_disposal_net(net);
+ cancel_work_sync(&nn->fcache_dispose_work);
+ nfsd_file_dispose_list(&nn->fcache_dispose_list);
}
void
diff --git a/fs/nfsd/lockd.c b/fs/nfsd/lockd.c
index b02886f38925..35b2a0541dc4 100644
--- a/fs/nfsd/lockd.c
+++ b/fs/nfsd/lockd.c
@@ -79,6 +79,7 @@ nlm_fclose(struct file *filp)
}
static const struct nlmsvc_binding nfsd_nlm_ops = {
+ .owner = THIS_MODULE,
.fopen = nlm_fopen, /* open file for locking */
.fclose = nlm_fclose, /* close file */
};
@@ -87,11 +88,12 @@ void
nfsd_lockd_init(void)
{
dprintk("nfsd: initializing lockd\n");
- nlmsvc_ops = &nfsd_nlm_ops;
+ rcu_assign_pointer(nlmsvc_ops, &nfsd_nlm_ops);
}
void
nfsd_lockd_shutdown(void)
{
- nlmsvc_ops = NULL;
+ RCU_INIT_POINTER(nlmsvc_ops, NULL);
+ synchronize_rcu();
}
diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h
index be0c8f1ce4e3..6291cd57f38e 100644
--- a/fs/nfsd/netns.h
+++ b/fs/nfsd/netns.h
@@ -67,6 +67,7 @@ struct nfsd_net {
bool grace_end_forced;
bool client_tracking_active;
time64_t boot_time;
+ time64_t boot_time_bt; /* same instant in CLOCK_BOOTTIME */
struct dentry *nfsd_client_dir;
@@ -206,7 +207,9 @@ struct nfsd_net {
/* utsname taken from the process that starts the server */
char nfsd_name[UNX_MAXNODENAME+1];
- struct nfsd_fcache_disposal *fcache_disposal;
+ spinlock_t fcache_dispose_lock;
+ struct list_head fcache_dispose_list;
+ struct work_struct fcache_dispose_work;
siphash_key_t siphash_key;
diff --git a/fs/nfsd/nfs2acl.c b/fs/nfsd/nfs2acl.c
index f8b91aacd817..cd6de7aca989 100644
--- a/fs/nfsd/nfs2acl.c
+++ b/fs/nfsd/nfs2acl.c
@@ -115,14 +115,19 @@ static __be32 nfsacld_proc_setacl(struct svc_rqst *rqstp)
inode_lock(inode);
- error = set_posix_acl(&init_user_ns, inode, ACL_TYPE_ACCESS,
- argp->acl_access);
- if (error)
- goto out_drop_lock;
- error = set_posix_acl(&init_user_ns, inode, ACL_TYPE_DEFAULT,
- argp->acl_default);
- if (error)
- goto out_drop_lock;
+ error = 0;
+ if (argp->mask & NFS_ACL) {
+ error = set_posix_acl(&init_user_ns, inode, ACL_TYPE_ACCESS,
+ argp->acl_access);
+ if (error)
+ goto out_drop_lock;
+ }
+ if (argp->mask & NFS_DFACL) {
+ error = set_posix_acl(&init_user_ns, inode, ACL_TYPE_DEFAULT,
+ argp->acl_default);
+ if (error)
+ goto out_drop_lock;
+ }
inode_unlock(inode);
diff --git a/fs/nfsd/nfs3acl.c b/fs/nfsd/nfs3acl.c
index d100a8a00e08..62435b137b9c 100644
--- a/fs/nfsd/nfs3acl.c
+++ b/fs/nfsd/nfs3acl.c
@@ -105,12 +105,17 @@ static __be32 nfsd3_proc_setacl(struct svc_rqst *rqstp)
inode_lock(inode);
- error = set_posix_acl(&init_user_ns, inode, ACL_TYPE_ACCESS,
- argp->acl_access);
- if (error)
- goto out_drop_lock;
- error = set_posix_acl(&init_user_ns, inode, ACL_TYPE_DEFAULT,
- argp->acl_default);
+ error = 0;
+ if (argp->mask & NFS_ACL) {
+ error = set_posix_acl(&init_user_ns, inode, ACL_TYPE_ACCESS,
+ argp->acl_access);
+ if (error)
+ goto out_drop_lock;
+ }
+ if (argp->mask & NFS_DFACL) {
+ error = set_posix_acl(&init_user_ns, inode, ACL_TYPE_DEFAULT,
+ argp->acl_default);
+ }
out_drop_lock:
inode_unlock(inode);
diff --git a/fs/nfsd/nfs3proc.c b/fs/nfsd/nfs3proc.c
index d01b29aba662..5b206463b11e 100644
--- a/fs/nfsd/nfs3proc.c
+++ b/fs/nfsd/nfs3proc.c
@@ -28,6 +28,25 @@ static int nfs3_ftypes[] = {
S_IFIFO, /* NF3FIFO */
};
+/*
+ * Reject a client-supplied atime or mtime whose nanoseconds field is out
+ * of range. Such a value is well-formed on the wire but is not a valid
+ * timespec64, and storing it verbatim can corrupt on-disk timestamps.
+ * tv_nsec is a long, so it is cast to unsigned long (the same width) to
+ * catch both an over-large value and one that became negative when an
+ * out-of-range u32 wire nseconds was assigned to a 32-bit long.
+ */
+static bool nfsd3_time_in_range(const struct iattr *iap)
+{
+ if ((iap->ia_valid & ATTR_ATIME_SET) &&
+ (unsigned long)iap->ia_atime.tv_nsec >= NSEC_PER_SEC)
+ return false;
+ if ((iap->ia_valid & ATTR_MTIME_SET) &&
+ (unsigned long)iap->ia_mtime.tv_nsec >= NSEC_PER_SEC)
+ return false;
+ return true;
+}
+
/*
* NULL call.
*/
@@ -76,6 +95,10 @@ nfsd3_proc_setattr(struct svc_rqst *rqstp)
SVCFH_fmt(&argp->fh));
fh_copy(&resp->fh, &argp->fh);
+ if (!nfsd3_time_in_range(&argp->attrs)) {
+ resp->status = nfserr_inval;
+ return rpc_success;
+ }
resp->status = nfsd_setattr(rqstp, &resp->fh, &attrs,
argp->check_guard, argp->guardtime);
return rpc_success;
@@ -245,6 +268,8 @@ nfsd3_create_file(struct svc_rqst *rqstp, struct svc_fh *fhp,
__be32 status;
int host_err;
+ if (!nfsd3_time_in_range(iap))
+ return nfserr_inval;
if (isdotent(argp->name, argp->len))
return nfserr_exist;
if (!(iap->ia_valid & ATTR_MODE))
@@ -389,6 +414,10 @@ nfsd3_proc_mkdir(struct svc_rqst *rqstp)
argp->attrs.ia_valid &= ~ATTR_SIZE;
fh_copy(&resp->dirfh, &argp->fh);
fh_init(&resp->fh, NFS3_FHSIZE);
+ if (!nfsd3_time_in_range(&argp->attrs)) {
+ resp->status = nfserr_inval;
+ return rpc_success;
+ }
resp->status = nfsd_create(rqstp, &resp->dirfh, argp->name, argp->len,
&attrs, S_IFDIR, 0, &resp->fh);
return rpc_success;
@@ -403,6 +432,10 @@ nfsd3_proc_symlink(struct svc_rqst *rqstp)
.na_iattr = &argp->attrs,
};
+ if (!nfsd3_time_in_range(&argp->attrs)) {
+ resp->status = nfserr_inval;
+ goto out;
+ }
if (argp->tlen == 0) {
resp->status = nfserr_inval;
goto out;
@@ -468,6 +501,11 @@ nfsd3_proc_mknod(struct svc_rqst *rqstp)
goto out;
}
+ if (!nfsd3_time_in_range(&argp->attrs)) {
+ resp->status = nfserr_inval;
+ goto out;
+ }
+
type = nfs3_ftypes[argp->ftype];
resp->status = nfsd_create(rqstp, &resp->dirfh, argp->name, argp->len,
&attrs, type, rdev, &resp->fh);
diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c
index 84bb200e24ad..03c8e71d1974 100644
--- a/fs/nfsd/nfs4layouts.c
+++ b/fs/nfsd/nfs4layouts.c
@@ -168,9 +168,12 @@ nfsd4_free_layout_stateid(struct nfs4_stid *stid)
list_del_init(&ls->ls_perfile);
spin_unlock(&fp->fi_lock);
- if (!nfsd4_layout_ops[ls->ls_layout_type]->disable_recalls)
- vfs_setlease(ls->ls_file->nf_file, F_UNLCK, NULL, (void **)&ls);
- nfsd_file_put(ls->ls_file);
+ if (ls->ls_file) {
+ if (!nfsd4_layout_ops[ls->ls_layout_type]->disable_recalls)
+ vfs_setlease(ls->ls_file->nf_file, F_UNLCK, NULL,
+ (void **)&ls);
+ nfsd_file_put(ls->ls_file);
+ }
if (ls->ls_recalled)
atomic_dec(&ls->ls_stid.sc_file->fi_lo_recalls);
@@ -235,11 +238,17 @@ nfsd4_alloc_layout_stateid(struct nfsd4_compound_state *cstate,
nfsd4_init_cb(&ls->ls_recall, clp, &nfsd4_cb_layout_ops,
NFSPROC4_CLNT_CB_LAYOUT);
- if (parent->sc_type == NFS4_DELEG_STID)
+ if (parent->sc_type == NFS4_DELEG_STID) {
+ spin_lock(&fp->fi_lock);
ls->ls_file = nfsd_file_get(fp->fi_deleg_file);
- else
+ spin_unlock(&fp->fi_lock);
+ } else {
ls->ls_file = find_any_file(fp);
- BUG_ON(!ls->ls_file);
+ }
+ if (!ls->ls_file) {
+ nfs4_put_stid(stp);
+ return NULL;
+ }
if (nfsd4_layout_setlease(ls)) {
nfs4_put_stid(stp);
diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c
index 20c3869a9ab9..00681c82bebd 100644
--- a/fs/nfsd/nfs4proc.c
+++ b/fs/nfsd/nfs4proc.c
@@ -761,7 +761,23 @@ nfsd4_create(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
if (status)
return status;
+ /* Sanitize cr_type to avoid returning ATTRNOTSUPP. */
+ switch (create->cr_type) {
+ case NF4LNK:
+ case NF4BLK:
+ case NF4CHR:
+ case NF4SOCK:
+ case NF4FIFO:
+ case NF4DIR:
+ break;
+ default:
+ status = nfserr_badtype;
+ goto out_aftermask;
+ }
+
status = nfsd4_acl_to_attr(create->cr_type, create->cr_acl, &attrs);
+ if (status != nfs_ok)
+ goto out_aftermask;
current->fs->umask = create->cr_umask;
switch (create->cr_type) {
case NF4LNK:
@@ -828,6 +844,7 @@ nfsd4_create(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
fh_put(&resfh);
out_umask:
current->fs->umask = 0;
+out_aftermask:
nfsd_attrs_free(&attrs);
return status;
}
@@ -1777,12 +1794,14 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
goto out;
}
status = nfsd4_setup_inter_ssc(rqstp, cstate, copy);
- if (status)
- return nfserr_offload_denied;
+ if (status) {
+ status = nfserr_offload_denied;
+ goto out;
+ }
} else {
status = nfsd4_setup_intra_ssc(rqstp, cstate, copy);
if (status)
- return status;
+ goto out;
}
memcpy(©->fh, &cstate->current_fh.fh_handle,
@@ -1801,11 +1820,14 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate,
async_copy->cp_src = kmalloc(sizeof(*async_copy->cp_src), GFP_KERNEL);
if (!async_copy->cp_src)
goto out_err;
- if (!nfs4_init_copy_state(nn, copy))
+ dup_copy_fields(copy, async_copy);
+
+ if (!nfs4_init_copy_state(nn, async_copy))
goto out_err;
- memcpy(&result->cb_stateid, ©->cp_stateid.cs_stid,
+ memcpy(&result->cb_stateid, &async_copy->cp_stateid.cs_stid,
+ sizeof(result->cb_stateid));
+ memcpy(&async_copy->cp_res.cb_stateid, &result->cb_stateid,
sizeof(result->cb_stateid));
- dup_copy_fields(copy, async_copy);
async_copy->copy_task = kthread_create(nfsd4_do_async_copy,
async_copy, "%s", "copy thread");
if (IS_ERR(async_copy->copy_task))
diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c
index 07c739f5f08d..ec37aca0949a 100644
--- a/fs/nfsd/nfs4state.c
+++ b/fs/nfsd/nfs4state.c
@@ -6029,7 +6029,7 @@ bool nfsd4_force_end_grace(struct nfsd_net *nn)
*/
static bool clients_still_reclaiming(struct nfsd_net *nn)
{
- time64_t double_grace_period_end = nn->boot_time +
+ time64_t double_grace_period_end = nn->boot_time_bt +
2 * nn->nfsd4_lease;
if (READ_ONCE(nn->grace_end_forced))
@@ -6270,6 +6270,7 @@ nfs4_laundromat(struct nfsd_net *nn)
.new_timeo = nn->nfsd4_lease
};
struct nfs4_cpntf_state *cps;
+ struct nfs4_client *clp;
copy_stateid_t *cps_t;
int i;
@@ -6295,6 +6296,18 @@ nfs4_laundromat(struct nfsd_net *nn)
dp = list_entry (pos, struct nfs4_delegation, dl_recall_lru);
if (!state_expired(<, dp->dl_time))
break;
+ clp = dp->dl_stid.sc_client;
+ spin_lock(&nn->client_lock);
+ if (is_client_expired(clp)) {
+ spin_unlock(&nn->client_lock);
+ continue;
+ }
+ /*
+ * Pin without reviving: get_client_locked() would
+ * flip a courtesy client back to NFSD4_ACTIVE.
+ */
+ atomic_inc(&clp->cl_rpc_users);
+ spin_unlock(&nn->client_lock);
WARN_ON(!unhash_delegation_locked(dp));
list_add(&dp->dl_recall_lru, &reaplist);
}
@@ -6302,8 +6315,18 @@ nfs4_laundromat(struct nfsd_net *nn)
while (!list_empty(&reaplist)) {
dp = list_first_entry(&reaplist, struct nfs4_delegation,
dl_recall_lru);
+ clp = dp->dl_stid.sc_client;
list_del_init(&dp->dl_recall_lru);
revoke_delegation(dp);
+ /*
+ * Unpin without renewing: put_client_renew() would
+ * renew the reaped client's lease.
+ */
+ if (atomic_dec_and_lock(&clp->cl_rpc_users, &nn->client_lock)) {
+ if (is_client_expired(clp))
+ wake_up_all(&expiry_wq);
+ spin_unlock(&nn->client_lock);
+ }
}
spin_lock(&nn->client_lock);
@@ -6313,11 +6336,18 @@ nfs4_laundromat(struct nfsd_net *nn)
if (!state_expired(<, oo->oo_time))
break;
list_del_init(&oo->oo_close_lru);
+ clp = oo->oo_owner.so_client;
+ if (is_client_expired(clp))
+ continue;
stp = oo->oo_last_closed_stid;
oo->oo_last_closed_stid = NULL;
+ atomic_inc(&clp->cl_rpc_users);
spin_unlock(&nn->client_lock);
nfs4_put_stid(&stp->st_stid);
spin_lock(&nn->client_lock);
+ if (atomic_dec_and_test(&clp->cl_rpc_users) &&
+ is_client_expired(clp))
+ wake_up_all(&expiry_wq);
}
spin_unlock(&nn->client_lock);
@@ -8277,6 +8307,7 @@ static int nfs4_state_create_net(struct net *net)
nn->conf_name_tree = RB_ROOT;
nn->unconf_name_tree = RB_ROOT;
nn->boot_time = ktime_get_real_seconds();
+ nn->boot_time_bt = ktime_get_boottime_seconds();
nn->grace_ended = false;
nn->grace_end_forced = false;
nn->client_tracking_active = false;
diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h
index 447b3483f94b..513245b3697c 100644
--- a/fs/nfsd/trace.h
+++ b/fs/nfsd/trace.h
@@ -201,7 +201,7 @@ TRACE_EVENT(nfsd_fh_verify,
TP_ARGS(rqstp, fhp, type, access),
TP_STRUCT__entry(
__field(unsigned int, netns_ino)
- __sockaddr(server, rqstp->rq_xprt->xpt_remotelen)
+ __sockaddr(server, rqstp->rq_xprt->xpt_locallen)
__sockaddr(client, rqstp->rq_xprt->xpt_remotelen)
__field(u32, xid)
__field(u32, fh_hash)
@@ -240,7 +240,7 @@ TRACE_EVENT_CONDITION(nfsd_fh_verify_err,
TP_CONDITION(error),
TP_STRUCT__entry(
__field(unsigned int, netns_ino)
- __sockaddr(server, rqstp->rq_xprt->xpt_remotelen)
+ __sockaddr(server, rqstp->rq_xprt->xpt_locallen)
__sockaddr(client, rqstp->rq_xprt->xpt_remotelen)
__field(u32, xid)
__field(u32, fh_hash)
diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c
index a69287613479..4ae834b9db87 100644
--- a/fs/nfsd/vfs.c
+++ b/fs/nfsd/vfs.c
@@ -130,8 +130,10 @@ nfsd_cross_mnt(struct svc_rqst *rqstp, struct dentry **dpp,
int err = 0;
err = follow_down(&path);
- if (err < 0)
+ if (err < 0) {
+ path_put(&path);
goto out;
+ }
if (path.mnt == exp->ex_path.mnt && path.dentry == dentry &&
nfsd_mountpoint(dentry, exp) == 2) {
/* This is only a mountpoint in some other namespace */
@@ -411,20 +413,22 @@ nfsd_sanitize_attrs(struct inode *inode, struct iattr *iap)
}
static __be32
-nfsd_get_write_access(struct svc_rqst *rqstp, struct svc_fh *fhp,
- struct iattr *iap)
+nfsd_may_truncate(struct svc_rqst *rqstp, struct svc_fh *fhp,
+ struct iattr *iap)
{
struct inode *inode = d_inode(fhp->fh_dentry);
- if (iap->ia_size < inode->i_size) {
- __be32 err;
+ if (iap->ia_size >= i_size_read(inode))
+ return nfs_ok;
- err = nfsd_permission(rqstp, fhp->fh_export, fhp->fh_dentry,
- NFSD_MAY_TRUNC | NFSD_MAY_OWNER_OVERRIDE);
- if (err)
- return err;
- }
- return nfserrno(get_write_access(inode));
+ return nfsd_permission(rqstp, fhp->fh_export, fhp->fh_dentry,
+ NFSD_MAY_TRUNC | NFSD_MAY_OWNER_OVERRIDE);
+}
+
+static __be32
+nfsd_get_write_access(struct svc_fh *fhp)
+{
+ return nfserrno(get_write_access(d_inode(fhp->fh_dentry)));
}
static int __nfsd_setattr(struct dentry *dentry, struct iattr *iap)
@@ -546,12 +550,20 @@ nfsd_setattr(struct svc_rqst *rqstp, struct svc_fh *fhp,
* setattr call.
*/
if (size_change) {
- err = nfsd_get_write_access(rqstp, fhp, iap);
+ err = nfsd_get_write_access(fhp);
if (err)
return err;
}
inode_lock(inode);
+ if (size_change) {
+ err = nfsd_may_truncate(rqstp, fhp, iap);
+ if (err) {
+ inode_unlock(inode);
+ put_write_access(inode);
+ return err;
+ }
+ }
fh_fill_pre_attrs(fhp);
for (retries = 1;;) {
struct iattr attrs;
diff --git a/fs/nilfs2/page.c b/fs/nilfs2/page.c
index bf090240f6ed..89cd1510bf71 100644
--- a/fs/nilfs2/page.c
+++ b/fs/nilfs2/page.c
@@ -241,6 +241,7 @@ static void nilfs_copy_page(struct page *dst, struct page *src, int copy_dirty)
int nilfs_copy_dirty_pages(struct address_space *dmap,
struct address_space *smap)
{
+ struct inode *smap_inode = smap->host;
struct pagevec pvec;
unsigned int i;
pgoff_t index = 0;
@@ -255,8 +256,19 @@ int nilfs_copy_dirty_pages(struct address_space *dmap,
struct page *page = pvec.pages[i], *dpage;
lock_page(page);
- if (unlikely(!PageDirty(page)))
- NILFS_PAGE_BUG(page, "inconsistent dirty state");
+ if (unlikely(!PageDirty(page))) {
+ if (WARN_ONCE(!sb_rdonly(smap_inode->i_sb),
+ "inconsistent dirty state\n"))
+ goto unlock_src_page;
+
+ /*
+ * If the filesystem has been forced to read-only
+ * due to metadata corruption.
+ */
+ unlock_page(page);
+ err = -EROFS;
+ break;
+ }
dpage = grab_cache_page(dmap, page->index);
if (unlikely(!dpage)) {
@@ -274,6 +286,7 @@ int nilfs_copy_dirty_pages(struct address_space *dmap,
unlock_page(dpage);
put_page(dpage);
+unlock_src_page:
unlock_page(page);
}
pagevec_release(&pvec);
diff --git a/fs/notify/mark.c b/fs/notify/mark.c
index b419a5ccf192..465724aafd49 100644
--- a/fs/notify/mark.c
+++ b/fs/notify/mark.c
@@ -153,7 +153,12 @@ static struct inode *fsnotify_update_iref(struct fsnotify_mark_connector *conn,
return inode;
}
-static void *__fsnotify_recalc_mask(struct fsnotify_mark_connector *conn)
+/*
+ * Calculate mask of events for a list of marks.
+ *
+ * Return true if any of the attached marks want to hold an inode reference.
+ */
+static bool __fsnotify_recalc_mask(struct fsnotify_mark_connector *conn)
{
u32 new_mask = 0;
bool want_iref = false;
@@ -173,6 +178,34 @@ static void *__fsnotify_recalc_mask(struct fsnotify_mark_connector *conn)
}
*fsnotify_conn_mask_p(conn) = new_mask;
+ return want_iref;
+}
+
+/*
+ * Calculate mask of events for a list of marks after attach/modify mark
+ * and get an inode reference for the connector if needed.
+ *
+ * A concurrent add of evictable mark and detach of non-evictable mark can
+ * lead to __fsnotify_recalc_mask() returning false want_iref, but in this
+ * case we defer clearing iref to fsnotify_recalc_mask_clear_iref() called
+ * from fsnotify_put_mark().
+ */
+static void fsnotify_recalc_mask_set_iref(struct fsnotify_mark_connector *conn)
+{
+ bool has_iref = conn->flags & FSNOTIFY_CONN_FLAG_HAS_IREF;
+ bool want_iref = __fsnotify_recalc_mask(conn) || has_iref;
+
+ (void) fsnotify_update_iref(conn, want_iref);
+}
+
+/*
+ * Calculate mask of events for a list of marks after detach mark
+ * and return the inode object if its reference is no longer needed.
+ */
+static void *fsnotify_recalc_mask_clear_iref(struct fsnotify_mark_connector *conn)
+{
+ bool want_iref = __fsnotify_recalc_mask(conn);
+
return fsnotify_update_iref(conn, want_iref);
}
@@ -209,7 +242,7 @@ void fsnotify_recalc_mask(struct fsnotify_mark_connector *conn)
spin_lock(&conn->lock);
update_children = !fsnotify_conn_watches_children(conn);
- __fsnotify_recalc_mask(conn);
+ fsnotify_recalc_mask_set_iref(conn);
update_children &= fsnotify_conn_watches_children(conn);
spin_unlock(&conn->lock);
/*
@@ -342,7 +375,7 @@ void fsnotify_put_mark(struct fsnotify_mark *mark)
objp = fsnotify_detach_connector_from_object(conn, &type);
free_conn = true;
} else {
- objp = __fsnotify_recalc_mask(conn);
+ objp = fsnotify_recalc_mask_clear_iref(conn);
type = conn->type;
}
WRITE_ONCE(mark->connector, NULL);
@@ -686,6 +719,11 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark,
goto out_err;
}
+ if (conn->flags & FSNOTIFY_CONN_FLAG_DESTROYING) {
+ err = -ENOENT;
+ goto out_err;
+ }
+
/* is mark the first mark? */
if (hlist_empty(&conn->list)) {
hlist_add_head_rcu(&mark->obj_list, &conn->list);
@@ -868,6 +906,12 @@ void fsnotify_destroy_marks(fsnotify_connp_t *connp)
conn = fsnotify_grab_connector(connp);
if (!conn)
return;
+ /*
+ * New marks could be inserted before our cursor and survive the final
+ * detachment of the connector from the object. Prevent additions to
+ * this connector before dropping its lock.
+ */
+ conn->flags |= FSNOTIFY_CONN_FLAG_DESTROYING;
/*
* We have to be careful since we can race with e.g.
* fsnotify_clear_marks_by_group() and once we drop the conn->lock, the
diff --git a/fs/ntfs/Kconfig b/fs/ntfs/Kconfig
deleted file mode 100644
index f93e69a61283..000000000000
--- a/fs/ntfs/Kconfig
+++ /dev/null
@@ -1,80 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0-only
-config NTFS_FS
- tristate "NTFS file system support"
- select NLS
- help
- NTFS is the file system of Microsoft Windows NT, 2000, XP and 2003.
-
- Saying Y or M here enables read support. There is partial, but
- safe, write support available. For write support you must also
- say Y to "NTFS write support" below.
-
- There are also a number of user-space tools available, called
- ntfsprogs. These include ntfsundelete and ntfsresize, that work
- without NTFS support enabled in the kernel.
-
- This is a rewrite from scratch of Linux NTFS support and replaced
- the old NTFS code starting with Linux 2.5.11. A backport to
- the Linux 2.4 kernel series is separately available as a patch
- from the project web site.
-
- For more information see <file:Documentation/filesystems/ntfs.rst>
- and <http://www.linux-ntfs.org/>.
-
- To compile this file system support as a module, choose M here: the
- module will be called ntfs.
-
- If you are not using Windows NT, 2000, XP or 2003 in addition to
- Linux on your computer it is safe to say N.
-
-config NTFS_DEBUG
- bool "NTFS debugging support"
- depends on NTFS_FS
- help
- If you are experiencing any problems with the NTFS file system, say
- Y here. This will result in additional consistency checks to be
- performed by the driver as well as additional debugging messages to
- be written to the system log. Note that debugging messages are
- disabled by default. To enable them, supply the option debug_msgs=1
- at the kernel command line when booting the kernel or as an option
- to insmod when loading the ntfs module. Once the driver is active,
- you can enable debugging messages by doing (as root):
- echo 1 > /proc/sys/fs/ntfs-debug
- Replacing the "1" with "0" would disable debug messages.
-
- If you leave debugging messages disabled, this results in little
- overhead, but enabling debug messages results in very significant
- slowdown of the system.
-
- When reporting bugs, please try to have available a full dump of
- debugging messages while the misbehaviour was occurring.
-
-config NTFS_RW
- bool "NTFS write support"
- depends on NTFS_FS
- depends on PAGE_SIZE_LESS_THAN_64KB
- help
- This enables the partial, but safe, write support in the NTFS driver.
-
- The only supported operation is overwriting existing files, without
- changing the file length. No file or directory creation, deletion or
- renaming is possible. Note only non-resident files can be written to
- so you may find that some very small files (<500 bytes or so) cannot
- be written to.
-
- While we cannot guarantee that it will not damage any data, we have
- so far not received a single report where the driver would have
- damaged someones data so we assume it is perfectly safe to use.
-
- Note: While write support is safe in this version (a rewrite from
- scratch of the NTFS support), it should be noted that the old NTFS
- write support, included in Linux 2.5.10 and before (since 1997),
- is not safe.
-
- This is currently useful with TopologiLinux. TopologiLinux is run
- on top of any DOS/Microsoft Windows system without partitioning your
- hard disk. Unlike other Linux distributions TopologiLinux does not
- need its own partition. For more information see
- <http://topologi-linux.sourceforge.net/>
-
- It is perfectly safe to say N here.
diff --git a/fs/ntfs/Makefile b/fs/ntfs/Makefile
deleted file mode 100644
index 3e736572ed00..000000000000
--- a/fs/ntfs/Makefile
+++ /dev/null
@@ -1,15 +0,0 @@
-# SPDX-License-Identifier: GPL-2.0
-# Rules for making the NTFS driver.
-
-obj-$(CONFIG_NTFS_FS) += ntfs.o
-
-ntfs-y := aops.o attrib.o collate.o compress.o debug.o dir.o file.o \
- index.o inode.o mft.o mst.o namei.o runlist.o super.o sysctl.o \
- unistr.o upcase.o
-
-ntfs-$(CONFIG_NTFS_RW) += bitmap.o lcnalloc.o logfile.o quota.o usnjrnl.o
-
-ccflags-y := -DNTFS_VERSION=\"2.1.32\"
-ccflags-$(CONFIG_NTFS_DEBUG) += -DDEBUG
-ccflags-$(CONFIG_NTFS_RW) += -DNTFS_RW
-
diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c
deleted file mode 100644
index 9364d35b4a10..000000000000
--- a/fs/ntfs/aops.c
+++ /dev/null
@@ -1,1761 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/**
- * aops.c - NTFS kernel address space operations and page cache handling.
- *
- * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc.
- * Copyright (c) 2002 Richard Russon
- */
-
-#include <linux/errno.h>
-#include <linux/fs.h>
-#include <linux/gfp.h>
-#include <linux/mm.h>
-#include <linux/pagemap.h>
-#include <linux/swap.h>
-#include <linux/buffer_head.h>
-#include <linux/writeback.h>
-#include <linux/bit_spinlock.h>
-#include <linux/bio.h>
-
-#include "aops.h"
-#include "attrib.h"
-#include "debug.h"
-#include "inode.h"
-#include "mft.h"
-#include "runlist.h"
-#include "types.h"
-#include "ntfs.h"
-
-/**
- * ntfs_end_buffer_async_read - async io completion for reading attributes
- * @bh: buffer head on which io is completed
- * @uptodate: whether @bh is now uptodate or not
- *
- * Asynchronous I/O completion handler for reading pages belonging to the
- * attribute address space of an inode. The inodes can either be files or
- * directories or they can be fake inodes describing some attribute.
- *
- * If NInoMstProtected(), perform the post read mst fixups when all IO on the
- * page has been completed and mark the page uptodate or set the error bit on
- * the page. To determine the size of the records that need fixing up, we
- * cheat a little bit by setting the index_block_size in ntfs_inode to the ntfs
- * record size, and index_block_size_bits, to the log(base 2) of the ntfs
- * record size.
- */
-static void ntfs_end_buffer_async_read(struct buffer_head *bh, int uptodate)
-{
- unsigned long flags;
- struct buffer_head *first, *tmp;
- struct page *page;
- struct inode *vi;
- ntfs_inode *ni;
- int page_uptodate = 1;
-
- page = bh->b_page;
- vi = page->mapping->host;
- ni = NTFS_I(vi);
-
- if (likely(uptodate)) {
- loff_t i_size;
- s64 file_ofs, init_size;
-
- set_buffer_uptodate(bh);
-
- file_ofs = ((s64)page->index << PAGE_SHIFT) +
- bh_offset(bh);
- read_lock_irqsave(&ni->size_lock, flags);
- init_size = ni->initialized_size;
- i_size = i_size_read(vi);
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (unlikely(init_size > i_size)) {
- /* Race with shrinking truncate. */
- init_size = i_size;
- }
- /* Check for the current buffer head overflowing. */
- if (unlikely(file_ofs + bh->b_size > init_size)) {
- int ofs;
- void *kaddr;
-
- ofs = 0;
- if (file_ofs < init_size)
- ofs = init_size - file_ofs;
- kaddr = kmap_atomic(page);
- memset(kaddr + bh_offset(bh) + ofs, 0,
- bh->b_size - ofs);
- flush_dcache_page(page);
- kunmap_atomic(kaddr);
- }
- } else {
- clear_buffer_uptodate(bh);
- SetPageError(page);
- ntfs_error(ni->vol->sb, "Buffer I/O error, logical block "
- "0x%llx.", (unsigned long long)bh->b_blocknr);
- }
- first = page_buffers(page);
- spin_lock_irqsave(&first->b_uptodate_lock, flags);
- clear_buffer_async_read(bh);
- unlock_buffer(bh);
- tmp = bh;
- do {
- if (!buffer_uptodate(tmp))
- page_uptodate = 0;
- if (buffer_async_read(tmp)) {
- if (likely(buffer_locked(tmp)))
- goto still_busy;
- /* Async buffers must be locked. */
- BUG();
- }
- tmp = tmp->b_this_page;
- } while (tmp != bh);
- spin_unlock_irqrestore(&first->b_uptodate_lock, flags);
- /*
- * If none of the buffers had errors then we can set the page uptodate,
- * but we first have to perform the post read mst fixups, if the
- * attribute is mst protected, i.e. if NInoMstProteced(ni) is true.
- * Note we ignore fixup errors as those are detected when
- * map_mft_record() is called which gives us per record granularity
- * rather than per page granularity.
- */
- if (!NInoMstProtected(ni)) {
- if (likely(page_uptodate && !PageError(page)))
- SetPageUptodate(page);
- } else {
- u8 *kaddr;
- unsigned int i, recs;
- u32 rec_size;
-
- rec_size = ni->itype.index.block_size;
- recs = PAGE_SIZE / rec_size;
- /* Should have been verified before we got here... */
- BUG_ON(!recs);
- kaddr = kmap_atomic(page);
- for (i = 0; i < recs; i++)
- post_read_mst_fixup((NTFS_RECORD*)(kaddr +
- i * rec_size), rec_size);
- kunmap_atomic(kaddr);
- flush_dcache_page(page);
- if (likely(page_uptodate && !PageError(page)))
- SetPageUptodate(page);
- }
- unlock_page(page);
- return;
-still_busy:
- spin_unlock_irqrestore(&first->b_uptodate_lock, flags);
- return;
-}
-
-/**
- * ntfs_read_block - fill a @page of an address space with data
- * @page: page cache page to fill with data
- *
- * Fill the page @page of the address space belonging to the @page->host inode.
- * We read each buffer asynchronously and when all buffers are read in, our io
- * completion handler ntfs_end_buffer_read_async(), if required, automatically
- * applies the mst fixups to the page before finally marking it uptodate and
- * unlocking it.
- *
- * We only enforce allocated_size limit because i_size is checked for in
- * generic_file_read().
- *
- * Return 0 on success and -errno on error.
- *
- * Contains an adapted version of fs/buffer.c::block_read_full_folio().
- */
-static int ntfs_read_block(struct page *page)
-{
- loff_t i_size;
- VCN vcn;
- LCN lcn;
- s64 init_size;
- struct inode *vi;
- ntfs_inode *ni;
- ntfs_volume *vol;
- runlist_element *rl;
- struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE];
- sector_t iblock, lblock, zblock;
- unsigned long flags;
- unsigned int blocksize, vcn_ofs;
- int i, nr;
- unsigned char blocksize_bits;
-
- vi = page->mapping->host;
- ni = NTFS_I(vi);
- vol = ni->vol;
-
- /* $MFT/$DATA must have its complete runlist in memory at all times. */
- BUG_ON(!ni->runlist.rl && !ni->mft_no && !NInoAttr(ni));
-
- blocksize = vol->sb->s_blocksize;
- blocksize_bits = vol->sb->s_blocksize_bits;
-
- if (!page_has_buffers(page)) {
- create_empty_buffers(page, blocksize, 0);
- if (unlikely(!page_has_buffers(page))) {
- unlock_page(page);
- return -ENOMEM;
- }
- }
- bh = head = page_buffers(page);
- BUG_ON(!bh);
-
- /*
- * We may be racing with truncate. To avoid some of the problems we
- * now take a snapshot of the various sizes and use those for the whole
- * of the function. In case of an extending truncate it just means we
- * may leave some buffers unmapped which are now allocated. This is
- * not a problem since these buffers will just get mapped when a write
- * occurs. In case of a shrinking truncate, we will detect this later
- * on due to the runlist being incomplete and if the page is being
- * fully truncated, truncate will throw it away as soon as we unlock
- * it so no need to worry what we do with it.
- */
- iblock = (s64)page->index << (PAGE_SHIFT - blocksize_bits);
- read_lock_irqsave(&ni->size_lock, flags);
- lblock = (ni->allocated_size + blocksize - 1) >> blocksize_bits;
- init_size = ni->initialized_size;
- i_size = i_size_read(vi);
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (unlikely(init_size > i_size)) {
- /* Race with shrinking truncate. */
- init_size = i_size;
- }
- zblock = (init_size + blocksize - 1) >> blocksize_bits;
-
- /* Loop through all the buffers in the page. */
- rl = NULL;
- nr = i = 0;
- do {
- int err = 0;
-
- if (unlikely(buffer_uptodate(bh)))
- continue;
- if (unlikely(buffer_mapped(bh))) {
- arr[nr++] = bh;
- continue;
- }
- bh->b_bdev = vol->sb->s_bdev;
- /* Is the block within the allowed limits? */
- if (iblock < lblock) {
- bool is_retry = false;
-
- /* Convert iblock into corresponding vcn and offset. */
- vcn = (VCN)iblock << blocksize_bits >>
- vol->cluster_size_bits;
- vcn_ofs = ((VCN)iblock << blocksize_bits) &
- vol->cluster_size_mask;
- if (!rl) {
-lock_retry_remap:
- down_read(&ni->runlist.lock);
- rl = ni->runlist.rl;
- }
- if (likely(rl != NULL)) {
- /* Seek to element containing target vcn. */
- while (rl->length && rl[1].vcn <= vcn)
- rl++;
- lcn = ntfs_rl_vcn_to_lcn(rl, vcn);
- } else
- lcn = LCN_RL_NOT_MAPPED;
- /* Successful remap. */
- if (lcn >= 0) {
- /* Setup buffer head to correct block. */
- bh->b_blocknr = ((lcn << vol->cluster_size_bits)
- + vcn_ofs) >> blocksize_bits;
- set_buffer_mapped(bh);
- /* Only read initialized data blocks. */
- if (iblock < zblock) {
- arr[nr++] = bh;
- continue;
- }
- /* Fully non-initialized data block, zero it. */
- goto handle_zblock;
- }
- /* It is a hole, need to zero it. */
- if (lcn == LCN_HOLE)
- goto handle_hole;
- /* If first try and runlist unmapped, map and retry. */
- if (!is_retry && lcn == LCN_RL_NOT_MAPPED) {
- is_retry = true;
- /*
- * Attempt to map runlist, dropping lock for
- * the duration.
- */
- up_read(&ni->runlist.lock);
- err = ntfs_map_runlist(ni, vcn);
- if (likely(!err))
- goto lock_retry_remap;
- rl = NULL;
- } else if (!rl)
- up_read(&ni->runlist.lock);
- /*
- * If buffer is outside the runlist, treat it as a
- * hole. This can happen due to concurrent truncate
- * for example.
- */
- if (err == -ENOENT || lcn == LCN_ENOENT) {
- err = 0;
- goto handle_hole;
- }
- /* Hard error, zero out region. */
- if (!err)
- err = -EIO;
- bh->b_blocknr = -1;
- SetPageError(page);
- ntfs_error(vol->sb, "Failed to read from inode 0x%lx, "
- "attribute type 0x%x, vcn 0x%llx, "
- "offset 0x%x because its location on "
- "disk could not be determined%s "
- "(error code %i).", ni->mft_no,
- ni->type, (unsigned long long)vcn,
- vcn_ofs, is_retry ? " even after "
- "retrying" : "", err);
- }
- /*
- * Either iblock was outside lblock limits or
- * ntfs_rl_vcn_to_lcn() returned error. Just zero that portion
- * of the page and set the buffer uptodate.
- */
-handle_hole:
- bh->b_blocknr = -1UL;
- clear_buffer_mapped(bh);
-handle_zblock:
- zero_user(page, i * blocksize, blocksize);
- if (likely(!err))
- set_buffer_uptodate(bh);
- } while (i++, iblock++, (bh = bh->b_this_page) != head);
-
- /* Release the lock if we took it. */
- if (rl)
- up_read(&ni->runlist.lock);
-
- /* Check we have at least one buffer ready for i/o. */
- if (nr) {
- struct buffer_head *tbh;
-
- /* Lock the buffers. */
- for (i = 0; i < nr; i++) {
- tbh = arr[i];
- lock_buffer(tbh);
- tbh->b_end_io = ntfs_end_buffer_async_read;
- set_buffer_async_read(tbh);
- }
- /* Finally, start i/o on the buffers. */
- for (i = 0; i < nr; i++) {
- tbh = arr[i];
- if (likely(!buffer_uptodate(tbh)))
- submit_bh(REQ_OP_READ, tbh);
- else
- ntfs_end_buffer_async_read(tbh, 1);
- }
- return 0;
- }
- /* No i/o was scheduled on any of the buffers. */
- if (likely(!PageError(page)))
- SetPageUptodate(page);
- else /* Signal synchronous i/o error. */
- nr = -EIO;
- unlock_page(page);
- return nr;
-}
-
-/**
- * ntfs_read_folio - fill a @folio of a @file with data from the device
- * @file: open file to which the folio @folio belongs or NULL
- * @folio: page cache folio to fill with data
- *
- * For non-resident attributes, ntfs_read_folio() fills the @folio of the open
- * file @file by calling the ntfs version of the generic block_read_full_folio()
- * function, ntfs_read_block(), which in turn creates and reads in the buffers
- * associated with the folio asynchronously.
- *
- * For resident attributes, OTOH, ntfs_read_folio() fills @folio by copying the
- * data from the mft record (which at this stage is most likely in memory) and
- * fills the remainder with zeroes. Thus, in this case, I/O is synchronous, as
- * even if the mft record is not cached at this point in time, we need to wait
- * for it to be read in before we can do the copy.
- *
- * Return 0 on success and -errno on error.
- */
-static int ntfs_read_folio(struct file *file, struct folio *folio)
-{
- struct page *page = &folio->page;
- loff_t i_size;
- struct inode *vi;
- ntfs_inode *ni, *base_ni;
- u8 *addr;
- ntfs_attr_search_ctx *ctx;
- MFT_RECORD *mrec;
- unsigned long flags;
- u32 attr_len;
- int err = 0;
-
-retry_readpage:
- BUG_ON(!PageLocked(page));
- vi = page->mapping->host;
- i_size = i_size_read(vi);
- /* Is the page fully outside i_size? (truncate in progress) */
- if (unlikely(page->index >= (i_size + PAGE_SIZE - 1) >>
- PAGE_SHIFT)) {
- zero_user(page, 0, PAGE_SIZE);
- ntfs_debug("Read outside i_size - truncated?");
- goto done;
- }
- /*
- * This can potentially happen because we clear PageUptodate() during
- * ntfs_writepage() of MstProtected() attributes.
- */
- if (PageUptodate(page)) {
- unlock_page(page);
- return 0;
- }
- ni = NTFS_I(vi);
- /*
- * Only $DATA attributes can be encrypted and only unnamed $DATA
- * attributes can be compressed. Index root can have the flags set but
- * this means to create compressed/encrypted files, not that the
- * attribute is compressed/encrypted. Note we need to check for
- * AT_INDEX_ALLOCATION since this is the type of both directory and
- * index inodes.
- */
- if (ni->type != AT_INDEX_ALLOCATION) {
- /* If attribute is encrypted, deny access, just like NT4. */
- if (NInoEncrypted(ni)) {
- BUG_ON(ni->type != AT_DATA);
- err = -EACCES;
- goto err_out;
- }
- /* Compressed data streams are handled in compress.c. */
- if (NInoNonResident(ni) && NInoCompressed(ni)) {
- BUG_ON(ni->type != AT_DATA);
- BUG_ON(ni->name_len);
- return ntfs_read_compressed_block(page);
- }
- }
- /* NInoNonResident() == NInoIndexAllocPresent() */
- if (NInoNonResident(ni)) {
- /* Normal, non-resident data stream. */
- return ntfs_read_block(page);
- }
- /*
- * Attribute is resident, implying it is not compressed or encrypted.
- * This also means the attribute is smaller than an mft record and
- * hence smaller than a page, so can simply zero out any pages with
- * index above 0. Note the attribute can actually be marked compressed
- * but if it is resident the actual data is not compressed so we are
- * ok to ignore the compressed flag here.
- */
- if (unlikely(page->index > 0)) {
- zero_user(page, 0, PAGE_SIZE);
- goto done;
- }
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- /* Map, pin, and lock the mft record. */
- mrec = map_mft_record(base_ni);
- if (IS_ERR(mrec)) {
- err = PTR_ERR(mrec);
- goto err_out;
- }
- /*
- * If a parallel write made the attribute non-resident, drop the mft
- * record and retry the read_folio.
- */
- if (unlikely(NInoNonResident(ni))) {
- unmap_mft_record(base_ni);
- goto retry_readpage;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, mrec);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto unm_err_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err))
- goto put_unm_err_out;
- attr_len = le32_to_cpu(ctx->attr->data.resident.value_length);
- read_lock_irqsave(&ni->size_lock, flags);
- if (unlikely(attr_len > ni->initialized_size))
- attr_len = ni->initialized_size;
- i_size = i_size_read(vi);
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (unlikely(attr_len > i_size)) {
- /* Race with shrinking truncate. */
- attr_len = i_size;
- }
- addr = kmap_atomic(page);
- /* Copy the data to the page. */
- memcpy(addr, (u8*)ctx->attr +
- le16_to_cpu(ctx->attr->data.resident.value_offset),
- attr_len);
- /* Zero the remainder of the page. */
- memset(addr + attr_len, 0, PAGE_SIZE - attr_len);
- flush_dcache_page(page);
- kunmap_atomic(addr);
-put_unm_err_out:
- ntfs_attr_put_search_ctx(ctx);
-unm_err_out:
- unmap_mft_record(base_ni);
-done:
- SetPageUptodate(page);
-err_out:
- unlock_page(page);
- return err;
-}
-
-#ifdef NTFS_RW
-
-/**
- * ntfs_write_block - write a @page to the backing store
- * @page: page cache page to write out
- * @wbc: writeback control structure
- *
- * This function is for writing pages belonging to non-resident, non-mst
- * protected attributes to their backing store.
- *
- * For a page with buffers, map and write the dirty buffers asynchronously
- * under page writeback. For a page without buffers, create buffers for the
- * page, then proceed as above.
- *
- * If a page doesn't have buffers the page dirty state is definitive. If a page
- * does have buffers, the page dirty state is just a hint, and the buffer dirty
- * state is definitive. (A hint which has rules: dirty buffers against a clean
- * page is illegal. Other combinations are legal and need to be handled. In
- * particular a dirty page containing clean buffers for example.)
- *
- * Return 0 on success and -errno on error.
- *
- * Based on ntfs_read_block() and __block_write_full_page().
- */
-static int ntfs_write_block(struct page *page, struct writeback_control *wbc)
-{
- VCN vcn;
- LCN lcn;
- s64 initialized_size;
- loff_t i_size;
- sector_t block, dblock, iblock;
- struct inode *vi;
- ntfs_inode *ni;
- ntfs_volume *vol;
- runlist_element *rl;
- struct buffer_head *bh, *head;
- unsigned long flags;
- unsigned int blocksize, vcn_ofs;
- int err;
- bool need_end_writeback;
- unsigned char blocksize_bits;
-
- vi = page->mapping->host;
- ni = NTFS_I(vi);
- vol = ni->vol;
-
- ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, page index "
- "0x%lx.", ni->mft_no, ni->type, page->index);
-
- BUG_ON(!NInoNonResident(ni));
- BUG_ON(NInoMstProtected(ni));
- blocksize = vol->sb->s_blocksize;
- blocksize_bits = vol->sb->s_blocksize_bits;
- if (!page_has_buffers(page)) {
- BUG_ON(!PageUptodate(page));
- create_empty_buffers(page, blocksize,
- (1 << BH_Uptodate) | (1 << BH_Dirty));
- if (unlikely(!page_has_buffers(page))) {
- ntfs_warning(vol->sb, "Error allocating page "
- "buffers. Redirtying page so we try "
- "again later.");
- /*
- * Put the page back on mapping->dirty_pages, but leave
- * its buffers' dirty state as-is.
- */
- redirty_page_for_writepage(wbc, page);
- unlock_page(page);
- return 0;
- }
- }
- bh = head = page_buffers(page);
- BUG_ON(!bh);
-
- /* NOTE: Different naming scheme to ntfs_read_block()! */
-
- /* The first block in the page. */
- block = (s64)page->index << (PAGE_SHIFT - blocksize_bits);
-
- read_lock_irqsave(&ni->size_lock, flags);
- i_size = i_size_read(vi);
- initialized_size = ni->initialized_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
-
- /* The first out of bounds block for the data size. */
- dblock = (i_size + blocksize - 1) >> blocksize_bits;
-
- /* The last (fully or partially) initialized block. */
- iblock = initialized_size >> blocksize_bits;
-
- /*
- * Be very careful. We have no exclusion from block_dirty_folio
- * here, and the (potentially unmapped) buffers may become dirty at
- * any time. If a buffer becomes dirty here after we've inspected it
- * then we just miss that fact, and the page stays dirty.
- *
- * Buffers outside i_size may be dirtied by block_dirty_folio;
- * handle that here by just cleaning them.
- */
-
- /*
- * Loop through all the buffers in the page, mapping all the dirty
- * buffers to disk addresses and handling any aliases from the
- * underlying block device's mapping.
- */
- rl = NULL;
- err = 0;
- do {
- bool is_retry = false;
-
- if (unlikely(block >= dblock)) {
- /*
- * Mapped buffers outside i_size will occur, because
- * this page can be outside i_size when there is a
- * truncate in progress. The contents of such buffers
- * were zeroed by ntfs_writepage().
- *
- * FIXME: What about the small race window where
- * ntfs_writepage() has not done any clearing because
- * the page was within i_size but before we get here,
- * vmtruncate() modifies i_size?
- */
- clear_buffer_dirty(bh);
- set_buffer_uptodate(bh);
- continue;
- }
-
- /* Clean buffers are not written out, so no need to map them. */
- if (!buffer_dirty(bh))
- continue;
-
- /* Make sure we have enough initialized size. */
- if (unlikely((block >= iblock) &&
- (initialized_size < i_size))) {
- /*
- * If this page is fully outside initialized
- * size, zero out all pages between the current
- * initialized size and the current page. Just
- * use ntfs_read_folio() to do the zeroing
- * transparently.
- */
- if (block > iblock) {
- // TODO:
- // For each page do:
- // - read_cache_page()
- // Again for each page do:
- // - wait_on_page_locked()
- // - Check (PageUptodate(page) &&
- // !PageError(page))
- // Update initialized size in the attribute and
- // in the inode.
- // Again, for each page do:
- // block_dirty_folio();
- // put_page()
- // We don't need to wait on the writes.
- // Update iblock.
- }
- /*
- * The current page straddles initialized size. Zero
- * all non-uptodate buffers and set them uptodate (and
- * dirty?). Note, there aren't any non-uptodate buffers
- * if the page is uptodate.
- * FIXME: For an uptodate page, the buffers may need to
- * be written out because they were not initialized on
- * disk before.
- */
- if (!PageUptodate(page)) {
- // TODO:
- // Zero any non-uptodate buffers up to i_size.
- // Set them uptodate and dirty.
- }
- // TODO:
- // Update initialized size in the attribute and in the
- // inode (up to i_size).
- // Update iblock.
- // FIXME: This is inefficient. Try to batch the two
- // size changes to happen in one go.
- ntfs_error(vol->sb, "Writing beyond initialized size "
- "is not supported yet. Sorry.");
- err = -EOPNOTSUPP;
- break;
- // Do NOT set_buffer_new() BUT DO clear buffer range
- // outside write request range.
- // set_buffer_uptodate() on complete buffers as well as
- // set_buffer_dirty().
- }
-
- /* No need to map buffers that are already mapped. */
- if (buffer_mapped(bh))
- continue;
-
- /* Unmapped, dirty buffer. Need to map it. */
- bh->b_bdev = vol->sb->s_bdev;
-
- /* Convert block into corresponding vcn and offset. */
- vcn = (VCN)block << blocksize_bits;
- vcn_ofs = vcn & vol->cluster_size_mask;
- vcn >>= vol->cluster_size_bits;
- if (!rl) {
-lock_retry_remap:
- down_read(&ni->runlist.lock);
- rl = ni->runlist.rl;
- }
- if (likely(rl != NULL)) {
- /* Seek to element containing target vcn. */
- while (rl->length && rl[1].vcn <= vcn)
- rl++;
- lcn = ntfs_rl_vcn_to_lcn(rl, vcn);
- } else
- lcn = LCN_RL_NOT_MAPPED;
- /* Successful remap. */
- if (lcn >= 0) {
- /* Setup buffer head to point to correct block. */
- bh->b_blocknr = ((lcn << vol->cluster_size_bits) +
- vcn_ofs) >> blocksize_bits;
- set_buffer_mapped(bh);
- continue;
- }
- /* It is a hole, need to instantiate it. */
- if (lcn == LCN_HOLE) {
- u8 *kaddr;
- unsigned long *bpos, *bend;
-
- /* Check if the buffer is zero. */
- kaddr = kmap_atomic(page);
- bpos = (unsigned long *)(kaddr + bh_offset(bh));
- bend = (unsigned long *)((u8*)bpos + blocksize);
- do {
- if (unlikely(*bpos))
- break;
- } while (likely(++bpos < bend));
- kunmap_atomic(kaddr);
- if (bpos == bend) {
- /*
- * Buffer is zero and sparse, no need to write
- * it.
- */
- bh->b_blocknr = -1;
- clear_buffer_dirty(bh);
- continue;
- }
- // TODO: Instantiate the hole.
- // clear_buffer_new(bh);
- // clean_bdev_bh_alias(bh);
- ntfs_error(vol->sb, "Writing into sparse regions is "
- "not supported yet. Sorry.");
- err = -EOPNOTSUPP;
- break;
- }
- /* If first try and runlist unmapped, map and retry. */
- if (!is_retry && lcn == LCN_RL_NOT_MAPPED) {
- is_retry = true;
- /*
- * Attempt to map runlist, dropping lock for
- * the duration.
- */
- up_read(&ni->runlist.lock);
- err = ntfs_map_runlist(ni, vcn);
- if (likely(!err))
- goto lock_retry_remap;
- rl = NULL;
- } else if (!rl)
- up_read(&ni->runlist.lock);
- /*
- * If buffer is outside the runlist, truncate has cut it out
- * of the runlist. Just clean and clear the buffer and set it
- * uptodate so it can get discarded by the VM.
- */
- if (err == -ENOENT || lcn == LCN_ENOENT) {
- bh->b_blocknr = -1;
- clear_buffer_dirty(bh);
- zero_user(page, bh_offset(bh), blocksize);
- set_buffer_uptodate(bh);
- err = 0;
- continue;
- }
- /* Failed to map the buffer, even after retrying. */
- if (!err)
- err = -EIO;
- bh->b_blocknr = -1;
- ntfs_error(vol->sb, "Failed to write to inode 0x%lx, "
- "attribute type 0x%x, vcn 0x%llx, offset 0x%x "
- "because its location on disk could not be "
- "determined%s (error code %i).", ni->mft_no,
- ni->type, (unsigned long long)vcn,
- vcn_ofs, is_retry ? " even after "
- "retrying" : "", err);
- break;
- } while (block++, (bh = bh->b_this_page) != head);
-
- /* Release the lock if we took it. */
- if (rl)
- up_read(&ni->runlist.lock);
-
- /* For the error case, need to reset bh to the beginning. */
- bh = head;
-
- /* Just an optimization, so ->read_folio() is not called later. */
- if (unlikely(!PageUptodate(page))) {
- int uptodate = 1;
- do {
- if (!buffer_uptodate(bh)) {
- uptodate = 0;
- bh = head;
- break;
- }
- } while ((bh = bh->b_this_page) != head);
- if (uptodate)
- SetPageUptodate(page);
- }
-
- /* Setup all mapped, dirty buffers for async write i/o. */
- do {
- if (buffer_mapped(bh) && buffer_dirty(bh)) {
- lock_buffer(bh);
- if (test_clear_buffer_dirty(bh)) {
- BUG_ON(!buffer_uptodate(bh));
- mark_buffer_async_write(bh);
- } else
- unlock_buffer(bh);
- } else if (unlikely(err)) {
- /*
- * For the error case. The buffer may have been set
- * dirty during attachment to a dirty page.
- */
- if (err != -ENOMEM)
- clear_buffer_dirty(bh);
- }
- } while ((bh = bh->b_this_page) != head);
-
- if (unlikely(err)) {
- // TODO: Remove the -EOPNOTSUPP check later on...
- if (unlikely(err == -EOPNOTSUPP))
- err = 0;
- else if (err == -ENOMEM) {
- ntfs_warning(vol->sb, "Error allocating memory. "
- "Redirtying page so we try again "
- "later.");
- /*
- * Put the page back on mapping->dirty_pages, but
- * leave its buffer's dirty state as-is.
- */
- redirty_page_for_writepage(wbc, page);
- err = 0;
- } else
- SetPageError(page);
- }
-
- BUG_ON(PageWriteback(page));
- set_page_writeback(page); /* Keeps try_to_free_buffers() away. */
-
- /* Submit the prepared buffers for i/o. */
- need_end_writeback = true;
- do {
- struct buffer_head *next = bh->b_this_page;
- if (buffer_async_write(bh)) {
- submit_bh(REQ_OP_WRITE, bh);
- need_end_writeback = false;
- }
- bh = next;
- } while (bh != head);
- unlock_page(page);
-
- /* If no i/o was started, need to end_page_writeback(). */
- if (unlikely(need_end_writeback))
- end_page_writeback(page);
-
- ntfs_debug("Done.");
- return err;
-}
-
-/**
- * ntfs_write_mst_block - write a @page to the backing store
- * @page: page cache page to write out
- * @wbc: writeback control structure
- *
- * This function is for writing pages belonging to non-resident, mst protected
- * attributes to their backing store. The only supported attributes are index
- * allocation and $MFT/$DATA. Both directory inodes and index inodes are
- * supported for the index allocation case.
- *
- * The page must remain locked for the duration of the write because we apply
- * the mst fixups, write, and then undo the fixups, so if we were to unlock the
- * page before undoing the fixups, any other user of the page will see the
- * page contents as corrupt.
- *
- * We clear the page uptodate flag for the duration of the function to ensure
- * exclusion for the $MFT/$DATA case against someone mapping an mft record we
- * are about to apply the mst fixups to.
- *
- * Return 0 on success and -errno on error.
- *
- * Based on ntfs_write_block(), ntfs_mft_writepage(), and
- * write_mft_record_nolock().
- */
-static int ntfs_write_mst_block(struct page *page,
- struct writeback_control *wbc)
-{
- sector_t block, dblock, rec_block;
- struct inode *vi = page->mapping->host;
- ntfs_inode *ni = NTFS_I(vi);
- ntfs_volume *vol = ni->vol;
- u8 *kaddr;
- unsigned int rec_size = ni->itype.index.block_size;
- ntfs_inode *locked_nis[PAGE_SIZE / NTFS_BLOCK_SIZE];
- struct buffer_head *bh, *head, *tbh, *rec_start_bh;
- struct buffer_head *bhs[MAX_BUF_PER_PAGE];
- runlist_element *rl;
- int i, nr_locked_nis, nr_recs, nr_bhs, max_bhs, bhs_per_rec, err, err2;
- unsigned bh_size, rec_size_bits;
- bool sync, is_mft, page_is_dirty, rec_is_dirty;
- unsigned char bh_size_bits;
-
- if (WARN_ON(rec_size < NTFS_BLOCK_SIZE))
- return -EINVAL;
-
- ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, page index "
- "0x%lx.", vi->i_ino, ni->type, page->index);
- BUG_ON(!NInoNonResident(ni));
- BUG_ON(!NInoMstProtected(ni));
- is_mft = (S_ISREG(vi->i_mode) && !vi->i_ino);
- /*
- * NOTE: ntfs_write_mst_block() would be called for $MFTMirr if a page
- * in its page cache were to be marked dirty. However this should
- * never happen with the current driver and considering we do not
- * handle this case here we do want to BUG(), at least for now.
- */
- BUG_ON(!(is_mft || S_ISDIR(vi->i_mode) ||
- (NInoAttr(ni) && ni->type == AT_INDEX_ALLOCATION)));
- bh_size = vol->sb->s_blocksize;
- bh_size_bits = vol->sb->s_blocksize_bits;
- max_bhs = PAGE_SIZE / bh_size;
- BUG_ON(!max_bhs);
- BUG_ON(max_bhs > MAX_BUF_PER_PAGE);
-
- /* Were we called for sync purposes? */
- sync = (wbc->sync_mode == WB_SYNC_ALL);
-
- /* Make sure we have mapped buffers. */
- bh = head = page_buffers(page);
- BUG_ON(!bh);
-
- rec_size_bits = ni->itype.index.block_size_bits;
- BUG_ON(!(PAGE_SIZE >> rec_size_bits));
- bhs_per_rec = rec_size >> bh_size_bits;
- BUG_ON(!bhs_per_rec);
-
- /* The first block in the page. */
- rec_block = block = (sector_t)page->index <<
- (PAGE_SHIFT - bh_size_bits);
-
- /* The first out of bounds block for the data size. */
- dblock = (i_size_read(vi) + bh_size - 1) >> bh_size_bits;
-
- rl = NULL;
- err = err2 = nr_bhs = nr_recs = nr_locked_nis = 0;
- page_is_dirty = rec_is_dirty = false;
- rec_start_bh = NULL;
- do {
- bool is_retry = false;
-
- if (likely(block < rec_block)) {
- if (unlikely(block >= dblock)) {
- clear_buffer_dirty(bh);
- set_buffer_uptodate(bh);
- continue;
- }
- /*
- * This block is not the first one in the record. We
- * ignore the buffer's dirty state because we could
- * have raced with a parallel mark_ntfs_record_dirty().
- */
- if (!rec_is_dirty)
- continue;
- if (unlikely(err2)) {
- if (err2 != -ENOMEM)
- clear_buffer_dirty(bh);
- continue;
- }
- } else /* if (block == rec_block) */ {
- BUG_ON(block > rec_block);
- /* This block is the first one in the record. */
- rec_block += bhs_per_rec;
- err2 = 0;
- if (unlikely(block >= dblock)) {
- clear_buffer_dirty(bh);
- continue;
- }
- if (!buffer_dirty(bh)) {
- /* Clean records are not written out. */
- rec_is_dirty = false;
- continue;
- }
- rec_is_dirty = true;
- rec_start_bh = bh;
- }
- /* Need to map the buffer if it is not mapped already. */
- if (unlikely(!buffer_mapped(bh))) {
- VCN vcn;
- LCN lcn;
- unsigned int vcn_ofs;
-
- bh->b_bdev = vol->sb->s_bdev;
- /* Obtain the vcn and offset of the current block. */
- vcn = (VCN)block << bh_size_bits;
- vcn_ofs = vcn & vol->cluster_size_mask;
- vcn >>= vol->cluster_size_bits;
- if (!rl) {
-lock_retry_remap:
- down_read(&ni->runlist.lock);
- rl = ni->runlist.rl;
- }
- if (likely(rl != NULL)) {
- /* Seek to element containing target vcn. */
- while (rl->length && rl[1].vcn <= vcn)
- rl++;
- lcn = ntfs_rl_vcn_to_lcn(rl, vcn);
- } else
- lcn = LCN_RL_NOT_MAPPED;
- /* Successful remap. */
- if (likely(lcn >= 0)) {
- /* Setup buffer head to correct block. */
- bh->b_blocknr = ((lcn <<
- vol->cluster_size_bits) +
- vcn_ofs) >> bh_size_bits;
- set_buffer_mapped(bh);
- } else {
- /*
- * Remap failed. Retry to map the runlist once
- * unless we are working on $MFT which always
- * has the whole of its runlist in memory.
- */
- if (!is_mft && !is_retry &&
- lcn == LCN_RL_NOT_MAPPED) {
- is_retry = true;
- /*
- * Attempt to map runlist, dropping
- * lock for the duration.
- */
- up_read(&ni->runlist.lock);
- err2 = ntfs_map_runlist(ni, vcn);
- if (likely(!err2))
- goto lock_retry_remap;
- if (err2 == -ENOMEM)
- page_is_dirty = true;
- lcn = err2;
- } else {
- err2 = -EIO;
- if (!rl)
- up_read(&ni->runlist.lock);
- }
- /* Hard error. Abort writing this record. */
- if (!err || err == -ENOMEM)
- err = err2;
- bh->b_blocknr = -1;
- ntfs_error(vol->sb, "Cannot write ntfs record "
- "0x%llx (inode 0x%lx, "
- "attribute type 0x%x) because "
- "its location on disk could "
- "not be determined (error "
- "code %lli).",
- (long long)block <<
- bh_size_bits >>
- vol->mft_record_size_bits,
- ni->mft_no, ni->type,
- (long long)lcn);
- /*
- * If this is not the first buffer, remove the
- * buffers in this record from the list of
- * buffers to write and clear their dirty bit
- * if not error -ENOMEM.
- */
- if (rec_start_bh != bh) {
- while (bhs[--nr_bhs] != rec_start_bh)
- ;
- if (err2 != -ENOMEM) {
- do {
- clear_buffer_dirty(
- rec_start_bh);
- } while ((rec_start_bh =
- rec_start_bh->
- b_this_page) !=
- bh);
- }
- }
- continue;
- }
- }
- BUG_ON(!buffer_uptodate(bh));
- BUG_ON(nr_bhs >= max_bhs);
- bhs[nr_bhs++] = bh;
- } while (block++, (bh = bh->b_this_page) != head);
- if (unlikely(rl))
- up_read(&ni->runlist.lock);
- /* If there were no dirty buffers, we are done. */
- if (!nr_bhs)
- goto done;
- /* Map the page so we can access its contents. */
- kaddr = kmap(page);
- /* Clear the page uptodate flag whilst the mst fixups are applied. */
- BUG_ON(!PageUptodate(page));
- ClearPageUptodate(page);
- for (i = 0; i < nr_bhs; i++) {
- unsigned int ofs;
-
- /* Skip buffers which are not at the beginning of records. */
- if (i % bhs_per_rec)
- continue;
- tbh = bhs[i];
- ofs = bh_offset(tbh);
- if (is_mft) {
- ntfs_inode *tni;
- unsigned long mft_no;
-
- /* Get the mft record number. */
- mft_no = (((s64)page->index << PAGE_SHIFT) + ofs)
- >> rec_size_bits;
- /* Check whether to write this mft record. */
- tni = NULL;
- if (!ntfs_may_write_mft_record(vol, mft_no,
- (MFT_RECORD*)(kaddr + ofs), &tni)) {
- /*
- * The record should not be written. This
- * means we need to redirty the page before
- * returning.
- */
- page_is_dirty = true;
- /*
- * Remove the buffers in this mft record from
- * the list of buffers to write.
- */
- do {
- bhs[i] = NULL;
- } while (++i % bhs_per_rec);
- continue;
- }
- /*
- * The record should be written. If a locked ntfs
- * inode was returned, add it to the array of locked
- * ntfs inodes.
- */
- if (tni)
- locked_nis[nr_locked_nis++] = tni;
- }
- /* Apply the mst protection fixups. */
- err2 = pre_write_mst_fixup((NTFS_RECORD*)(kaddr + ofs),
- rec_size);
- if (unlikely(err2)) {
- if (!err || err == -ENOMEM)
- err = -EIO;
- ntfs_error(vol->sb, "Failed to apply mst fixups "
- "(inode 0x%lx, attribute type 0x%x, "
- "page index 0x%lx, page offset 0x%x)!"
- " Unmount and run chkdsk.", vi->i_ino,
- ni->type, page->index, ofs);
- /*
- * Mark all the buffers in this record clean as we do
- * not want to write corrupt data to disk.
- */
- do {
- clear_buffer_dirty(bhs[i]);
- bhs[i] = NULL;
- } while (++i % bhs_per_rec);
- continue;
- }
- nr_recs++;
- }
- /* If no records are to be written out, we are done. */
- if (!nr_recs)
- goto unm_done;
- flush_dcache_page(page);
- /* Lock buffers and start synchronous write i/o on them. */
- for (i = 0; i < nr_bhs; i++) {
- tbh = bhs[i];
- if (!tbh)
- continue;
- if (!trylock_buffer(tbh))
- BUG();
- /* The buffer dirty state is now irrelevant, just clean it. */
- clear_buffer_dirty(tbh);
- BUG_ON(!buffer_uptodate(tbh));
- BUG_ON(!buffer_mapped(tbh));
- get_bh(tbh);
- tbh->b_end_io = end_buffer_write_sync;
- submit_bh(REQ_OP_WRITE, tbh);
- }
- /* Synchronize the mft mirror now if not @sync. */
- if (is_mft && !sync)
- goto do_mirror;
-do_wait:
- /* Wait on i/o completion of buffers. */
- for (i = 0; i < nr_bhs; i++) {
- tbh = bhs[i];
- if (!tbh)
- continue;
- wait_on_buffer(tbh);
- if (unlikely(!buffer_uptodate(tbh))) {
- ntfs_error(vol->sb, "I/O error while writing ntfs "
- "record buffer (inode 0x%lx, "
- "attribute type 0x%x, page index "
- "0x%lx, page offset 0x%lx)! Unmount "
- "and run chkdsk.", vi->i_ino, ni->type,
- page->index, bh_offset(tbh));
- if (!err || err == -ENOMEM)
- err = -EIO;
- /*
- * Set the buffer uptodate so the page and buffer
- * states do not become out of sync.
- */
- set_buffer_uptodate(tbh);
- }
- }
- /* If @sync, now synchronize the mft mirror. */
- if (is_mft && sync) {
-do_mirror:
- for (i = 0; i < nr_bhs; i++) {
- unsigned long mft_no;
- unsigned int ofs;
-
- /*
- * Skip buffers which are not at the beginning of
- * records.
- */
- if (i % bhs_per_rec)
- continue;
- tbh = bhs[i];
- /* Skip removed buffers (and hence records). */
- if (!tbh)
- continue;
- ofs = bh_offset(tbh);
- /* Get the mft record number. */
- mft_no = (((s64)page->index << PAGE_SHIFT) + ofs)
- >> rec_size_bits;
- if (mft_no < vol->mftmirr_size)
- ntfs_sync_mft_mirror(vol, mft_no,
- (MFT_RECORD*)(kaddr + ofs),
- sync);
- }
- if (!sync)
- goto do_wait;
- }
- /* Remove the mst protection fixups again. */
- for (i = 0; i < nr_bhs; i++) {
- if (!(i % bhs_per_rec)) {
- tbh = bhs[i];
- if (!tbh)
- continue;
- post_write_mst_fixup((NTFS_RECORD*)(kaddr +
- bh_offset(tbh)));
- }
- }
- flush_dcache_page(page);
-unm_done:
- /* Unlock any locked inodes. */
- while (nr_locked_nis-- > 0) {
- ntfs_inode *tni, *base_tni;
-
- tni = locked_nis[nr_locked_nis];
- /* Get the base inode. */
- mutex_lock(&tni->extent_lock);
- if (tni->nr_extents >= 0)
- base_tni = tni;
- else {
- base_tni = tni->ext.base_ntfs_ino;
- BUG_ON(!base_tni);
- }
- mutex_unlock(&tni->extent_lock);
- ntfs_debug("Unlocking %s inode 0x%lx.",
- tni == base_tni ? "base" : "extent",
- tni->mft_no);
- mutex_unlock(&tni->mrec_lock);
- atomic_dec(&tni->count);
- iput(VFS_I(base_tni));
- }
- SetPageUptodate(page);
- kunmap(page);
-done:
- if (unlikely(err && err != -ENOMEM)) {
- /*
- * Set page error if there is only one ntfs record in the page.
- * Otherwise we would loose per-record granularity.
- */
- if (ni->itype.index.block_size == PAGE_SIZE)
- SetPageError(page);
- NVolSetErrors(vol);
- }
- if (page_is_dirty) {
- ntfs_debug("Page still contains one or more dirty ntfs "
- "records. Redirtying the page starting at "
- "record 0x%lx.", page->index <<
- (PAGE_SHIFT - rec_size_bits));
- redirty_page_for_writepage(wbc, page);
- unlock_page(page);
- } else {
- /*
- * Keep the VM happy. This must be done otherwise the
- * radix-tree tag PAGECACHE_TAG_DIRTY remains set even though
- * the page is clean.
- */
- BUG_ON(PageWriteback(page));
- set_page_writeback(page);
- unlock_page(page);
- end_page_writeback(page);
- }
- if (likely(!err))
- ntfs_debug("Done.");
- return err;
-}
-
-/**
- * ntfs_writepage - write a @page to the backing store
- * @page: page cache page to write out
- * @wbc: writeback control structure
- *
- * This is called from the VM when it wants to have a dirty ntfs page cache
- * page cleaned. The VM has already locked the page and marked it clean.
- *
- * For non-resident attributes, ntfs_writepage() writes the @page by calling
- * the ntfs version of the generic block_write_full_page() function,
- * ntfs_write_block(), which in turn if necessary creates and writes the
- * buffers associated with the page asynchronously.
- *
- * For resident attributes, OTOH, ntfs_writepage() writes the @page by copying
- * the data to the mft record (which at this stage is most likely in memory).
- * The mft record is then marked dirty and written out asynchronously via the
- * vfs inode dirty code path for the inode the mft record belongs to or via the
- * vm page dirty code path for the page the mft record is in.
- *
- * Based on ntfs_read_folio() and fs/buffer.c::block_write_full_page().
- *
- * Return 0 on success and -errno on error.
- */
-static int ntfs_writepage(struct page *page, struct writeback_control *wbc)
-{
- loff_t i_size;
- struct inode *vi = page->mapping->host;
- ntfs_inode *base_ni = NULL, *ni = NTFS_I(vi);
- char *addr;
- ntfs_attr_search_ctx *ctx = NULL;
- MFT_RECORD *m = NULL;
- u32 attr_len;
- int err;
-
-retry_writepage:
- BUG_ON(!PageLocked(page));
- i_size = i_size_read(vi);
- /* Is the page fully outside i_size? (truncate in progress) */
- if (unlikely(page->index >= (i_size + PAGE_SIZE - 1) >>
- PAGE_SHIFT)) {
- struct folio *folio = page_folio(page);
- /*
- * The page may have dirty, unmapped buffers. Make them
- * freeable here, so the page does not leak.
- */
- block_invalidate_folio(folio, 0, folio_size(folio));
- folio_unlock(folio);
- ntfs_debug("Write outside i_size - truncated?");
- return 0;
- }
- /*
- * Only $DATA attributes can be encrypted and only unnamed $DATA
- * attributes can be compressed. Index root can have the flags set but
- * this means to create compressed/encrypted files, not that the
- * attribute is compressed/encrypted. Note we need to check for
- * AT_INDEX_ALLOCATION since this is the type of both directory and
- * index inodes.
- */
- if (ni->type != AT_INDEX_ALLOCATION) {
- /* If file is encrypted, deny access, just like NT4. */
- if (NInoEncrypted(ni)) {
- unlock_page(page);
- BUG_ON(ni->type != AT_DATA);
- ntfs_debug("Denying write access to encrypted file.");
- return -EACCES;
- }
- /* Compressed data streams are handled in compress.c. */
- if (NInoNonResident(ni) && NInoCompressed(ni)) {
- BUG_ON(ni->type != AT_DATA);
- BUG_ON(ni->name_len);
- // TODO: Implement and replace this with
- // return ntfs_write_compressed_block(page);
- unlock_page(page);
- ntfs_error(vi->i_sb, "Writing to compressed files is "
- "not supported yet. Sorry.");
- return -EOPNOTSUPP;
- }
- // TODO: Implement and remove this check.
- if (NInoNonResident(ni) && NInoSparse(ni)) {
- unlock_page(page);
- ntfs_error(vi->i_sb, "Writing to sparse files is not "
- "supported yet. Sorry.");
- return -EOPNOTSUPP;
- }
- }
- /* NInoNonResident() == NInoIndexAllocPresent() */
- if (NInoNonResident(ni)) {
- /* We have to zero every time due to mmap-at-end-of-file. */
- if (page->index >= (i_size >> PAGE_SHIFT)) {
- /* The page straddles i_size. */
- unsigned int ofs = i_size & ~PAGE_MASK;
- zero_user_segment(page, ofs, PAGE_SIZE);
- }
- /* Handle mst protected attributes. */
- if (NInoMstProtected(ni))
- return ntfs_write_mst_block(page, wbc);
- /* Normal, non-resident data stream. */
- return ntfs_write_block(page, wbc);
- }
- /*
- * Attribute is resident, implying it is not compressed, encrypted, or
- * mst protected. This also means the attribute is smaller than an mft
- * record and hence smaller than a page, so can simply return error on
- * any pages with index above 0. Note the attribute can actually be
- * marked compressed but if it is resident the actual data is not
- * compressed so we are ok to ignore the compressed flag here.
- */
- BUG_ON(page_has_buffers(page));
- BUG_ON(!PageUptodate(page));
- if (unlikely(page->index > 0)) {
- ntfs_error(vi->i_sb, "BUG()! page->index (0x%lx) > 0. "
- "Aborting write.", page->index);
- BUG_ON(PageWriteback(page));
- set_page_writeback(page);
- unlock_page(page);
- end_page_writeback(page);
- return -EIO;
- }
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- /* Map, pin, and lock the mft record. */
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- ctx = NULL;
- goto err_out;
- }
- /*
- * If a parallel write made the attribute non-resident, drop the mft
- * record and retry the writepage.
- */
- if (unlikely(NInoNonResident(ni))) {
- unmap_mft_record(base_ni);
- goto retry_writepage;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err))
- goto err_out;
- /*
- * Keep the VM happy. This must be done otherwise the radix-tree tag
- * PAGECACHE_TAG_DIRTY remains set even though the page is clean.
- */
- BUG_ON(PageWriteback(page));
- set_page_writeback(page);
- unlock_page(page);
- attr_len = le32_to_cpu(ctx->attr->data.resident.value_length);
- i_size = i_size_read(vi);
- if (unlikely(attr_len > i_size)) {
- /* Race with shrinking truncate or a failed truncate. */
- attr_len = i_size;
- /*
- * If the truncate failed, fix it up now. If a concurrent
- * truncate, we do its job, so it does not have to do anything.
- */
- err = ntfs_resident_attr_value_resize(ctx->mrec, ctx->attr,
- attr_len);
- /* Shrinking cannot fail. */
- BUG_ON(err);
- }
- addr = kmap_atomic(page);
- /* Copy the data from the page to the mft record. */
- memcpy((u8*)ctx->attr +
- le16_to_cpu(ctx->attr->data.resident.value_offset),
- addr, attr_len);
- /* Zero out of bounds area in the page cache page. */
- memset(addr + attr_len, 0, PAGE_SIZE - attr_len);
- kunmap_atomic(addr);
- flush_dcache_page(page);
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- /* We are done with the page. */
- end_page_writeback(page);
- /* Finally, mark the mft record dirty, so it gets written back. */
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- return 0;
-err_out:
- if (err == -ENOMEM) {
- ntfs_warning(vi->i_sb, "Error allocating memory. Redirtying "
- "page so we try again later.");
- /*
- * Put the page back on mapping->dirty_pages, but leave its
- * buffers' dirty state as-is.
- */
- redirty_page_for_writepage(wbc, page);
- err = 0;
- } else {
- ntfs_error(vi->i_sb, "Resident attribute write failed with "
- "error %i.", err);
- SetPageError(page);
- NVolSetErrors(ni->vol);
- }
- unlock_page(page);
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
- return err;
-}
-
-#endif /* NTFS_RW */
-
-/**
- * ntfs_bmap - map logical file block to physical device block
- * @mapping: address space mapping to which the block to be mapped belongs
- * @block: logical block to map to its physical device block
- *
- * For regular, non-resident files (i.e. not compressed and not encrypted), map
- * the logical @block belonging to the file described by the address space
- * mapping @mapping to its physical device block.
- *
- * The size of the block is equal to the @s_blocksize field of the super block
- * of the mounted file system which is guaranteed to be smaller than or equal
- * to the cluster size thus the block is guaranteed to fit entirely inside the
- * cluster which means we do not need to care how many contiguous bytes are
- * available after the beginning of the block.
- *
- * Return the physical device block if the mapping succeeded or 0 if the block
- * is sparse or there was an error.
- *
- * Note: This is a problem if someone tries to run bmap() on $Boot system file
- * as that really is in block zero but there is nothing we can do. bmap() is
- * just broken in that respect (just like it cannot distinguish sparse from
- * not available or error).
- */
-static sector_t ntfs_bmap(struct address_space *mapping, sector_t block)
-{
- s64 ofs, size;
- loff_t i_size;
- LCN lcn;
- unsigned long blocksize, flags;
- ntfs_inode *ni = NTFS_I(mapping->host);
- ntfs_volume *vol = ni->vol;
- unsigned delta;
- unsigned char blocksize_bits, cluster_size_shift;
-
- ntfs_debug("Entering for mft_no 0x%lx, logical block 0x%llx.",
- ni->mft_no, (unsigned long long)block);
- if (ni->type != AT_DATA || !NInoNonResident(ni) || NInoEncrypted(ni)) {
- ntfs_error(vol->sb, "BMAP does not make sense for %s "
- "attributes, returning 0.",
- (ni->type != AT_DATA) ? "non-data" :
- (!NInoNonResident(ni) ? "resident" :
- "encrypted"));
- return 0;
- }
- /* None of these can happen. */
- BUG_ON(NInoCompressed(ni));
- BUG_ON(NInoMstProtected(ni));
- blocksize = vol->sb->s_blocksize;
- blocksize_bits = vol->sb->s_blocksize_bits;
- ofs = (s64)block << blocksize_bits;
- read_lock_irqsave(&ni->size_lock, flags);
- size = ni->initialized_size;
- i_size = i_size_read(VFS_I(ni));
- read_unlock_irqrestore(&ni->size_lock, flags);
- /*
- * If the offset is outside the initialized size or the block straddles
- * the initialized size then pretend it is a hole unless the
- * initialized size equals the file size.
- */
- if (unlikely(ofs >= size || (ofs + blocksize > size && size < i_size)))
- goto hole;
- cluster_size_shift = vol->cluster_size_bits;
- down_read(&ni->runlist.lock);
- lcn = ntfs_attr_vcn_to_lcn_nolock(ni, ofs >> cluster_size_shift, false);
- up_read(&ni->runlist.lock);
- if (unlikely(lcn < LCN_HOLE)) {
- /*
- * Step down to an integer to avoid gcc doing a long long
- * comparision in the switch when we know @lcn is between
- * LCN_HOLE and LCN_EIO (i.e. -1 to -5).
- *
- * Otherwise older gcc (at least on some architectures) will
- * try to use __cmpdi2() which is of course not available in
- * the kernel.
- */
- switch ((int)lcn) {
- case LCN_ENOENT:
- /*
- * If the offset is out of bounds then pretend it is a
- * hole.
- */
- goto hole;
- case LCN_ENOMEM:
- ntfs_error(vol->sb, "Not enough memory to complete "
- "mapping for inode 0x%lx. "
- "Returning 0.", ni->mft_no);
- break;
- default:
- ntfs_error(vol->sb, "Failed to complete mapping for "
- "inode 0x%lx. Run chkdsk. "
- "Returning 0.", ni->mft_no);
- break;
- }
- return 0;
- }
- if (lcn < 0) {
- /* It is a hole. */
-hole:
- ntfs_debug("Done (returning hole).");
- return 0;
- }
- /*
- * The block is really allocated and fullfils all our criteria.
- * Convert the cluster to units of block size and return the result.
- */
- delta = ofs & vol->cluster_size_mask;
- if (unlikely(sizeof(block) < sizeof(lcn))) {
- block = lcn = ((lcn << cluster_size_shift) + delta) >>
- blocksize_bits;
- /* If the block number was truncated return 0. */
- if (unlikely(block != lcn)) {
- ntfs_error(vol->sb, "Physical block 0x%llx is too "
- "large to be returned, returning 0.",
- (long long)lcn);
- return 0;
- }
- } else
- block = ((lcn << cluster_size_shift) + delta) >>
- blocksize_bits;
- ntfs_debug("Done (returning block 0x%llx).", (unsigned long long)lcn);
- return block;
-}
-
-/**
- * ntfs_normal_aops - address space operations for normal inodes and attributes
- *
- * Note these are not used for compressed or mst protected inodes and
- * attributes.
- */
-const struct address_space_operations ntfs_normal_aops = {
- .read_folio = ntfs_read_folio,
-#ifdef NTFS_RW
- .writepage = ntfs_writepage,
- .dirty_folio = block_dirty_folio,
-#endif /* NTFS_RW */
- .bmap = ntfs_bmap,
- .migrate_folio = buffer_migrate_folio,
- .is_partially_uptodate = block_is_partially_uptodate,
- .error_remove_page = generic_error_remove_page,
-};
-
-/**
- * ntfs_compressed_aops - address space operations for compressed inodes
- */
-const struct address_space_operations ntfs_compressed_aops = {
- .read_folio = ntfs_read_folio,
-#ifdef NTFS_RW
- .writepage = ntfs_writepage,
- .dirty_folio = block_dirty_folio,
-#endif /* NTFS_RW */
- .migrate_folio = buffer_migrate_folio,
- .is_partially_uptodate = block_is_partially_uptodate,
- .error_remove_page = generic_error_remove_page,
-};
-
-/**
- * ntfs_mst_aops - general address space operations for mst protecteed inodes
- * and attributes
- */
-const struct address_space_operations ntfs_mst_aops = {
- .read_folio = ntfs_read_folio, /* Fill page with data. */
-#ifdef NTFS_RW
- .writepage = ntfs_writepage, /* Write dirty page to disk. */
- .dirty_folio = filemap_dirty_folio,
-#endif /* NTFS_RW */
- .migrate_folio = buffer_migrate_folio,
- .is_partially_uptodate = block_is_partially_uptodate,
- .error_remove_page = generic_error_remove_page,
-};
-
-#ifdef NTFS_RW
-
-/**
- * mark_ntfs_record_dirty - mark an ntfs record dirty
- * @page: page containing the ntfs record to mark dirty
- * @ofs: byte offset within @page at which the ntfs record begins
- *
- * Set the buffers and the page in which the ntfs record is located dirty.
- *
- * The latter also marks the vfs inode the ntfs record belongs to dirty
- * (I_DIRTY_PAGES only).
- *
- * If the page does not have buffers, we create them and set them uptodate.
- * The page may not be locked which is why we need to handle the buffers under
- * the mapping->private_lock. Once the buffers are marked dirty we no longer
- * need the lock since try_to_free_buffers() does not free dirty buffers.
- */
-void mark_ntfs_record_dirty(struct page *page, const unsigned int ofs) {
- struct address_space *mapping = page->mapping;
- ntfs_inode *ni = NTFS_I(mapping->host);
- struct buffer_head *bh, *head, *buffers_to_free = NULL;
- unsigned int end, bh_size, bh_ofs;
-
- BUG_ON(!PageUptodate(page));
- end = ofs + ni->itype.index.block_size;
- bh_size = VFS_I(ni)->i_sb->s_blocksize;
- spin_lock(&mapping->private_lock);
- if (unlikely(!page_has_buffers(page))) {
- spin_unlock(&mapping->private_lock);
- bh = head = alloc_page_buffers(page, bh_size, true);
- spin_lock(&mapping->private_lock);
- if (likely(!page_has_buffers(page))) {
- struct buffer_head *tail;
-
- do {
- set_buffer_uptodate(bh);
- tail = bh;
- bh = bh->b_this_page;
- } while (bh);
- tail->b_this_page = head;
- attach_page_private(page, head);
- } else
- buffers_to_free = bh;
- }
- bh = head = page_buffers(page);
- BUG_ON(!bh);
- do {
- bh_ofs = bh_offset(bh);
- if (bh_ofs + bh_size <= ofs)
- continue;
- if (unlikely(bh_ofs >= end))
- break;
- set_buffer_dirty(bh);
- } while ((bh = bh->b_this_page) != head);
- spin_unlock(&mapping->private_lock);
- filemap_dirty_folio(mapping, page_folio(page));
- if (unlikely(buffers_to_free)) {
- do {
- bh = buffers_to_free->b_this_page;
- free_buffer_head(buffers_to_free);
- buffers_to_free = bh;
- } while (buffers_to_free);
- }
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/aops.h b/fs/ntfs/aops.h
deleted file mode 100644
index 0cac5458c023..000000000000
--- a/fs/ntfs/aops.h
+++ /dev/null
@@ -1,88 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/**
- * aops.h - Defines for NTFS kernel address space operations and page cache
- * handling. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2004 Anton Altaparmakov
- * Copyright (c) 2002 Richard Russon
- */
-
-#ifndef _LINUX_NTFS_AOPS_H
-#define _LINUX_NTFS_AOPS_H
-
-#include <linux/mm.h>
-#include <linux/highmem.h>
-#include <linux/pagemap.h>
-#include <linux/fs.h>
-
-#include "inode.h"
-
-/**
- * ntfs_unmap_page - release a page that was mapped using ntfs_map_page()
- * @page: the page to release
- *
- * Unpin, unmap and release a page that was obtained from ntfs_map_page().
- */
-static inline void ntfs_unmap_page(struct page *page)
-{
- kunmap(page);
- put_page(page);
-}
-
-/**
- * ntfs_map_page - map a page into accessible memory, reading it if necessary
- * @mapping: address space for which to obtain the page
- * @index: index into the page cache for @mapping of the page to map
- *
- * Read a page from the page cache of the address space @mapping at position
- * @index, where @index is in units of PAGE_SIZE, and not in bytes.
- *
- * If the page is not in memory it is loaded from disk first using the
- * read_folio method defined in the address space operations of @mapping
- * and the page is added to the page cache of @mapping in the process.
- *
- * If the page belongs to an mst protected attribute and it is marked as such
- * in its ntfs inode (NInoMstProtected()) the mst fixups are applied but no
- * error checking is performed. This means the caller has to verify whether
- * the ntfs record(s) contained in the page are valid or not using one of the
- * ntfs_is_XXXX_record{,p}() macros, where XXXX is the record type you are
- * expecting to see. (For details of the macros, see fs/ntfs/layout.h.)
- *
- * If the page is in high memory it is mapped into memory directly addressible
- * by the kernel.
- *
- * Finally the page count is incremented, thus pinning the page into place.
- *
- * The above means that page_address(page) can be used on all pages obtained
- * with ntfs_map_page() to get the kernel virtual address of the page.
- *
- * When finished with the page, the caller has to call ntfs_unmap_page() to
- * unpin, unmap and release the page.
- *
- * Note this does not grant exclusive access. If such is desired, the caller
- * must provide it independently of the ntfs_{un}map_page() calls by using
- * a {rw_}semaphore or other means of serialization. A spin lock cannot be
- * used as ntfs_map_page() can block.
- *
- * The unlocked and uptodate page is returned on success or an encoded error
- * on failure. Caller has to test for error using the IS_ERR() macro on the
- * return value. If that evaluates to 'true', the negative error code can be
- * obtained using PTR_ERR() on the return value of ntfs_map_page().
- */
-static inline struct page *ntfs_map_page(struct address_space *mapping,
- unsigned long index)
-{
- struct page *page = read_mapping_page(mapping, index, NULL);
-
- if (!IS_ERR(page))
- kmap(page);
- return page;
-}
-
-#ifdef NTFS_RW
-
-extern void mark_ntfs_record_dirty(struct page *page, const unsigned int ofs);
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_AOPS_H */
diff --git a/fs/ntfs/attrib.c b/fs/ntfs/attrib.c
deleted file mode 100644
index a3865bc4a0c6..000000000000
--- a/fs/ntfs/attrib.c
+++ /dev/null
@@ -1,2624 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * attrib.c - NTFS attribute operations. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2012 Anton Altaparmakov and Tuxera Inc.
- * Copyright (c) 2002 Richard Russon
- */
-
-#include <linux/buffer_head.h>
-#include <linux/sched.h>
-#include <linux/slab.h>
-#include <linux/swap.h>
-#include <linux/writeback.h>
-
-#include "attrib.h"
-#include "debug.h"
-#include "layout.h"
-#include "lcnalloc.h"
-#include "malloc.h"
-#include "mft.h"
-#include "ntfs.h"
-#include "types.h"
-
-/**
- * ntfs_map_runlist_nolock - map (a part of) a runlist of an ntfs inode
- * @ni: ntfs inode for which to map (part of) a runlist
- * @vcn: map runlist part containing this vcn
- * @ctx: active attribute search context if present or NULL if not
- *
- * Map the part of a runlist containing the @vcn of the ntfs inode @ni.
- *
- * If @ctx is specified, it is an active search context of @ni and its base mft
- * record. This is needed when ntfs_map_runlist_nolock() encounters unmapped
- * runlist fragments and allows their mapping. If you do not have the mft
- * record mapped, you can specify @ctx as NULL and ntfs_map_runlist_nolock()
- * will perform the necessary mapping and unmapping.
- *
- * Note, ntfs_map_runlist_nolock() saves the state of @ctx on entry and
- * restores it before returning. Thus, @ctx will be left pointing to the same
- * attribute on return as on entry. However, the actual pointers in @ctx may
- * point to different memory locations on return, so you must remember to reset
- * any cached pointers from the @ctx, i.e. after the call to
- * ntfs_map_runlist_nolock(), you will probably want to do:
- * m = ctx->mrec;
- * a = ctx->attr;
- * Assuming you cache ctx->attr in a variable @a of type ATTR_RECORD * and that
- * you cache ctx->mrec in a variable @m of type MFT_RECORD *.
- *
- * Return 0 on success and -errno on error. There is one special error code
- * which is not an error as such. This is -ENOENT. It means that @vcn is out
- * of bounds of the runlist.
- *
- * Note the runlist can be NULL after this function returns if @vcn is zero and
- * the attribute has zero allocated size, i.e. there simply is no runlist.
- *
- * WARNING: If @ctx is supplied, regardless of whether success or failure is
- * returned, you need to check IS_ERR(@ctx->mrec) and if 'true' the @ctx
- * is no longer valid, i.e. you need to either call
- * ntfs_attr_reinit_search_ctx() or ntfs_attr_put_search_ctx() on it.
- * In that case PTR_ERR(@ctx->mrec) will give you the error code for
- * why the mapping of the old inode failed.
- *
- * Locking: - The runlist described by @ni must be locked for writing on entry
- * and is locked on return. Note the runlist will be modified.
- * - If @ctx is NULL, the base mft record of @ni must not be mapped on
- * entry and it will be left unmapped on return.
- * - If @ctx is not NULL, the base mft record must be mapped on entry
- * and it will be left mapped on return.
- */
-int ntfs_map_runlist_nolock(ntfs_inode *ni, VCN vcn, ntfs_attr_search_ctx *ctx)
-{
- VCN end_vcn;
- unsigned long flags;
- ntfs_inode *base_ni;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- runlist_element *rl;
- struct page *put_this_page = NULL;
- int err = 0;
- bool ctx_is_temporary, ctx_needs_reset;
- ntfs_attr_search_ctx old_ctx = { NULL, };
-
- ntfs_debug("Mapping runlist part containing vcn 0x%llx.",
- (unsigned long long)vcn);
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- if (!ctx) {
- ctx_is_temporary = ctx_needs_reset = true;
- m = map_mft_record(base_ni);
- if (IS_ERR(m))
- return PTR_ERR(m);
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- } else {
- VCN allocated_size_vcn;
-
- BUG_ON(IS_ERR(ctx->mrec));
- a = ctx->attr;
- BUG_ON(!a->non_resident);
- ctx_is_temporary = false;
- end_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn);
- read_lock_irqsave(&ni->size_lock, flags);
- allocated_size_vcn = ni->allocated_size >>
- ni->vol->cluster_size_bits;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (!a->data.non_resident.lowest_vcn && end_vcn <= 0)
- end_vcn = allocated_size_vcn - 1;
- /*
- * If we already have the attribute extent containing @vcn in
- * @ctx, no need to look it up again. We slightly cheat in
- * that if vcn exceeds the allocated size, we will refuse to
- * map the runlist below, so there is definitely no need to get
- * the right attribute extent.
- */
- if (vcn >= allocated_size_vcn || (a->type == ni->type &&
- a->name_length == ni->name_len &&
- !memcmp((u8*)a + le16_to_cpu(a->name_offset),
- ni->name, ni->name_len) &&
- sle64_to_cpu(a->data.non_resident.lowest_vcn)
- <= vcn && end_vcn >= vcn))
- ctx_needs_reset = false;
- else {
- /* Save the old search context. */
- old_ctx = *ctx;
- /*
- * If the currently mapped (extent) inode is not the
- * base inode we will unmap it when we reinitialize the
- * search context which means we need to get a
- * reference to the page containing the mapped mft
- * record so we do not accidentally drop changes to the
- * mft record when it has not been marked dirty yet.
- */
- if (old_ctx.base_ntfs_ino && old_ctx.ntfs_ino !=
- old_ctx.base_ntfs_ino) {
- put_this_page = old_ctx.ntfs_ino->page;
- get_page(put_this_page);
- }
- /*
- * Reinitialize the search context so we can lookup the
- * needed attribute extent.
- */
- ntfs_attr_reinit_search_ctx(ctx);
- ctx_needs_reset = true;
- }
- }
- if (ctx_needs_reset) {
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, vcn, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- goto err_out;
- }
- BUG_ON(!ctx->attr->non_resident);
- }
- a = ctx->attr;
- /*
- * Only decompress the mapping pairs if @vcn is inside it. Otherwise
- * we get into problems when we try to map an out of bounds vcn because
- * we then try to map the already mapped runlist fragment and
- * ntfs_mapping_pairs_decompress() fails.
- */
- end_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn) + 1;
- if (unlikely(vcn && vcn >= end_vcn)) {
- err = -ENOENT;
- goto err_out;
- }
- rl = ntfs_mapping_pairs_decompress(ni->vol, a, ni->runlist.rl);
- if (IS_ERR(rl))
- err = PTR_ERR(rl);
- else
- ni->runlist.rl = rl;
-err_out:
- if (ctx_is_temporary) {
- if (likely(ctx))
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- } else if (ctx_needs_reset) {
- /*
- * If there is no attribute list, restoring the search context
- * is accomplished simply by copying the saved context back over
- * the caller supplied context. If there is an attribute list,
- * things are more complicated as we need to deal with mapping
- * of mft records and resulting potential changes in pointers.
- */
- if (NInoAttrList(base_ni)) {
- /*
- * If the currently mapped (extent) inode is not the
- * one we had before, we need to unmap it and map the
- * old one.
- */
- if (ctx->ntfs_ino != old_ctx.ntfs_ino) {
- /*
- * If the currently mapped inode is not the
- * base inode, unmap it.
- */
- if (ctx->base_ntfs_ino && ctx->ntfs_ino !=
- ctx->base_ntfs_ino) {
- unmap_extent_mft_record(ctx->ntfs_ino);
- ctx->mrec = ctx->base_mrec;
- BUG_ON(!ctx->mrec);
- }
- /*
- * If the old mapped inode is not the base
- * inode, map it.
- */
- if (old_ctx.base_ntfs_ino &&
- old_ctx.ntfs_ino !=
- old_ctx.base_ntfs_ino) {
-retry_map:
- ctx->mrec = map_mft_record(
- old_ctx.ntfs_ino);
- /*
- * Something bad has happened. If out
- * of memory retry till it succeeds.
- * Any other errors are fatal and we
- * return the error code in ctx->mrec.
- * Let the caller deal with it... We
- * just need to fudge things so the
- * caller can reinit and/or put the
- * search context safely.
- */
- if (IS_ERR(ctx->mrec)) {
- if (PTR_ERR(ctx->mrec) ==
- -ENOMEM) {
- schedule();
- goto retry_map;
- } else
- old_ctx.ntfs_ino =
- old_ctx.
- base_ntfs_ino;
- }
- }
- }
- /* Update the changed pointers in the saved context. */
- if (ctx->mrec != old_ctx.mrec) {
- if (!IS_ERR(ctx->mrec))
- old_ctx.attr = (ATTR_RECORD*)(
- (u8*)ctx->mrec +
- ((u8*)old_ctx.attr -
- (u8*)old_ctx.mrec));
- old_ctx.mrec = ctx->mrec;
- }
- }
- /* Restore the search context to the saved one. */
- *ctx = old_ctx;
- /*
- * We drop the reference on the page we took earlier. In the
- * case that IS_ERR(ctx->mrec) is true this means we might lose
- * some changes to the mft record that had been made between
- * the last time it was marked dirty/written out and now. This
- * at this stage is not a problem as the mapping error is fatal
- * enough that the mft record cannot be written out anyway and
- * the caller is very likely to shutdown the whole inode
- * immediately and mark the volume dirty for chkdsk to pick up
- * the pieces anyway.
- */
- if (put_this_page)
- put_page(put_this_page);
- }
- return err;
-}
-
-/**
- * ntfs_map_runlist - map (a part of) a runlist of an ntfs inode
- * @ni: ntfs inode for which to map (part of) a runlist
- * @vcn: map runlist part containing this vcn
- *
- * Map the part of a runlist containing the @vcn of the ntfs inode @ni.
- *
- * Return 0 on success and -errno on error. There is one special error code
- * which is not an error as such. This is -ENOENT. It means that @vcn is out
- * of bounds of the runlist.
- *
- * Locking: - The runlist must be unlocked on entry and is unlocked on return.
- * - This function takes the runlist lock for writing and may modify
- * the runlist.
- */
-int ntfs_map_runlist(ntfs_inode *ni, VCN vcn)
-{
- int err = 0;
-
- down_write(&ni->runlist.lock);
- /* Make sure someone else didn't do the work while we were sleeping. */
- if (likely(ntfs_rl_vcn_to_lcn(ni->runlist.rl, vcn) <=
- LCN_RL_NOT_MAPPED))
- err = ntfs_map_runlist_nolock(ni, vcn, NULL);
- up_write(&ni->runlist.lock);
- return err;
-}
-
-/**
- * ntfs_attr_vcn_to_lcn_nolock - convert a vcn into a lcn given an ntfs inode
- * @ni: ntfs inode of the attribute whose runlist to search
- * @vcn: vcn to convert
- * @write_locked: true if the runlist is locked for writing
- *
- * Find the virtual cluster number @vcn in the runlist of the ntfs attribute
- * described by the ntfs inode @ni and return the corresponding logical cluster
- * number (lcn).
- *
- * If the @vcn is not mapped yet, the attempt is made to map the attribute
- * extent containing the @vcn and the vcn to lcn conversion is retried.
- *
- * If @write_locked is true the caller has locked the runlist for writing and
- * if false for reading.
- *
- * Since lcns must be >= 0, we use negative return codes with special meaning:
- *
- * Return code Meaning / Description
- * ==========================================
- * LCN_HOLE Hole / not allocated on disk.
- * LCN_ENOENT There is no such vcn in the runlist, i.e. @vcn is out of bounds.
- * LCN_ENOMEM Not enough memory to map runlist.
- * LCN_EIO Critical error (runlist/file is corrupt, i/o error, etc).
- *
- * Locking: - The runlist must be locked on entry and is left locked on return.
- * - If @write_locked is 'false', i.e. the runlist is locked for reading,
- * the lock may be dropped inside the function so you cannot rely on
- * the runlist still being the same when this function returns.
- */
-LCN ntfs_attr_vcn_to_lcn_nolock(ntfs_inode *ni, const VCN vcn,
- const bool write_locked)
-{
- LCN lcn;
- unsigned long flags;
- bool is_retry = false;
-
- BUG_ON(!ni);
- ntfs_debug("Entering for i_ino 0x%lx, vcn 0x%llx, %s_locked.",
- ni->mft_no, (unsigned long long)vcn,
- write_locked ? "write" : "read");
- BUG_ON(!NInoNonResident(ni));
- BUG_ON(vcn < 0);
- if (!ni->runlist.rl) {
- read_lock_irqsave(&ni->size_lock, flags);
- if (!ni->allocated_size) {
- read_unlock_irqrestore(&ni->size_lock, flags);
- return LCN_ENOENT;
- }
- read_unlock_irqrestore(&ni->size_lock, flags);
- }
-retry_remap:
- /* Convert vcn to lcn. If that fails map the runlist and retry once. */
- lcn = ntfs_rl_vcn_to_lcn(ni->runlist.rl, vcn);
- if (likely(lcn >= LCN_HOLE)) {
- ntfs_debug("Done, lcn 0x%llx.", (long long)lcn);
- return lcn;
- }
- if (lcn != LCN_RL_NOT_MAPPED) {
- if (lcn != LCN_ENOENT)
- lcn = LCN_EIO;
- } else if (!is_retry) {
- int err;
-
- if (!write_locked) {
- up_read(&ni->runlist.lock);
- down_write(&ni->runlist.lock);
- if (unlikely(ntfs_rl_vcn_to_lcn(ni->runlist.rl, vcn) !=
- LCN_RL_NOT_MAPPED)) {
- up_write(&ni->runlist.lock);
- down_read(&ni->runlist.lock);
- goto retry_remap;
- }
- }
- err = ntfs_map_runlist_nolock(ni, vcn, NULL);
- if (!write_locked) {
- up_write(&ni->runlist.lock);
- down_read(&ni->runlist.lock);
- }
- if (likely(!err)) {
- is_retry = true;
- goto retry_remap;
- }
- if (err == -ENOENT)
- lcn = LCN_ENOENT;
- else if (err == -ENOMEM)
- lcn = LCN_ENOMEM;
- else
- lcn = LCN_EIO;
- }
- if (lcn != LCN_ENOENT)
- ntfs_error(ni->vol->sb, "Failed with error code %lli.",
- (long long)lcn);
- return lcn;
-}
-
-/**
- * ntfs_attr_find_vcn_nolock - find a vcn in the runlist of an ntfs inode
- * @ni: ntfs inode describing the runlist to search
- * @vcn: vcn to find
- * @ctx: active attribute search context if present or NULL if not
- *
- * Find the virtual cluster number @vcn in the runlist described by the ntfs
- * inode @ni and return the address of the runlist element containing the @vcn.
- *
- * If the @vcn is not mapped yet, the attempt is made to map the attribute
- * extent containing the @vcn and the vcn to lcn conversion is retried.
- *
- * If @ctx is specified, it is an active search context of @ni and its base mft
- * record. This is needed when ntfs_attr_find_vcn_nolock() encounters unmapped
- * runlist fragments and allows their mapping. If you do not have the mft
- * record mapped, you can specify @ctx as NULL and ntfs_attr_find_vcn_nolock()
- * will perform the necessary mapping and unmapping.
- *
- * Note, ntfs_attr_find_vcn_nolock() saves the state of @ctx on entry and
- * restores it before returning. Thus, @ctx will be left pointing to the same
- * attribute on return as on entry. However, the actual pointers in @ctx may
- * point to different memory locations on return, so you must remember to reset
- * any cached pointers from the @ctx, i.e. after the call to
- * ntfs_attr_find_vcn_nolock(), you will probably want to do:
- * m = ctx->mrec;
- * a = ctx->attr;
- * Assuming you cache ctx->attr in a variable @a of type ATTR_RECORD * and that
- * you cache ctx->mrec in a variable @m of type MFT_RECORD *.
- * Note you need to distinguish between the lcn of the returned runlist element
- * being >= 0 and LCN_HOLE. In the later case you have to return zeroes on
- * read and allocate clusters on write.
- *
- * Return the runlist element containing the @vcn on success and
- * ERR_PTR(-errno) on error. You need to test the return value with IS_ERR()
- * to decide if the return is success or failure and PTR_ERR() to get to the
- * error code if IS_ERR() is true.
- *
- * The possible error return codes are:
- * -ENOENT - No such vcn in the runlist, i.e. @vcn is out of bounds.
- * -ENOMEM - Not enough memory to map runlist.
- * -EIO - Critical error (runlist/file is corrupt, i/o error, etc).
- *
- * WARNING: If @ctx is supplied, regardless of whether success or failure is
- * returned, you need to check IS_ERR(@ctx->mrec) and if 'true' the @ctx
- * is no longer valid, i.e. you need to either call
- * ntfs_attr_reinit_search_ctx() or ntfs_attr_put_search_ctx() on it.
- * In that case PTR_ERR(@ctx->mrec) will give you the error code for
- * why the mapping of the old inode failed.
- *
- * Locking: - The runlist described by @ni must be locked for writing on entry
- * and is locked on return. Note the runlist may be modified when
- * needed runlist fragments need to be mapped.
- * - If @ctx is NULL, the base mft record of @ni must not be mapped on
- * entry and it will be left unmapped on return.
- * - If @ctx is not NULL, the base mft record must be mapped on entry
- * and it will be left mapped on return.
- */
-runlist_element *ntfs_attr_find_vcn_nolock(ntfs_inode *ni, const VCN vcn,
- ntfs_attr_search_ctx *ctx)
-{
- unsigned long flags;
- runlist_element *rl;
- int err = 0;
- bool is_retry = false;
-
- BUG_ON(!ni);
- ntfs_debug("Entering for i_ino 0x%lx, vcn 0x%llx, with%s ctx.",
- ni->mft_no, (unsigned long long)vcn, ctx ? "" : "out");
- BUG_ON(!NInoNonResident(ni));
- BUG_ON(vcn < 0);
- if (!ni->runlist.rl) {
- read_lock_irqsave(&ni->size_lock, flags);
- if (!ni->allocated_size) {
- read_unlock_irqrestore(&ni->size_lock, flags);
- return ERR_PTR(-ENOENT);
- }
- read_unlock_irqrestore(&ni->size_lock, flags);
- }
-retry_remap:
- rl = ni->runlist.rl;
- if (likely(rl && vcn >= rl[0].vcn)) {
- while (likely(rl->length)) {
- if (unlikely(vcn < rl[1].vcn)) {
- if (likely(rl->lcn >= LCN_HOLE)) {
- ntfs_debug("Done.");
- return rl;
- }
- break;
- }
- rl++;
- }
- if (likely(rl->lcn != LCN_RL_NOT_MAPPED)) {
- if (likely(rl->lcn == LCN_ENOENT))
- err = -ENOENT;
- else
- err = -EIO;
- }
- }
- if (!err && !is_retry) {
- /*
- * If the search context is invalid we cannot map the unmapped
- * region.
- */
- if (IS_ERR(ctx->mrec))
- err = PTR_ERR(ctx->mrec);
- else {
- /*
- * The @vcn is in an unmapped region, map the runlist
- * and retry.
- */
- err = ntfs_map_runlist_nolock(ni, vcn, ctx);
- if (likely(!err)) {
- is_retry = true;
- goto retry_remap;
- }
- }
- if (err == -EINVAL)
- err = -EIO;
- } else if (!err)
- err = -EIO;
- if (err != -ENOENT)
- ntfs_error(ni->vol->sb, "Failed with error code %i.", err);
- return ERR_PTR(err);
-}
-
-/**
- * ntfs_attr_find - find (next) attribute in mft record
- * @type: attribute type to find
- * @name: attribute name to find (optional, i.e. NULL means don't care)
- * @name_len: attribute name length (only needed if @name present)
- * @ic: IGNORE_CASE or CASE_SENSITIVE (ignored if @name not present)
- * @val: attribute value to find (optional, resident attributes only)
- * @val_len: attribute value length
- * @ctx: search context with mft record and attribute to search from
- *
- * You should not need to call this function directly. Use ntfs_attr_lookup()
- * instead.
- *
- * ntfs_attr_find() takes a search context @ctx as parameter and searches the
- * mft record specified by @ctx->mrec, beginning at @ctx->attr, for an
- * attribute of @type, optionally @name and @val.
- *
- * If the attribute is found, ntfs_attr_find() returns 0 and @ctx->attr will
- * point to the found attribute.
- *
- * If the attribute is not found, ntfs_attr_find() returns -ENOENT and
- * @ctx->attr will point to the attribute before which the attribute being
- * searched for would need to be inserted if such an action were to be desired.
- *
- * On actual error, ntfs_attr_find() returns -EIO. In this case @ctx->attr is
- * undefined and in particular do not rely on it not changing.
- *
- * If @ctx->is_first is 'true', the search begins with @ctx->attr itself. If it
- * is 'false', the search begins after @ctx->attr.
- *
- * If @ic is IGNORE_CASE, the @name comparisson is not case sensitive and
- * @ctx->ntfs_ino must be set to the ntfs inode to which the mft record
- * @ctx->mrec belongs. This is so we can get at the ntfs volume and hence at
- * the upcase table. If @ic is CASE_SENSITIVE, the comparison is case
- * sensitive. When @name is present, @name_len is the @name length in Unicode
- * characters.
- *
- * If @name is not present (NULL), we assume that the unnamed attribute is
- * being searched for.
- *
- * Finally, the resident attribute value @val is looked for, if present. If
- * @val is not present (NULL), @val_len is ignored.
- *
- * ntfs_attr_find() only searches the specified mft record and it ignores the
- * presence of an attribute list attribute (unless it is the one being searched
- * for, obviously). If you need to take attribute lists into consideration,
- * use ntfs_attr_lookup() instead (see below). This also means that you cannot
- * use ntfs_attr_find() to search for extent records of non-resident
- * attributes, as extents with lowest_vcn != 0 are usually described by the
- * attribute list attribute only. - Note that it is possible that the first
- * extent is only in the attribute list while the last extent is in the base
- * mft record, so do not rely on being able to find the first extent in the
- * base mft record.
- *
- * Warning: Never use @val when looking for attribute types which can be
- * non-resident as this most likely will result in a crash!
- */
-static int ntfs_attr_find(const ATTR_TYPE type, const ntfschar *name,
- const u32 name_len, const IGNORE_CASE_BOOL ic,
- const u8 *val, const u32 val_len, ntfs_attr_search_ctx *ctx)
-{
- ATTR_RECORD *a;
- ntfs_volume *vol = ctx->ntfs_ino->vol;
- ntfschar *upcase = vol->upcase;
- u32 upcase_len = vol->upcase_len;
-
- /*
- * Iterate over attributes in mft record starting at @ctx->attr, or the
- * attribute following that, if @ctx->is_first is 'true'.
- */
- if (ctx->is_first) {
- a = ctx->attr;
- ctx->is_first = false;
- } else
- a = (ATTR_RECORD*)((u8*)ctx->attr +
- le32_to_cpu(ctx->attr->length));
- for (;; a = (ATTR_RECORD*)((u8*)a + le32_to_cpu(a->length))) {
- u8 *mrec_end = (u8 *)ctx->mrec +
- le32_to_cpu(ctx->mrec->bytes_allocated);
- u8 *name_end;
-
- /* check whether ATTR_RECORD wrap */
- if ((u8 *)a < (u8 *)ctx->mrec)
- break;
-
- /* check whether Attribute Record Header is within bounds */
- if ((u8 *)a > mrec_end ||
- (u8 *)a + sizeof(ATTR_RECORD) > mrec_end)
- break;
-
- /* check whether ATTR_RECORD's name is within bounds */
- name_end = (u8 *)a + le16_to_cpu(a->name_offset) +
- a->name_length * sizeof(ntfschar);
- if (name_end > mrec_end)
- break;
-
- ctx->attr = a;
- if (unlikely(le32_to_cpu(a->type) > le32_to_cpu(type) ||
- a->type == AT_END))
- return -ENOENT;
- if (unlikely(!a->length))
- break;
-
- /* check whether ATTR_RECORD's length wrap */
- if ((u8 *)a + le32_to_cpu(a->length) < (u8 *)a)
- break;
- /* check whether ATTR_RECORD's length is within bounds */
- if ((u8 *)a + le32_to_cpu(a->length) > mrec_end)
- break;
-
- if (a->type != type)
- continue;
- /*
- * If @name is present, compare the two names. If @name is
- * missing, assume we want an unnamed attribute.
- */
- if (!name) {
- /* The search failed if the found attribute is named. */
- if (a->name_length)
- return -ENOENT;
- } else if (!ntfs_are_names_equal(name, name_len,
- (ntfschar*)((u8*)a + le16_to_cpu(a->name_offset)),
- a->name_length, ic, upcase, upcase_len)) {
- register int rc;
-
- rc = ntfs_collate_names(name, name_len,
- (ntfschar*)((u8*)a +
- le16_to_cpu(a->name_offset)),
- a->name_length, 1, IGNORE_CASE,
- upcase, upcase_len);
- /*
- * If @name collates before a->name, there is no
- * matching attribute.
- */
- if (rc == -1)
- return -ENOENT;
- /* If the strings are not equal, continue search. */
- if (rc)
- continue;
- rc = ntfs_collate_names(name, name_len,
- (ntfschar*)((u8*)a +
- le16_to_cpu(a->name_offset)),
- a->name_length, 1, CASE_SENSITIVE,
- upcase, upcase_len);
- if (rc == -1)
- return -ENOENT;
- if (rc)
- continue;
- }
- /*
- * The names match or @name not present and attribute is
- * unnamed. If no @val specified, we have found the attribute
- * and are done.
- */
- if (!val)
- return 0;
- /* @val is present; compare values. */
- else {
- register int rc;
-
- rc = memcmp(val, (u8*)a + le16_to_cpu(
- a->data.resident.value_offset),
- min_t(u32, val_len, le32_to_cpu(
- a->data.resident.value_length)));
- /*
- * If @val collates before the current attribute's
- * value, there is no matching attribute.
- */
- if (!rc) {
- register u32 avl;
-
- avl = le32_to_cpu(
- a->data.resident.value_length);
- if (val_len == avl)
- return 0;
- if (val_len < avl)
- return -ENOENT;
- } else if (rc < 0)
- return -ENOENT;
- }
- }
- ntfs_error(vol->sb, "Inode is corrupt. Run chkdsk.");
- NVolSetErrors(vol);
- return -EIO;
-}
-
-/**
- * load_attribute_list - load an attribute list into memory
- * @vol: ntfs volume from which to read
- * @runlist: runlist of the attribute list
- * @al_start: destination buffer
- * @size: size of the destination buffer in bytes
- * @initialized_size: initialized size of the attribute list
- *
- * Walk the runlist @runlist and load all clusters from it copying them into
- * the linear buffer @al. The maximum number of bytes copied to @al is @size
- * bytes. Note, @size does not need to be a multiple of the cluster size. If
- * @initialized_size is less than @size, the region in @al between
- * @initialized_size and @size will be zeroed and not read from disk.
- *
- * Return 0 on success or -errno on error.
- */
-int load_attribute_list(ntfs_volume *vol, runlist *runlist, u8 *al_start,
- const s64 size, const s64 initialized_size)
-{
- LCN lcn;
- u8 *al = al_start;
- u8 *al_end = al + initialized_size;
- runlist_element *rl;
- struct buffer_head *bh;
- struct super_block *sb;
- unsigned long block_size;
- unsigned long block, max_block;
- int err = 0;
- unsigned char block_size_bits;
-
- ntfs_debug("Entering.");
- if (!vol || !runlist || !al || size <= 0 || initialized_size < 0 ||
- initialized_size > size)
- return -EINVAL;
- if (!initialized_size) {
- memset(al, 0, size);
- return 0;
- }
- sb = vol->sb;
- block_size = sb->s_blocksize;
- block_size_bits = sb->s_blocksize_bits;
- down_read(&runlist->lock);
- rl = runlist->rl;
- if (!rl) {
- ntfs_error(sb, "Cannot read attribute list since runlist is "
- "missing.");
- goto err_out;
- }
- /* Read all clusters specified by the runlist one run at a time. */
- while (rl->length) {
- lcn = ntfs_rl_vcn_to_lcn(rl, rl->vcn);
- ntfs_debug("Reading vcn = 0x%llx, lcn = 0x%llx.",
- (unsigned long long)rl->vcn,
- (unsigned long long)lcn);
- /* The attribute list cannot be sparse. */
- if (lcn < 0) {
- ntfs_error(sb, "ntfs_rl_vcn_to_lcn() failed. Cannot "
- "read attribute list.");
- goto err_out;
- }
- block = lcn << vol->cluster_size_bits >> block_size_bits;
- /* Read the run from device in chunks of block_size bytes. */
- max_block = block + (rl->length << vol->cluster_size_bits >>
- block_size_bits);
- ntfs_debug("max_block = 0x%lx.", max_block);
- do {
- ntfs_debug("Reading block = 0x%lx.", block);
- bh = sb_bread(sb, block);
- if (!bh) {
- ntfs_error(sb, "sb_bread() failed. Cannot "
- "read attribute list.");
- goto err_out;
- }
- if (al + block_size >= al_end)
- goto do_final;
- memcpy(al, bh->b_data, block_size);
- brelse(bh);
- al += block_size;
- } while (++block < max_block);
- rl++;
- }
- if (initialized_size < size) {
-initialize:
- memset(al_start + initialized_size, 0, size - initialized_size);
- }
-done:
- up_read(&runlist->lock);
- return err;
-do_final:
- if (al < al_end) {
- /*
- * Partial block.
- *
- * Note: The attribute list can be smaller than its allocation
- * by multiple clusters. This has been encountered by at least
- * two people running Windows XP, thus we cannot do any
- * truncation sanity checking here. (AIA)
- */
- memcpy(al, bh->b_data, al_end - al);
- brelse(bh);
- if (initialized_size < size)
- goto initialize;
- goto done;
- }
- brelse(bh);
- /* Real overflow! */
- ntfs_error(sb, "Attribute list buffer overflow. Read attribute list "
- "is truncated.");
-err_out:
- err = -EIO;
- goto done;
-}
-
-/**
- * ntfs_external_attr_find - find an attribute in the attribute list of an inode
- * @type: attribute type to find
- * @name: attribute name to find (optional, i.e. NULL means don't care)
- * @name_len: attribute name length (only needed if @name present)
- * @ic: IGNORE_CASE or CASE_SENSITIVE (ignored if @name not present)
- * @lowest_vcn: lowest vcn to find (optional, non-resident attributes only)
- * @val: attribute value to find (optional, resident attributes only)
- * @val_len: attribute value length
- * @ctx: search context with mft record and attribute to search from
- *
- * You should not need to call this function directly. Use ntfs_attr_lookup()
- * instead.
- *
- * Find an attribute by searching the attribute list for the corresponding
- * attribute list entry. Having found the entry, map the mft record if the
- * attribute is in a different mft record/inode, ntfs_attr_find() the attribute
- * in there and return it.
- *
- * On first search @ctx->ntfs_ino must be the base mft record and @ctx must
- * have been obtained from a call to ntfs_attr_get_search_ctx(). On subsequent
- * calls @ctx->ntfs_ino can be any extent inode, too (@ctx->base_ntfs_ino is
- * then the base inode).
- *
- * After finishing with the attribute/mft record you need to call
- * ntfs_attr_put_search_ctx() to cleanup the search context (unmapping any
- * mapped inodes, etc).
- *
- * If the attribute is found, ntfs_external_attr_find() returns 0 and
- * @ctx->attr will point to the found attribute. @ctx->mrec will point to the
- * mft record in which @ctx->attr is located and @ctx->al_entry will point to
- * the attribute list entry for the attribute.
- *
- * If the attribute is not found, ntfs_external_attr_find() returns -ENOENT and
- * @ctx->attr will point to the attribute in the base mft record before which
- * the attribute being searched for would need to be inserted if such an action
- * were to be desired. @ctx->mrec will point to the mft record in which
- * @ctx->attr is located and @ctx->al_entry will point to the attribute list
- * entry of the attribute before which the attribute being searched for would
- * need to be inserted if such an action were to be desired.
- *
- * Thus to insert the not found attribute, one wants to add the attribute to
- * @ctx->mrec (the base mft record) and if there is not enough space, the
- * attribute should be placed in a newly allocated extent mft record. The
- * attribute list entry for the inserted attribute should be inserted in the
- * attribute list attribute at @ctx->al_entry.
- *
- * On actual error, ntfs_external_attr_find() returns -EIO. In this case
- * @ctx->attr is undefined and in particular do not rely on it not changing.
- */
-static int ntfs_external_attr_find(const ATTR_TYPE type,
- const ntfschar *name, const u32 name_len,
- const IGNORE_CASE_BOOL ic, const VCN lowest_vcn,
- const u8 *val, const u32 val_len, ntfs_attr_search_ctx *ctx)
-{
- ntfs_inode *base_ni, *ni;
- ntfs_volume *vol;
- ATTR_LIST_ENTRY *al_entry, *next_al_entry;
- u8 *al_start, *al_end;
- ATTR_RECORD *a;
- ntfschar *al_name;
- u32 al_name_len;
- int err = 0;
- static const char *es = " Unmount and run chkdsk.";
-
- ni = ctx->ntfs_ino;
- base_ni = ctx->base_ntfs_ino;
- ntfs_debug("Entering for inode 0x%lx, type 0x%x.", ni->mft_no, type);
- if (!base_ni) {
- /* First call happens with the base mft record. */
- base_ni = ctx->base_ntfs_ino = ctx->ntfs_ino;
- ctx->base_mrec = ctx->mrec;
- }
- if (ni == base_ni)
- ctx->base_attr = ctx->attr;
- if (type == AT_END)
- goto not_found;
- vol = base_ni->vol;
- al_start = base_ni->attr_list;
- al_end = al_start + base_ni->attr_list_size;
- if (!ctx->al_entry)
- ctx->al_entry = (ATTR_LIST_ENTRY*)al_start;
- /*
- * Iterate over entries in attribute list starting at @ctx->al_entry,
- * or the entry following that, if @ctx->is_first is 'true'.
- */
- if (ctx->is_first) {
- al_entry = ctx->al_entry;
- ctx->is_first = false;
- } else
- al_entry = (ATTR_LIST_ENTRY*)((u8*)ctx->al_entry +
- le16_to_cpu(ctx->al_entry->length));
- for (;; al_entry = next_al_entry) {
- /* Out of bounds check. */
- if ((u8*)al_entry < base_ni->attr_list ||
- (u8*)al_entry > al_end)
- break; /* Inode is corrupt. */
- ctx->al_entry = al_entry;
- /* Catch the end of the attribute list. */
- if ((u8*)al_entry == al_end)
- goto not_found;
- if (!al_entry->length)
- break;
- if ((u8*)al_entry + 6 > al_end || (u8*)al_entry +
- le16_to_cpu(al_entry->length) > al_end)
- break;
- next_al_entry = (ATTR_LIST_ENTRY*)((u8*)al_entry +
- le16_to_cpu(al_entry->length));
- if (le32_to_cpu(al_entry->type) > le32_to_cpu(type))
- goto not_found;
- if (type != al_entry->type)
- continue;
- /*
- * If @name is present, compare the two names. If @name is
- * missing, assume we want an unnamed attribute.
- */
- al_name_len = al_entry->name_length;
- al_name = (ntfschar*)((u8*)al_entry + al_entry->name_offset);
- if (!name) {
- if (al_name_len)
- goto not_found;
- } else if (!ntfs_are_names_equal(al_name, al_name_len, name,
- name_len, ic, vol->upcase, vol->upcase_len)) {
- register int rc;
-
- rc = ntfs_collate_names(name, name_len, al_name,
- al_name_len, 1, IGNORE_CASE,
- vol->upcase, vol->upcase_len);
- /*
- * If @name collates before al_name, there is no
- * matching attribute.
- */
- if (rc == -1)
- goto not_found;
- /* If the strings are not equal, continue search. */
- if (rc)
- continue;
- /*
- * FIXME: Reverse engineering showed 0, IGNORE_CASE but
- * that is inconsistent with ntfs_attr_find(). The
- * subsequent rc checks were also different. Perhaps I
- * made a mistake in one of the two. Need to recheck
- * which is correct or at least see what is going on...
- * (AIA)
- */
- rc = ntfs_collate_names(name, name_len, al_name,
- al_name_len, 1, CASE_SENSITIVE,
- vol->upcase, vol->upcase_len);
- if (rc == -1)
- goto not_found;
- if (rc)
- continue;
- }
- /*
- * The names match or @name not present and attribute is
- * unnamed. Now check @lowest_vcn. Continue search if the
- * next attribute list entry still fits @lowest_vcn. Otherwise
- * we have reached the right one or the search has failed.
- */
- if (lowest_vcn && (u8*)next_al_entry >= al_start &&
- (u8*)next_al_entry + 6 < al_end &&
- (u8*)next_al_entry + le16_to_cpu(
- next_al_entry->length) <= al_end &&
- sle64_to_cpu(next_al_entry->lowest_vcn) <=
- lowest_vcn &&
- next_al_entry->type == al_entry->type &&
- next_al_entry->name_length == al_name_len &&
- ntfs_are_names_equal((ntfschar*)((u8*)
- next_al_entry +
- next_al_entry->name_offset),
- next_al_entry->name_length,
- al_name, al_name_len, CASE_SENSITIVE,
- vol->upcase, vol->upcase_len))
- continue;
- if (MREF_LE(al_entry->mft_reference) == ni->mft_no) {
- if (MSEQNO_LE(al_entry->mft_reference) != ni->seq_no) {
- ntfs_error(vol->sb, "Found stale mft "
- "reference in attribute list "
- "of base inode 0x%lx.%s",
- base_ni->mft_no, es);
- err = -EIO;
- break;
- }
- } else { /* Mft references do not match. */
- /* If there is a mapped record unmap it first. */
- if (ni != base_ni)
- unmap_extent_mft_record(ni);
- /* Do we want the base record back? */
- if (MREF_LE(al_entry->mft_reference) ==
- base_ni->mft_no) {
- ni = ctx->ntfs_ino = base_ni;
- ctx->mrec = ctx->base_mrec;
- } else {
- /* We want an extent record. */
- ctx->mrec = map_extent_mft_record(base_ni,
- le64_to_cpu(
- al_entry->mft_reference), &ni);
- if (IS_ERR(ctx->mrec)) {
- ntfs_error(vol->sb, "Failed to map "
- "extent mft record "
- "0x%lx of base inode "
- "0x%lx.%s",
- MREF_LE(al_entry->
- mft_reference),
- base_ni->mft_no, es);
- err = PTR_ERR(ctx->mrec);
- if (err == -ENOENT)
- err = -EIO;
- /* Cause @ctx to be sanitized below. */
- ni = NULL;
- break;
- }
- ctx->ntfs_ino = ni;
- }
- ctx->attr = (ATTR_RECORD*)((u8*)ctx->mrec +
- le16_to_cpu(ctx->mrec->attrs_offset));
- }
- /*
- * ctx->vfs_ino, ctx->mrec, and ctx->attr now point to the
- * mft record containing the attribute represented by the
- * current al_entry.
- */
- /*
- * We could call into ntfs_attr_find() to find the right
- * attribute in this mft record but this would be less
- * efficient and not quite accurate as ntfs_attr_find() ignores
- * the attribute instance numbers for example which become
- * important when one plays with attribute lists. Also,
- * because a proper match has been found in the attribute list
- * entry above, the comparison can now be optimized. So it is
- * worth re-implementing a simplified ntfs_attr_find() here.
- */
- a = ctx->attr;
- /*
- * Use a manual loop so we can still use break and continue
- * with the same meanings as above.
- */
-do_next_attr_loop:
- if ((u8*)a < (u8*)ctx->mrec || (u8*)a > (u8*)ctx->mrec +
- le32_to_cpu(ctx->mrec->bytes_allocated))
- break;
- if (a->type == AT_END)
- break;
- if (!a->length)
- break;
- if (al_entry->instance != a->instance)
- goto do_next_attr;
- /*
- * If the type and/or the name are mismatched between the
- * attribute list entry and the attribute record, there is
- * corruption so we break and return error EIO.
- */
- if (al_entry->type != a->type)
- break;
- if (!ntfs_are_names_equal((ntfschar*)((u8*)a +
- le16_to_cpu(a->name_offset)), a->name_length,
- al_name, al_name_len, CASE_SENSITIVE,
- vol->upcase, vol->upcase_len))
- break;
- ctx->attr = a;
- /*
- * If no @val specified or @val specified and it matches, we
- * have found it!
- */
- if (!val || (!a->non_resident && le32_to_cpu(
- a->data.resident.value_length) == val_len &&
- !memcmp((u8*)a +
- le16_to_cpu(a->data.resident.value_offset),
- val, val_len))) {
- ntfs_debug("Done, found.");
- return 0;
- }
-do_next_attr:
- /* Proceed to the next attribute in the current mft record. */
- a = (ATTR_RECORD*)((u8*)a + le32_to_cpu(a->length));
- goto do_next_attr_loop;
- }
- if (!err) {
- ntfs_error(vol->sb, "Base inode 0x%lx contains corrupt "
- "attribute list attribute.%s", base_ni->mft_no,
- es);
- err = -EIO;
- }
- if (ni != base_ni) {
- if (ni)
- unmap_extent_mft_record(ni);
- ctx->ntfs_ino = base_ni;
- ctx->mrec = ctx->base_mrec;
- ctx->attr = ctx->base_attr;
- }
- if (err != -ENOMEM)
- NVolSetErrors(vol);
- return err;
-not_found:
- /*
- * If we were looking for AT_END, we reset the search context @ctx and
- * use ntfs_attr_find() to seek to the end of the base mft record.
- */
- if (type == AT_END) {
- ntfs_attr_reinit_search_ctx(ctx);
- return ntfs_attr_find(AT_END, name, name_len, ic, val, val_len,
- ctx);
- }
- /*
- * The attribute was not found. Before we return, we want to ensure
- * @ctx->mrec and @ctx->attr indicate the position at which the
- * attribute should be inserted in the base mft record. Since we also
- * want to preserve @ctx->al_entry we cannot reinitialize the search
- * context using ntfs_attr_reinit_search_ctx() as this would set
- * @ctx->al_entry to NULL. Thus we do the necessary bits manually (see
- * ntfs_attr_init_search_ctx() below). Note, we _only_ preserve
- * @ctx->al_entry as the remaining fields (base_*) are identical to
- * their non base_ counterparts and we cannot set @ctx->base_attr
- * correctly yet as we do not know what @ctx->attr will be set to by
- * the call to ntfs_attr_find() below.
- */
- if (ni != base_ni)
- unmap_extent_mft_record(ni);
- ctx->mrec = ctx->base_mrec;
- ctx->attr = (ATTR_RECORD*)((u8*)ctx->mrec +
- le16_to_cpu(ctx->mrec->attrs_offset));
- ctx->is_first = true;
- ctx->ntfs_ino = base_ni;
- ctx->base_ntfs_ino = NULL;
- ctx->base_mrec = NULL;
- ctx->base_attr = NULL;
- /*
- * In case there are multiple matches in the base mft record, need to
- * keep enumerating until we get an attribute not found response (or
- * another error), otherwise we would keep returning the same attribute
- * over and over again and all programs using us for enumeration would
- * lock up in a tight loop.
- */
- do {
- err = ntfs_attr_find(type, name, name_len, ic, val, val_len,
- ctx);
- } while (!err);
- ntfs_debug("Done, not found.");
- return err;
-}
-
-/**
- * ntfs_attr_lookup - find an attribute in an ntfs inode
- * @type: attribute type to find
- * @name: attribute name to find (optional, i.e. NULL means don't care)
- * @name_len: attribute name length (only needed if @name present)
- * @ic: IGNORE_CASE or CASE_SENSITIVE (ignored if @name not present)
- * @lowest_vcn: lowest vcn to find (optional, non-resident attributes only)
- * @val: attribute value to find (optional, resident attributes only)
- * @val_len: attribute value length
- * @ctx: search context with mft record and attribute to search from
- *
- * Find an attribute in an ntfs inode. On first search @ctx->ntfs_ino must
- * be the base mft record and @ctx must have been obtained from a call to
- * ntfs_attr_get_search_ctx().
- *
- * This function transparently handles attribute lists and @ctx is used to
- * continue searches where they were left off at.
- *
- * After finishing with the attribute/mft record you need to call
- * ntfs_attr_put_search_ctx() to cleanup the search context (unmapping any
- * mapped inodes, etc).
- *
- * Return 0 if the search was successful and -errno if not.
- *
- * When 0, @ctx->attr is the found attribute and it is in mft record
- * @ctx->mrec. If an attribute list attribute is present, @ctx->al_entry is
- * the attribute list entry of the found attribute.
- *
- * When -ENOENT, @ctx->attr is the attribute which collates just after the
- * attribute being searched for, i.e. if one wants to add the attribute to the
- * mft record this is the correct place to insert it into. If an attribute
- * list attribute is present, @ctx->al_entry is the attribute list entry which
- * collates just after the attribute list entry of the attribute being searched
- * for, i.e. if one wants to add the attribute to the mft record this is the
- * correct place to insert its attribute list entry into.
- *
- * When -errno != -ENOENT, an error occurred during the lookup. @ctx->attr is
- * then undefined and in particular you should not rely on it not changing.
- */
-int ntfs_attr_lookup(const ATTR_TYPE type, const ntfschar *name,
- const u32 name_len, const IGNORE_CASE_BOOL ic,
- const VCN lowest_vcn, const u8 *val, const u32 val_len,
- ntfs_attr_search_ctx *ctx)
-{
- ntfs_inode *base_ni;
-
- ntfs_debug("Entering.");
- BUG_ON(IS_ERR(ctx->mrec));
- if (ctx->base_ntfs_ino)
- base_ni = ctx->base_ntfs_ino;
- else
- base_ni = ctx->ntfs_ino;
- /* Sanity check, just for debugging really. */
- BUG_ON(!base_ni);
- if (!NInoAttrList(base_ni) || type == AT_ATTRIBUTE_LIST)
- return ntfs_attr_find(type, name, name_len, ic, val, val_len,
- ctx);
- return ntfs_external_attr_find(type, name, name_len, ic, lowest_vcn,
- val, val_len, ctx);
-}
-
-/**
- * ntfs_attr_init_search_ctx - initialize an attribute search context
- * @ctx: attribute search context to initialize
- * @ni: ntfs inode with which to initialize the search context
- * @mrec: mft record with which to initialize the search context
- *
- * Initialize the attribute search context @ctx with @ni and @mrec.
- */
-static inline void ntfs_attr_init_search_ctx(ntfs_attr_search_ctx *ctx,
- ntfs_inode *ni, MFT_RECORD *mrec)
-{
- *ctx = (ntfs_attr_search_ctx) {
- .mrec = mrec,
- /* Sanity checks are performed elsewhere. */
- .attr = (ATTR_RECORD*)((u8*)mrec +
- le16_to_cpu(mrec->attrs_offset)),
- .is_first = true,
- .ntfs_ino = ni,
- };
-}
-
-/**
- * ntfs_attr_reinit_search_ctx - reinitialize an attribute search context
- * @ctx: attribute search context to reinitialize
- *
- * Reinitialize the attribute search context @ctx, unmapping an associated
- * extent mft record if present, and initialize the search context again.
- *
- * This is used when a search for a new attribute is being started to reset
- * the search context to the beginning.
- */
-void ntfs_attr_reinit_search_ctx(ntfs_attr_search_ctx *ctx)
-{
- if (likely(!ctx->base_ntfs_ino)) {
- /* No attribute list. */
- ctx->is_first = true;
- /* Sanity checks are performed elsewhere. */
- ctx->attr = (ATTR_RECORD*)((u8*)ctx->mrec +
- le16_to_cpu(ctx->mrec->attrs_offset));
- /*
- * This needs resetting due to ntfs_external_attr_find() which
- * can leave it set despite having zeroed ctx->base_ntfs_ino.
- */
- ctx->al_entry = NULL;
- return;
- } /* Attribute list. */
- if (ctx->ntfs_ino != ctx->base_ntfs_ino)
- unmap_extent_mft_record(ctx->ntfs_ino);
- ntfs_attr_init_search_ctx(ctx, ctx->base_ntfs_ino, ctx->base_mrec);
- return;
-}
-
-/**
- * ntfs_attr_get_search_ctx - allocate/initialize a new attribute search context
- * @ni: ntfs inode with which to initialize the search context
- * @mrec: mft record with which to initialize the search context
- *
- * Allocate a new attribute search context, initialize it with @ni and @mrec,
- * and return it. Return NULL if allocation failed.
- */
-ntfs_attr_search_ctx *ntfs_attr_get_search_ctx(ntfs_inode *ni, MFT_RECORD *mrec)
-{
- ntfs_attr_search_ctx *ctx;
-
- ctx = kmem_cache_alloc(ntfs_attr_ctx_cache, GFP_NOFS);
- if (ctx)
- ntfs_attr_init_search_ctx(ctx, ni, mrec);
- return ctx;
-}
-
-/**
- * ntfs_attr_put_search_ctx - release an attribute search context
- * @ctx: attribute search context to free
- *
- * Release the attribute search context @ctx, unmapping an associated extent
- * mft record if present.
- */
-void ntfs_attr_put_search_ctx(ntfs_attr_search_ctx *ctx)
-{
- if (ctx->base_ntfs_ino && ctx->ntfs_ino != ctx->base_ntfs_ino)
- unmap_extent_mft_record(ctx->ntfs_ino);
- kmem_cache_free(ntfs_attr_ctx_cache, ctx);
- return;
-}
-
-#ifdef NTFS_RW
-
-/**
- * ntfs_attr_find_in_attrdef - find an attribute in the $AttrDef system file
- * @vol: ntfs volume to which the attribute belongs
- * @type: attribute type which to find
- *
- * Search for the attribute definition record corresponding to the attribute
- * @type in the $AttrDef system file.
- *
- * Return the attribute type definition record if found and NULL if not found.
- */
-static ATTR_DEF *ntfs_attr_find_in_attrdef(const ntfs_volume *vol,
- const ATTR_TYPE type)
-{
- ATTR_DEF *ad;
-
- BUG_ON(!vol->attrdef);
- BUG_ON(!type);
- for (ad = vol->attrdef; (u8*)ad - (u8*)vol->attrdef <
- vol->attrdef_size && ad->type; ++ad) {
- /* We have not found it yet, carry on searching. */
- if (likely(le32_to_cpu(ad->type) < le32_to_cpu(type)))
- continue;
- /* We found the attribute; return it. */
- if (likely(ad->type == type))
- return ad;
- /* We have gone too far already. No point in continuing. */
- break;
- }
- /* Attribute not found. */
- ntfs_debug("Attribute type 0x%x not found in $AttrDef.",
- le32_to_cpu(type));
- return NULL;
-}
-
-/**
- * ntfs_attr_size_bounds_check - check a size of an attribute type for validity
- * @vol: ntfs volume to which the attribute belongs
- * @type: attribute type which to check
- * @size: size which to check
- *
- * Check whether the @size in bytes is valid for an attribute of @type on the
- * ntfs volume @vol. This information is obtained from $AttrDef system file.
- *
- * Return 0 if valid, -ERANGE if not valid, or -ENOENT if the attribute is not
- * listed in $AttrDef.
- */
-int ntfs_attr_size_bounds_check(const ntfs_volume *vol, const ATTR_TYPE type,
- const s64 size)
-{
- ATTR_DEF *ad;
-
- BUG_ON(size < 0);
- /*
- * $ATTRIBUTE_LIST has a maximum size of 256kiB, but this is not
- * listed in $AttrDef.
- */
- if (unlikely(type == AT_ATTRIBUTE_LIST && size > 256 * 1024))
- return -ERANGE;
- /* Get the $AttrDef entry for the attribute @type. */
- ad = ntfs_attr_find_in_attrdef(vol, type);
- if (unlikely(!ad))
- return -ENOENT;
- /* Do the bounds check. */
- if (((sle64_to_cpu(ad->min_size) > 0) &&
- size < sle64_to_cpu(ad->min_size)) ||
- ((sle64_to_cpu(ad->max_size) > 0) && size >
- sle64_to_cpu(ad->max_size)))
- return -ERANGE;
- return 0;
-}
-
-/**
- * ntfs_attr_can_be_non_resident - check if an attribute can be non-resident
- * @vol: ntfs volume to which the attribute belongs
- * @type: attribute type which to check
- *
- * Check whether the attribute of @type on the ntfs volume @vol is allowed to
- * be non-resident. This information is obtained from $AttrDef system file.
- *
- * Return 0 if the attribute is allowed to be non-resident, -EPERM if not, and
- * -ENOENT if the attribute is not listed in $AttrDef.
- */
-int ntfs_attr_can_be_non_resident(const ntfs_volume *vol, const ATTR_TYPE type)
-{
- ATTR_DEF *ad;
-
- /* Find the attribute definition record in $AttrDef. */
- ad = ntfs_attr_find_in_attrdef(vol, type);
- if (unlikely(!ad))
- return -ENOENT;
- /* Check the flags and return the result. */
- if (ad->flags & ATTR_DEF_RESIDENT)
- return -EPERM;
- return 0;
-}
-
-/**
- * ntfs_attr_can_be_resident - check if an attribute can be resident
- * @vol: ntfs volume to which the attribute belongs
- * @type: attribute type which to check
- *
- * Check whether the attribute of @type on the ntfs volume @vol is allowed to
- * be resident. This information is derived from our ntfs knowledge and may
- * not be completely accurate, especially when user defined attributes are
- * present. Basically we allow everything to be resident except for index
- * allocation and $EA attributes.
- *
- * Return 0 if the attribute is allowed to be non-resident and -EPERM if not.
- *
- * Warning: In the system file $MFT the attribute $Bitmap must be non-resident
- * otherwise windows will not boot (blue screen of death)! We cannot
- * check for this here as we do not know which inode's $Bitmap is
- * being asked about so the caller needs to special case this.
- */
-int ntfs_attr_can_be_resident(const ntfs_volume *vol, const ATTR_TYPE type)
-{
- if (type == AT_INDEX_ALLOCATION)
- return -EPERM;
- return 0;
-}
-
-/**
- * ntfs_attr_record_resize - resize an attribute record
- * @m: mft record containing attribute record
- * @a: attribute record to resize
- * @new_size: new size in bytes to which to resize the attribute record @a
- *
- * Resize the attribute record @a, i.e. the resident part of the attribute, in
- * the mft record @m to @new_size bytes.
- *
- * Return 0 on success and -errno on error. The following error codes are
- * defined:
- * -ENOSPC - Not enough space in the mft record @m to perform the resize.
- *
- * Note: On error, no modifications have been performed whatsoever.
- *
- * Warning: If you make a record smaller without having copied all the data you
- * are interested in the data may be overwritten.
- */
-int ntfs_attr_record_resize(MFT_RECORD *m, ATTR_RECORD *a, u32 new_size)
-{
- ntfs_debug("Entering for new_size %u.", new_size);
- /* Align to 8 bytes if it is not already done. */
- if (new_size & 7)
- new_size = (new_size + 7) & ~7;
- /* If the actual attribute length has changed, move things around. */
- if (new_size != le32_to_cpu(a->length)) {
- u32 new_muse = le32_to_cpu(m->bytes_in_use) -
- le32_to_cpu(a->length) + new_size;
- /* Not enough space in this mft record. */
- if (new_muse > le32_to_cpu(m->bytes_allocated))
- return -ENOSPC;
- /* Move attributes following @a to their new location. */
- memmove((u8*)a + new_size, (u8*)a + le32_to_cpu(a->length),
- le32_to_cpu(m->bytes_in_use) - ((u8*)a -
- (u8*)m) - le32_to_cpu(a->length));
- /* Adjust @m to reflect the change in used space. */
- m->bytes_in_use = cpu_to_le32(new_muse);
- /* Adjust @a to reflect the new size. */
- if (new_size >= offsetof(ATTR_REC, length) + sizeof(a->length))
- a->length = cpu_to_le32(new_size);
- }
- return 0;
-}
-
-/**
- * ntfs_resident_attr_value_resize - resize the value of a resident attribute
- * @m: mft record containing attribute record
- * @a: attribute record whose value to resize
- * @new_size: new size in bytes to which to resize the attribute value of @a
- *
- * Resize the value of the attribute @a in the mft record @m to @new_size bytes.
- * If the value is made bigger, the newly allocated space is cleared.
- *
- * Return 0 on success and -errno on error. The following error codes are
- * defined:
- * -ENOSPC - Not enough space in the mft record @m to perform the resize.
- *
- * Note: On error, no modifications have been performed whatsoever.
- *
- * Warning: If you make a record smaller without having copied all the data you
- * are interested in the data may be overwritten.
- */
-int ntfs_resident_attr_value_resize(MFT_RECORD *m, ATTR_RECORD *a,
- const u32 new_size)
-{
- u32 old_size;
-
- /* Resize the resident part of the attribute record. */
- if (ntfs_attr_record_resize(m, a,
- le16_to_cpu(a->data.resident.value_offset) + new_size))
- return -ENOSPC;
- /*
- * The resize succeeded! If we made the attribute value bigger, clear
- * the area between the old size and @new_size.
- */
- old_size = le32_to_cpu(a->data.resident.value_length);
- if (new_size > old_size)
- memset((u8*)a + le16_to_cpu(a->data.resident.value_offset) +
- old_size, 0, new_size - old_size);
- /* Finally update the length of the attribute value. */
- a->data.resident.value_length = cpu_to_le32(new_size);
- return 0;
-}
-
-/**
- * ntfs_attr_make_non_resident - convert a resident to a non-resident attribute
- * @ni: ntfs inode describing the attribute to convert
- * @data_size: size of the resident data to copy to the non-resident attribute
- *
- * Convert the resident ntfs attribute described by the ntfs inode @ni to a
- * non-resident one.
- *
- * @data_size must be equal to the attribute value size. This is needed since
- * we need to know the size before we can map the mft record and our callers
- * always know it. The reason we cannot simply read the size from the vfs
- * inode i_size is that this is not necessarily uptodate. This happens when
- * ntfs_attr_make_non_resident() is called in the ->truncate call path(s).
- *
- * Return 0 on success and -errno on error. The following error return codes
- * are defined:
- * -EPERM - The attribute is not allowed to be non-resident.
- * -ENOMEM - Not enough memory.
- * -ENOSPC - Not enough disk space.
- * -EINVAL - Attribute not defined on the volume.
- * -EIO - I/o error or other error.
- * Note that -ENOSPC is also returned in the case that there is not enough
- * space in the mft record to do the conversion. This can happen when the mft
- * record is already very full. The caller is responsible for trying to make
- * space in the mft record and trying again. FIXME: Do we need a separate
- * error return code for this kind of -ENOSPC or is it always worth trying
- * again in case the attribute may then fit in a resident state so no need to
- * make it non-resident at all? Ho-hum... (AIA)
- *
- * NOTE to self: No changes in the attribute list are required to move from
- * a resident to a non-resident attribute.
- *
- * Locking: - The caller must hold i_mutex on the inode.
- */
-int ntfs_attr_make_non_resident(ntfs_inode *ni, const u32 data_size)
-{
- s64 new_size;
- struct inode *vi = VFS_I(ni);
- ntfs_volume *vol = ni->vol;
- ntfs_inode *base_ni;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- ntfs_attr_search_ctx *ctx;
- struct page *page;
- runlist_element *rl;
- u8 *kaddr;
- unsigned long flags;
- int mp_size, mp_ofs, name_ofs, arec_size, err, err2;
- u32 attr_size;
- u8 old_res_attr_flags;
-
- /* Check that the attribute is allowed to be non-resident. */
- err = ntfs_attr_can_be_non_resident(vol, ni->type);
- if (unlikely(err)) {
- if (err == -EPERM)
- ntfs_debug("Attribute is not allowed to be "
- "non-resident.");
- else
- ntfs_debug("Attribute not defined on the NTFS "
- "volume!");
- return err;
- }
- /*
- * FIXME: Compressed and encrypted attributes are not supported when
- * writing and we should never have gotten here for them.
- */
- BUG_ON(NInoCompressed(ni));
- BUG_ON(NInoEncrypted(ni));
- /*
- * The size needs to be aligned to a cluster boundary for allocation
- * purposes.
- */
- new_size = (data_size + vol->cluster_size - 1) &
- ~(vol->cluster_size - 1);
- if (new_size > 0) {
- /*
- * Will need the page later and since the page lock nests
- * outside all ntfs locks, we need to get the page now.
- */
- page = find_or_create_page(vi->i_mapping, 0,
- mapping_gfp_mask(vi->i_mapping));
- if (unlikely(!page))
- return -ENOMEM;
- /* Start by allocating clusters to hold the attribute value. */
- rl = ntfs_cluster_alloc(vol, 0, new_size >>
- vol->cluster_size_bits, -1, DATA_ZONE, true);
- if (IS_ERR(rl)) {
- err = PTR_ERR(rl);
- ntfs_debug("Failed to allocate cluster%s, error code "
- "%i.", (new_size >>
- vol->cluster_size_bits) > 1 ? "s" : "",
- err);
- goto page_err_out;
- }
- } else {
- rl = NULL;
- page = NULL;
- }
- /* Determine the size of the mapping pairs array. */
- mp_size = ntfs_get_size_for_mapping_pairs(vol, rl, 0, -1);
- if (unlikely(mp_size < 0)) {
- err = mp_size;
- ntfs_debug("Failed to get size for mapping pairs array, error "
- "code %i.", err);
- goto rl_err_out;
- }
- down_write(&ni->runlist.lock);
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- ctx = NULL;
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- goto err_out;
- }
- m = ctx->mrec;
- a = ctx->attr;
- BUG_ON(NInoNonResident(ni));
- BUG_ON(a->non_resident);
- /*
- * Calculate new offsets for the name and the mapping pairs array.
- */
- if (NInoSparse(ni) || NInoCompressed(ni))
- name_ofs = (offsetof(ATTR_REC,
- data.non_resident.compressed_size) +
- sizeof(a->data.non_resident.compressed_size) +
- 7) & ~7;
- else
- name_ofs = (offsetof(ATTR_REC,
- data.non_resident.compressed_size) + 7) & ~7;
- mp_ofs = (name_ofs + a->name_length * sizeof(ntfschar) + 7) & ~7;
- /*
- * Determine the size of the resident part of the now non-resident
- * attribute record.
- */
- arec_size = (mp_ofs + mp_size + 7) & ~7;
- /*
- * If the page is not uptodate bring it uptodate by copying from the
- * attribute value.
- */
- attr_size = le32_to_cpu(a->data.resident.value_length);
- BUG_ON(attr_size != data_size);
- if (page && !PageUptodate(page)) {
- kaddr = kmap_atomic(page);
- memcpy(kaddr, (u8*)a +
- le16_to_cpu(a->data.resident.value_offset),
- attr_size);
- memset(kaddr + attr_size, 0, PAGE_SIZE - attr_size);
- kunmap_atomic(kaddr);
- flush_dcache_page(page);
- SetPageUptodate(page);
- }
- /* Backup the attribute flag. */
- old_res_attr_flags = a->data.resident.flags;
- /* Resize the resident part of the attribute record. */
- err = ntfs_attr_record_resize(m, a, arec_size);
- if (unlikely(err))
- goto err_out;
- /*
- * Convert the resident part of the attribute record to describe a
- * non-resident attribute.
- */
- a->non_resident = 1;
- /* Move the attribute name if it exists and update the offset. */
- if (a->name_length)
- memmove((u8*)a + name_ofs, (u8*)a + le16_to_cpu(a->name_offset),
- a->name_length * sizeof(ntfschar));
- a->name_offset = cpu_to_le16(name_ofs);
- /* Setup the fields specific to non-resident attributes. */
- a->data.non_resident.lowest_vcn = 0;
- a->data.non_resident.highest_vcn = cpu_to_sle64((new_size - 1) >>
- vol->cluster_size_bits);
- a->data.non_resident.mapping_pairs_offset = cpu_to_le16(mp_ofs);
- memset(&a->data.non_resident.reserved, 0,
- sizeof(a->data.non_resident.reserved));
- a->data.non_resident.allocated_size = cpu_to_sle64(new_size);
- a->data.non_resident.data_size =
- a->data.non_resident.initialized_size =
- cpu_to_sle64(attr_size);
- if (NInoSparse(ni) || NInoCompressed(ni)) {
- a->data.non_resident.compression_unit = 0;
- if (NInoCompressed(ni) || vol->major_ver < 3)
- a->data.non_resident.compression_unit = 4;
- a->data.non_resident.compressed_size =
- a->data.non_resident.allocated_size;
- } else
- a->data.non_resident.compression_unit = 0;
- /* Generate the mapping pairs array into the attribute record. */
- err = ntfs_mapping_pairs_build(vol, (u8*)a + mp_ofs,
- arec_size - mp_ofs, rl, 0, -1, NULL);
- if (unlikely(err)) {
- ntfs_debug("Failed to build mapping pairs, error code %i.",
- err);
- goto undo_err_out;
- }
- /* Setup the in-memory attribute structure to be non-resident. */
- ni->runlist.rl = rl;
- write_lock_irqsave(&ni->size_lock, flags);
- ni->allocated_size = new_size;
- if (NInoSparse(ni) || NInoCompressed(ni)) {
- ni->itype.compressed.size = ni->allocated_size;
- if (a->data.non_resident.compression_unit) {
- ni->itype.compressed.block_size = 1U << (a->data.
- non_resident.compression_unit +
- vol->cluster_size_bits);
- ni->itype.compressed.block_size_bits =
- ffs(ni->itype.compressed.block_size) -
- 1;
- ni->itype.compressed.block_clusters = 1U <<
- a->data.non_resident.compression_unit;
- } else {
- ni->itype.compressed.block_size = 0;
- ni->itype.compressed.block_size_bits = 0;
- ni->itype.compressed.block_clusters = 0;
- }
- vi->i_blocks = ni->itype.compressed.size >> 9;
- } else
- vi->i_blocks = ni->allocated_size >> 9;
- write_unlock_irqrestore(&ni->size_lock, flags);
- /*
- * This needs to be last since the address space operations ->read_folio
- * and ->writepage can run concurrently with us as they are not
- * serialized on i_mutex. Note, we are not allowed to fail once we flip
- * this switch, which is another reason to do this last.
- */
- NInoSetNonResident(ni);
- /* Mark the mft record dirty, so it gets written back. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
- if (page) {
- set_page_dirty(page);
- unlock_page(page);
- put_page(page);
- }
- ntfs_debug("Done.");
- return 0;
-undo_err_out:
- /* Convert the attribute back into a resident attribute. */
- a->non_resident = 0;
- /* Move the attribute name if it exists and update the offset. */
- name_ofs = (offsetof(ATTR_RECORD, data.resident.reserved) +
- sizeof(a->data.resident.reserved) + 7) & ~7;
- if (a->name_length)
- memmove((u8*)a + name_ofs, (u8*)a + le16_to_cpu(a->name_offset),
- a->name_length * sizeof(ntfschar));
- mp_ofs = (name_ofs + a->name_length * sizeof(ntfschar) + 7) & ~7;
- a->name_offset = cpu_to_le16(name_ofs);
- arec_size = (mp_ofs + attr_size + 7) & ~7;
- /* Resize the resident part of the attribute record. */
- err2 = ntfs_attr_record_resize(m, a, arec_size);
- if (unlikely(err2)) {
- /*
- * This cannot happen (well if memory corruption is at work it
- * could happen in theory), but deal with it as well as we can.
- * If the old size is too small, truncate the attribute,
- * otherwise simply give it a larger allocated size.
- * FIXME: Should check whether chkdsk complains when the
- * allocated size is much bigger than the resident value size.
- */
- arec_size = le32_to_cpu(a->length);
- if ((mp_ofs + attr_size) > arec_size) {
- err2 = attr_size;
- attr_size = arec_size - mp_ofs;
- ntfs_error(vol->sb, "Failed to undo partial resident "
- "to non-resident attribute "
- "conversion. Truncating inode 0x%lx, "
- "attribute type 0x%x from %i bytes to "
- "%i bytes to maintain metadata "
- "consistency. THIS MEANS YOU ARE "
- "LOSING %i BYTES DATA FROM THIS %s.",
- vi->i_ino,
- (unsigned)le32_to_cpu(ni->type),
- err2, attr_size, err2 - attr_size,
- ((ni->type == AT_DATA) &&
- !ni->name_len) ? "FILE": "ATTRIBUTE");
- write_lock_irqsave(&ni->size_lock, flags);
- ni->initialized_size = attr_size;
- i_size_write(vi, attr_size);
- write_unlock_irqrestore(&ni->size_lock, flags);
- }
- }
- /* Setup the fields specific to resident attributes. */
- a->data.resident.value_length = cpu_to_le32(attr_size);
- a->data.resident.value_offset = cpu_to_le16(mp_ofs);
- a->data.resident.flags = old_res_attr_flags;
- memset(&a->data.resident.reserved, 0,
- sizeof(a->data.resident.reserved));
- /* Copy the data from the page back to the attribute value. */
- if (page) {
- kaddr = kmap_atomic(page);
- memcpy((u8*)a + mp_ofs, kaddr, attr_size);
- kunmap_atomic(kaddr);
- }
- /* Setup the allocated size in the ntfs inode in case it changed. */
- write_lock_irqsave(&ni->size_lock, flags);
- ni->allocated_size = arec_size - mp_ofs;
- write_unlock_irqrestore(&ni->size_lock, flags);
- /* Mark the mft record dirty, so it gets written back. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
-err_out:
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
- ni->runlist.rl = NULL;
- up_write(&ni->runlist.lock);
-rl_err_out:
- if (rl) {
- if (ntfs_cluster_free_from_rl(vol, rl) < 0) {
- ntfs_error(vol->sb, "Failed to release allocated "
- "cluster(s) in error code path. Run "
- "chkdsk to recover the lost "
- "cluster(s).");
- NVolSetErrors(vol);
- }
- ntfs_free(rl);
-page_err_out:
- unlock_page(page);
- put_page(page);
- }
- if (err == -EINVAL)
- err = -EIO;
- return err;
-}
-
-/**
- * ntfs_attr_extend_allocation - extend the allocated space of an attribute
- * @ni: ntfs inode of the attribute whose allocation to extend
- * @new_alloc_size: new size in bytes to which to extend the allocation to
- * @new_data_size: new size in bytes to which to extend the data to
- * @data_start: beginning of region which is required to be non-sparse
- *
- * Extend the allocated space of an attribute described by the ntfs inode @ni
- * to @new_alloc_size bytes. If @data_start is -1, the whole extension may be
- * implemented as a hole in the file (as long as both the volume and the ntfs
- * inode @ni have sparse support enabled). If @data_start is >= 0, then the
- * region between the old allocated size and @data_start - 1 may be made sparse
- * but the regions between @data_start and @new_alloc_size must be backed by
- * actual clusters.
- *
- * If @new_data_size is -1, it is ignored. If it is >= 0, then the data size
- * of the attribute is extended to @new_data_size. Note that the i_size of the
- * vfs inode is not updated. Only the data size in the base attribute record
- * is updated. The caller has to update i_size separately if this is required.
- * WARNING: It is a BUG() for @new_data_size to be smaller than the old data
- * size as well as for @new_data_size to be greater than @new_alloc_size.
- *
- * For resident attributes this involves resizing the attribute record and if
- * necessary moving it and/or other attributes into extent mft records and/or
- * converting the attribute to a non-resident attribute which in turn involves
- * extending the allocation of a non-resident attribute as described below.
- *
- * For non-resident attributes this involves allocating clusters in the data
- * zone on the volume (except for regions that are being made sparse) and
- * extending the run list to describe the allocated clusters as well as
- * updating the mapping pairs array of the attribute. This in turn involves
- * resizing the attribute record and if necessary moving it and/or other
- * attributes into extent mft records and/or splitting the attribute record
- * into multiple extent attribute records.
- *
- * Also, the attribute list attribute is updated if present and in some of the
- * above cases (the ones where extent mft records/attributes come into play),
- * an attribute list attribute is created if not already present.
- *
- * Return the new allocated size on success and -errno on error. In the case
- * that an error is encountered but a partial extension at least up to
- * @data_start (if present) is possible, the allocation is partially extended
- * and this is returned. This means the caller must check the returned size to
- * determine if the extension was partial. If @data_start is -1 then partial
- * allocations are not performed.
- *
- * WARNING: Do not call ntfs_attr_extend_allocation() for $MFT/$DATA.
- *
- * Locking: This function takes the runlist lock of @ni for writing as well as
- * locking the mft record of the base ntfs inode. These locks are maintained
- * throughout execution of the function. These locks are required so that the
- * attribute can be resized safely and so that it can for example be converted
- * from resident to non-resident safely.
- *
- * TODO: At present attribute list attribute handling is not implemented.
- *
- * TODO: At present it is not safe to call this function for anything other
- * than the $DATA attribute(s) of an uncompressed and unencrypted file.
- */
-s64 ntfs_attr_extend_allocation(ntfs_inode *ni, s64 new_alloc_size,
- const s64 new_data_size, const s64 data_start)
-{
- VCN vcn;
- s64 ll, allocated_size, start = data_start;
- struct inode *vi = VFS_I(ni);
- ntfs_volume *vol = ni->vol;
- ntfs_inode *base_ni;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- ntfs_attr_search_ctx *ctx;
- runlist_element *rl, *rl2;
- unsigned long flags;
- int err, mp_size;
- u32 attr_len = 0; /* Silence stupid gcc warning. */
- bool mp_rebuilt;
-
-#ifdef DEBUG
- read_lock_irqsave(&ni->size_lock, flags);
- allocated_size = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- ntfs_debug("Entering for i_ino 0x%lx, attribute type 0x%x, "
- "old_allocated_size 0x%llx, "
- "new_allocated_size 0x%llx, new_data_size 0x%llx, "
- "data_start 0x%llx.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type),
- (unsigned long long)allocated_size,
- (unsigned long long)new_alloc_size,
- (unsigned long long)new_data_size,
- (unsigned long long)start);
-#endif
-retry_extend:
- /*
- * For non-resident attributes, @start and @new_size need to be aligned
- * to cluster boundaries for allocation purposes.
- */
- if (NInoNonResident(ni)) {
- if (start > 0)
- start &= ~(s64)vol->cluster_size_mask;
- new_alloc_size = (new_alloc_size + vol->cluster_size - 1) &
- ~(s64)vol->cluster_size_mask;
- }
- BUG_ON(new_data_size >= 0 && new_data_size > new_alloc_size);
- /* Check if new size is allowed in $AttrDef. */
- err = ntfs_attr_size_bounds_check(vol, ni->type, new_alloc_size);
- if (unlikely(err)) {
- /* Only emit errors when the write will fail completely. */
- read_lock_irqsave(&ni->size_lock, flags);
- allocated_size = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (start < 0 || start >= allocated_size) {
- if (err == -ERANGE) {
- ntfs_error(vol->sb, "Cannot extend allocation "
- "of inode 0x%lx, attribute "
- "type 0x%x, because the new "
- "allocation would exceed the "
- "maximum allowed size for "
- "this attribute type.",
- vi->i_ino, (unsigned)
- le32_to_cpu(ni->type));
- } else {
- ntfs_error(vol->sb, "Cannot extend allocation "
- "of inode 0x%lx, attribute "
- "type 0x%x, because this "
- "attribute type is not "
- "defined on the NTFS volume. "
- "Possible corruption! You "
- "should run chkdsk!",
- vi->i_ino, (unsigned)
- le32_to_cpu(ni->type));
- }
- }
- /* Translate error code to be POSIX conformant for write(2). */
- if (err == -ERANGE)
- err = -EFBIG;
- else
- err = -EIO;
- return err;
- }
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- /*
- * We will be modifying both the runlist (if non-resident) and the mft
- * record so lock them both down.
- */
- down_write(&ni->runlist.lock);
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- ctx = NULL;
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- read_lock_irqsave(&ni->size_lock, flags);
- allocated_size = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- /*
- * If non-resident, seek to the last extent. If resident, there is
- * only one extent, so seek to that.
- */
- vcn = NInoNonResident(ni) ? allocated_size >> vol->cluster_size_bits :
- 0;
- /*
- * Abort if someone did the work whilst we waited for the locks. If we
- * just converted the attribute from resident to non-resident it is
- * likely that exactly this has happened already. We cannot quite
- * abort if we need to update the data size.
- */
- if (unlikely(new_alloc_size <= allocated_size)) {
- ntfs_debug("Allocated size already exceeds requested size.");
- new_alloc_size = allocated_size;
- if (new_data_size < 0)
- goto done;
- /*
- * We want the first attribute extent so that we can update the
- * data size.
- */
- vcn = 0;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, vcn, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- goto err_out;
- }
- m = ctx->mrec;
- a = ctx->attr;
- /* Use goto to reduce indentation. */
- if (a->non_resident)
- goto do_non_resident_extend;
- BUG_ON(NInoNonResident(ni));
- /* The total length of the attribute value. */
- attr_len = le32_to_cpu(a->data.resident.value_length);
- /*
- * Extend the attribute record to be able to store the new attribute
- * size. ntfs_attr_record_resize() will not do anything if the size is
- * not changing.
- */
- if (new_alloc_size < vol->mft_record_size &&
- !ntfs_attr_record_resize(m, a,
- le16_to_cpu(a->data.resident.value_offset) +
- new_alloc_size)) {
- /* The resize succeeded! */
- write_lock_irqsave(&ni->size_lock, flags);
- ni->allocated_size = le32_to_cpu(a->length) -
- le16_to_cpu(a->data.resident.value_offset);
- write_unlock_irqrestore(&ni->size_lock, flags);
- if (new_data_size >= 0) {
- BUG_ON(new_data_size < attr_len);
- a->data.resident.value_length =
- cpu_to_le32((u32)new_data_size);
- }
- goto flush_done;
- }
- /*
- * We have to drop all the locks so we can call
- * ntfs_attr_make_non_resident(). This could be optimised by try-
- * locking the first page cache page and only if that fails dropping
- * the locks, locking the page, and redoing all the locking and
- * lookups. While this would be a huge optimisation, it is not worth
- * it as this is definitely a slow code path.
- */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
- /*
- * Not enough space in the mft record, try to make the attribute
- * non-resident and if successful restart the extension process.
- */
- err = ntfs_attr_make_non_resident(ni, attr_len);
- if (likely(!err))
- goto retry_extend;
- /*
- * Could not make non-resident. If this is due to this not being
- * permitted for this attribute type or there not being enough space,
- * try to make other attributes non-resident. Otherwise fail.
- */
- if (unlikely(err != -EPERM && err != -ENOSPC)) {
- /* Only emit errors when the write will fail completely. */
- read_lock_irqsave(&ni->size_lock, flags);
- allocated_size = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (start < 0 || start >= allocated_size)
- ntfs_error(vol->sb, "Cannot extend allocation of "
- "inode 0x%lx, attribute type 0x%x, "
- "because the conversion from resident "
- "to non-resident attribute failed "
- "with error code %i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- if (err != -ENOMEM)
- err = -EIO;
- goto conv_err_out;
- }
- /* TODO: Not implemented from here, abort. */
- read_lock_irqsave(&ni->size_lock, flags);
- allocated_size = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (start < 0 || start >= allocated_size) {
- if (err == -ENOSPC)
- ntfs_error(vol->sb, "Not enough space in the mft "
- "record/on disk for the non-resident "
- "attribute value. This case is not "
- "implemented yet.");
- else /* if (err == -EPERM) */
- ntfs_error(vol->sb, "This attribute type may not be "
- "non-resident. This case is not "
- "implemented yet.");
- }
- err = -EOPNOTSUPP;
- goto conv_err_out;
-#if 0
- // TODO: Attempt to make other attributes non-resident.
- if (!err)
- goto do_resident_extend;
- /*
- * Both the attribute list attribute and the standard information
- * attribute must remain in the base inode. Thus, if this is one of
- * these attributes, we have to try to move other attributes out into
- * extent mft records instead.
- */
- if (ni->type == AT_ATTRIBUTE_LIST ||
- ni->type == AT_STANDARD_INFORMATION) {
- // TODO: Attempt to move other attributes into extent mft
- // records.
- err = -EOPNOTSUPP;
- if (!err)
- goto do_resident_extend;
- goto err_out;
- }
- // TODO: Attempt to move this attribute to an extent mft record, but
- // only if it is not already the only attribute in an mft record in
- // which case there would be nothing to gain.
- err = -EOPNOTSUPP;
- if (!err)
- goto do_resident_extend;
- /* There is nothing we can do to make enough space. )-: */
- goto err_out;
-#endif
-do_non_resident_extend:
- BUG_ON(!NInoNonResident(ni));
- if (new_alloc_size == allocated_size) {
- BUG_ON(vcn);
- goto alloc_done;
- }
- /*
- * If the data starts after the end of the old allocation, this is a
- * $DATA attribute and sparse attributes are enabled on the volume and
- * for this inode, then create a sparse region between the old
- * allocated size and the start of the data. Otherwise simply proceed
- * with filling the whole space between the old allocated size and the
- * new allocated size with clusters.
- */
- if ((start >= 0 && start <= allocated_size) || ni->type != AT_DATA ||
- !NVolSparseEnabled(vol) || NInoSparseDisabled(ni))
- goto skip_sparse;
- // TODO: This is not implemented yet. We just fill in with real
- // clusters for now...
- ntfs_debug("Inserting holes is not-implemented yet. Falling back to "
- "allocating real clusters instead.");
-skip_sparse:
- rl = ni->runlist.rl;
- if (likely(rl)) {
- /* Seek to the end of the runlist. */
- while (rl->length)
- rl++;
- }
- /* If this attribute extent is not mapped, map it now. */
- if (unlikely(!rl || rl->lcn == LCN_RL_NOT_MAPPED ||
- (rl->lcn == LCN_ENOENT && rl > ni->runlist.rl &&
- (rl-1)->lcn == LCN_RL_NOT_MAPPED))) {
- if (!rl && !allocated_size)
- goto first_alloc;
- rl = ntfs_mapping_pairs_decompress(vol, a, ni->runlist.rl);
- if (IS_ERR(rl)) {
- err = PTR_ERR(rl);
- if (start < 0 || start >= allocated_size)
- ntfs_error(vol->sb, "Cannot extend allocation "
- "of inode 0x%lx, attribute "
- "type 0x%x, because the "
- "mapping of a runlist "
- "fragment failed with error "
- "code %i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type),
- err);
- if (err != -ENOMEM)
- err = -EIO;
- goto err_out;
- }
- ni->runlist.rl = rl;
- /* Seek to the end of the runlist. */
- while (rl->length)
- rl++;
- }
- /*
- * We now know the runlist of the last extent is mapped and @rl is at
- * the end of the runlist. We want to begin allocating clusters
- * starting at the last allocated cluster to reduce fragmentation. If
- * there are no valid LCNs in the attribute we let the cluster
- * allocator choose the starting cluster.
- */
- /* If the last LCN is a hole or simillar seek back to last real LCN. */
- while (rl->lcn < 0 && rl > ni->runlist.rl)
- rl--;
-first_alloc:
- // FIXME: Need to implement partial allocations so at least part of the
- // write can be performed when start >= 0. (Needed for POSIX write(2)
- // conformance.)
- rl2 = ntfs_cluster_alloc(vol, allocated_size >> vol->cluster_size_bits,
- (new_alloc_size - allocated_size) >>
- vol->cluster_size_bits, (rl && (rl->lcn >= 0)) ?
- rl->lcn + rl->length : -1, DATA_ZONE, true);
- if (IS_ERR(rl2)) {
- err = PTR_ERR(rl2);
- if (start < 0 || start >= allocated_size)
- ntfs_error(vol->sb, "Cannot extend allocation of "
- "inode 0x%lx, attribute type 0x%x, "
- "because the allocation of clusters "
- "failed with error code %i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- if (err != -ENOMEM && err != -ENOSPC)
- err = -EIO;
- goto err_out;
- }
- rl = ntfs_runlists_merge(ni->runlist.rl, rl2);
- if (IS_ERR(rl)) {
- err = PTR_ERR(rl);
- if (start < 0 || start >= allocated_size)
- ntfs_error(vol->sb, "Cannot extend allocation of "
- "inode 0x%lx, attribute type 0x%x, "
- "because the runlist merge failed "
- "with error code %i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- if (err != -ENOMEM)
- err = -EIO;
- if (ntfs_cluster_free_from_rl(vol, rl2)) {
- ntfs_error(vol->sb, "Failed to release allocated "
- "cluster(s) in error code path. Run "
- "chkdsk to recover the lost "
- "cluster(s).");
- NVolSetErrors(vol);
- }
- ntfs_free(rl2);
- goto err_out;
- }
- ni->runlist.rl = rl;
- ntfs_debug("Allocated 0x%llx clusters.", (long long)(new_alloc_size -
- allocated_size) >> vol->cluster_size_bits);
- /* Find the runlist element with which the attribute extent starts. */
- ll = sle64_to_cpu(a->data.non_resident.lowest_vcn);
- rl2 = ntfs_rl_find_vcn_nolock(rl, ll);
- BUG_ON(!rl2);
- BUG_ON(!rl2->length);
- BUG_ON(rl2->lcn < LCN_HOLE);
- mp_rebuilt = false;
- /* Get the size for the new mapping pairs array for this extent. */
- mp_size = ntfs_get_size_for_mapping_pairs(vol, rl2, ll, -1);
- if (unlikely(mp_size <= 0)) {
- err = mp_size;
- if (start < 0 || start >= allocated_size)
- ntfs_error(vol->sb, "Cannot extend allocation of "
- "inode 0x%lx, attribute type 0x%x, "
- "because determining the size for the "
- "mapping pairs failed with error code "
- "%i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- err = -EIO;
- goto undo_alloc;
- }
- /* Extend the attribute record to fit the bigger mapping pairs array. */
- attr_len = le32_to_cpu(a->length);
- err = ntfs_attr_record_resize(m, a, mp_size +
- le16_to_cpu(a->data.non_resident.mapping_pairs_offset));
- if (unlikely(err)) {
- BUG_ON(err != -ENOSPC);
- // TODO: Deal with this by moving this extent to a new mft
- // record or by starting a new extent in a new mft record,
- // possibly by extending this extent partially and filling it
- // and creating a new extent for the remainder, or by making
- // other attributes non-resident and/or by moving other
- // attributes out of this mft record.
- if (start < 0 || start >= allocated_size)
- ntfs_error(vol->sb, "Not enough space in the mft "
- "record for the extended attribute "
- "record. This case is not "
- "implemented yet.");
- err = -EOPNOTSUPP;
- goto undo_alloc;
- }
- mp_rebuilt = true;
- /* Generate the mapping pairs array directly into the attr record. */
- err = ntfs_mapping_pairs_build(vol, (u8*)a +
- le16_to_cpu(a->data.non_resident.mapping_pairs_offset),
- mp_size, rl2, ll, -1, NULL);
- if (unlikely(err)) {
- if (start < 0 || start >= allocated_size)
- ntfs_error(vol->sb, "Cannot extend allocation of "
- "inode 0x%lx, attribute type 0x%x, "
- "because building the mapping pairs "
- "failed with error code %i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- err = -EIO;
- goto undo_alloc;
- }
- /* Update the highest_vcn. */
- a->data.non_resident.highest_vcn = cpu_to_sle64((new_alloc_size >>
- vol->cluster_size_bits) - 1);
- /*
- * We now have extended the allocated size of the attribute. Reflect
- * this in the ntfs_inode structure and the attribute record.
- */
- if (a->data.non_resident.lowest_vcn) {
- /*
- * We are not in the first attribute extent, switch to it, but
- * first ensure the changes will make it to disk later.
- */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_reinit_search_ctx(ctx);
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err))
- goto restore_undo_alloc;
- /* @m is not used any more so no need to set it. */
- a = ctx->attr;
- }
- write_lock_irqsave(&ni->size_lock, flags);
- ni->allocated_size = new_alloc_size;
- a->data.non_resident.allocated_size = cpu_to_sle64(new_alloc_size);
- /*
- * FIXME: This would fail if @ni is a directory, $MFT, or an index,
- * since those can have sparse/compressed set. For example can be
- * set compressed even though it is not compressed itself and in that
- * case the bit means that files are to be created compressed in the
- * directory... At present this is ok as this code is only called for
- * regular files, and only for their $DATA attribute(s).
- * FIXME: The calculation is wrong if we created a hole above. For now
- * it does not matter as we never create holes.
- */
- if (NInoSparse(ni) || NInoCompressed(ni)) {
- ni->itype.compressed.size += new_alloc_size - allocated_size;
- a->data.non_resident.compressed_size =
- cpu_to_sle64(ni->itype.compressed.size);
- vi->i_blocks = ni->itype.compressed.size >> 9;
- } else
- vi->i_blocks = new_alloc_size >> 9;
- write_unlock_irqrestore(&ni->size_lock, flags);
-alloc_done:
- if (new_data_size >= 0) {
- BUG_ON(new_data_size <
- sle64_to_cpu(a->data.non_resident.data_size));
- a->data.non_resident.data_size = cpu_to_sle64(new_data_size);
- }
-flush_done:
- /* Ensure the changes make it to disk. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
-done:
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
- ntfs_debug("Done, new_allocated_size 0x%llx.",
- (unsigned long long)new_alloc_size);
- return new_alloc_size;
-restore_undo_alloc:
- if (start < 0 || start >= allocated_size)
- ntfs_error(vol->sb, "Cannot complete extension of allocation "
- "of inode 0x%lx, attribute type 0x%x, because "
- "lookup of first attribute extent failed with "
- "error code %i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- if (err == -ENOENT)
- err = -EIO;
- ntfs_attr_reinit_search_ctx(ctx);
- if (ntfs_attr_lookup(ni->type, ni->name, ni->name_len, CASE_SENSITIVE,
- allocated_size >> vol->cluster_size_bits, NULL, 0,
- ctx)) {
- ntfs_error(vol->sb, "Failed to find last attribute extent of "
- "attribute in error code path. Run chkdsk to "
- "recover.");
- write_lock_irqsave(&ni->size_lock, flags);
- ni->allocated_size = new_alloc_size;
- /*
- * FIXME: This would fail if @ni is a directory... See above.
- * FIXME: The calculation is wrong if we created a hole above.
- * For now it does not matter as we never create holes.
- */
- if (NInoSparse(ni) || NInoCompressed(ni)) {
- ni->itype.compressed.size += new_alloc_size -
- allocated_size;
- vi->i_blocks = ni->itype.compressed.size >> 9;
- } else
- vi->i_blocks = new_alloc_size >> 9;
- write_unlock_irqrestore(&ni->size_lock, flags);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
- /*
- * The only thing that is now wrong is the allocated size of the
- * base attribute extent which chkdsk should be able to fix.
- */
- NVolSetErrors(vol);
- return err;
- }
- ctx->attr->data.non_resident.highest_vcn = cpu_to_sle64(
- (allocated_size >> vol->cluster_size_bits) - 1);
-undo_alloc:
- ll = allocated_size >> vol->cluster_size_bits;
- if (ntfs_cluster_free(ni, ll, -1, ctx) < 0) {
- ntfs_error(vol->sb, "Failed to release allocated cluster(s) "
- "in error code path. Run chkdsk to recover "
- "the lost cluster(s).");
- NVolSetErrors(vol);
- }
- m = ctx->mrec;
- a = ctx->attr;
- /*
- * If the runlist truncation fails and/or the search context is no
- * longer valid, we cannot resize the attribute record or build the
- * mapping pairs array thus we mark the inode bad so that no access to
- * the freed clusters can happen.
- */
- if (ntfs_rl_truncate_nolock(vol, &ni->runlist, ll) || IS_ERR(m)) {
- ntfs_error(vol->sb, "Failed to %s in error code path. Run "
- "chkdsk to recover.", IS_ERR(m) ?
- "restore attribute search context" :
- "truncate attribute runlist");
- NVolSetErrors(vol);
- } else if (mp_rebuilt) {
- if (ntfs_attr_record_resize(m, a, attr_len)) {
- ntfs_error(vol->sb, "Failed to restore attribute "
- "record in error code path. Run "
- "chkdsk to recover.");
- NVolSetErrors(vol);
- } else /* if (success) */ {
- if (ntfs_mapping_pairs_build(vol, (u8*)a + le16_to_cpu(
- a->data.non_resident.
- mapping_pairs_offset), attr_len -
- le16_to_cpu(a->data.non_resident.
- mapping_pairs_offset), rl2, ll, -1,
- NULL)) {
- ntfs_error(vol->sb, "Failed to restore "
- "mapping pairs array in error "
- "code path. Run chkdsk to "
- "recover.");
- NVolSetErrors(vol);
- }
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- }
- }
-err_out:
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
-conv_err_out:
- ntfs_debug("Failed. Returning error code %i.", err);
- return err;
-}
-
-/**
- * ntfs_attr_set - fill (a part of) an attribute with a byte
- * @ni: ntfs inode describing the attribute to fill
- * @ofs: offset inside the attribute at which to start to fill
- * @cnt: number of bytes to fill
- * @val: the unsigned 8-bit value with which to fill the attribute
- *
- * Fill @cnt bytes of the attribute described by the ntfs inode @ni starting at
- * byte offset @ofs inside the attribute with the constant byte @val.
- *
- * This function is effectively like memset() applied to an ntfs attribute.
- * Note thie function actually only operates on the page cache pages belonging
- * to the ntfs attribute and it marks them dirty after doing the memset().
- * Thus it relies on the vm dirty page write code paths to cause the modified
- * pages to be written to the mft record/disk.
- *
- * Return 0 on success and -errno on error. An error code of -ESPIPE means
- * that @ofs + @cnt were outside the end of the attribute and no write was
- * performed.
- */
-int ntfs_attr_set(ntfs_inode *ni, const s64 ofs, const s64 cnt, const u8 val)
-{
- ntfs_volume *vol = ni->vol;
- struct address_space *mapping;
- struct page *page;
- u8 *kaddr;
- pgoff_t idx, end;
- unsigned start_ofs, end_ofs, size;
-
- ntfs_debug("Entering for ofs 0x%llx, cnt 0x%llx, val 0x%hx.",
- (long long)ofs, (long long)cnt, val);
- BUG_ON(ofs < 0);
- BUG_ON(cnt < 0);
- if (!cnt)
- goto done;
- /*
- * FIXME: Compressed and encrypted attributes are not supported when
- * writing and we should never have gotten here for them.
- */
- BUG_ON(NInoCompressed(ni));
- BUG_ON(NInoEncrypted(ni));
- mapping = VFS_I(ni)->i_mapping;
- /* Work out the starting index and page offset. */
- idx = ofs >> PAGE_SHIFT;
- start_ofs = ofs & ~PAGE_MASK;
- /* Work out the ending index and page offset. */
- end = ofs + cnt;
- end_ofs = end & ~PAGE_MASK;
- /* If the end is outside the inode size return -ESPIPE. */
- if (unlikely(end > i_size_read(VFS_I(ni)))) {
- ntfs_error(vol->sb, "Request exceeds end of attribute.");
- return -ESPIPE;
- }
- end >>= PAGE_SHIFT;
- /* If there is a first partial page, need to do it the slow way. */
- if (start_ofs) {
- page = read_mapping_page(mapping, idx, NULL);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to read first partial "
- "page (error, index 0x%lx).", idx);
- return PTR_ERR(page);
- }
- /*
- * If the last page is the same as the first page, need to
- * limit the write to the end offset.
- */
- size = PAGE_SIZE;
- if (idx == end)
- size = end_ofs;
- kaddr = kmap_atomic(page);
- memset(kaddr + start_ofs, val, size - start_ofs);
- flush_dcache_page(page);
- kunmap_atomic(kaddr);
- set_page_dirty(page);
- put_page(page);
- balance_dirty_pages_ratelimited(mapping);
- cond_resched();
- if (idx == end)
- goto done;
- idx++;
- }
- /* Do the whole pages the fast way. */
- for (; idx < end; idx++) {
- /* Find or create the current page. (The page is locked.) */
- page = grab_cache_page(mapping, idx);
- if (unlikely(!page)) {
- ntfs_error(vol->sb, "Insufficient memory to grab "
- "page (index 0x%lx).", idx);
- return -ENOMEM;
- }
- kaddr = kmap_atomic(page);
- memset(kaddr, val, PAGE_SIZE);
- flush_dcache_page(page);
- kunmap_atomic(kaddr);
- /*
- * If the page has buffers, mark them uptodate since buffer
- * state and not page state is definitive in 2.6 kernels.
- */
- if (page_has_buffers(page)) {
- struct buffer_head *bh, *head;
-
- bh = head = page_buffers(page);
- do {
- set_buffer_uptodate(bh);
- } while ((bh = bh->b_this_page) != head);
- }
- /* Now that buffers are uptodate, set the page uptodate, too. */
- SetPageUptodate(page);
- /*
- * Set the page and all its buffers dirty and mark the inode
- * dirty, too. The VM will write the page later on.
- */
- set_page_dirty(page);
- /* Finally unlock and release the page. */
- unlock_page(page);
- put_page(page);
- balance_dirty_pages_ratelimited(mapping);
- cond_resched();
- }
- /* If there is a last partial page, need to do it the slow way. */
- if (end_ofs) {
- page = read_mapping_page(mapping, idx, NULL);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to read last partial page "
- "(error, index 0x%lx).", idx);
- return PTR_ERR(page);
- }
- kaddr = kmap_atomic(page);
- memset(kaddr, val, end_ofs);
- flush_dcache_page(page);
- kunmap_atomic(kaddr);
- set_page_dirty(page);
- put_page(page);
- balance_dirty_pages_ratelimited(mapping);
- cond_resched();
- }
-done:
- ntfs_debug("Done.");
- return 0;
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/attrib.h b/fs/ntfs/attrib.h
deleted file mode 100644
index fe0890d3d072..000000000000
--- a/fs/ntfs/attrib.h
+++ /dev/null
@@ -1,102 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * attrib.h - Defines for attribute handling in NTFS Linux kernel driver.
- * Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2005 Anton Altaparmakov
- * Copyright (c) 2002 Richard Russon
- */
-
-#ifndef _LINUX_NTFS_ATTRIB_H
-#define _LINUX_NTFS_ATTRIB_H
-
-#include "endian.h"
-#include "types.h"
-#include "layout.h"
-#include "inode.h"
-#include "runlist.h"
-#include "volume.h"
-
-/**
- * ntfs_attr_search_ctx - used in attribute search functions
- * @mrec: buffer containing mft record to search
- * @attr: attribute record in @mrec where to begin/continue search
- * @is_first: if true ntfs_attr_lookup() begins search with @attr, else after
- *
- * Structure must be initialized to zero before the first call to one of the
- * attribute search functions. Initialize @mrec to point to the mft record to
- * search, and @attr to point to the first attribute within @mrec (not necessary
- * if calling the _first() functions), and set @is_first to 'true' (not necessary
- * if calling the _first() functions).
- *
- * If @is_first is 'true', the search begins with @attr. If @is_first is 'false',
- * the search begins after @attr. This is so that, after the first call to one
- * of the search attribute functions, we can call the function again, without
- * any modification of the search context, to automagically get the next
- * matching attribute.
- */
-typedef struct {
- MFT_RECORD *mrec;
- ATTR_RECORD *attr;
- bool is_first;
- ntfs_inode *ntfs_ino;
- ATTR_LIST_ENTRY *al_entry;
- ntfs_inode *base_ntfs_ino;
- MFT_RECORD *base_mrec;
- ATTR_RECORD *base_attr;
-} ntfs_attr_search_ctx;
-
-extern int ntfs_map_runlist_nolock(ntfs_inode *ni, VCN vcn,
- ntfs_attr_search_ctx *ctx);
-extern int ntfs_map_runlist(ntfs_inode *ni, VCN vcn);
-
-extern LCN ntfs_attr_vcn_to_lcn_nolock(ntfs_inode *ni, const VCN vcn,
- const bool write_locked);
-
-extern runlist_element *ntfs_attr_find_vcn_nolock(ntfs_inode *ni,
- const VCN vcn, ntfs_attr_search_ctx *ctx);
-
-int ntfs_attr_lookup(const ATTR_TYPE type, const ntfschar *name,
- const u32 name_len, const IGNORE_CASE_BOOL ic,
- const VCN lowest_vcn, const u8 *val, const u32 val_len,
- ntfs_attr_search_ctx *ctx);
-
-extern int load_attribute_list(ntfs_volume *vol, runlist *rl, u8 *al_start,
- const s64 size, const s64 initialized_size);
-
-static inline s64 ntfs_attr_size(const ATTR_RECORD *a)
-{
- if (!a->non_resident)
- return (s64)le32_to_cpu(a->data.resident.value_length);
- return sle64_to_cpu(a->data.non_resident.data_size);
-}
-
-extern void ntfs_attr_reinit_search_ctx(ntfs_attr_search_ctx *ctx);
-extern ntfs_attr_search_ctx *ntfs_attr_get_search_ctx(ntfs_inode *ni,
- MFT_RECORD *mrec);
-extern void ntfs_attr_put_search_ctx(ntfs_attr_search_ctx *ctx);
-
-#ifdef NTFS_RW
-
-extern int ntfs_attr_size_bounds_check(const ntfs_volume *vol,
- const ATTR_TYPE type, const s64 size);
-extern int ntfs_attr_can_be_non_resident(const ntfs_volume *vol,
- const ATTR_TYPE type);
-extern int ntfs_attr_can_be_resident(const ntfs_volume *vol,
- const ATTR_TYPE type);
-
-extern int ntfs_attr_record_resize(MFT_RECORD *m, ATTR_RECORD *a, u32 new_size);
-extern int ntfs_resident_attr_value_resize(MFT_RECORD *m, ATTR_RECORD *a,
- const u32 new_size);
-
-extern int ntfs_attr_make_non_resident(ntfs_inode *ni, const u32 data_size);
-
-extern s64 ntfs_attr_extend_allocation(ntfs_inode *ni, s64 new_alloc_size,
- const s64 new_data_size, const s64 data_start);
-
-extern int ntfs_attr_set(ntfs_inode *ni, const s64 ofs, const s64 cnt,
- const u8 val);
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_ATTRIB_H */
diff --git a/fs/ntfs/bitmap.c b/fs/ntfs/bitmap.c
deleted file mode 100644
index 0675b2400873..000000000000
--- a/fs/ntfs/bitmap.c
+++ /dev/null
@@ -1,179 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * bitmap.c - NTFS kernel bitmap handling. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2004-2005 Anton Altaparmakov
- */
-
-#ifdef NTFS_RW
-
-#include <linux/pagemap.h>
-
-#include "bitmap.h"
-#include "debug.h"
-#include "aops.h"
-#include "ntfs.h"
-
-/**
- * __ntfs_bitmap_set_bits_in_run - set a run of bits in a bitmap to a value
- * @vi: vfs inode describing the bitmap
- * @start_bit: first bit to set
- * @count: number of bits to set
- * @value: value to set the bits to (i.e. 0 or 1)
- * @is_rollback: if 'true' this is a rollback operation
- *
- * Set @count bits starting at bit @start_bit in the bitmap described by the
- * vfs inode @vi to @value, where @value is either 0 or 1.
- *
- * @is_rollback should always be 'false', it is for internal use to rollback
- * errors. You probably want to use ntfs_bitmap_set_bits_in_run() instead.
- *
- * Return 0 on success and -errno on error.
- */
-int __ntfs_bitmap_set_bits_in_run(struct inode *vi, const s64 start_bit,
- const s64 count, const u8 value, const bool is_rollback)
-{
- s64 cnt = count;
- pgoff_t index, end_index;
- struct address_space *mapping;
- struct page *page;
- u8 *kaddr;
- int pos, len;
- u8 bit;
-
- BUG_ON(!vi);
- ntfs_debug("Entering for i_ino 0x%lx, start_bit 0x%llx, count 0x%llx, "
- "value %u.%s", vi->i_ino, (unsigned long long)start_bit,
- (unsigned long long)cnt, (unsigned int)value,
- is_rollback ? " (rollback)" : "");
- BUG_ON(start_bit < 0);
- BUG_ON(cnt < 0);
- BUG_ON(value > 1);
- /*
- * Calculate the indices for the pages containing the first and last
- * bits, i.e. @start_bit and @start_bit + @cnt - 1, respectively.
- */
- index = start_bit >> (3 + PAGE_SHIFT);
- end_index = (start_bit + cnt - 1) >> (3 + PAGE_SHIFT);
-
- /* Get the page containing the first bit (@start_bit). */
- mapping = vi->i_mapping;
- page = ntfs_map_page(mapping, index);
- if (IS_ERR(page)) {
- if (!is_rollback)
- ntfs_error(vi->i_sb, "Failed to map first page (error "
- "%li), aborting.", PTR_ERR(page));
- return PTR_ERR(page);
- }
- kaddr = page_address(page);
-
- /* Set @pos to the position of the byte containing @start_bit. */
- pos = (start_bit >> 3) & ~PAGE_MASK;
-
- /* Calculate the position of @start_bit in the first byte. */
- bit = start_bit & 7;
-
- /* If the first byte is partial, modify the appropriate bits in it. */
- if (bit) {
- u8 *byte = kaddr + pos;
- while ((bit & 7) && cnt) {
- cnt--;
- if (value)
- *byte |= 1 << bit++;
- else
- *byte &= ~(1 << bit++);
- }
- /* If we are done, unmap the page and return success. */
- if (!cnt)
- goto done;
-
- /* Update @pos to the new position. */
- pos++;
- }
- /*
- * Depending on @value, modify all remaining whole bytes in the page up
- * to @cnt.
- */
- len = min_t(s64, cnt >> 3, PAGE_SIZE - pos);
- memset(kaddr + pos, value ? 0xff : 0, len);
- cnt -= len << 3;
-
- /* Update @len to point to the first not-done byte in the page. */
- if (cnt < 8)
- len += pos;
-
- /* If we are not in the last page, deal with all subsequent pages. */
- while (index < end_index) {
- BUG_ON(cnt <= 0);
-
- /* Update @index and get the next page. */
- flush_dcache_page(page);
- set_page_dirty(page);
- ntfs_unmap_page(page);
- page = ntfs_map_page(mapping, ++index);
- if (IS_ERR(page))
- goto rollback;
- kaddr = page_address(page);
- /*
- * Depending on @value, modify all remaining whole bytes in the
- * page up to @cnt.
- */
- len = min_t(s64, cnt >> 3, PAGE_SIZE);
- memset(kaddr, value ? 0xff : 0, len);
- cnt -= len << 3;
- }
- /*
- * The currently mapped page is the last one. If the last byte is
- * partial, modify the appropriate bits in it. Note, @len is the
- * position of the last byte inside the page.
- */
- if (cnt) {
- u8 *byte;
-
- BUG_ON(cnt > 7);
-
- bit = cnt;
- byte = kaddr + len;
- while (bit--) {
- if (value)
- *byte |= 1 << bit;
- else
- *byte &= ~(1 << bit);
- }
- }
-done:
- /* We are done. Unmap the page and return success. */
- flush_dcache_page(page);
- set_page_dirty(page);
- ntfs_unmap_page(page);
- ntfs_debug("Done.");
- return 0;
-rollback:
- /*
- * Current state:
- * - no pages are mapped
- * - @count - @cnt is the number of bits that have been modified
- */
- if (is_rollback)
- return PTR_ERR(page);
- if (count != cnt)
- pos = __ntfs_bitmap_set_bits_in_run(vi, start_bit, count - cnt,
- value ? 0 : 1, true);
- else
- pos = 0;
- if (!pos) {
- /* Rollback was successful. */
- ntfs_error(vi->i_sb, "Failed to map subsequent page (error "
- "%li), aborting.", PTR_ERR(page));
- } else {
- /* Rollback failed. */
- ntfs_error(vi->i_sb, "Failed to map subsequent page (error "
- "%li) and rollback failed (error %i). "
- "Aborting and leaving inconsistent metadata. "
- "Unmount and run chkdsk.", PTR_ERR(page), pos);
- NVolSetErrors(NTFS_SB(vi->i_sb));
- }
- return PTR_ERR(page);
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/bitmap.h b/fs/ntfs/bitmap.h
deleted file mode 100644
index 9dd2224ca9c4..000000000000
--- a/fs/ntfs/bitmap.h
+++ /dev/null
@@ -1,104 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * bitmap.h - Defines for NTFS kernel bitmap handling. Part of the Linux-NTFS
- * project.
- *
- * Copyright (c) 2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_BITMAP_H
-#define _LINUX_NTFS_BITMAP_H
-
-#ifdef NTFS_RW
-
-#include <linux/fs.h>
-
-#include "types.h"
-
-extern int __ntfs_bitmap_set_bits_in_run(struct inode *vi, const s64 start_bit,
- const s64 count, const u8 value, const bool is_rollback);
-
-/**
- * ntfs_bitmap_set_bits_in_run - set a run of bits in a bitmap to a value
- * @vi: vfs inode describing the bitmap
- * @start_bit: first bit to set
- * @count: number of bits to set
- * @value: value to set the bits to (i.e. 0 or 1)
- *
- * Set @count bits starting at bit @start_bit in the bitmap described by the
- * vfs inode @vi to @value, where @value is either 0 or 1.
- *
- * Return 0 on success and -errno on error.
- */
-static inline int ntfs_bitmap_set_bits_in_run(struct inode *vi,
- const s64 start_bit, const s64 count, const u8 value)
-{
- return __ntfs_bitmap_set_bits_in_run(vi, start_bit, count, value,
- false);
-}
-
-/**
- * ntfs_bitmap_set_run - set a run of bits in a bitmap
- * @vi: vfs inode describing the bitmap
- * @start_bit: first bit to set
- * @count: number of bits to set
- *
- * Set @count bits starting at bit @start_bit in the bitmap described by the
- * vfs inode @vi.
- *
- * Return 0 on success and -errno on error.
- */
-static inline int ntfs_bitmap_set_run(struct inode *vi, const s64 start_bit,
- const s64 count)
-{
- return ntfs_bitmap_set_bits_in_run(vi, start_bit, count, 1);
-}
-
-/**
- * ntfs_bitmap_clear_run - clear a run of bits in a bitmap
- * @vi: vfs inode describing the bitmap
- * @start_bit: first bit to clear
- * @count: number of bits to clear
- *
- * Clear @count bits starting at bit @start_bit in the bitmap described by the
- * vfs inode @vi.
- *
- * Return 0 on success and -errno on error.
- */
-static inline int ntfs_bitmap_clear_run(struct inode *vi, const s64 start_bit,
- const s64 count)
-{
- return ntfs_bitmap_set_bits_in_run(vi, start_bit, count, 0);
-}
-
-/**
- * ntfs_bitmap_set_bit - set a bit in a bitmap
- * @vi: vfs inode describing the bitmap
- * @bit: bit to set
- *
- * Set bit @bit in the bitmap described by the vfs inode @vi.
- *
- * Return 0 on success and -errno on error.
- */
-static inline int ntfs_bitmap_set_bit(struct inode *vi, const s64 bit)
-{
- return ntfs_bitmap_set_run(vi, bit, 1);
-}
-
-/**
- * ntfs_bitmap_clear_bit - clear a bit in a bitmap
- * @vi: vfs inode describing the bitmap
- * @bit: bit to clear
- *
- * Clear bit @bit in the bitmap described by the vfs inode @vi.
- *
- * Return 0 on success and -errno on error.
- */
-static inline int ntfs_bitmap_clear_bit(struct inode *vi, const s64 bit)
-{
- return ntfs_bitmap_clear_run(vi, bit, 1);
-}
-
-#endif /* NTFS_RW */
-
-#endif /* defined _LINUX_NTFS_BITMAP_H */
diff --git a/fs/ntfs/collate.c b/fs/ntfs/collate.c
deleted file mode 100644
index 3ab6ec96abfe..000000000000
--- a/fs/ntfs/collate.c
+++ /dev/null
@@ -1,110 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * collate.c - NTFS kernel collation handling. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2004 Anton Altaparmakov
- */
-
-#include "collate.h"
-#include "debug.h"
-#include "ntfs.h"
-
-static int ntfs_collate_binary(ntfs_volume *vol,
- const void *data1, const int data1_len,
- const void *data2, const int data2_len)
-{
- int rc;
-
- ntfs_debug("Entering.");
- rc = memcmp(data1, data2, min(data1_len, data2_len));
- if (!rc && (data1_len != data2_len)) {
- if (data1_len < data2_len)
- rc = -1;
- else
- rc = 1;
- }
- ntfs_debug("Done, returning %i", rc);
- return rc;
-}
-
-static int ntfs_collate_ntofs_ulong(ntfs_volume *vol,
- const void *data1, const int data1_len,
- const void *data2, const int data2_len)
-{
- int rc;
- u32 d1, d2;
-
- ntfs_debug("Entering.");
- // FIXME: We don't really want to bug here.
- BUG_ON(data1_len != data2_len);
- BUG_ON(data1_len != 4);
- d1 = le32_to_cpup(data1);
- d2 = le32_to_cpup(data2);
- if (d1 < d2)
- rc = -1;
- else {
- if (d1 == d2)
- rc = 0;
- else
- rc = 1;
- }
- ntfs_debug("Done, returning %i", rc);
- return rc;
-}
-
-typedef int (*ntfs_collate_func_t)(ntfs_volume *, const void *, const int,
- const void *, const int);
-
-static ntfs_collate_func_t ntfs_do_collate0x0[3] = {
- ntfs_collate_binary,
- NULL/*ntfs_collate_file_name*/,
- NULL/*ntfs_collate_unicode_string*/,
-};
-
-static ntfs_collate_func_t ntfs_do_collate0x1[4] = {
- ntfs_collate_ntofs_ulong,
- NULL/*ntfs_collate_ntofs_sid*/,
- NULL/*ntfs_collate_ntofs_security_hash*/,
- NULL/*ntfs_collate_ntofs_ulongs*/,
-};
-
-/**
- * ntfs_collate - collate two data items using a specified collation rule
- * @vol: ntfs volume to which the data items belong
- * @cr: collation rule to use when comparing the items
- * @data1: first data item to collate
- * @data1_len: length in bytes of @data1
- * @data2: second data item to collate
- * @data2_len: length in bytes of @data2
- *
- * Collate the two data items @data1 and @data2 using the collation rule @cr
- * and return -1, 0, ir 1 if @data1 is found, respectively, to collate before,
- * to match, or to collate after @data2.
- *
- * For speed we use the collation rule @cr as an index into two tables of
- * function pointers to call the appropriate collation function.
- */
-int ntfs_collate(ntfs_volume *vol, COLLATION_RULE cr,
- const void *data1, const int data1_len,
- const void *data2, const int data2_len) {
- int i;
-
- ntfs_debug("Entering.");
- /*
- * FIXME: At the moment we only support COLLATION_BINARY and
- * COLLATION_NTOFS_ULONG, so we BUG() for everything else for now.
- */
- BUG_ON(cr != COLLATION_BINARY && cr != COLLATION_NTOFS_ULONG);
- i = le32_to_cpu(cr);
- BUG_ON(i < 0);
- if (i <= 0x02)
- return ntfs_do_collate0x0[i](vol, data1, data1_len,
- data2, data2_len);
- BUG_ON(i < 0x10);
- i -= 0x10;
- if (likely(i <= 3))
- return ntfs_do_collate0x1[i](vol, data1, data1_len,
- data2, data2_len);
- BUG();
- return 0;
-}
diff --git a/fs/ntfs/collate.h b/fs/ntfs/collate.h
deleted file mode 100644
index f2255619b4f4..000000000000
--- a/fs/ntfs/collate.h
+++ /dev/null
@@ -1,36 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * collate.h - Defines for NTFS kernel collation handling. Part of the
- * Linux-NTFS project.
- *
- * Copyright (c) 2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_COLLATE_H
-#define _LINUX_NTFS_COLLATE_H
-
-#include "types.h"
-#include "volume.h"
-
-static inline bool ntfs_is_collation_rule_supported(COLLATION_RULE cr) {
- int i;
-
- /*
- * FIXME: At the moment we only support COLLATION_BINARY and
- * COLLATION_NTOFS_ULONG, so we return false for everything else for
- * now.
- */
- if (unlikely(cr != COLLATION_BINARY && cr != COLLATION_NTOFS_ULONG))
- return false;
- i = le32_to_cpu(cr);
- if (likely(((i >= 0) && (i <= 0x02)) ||
- ((i >= 0x10) && (i <= 0x13))))
- return true;
- return false;
-}
-
-extern int ntfs_collate(ntfs_volume *vol, COLLATION_RULE cr,
- const void *data1, const int data1_len,
- const void *data2, const int data2_len);
-
-#endif /* _LINUX_NTFS_COLLATE_H */
diff --git a/fs/ntfs/compress.c b/fs/ntfs/compress.c
deleted file mode 100644
index 587e9b187873..000000000000
--- a/fs/ntfs/compress.c
+++ /dev/null
@@ -1,950 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/**
- * compress.c - NTFS kernel compressed attributes handling.
- * Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2004 Anton Altaparmakov
- * Copyright (c) 2002 Richard Russon
- */
-
-#include <linux/fs.h>
-#include <linux/buffer_head.h>
-#include <linux/blkdev.h>
-#include <linux/vmalloc.h>
-#include <linux/slab.h>
-
-#include "attrib.h"
-#include "inode.h"
-#include "debug.h"
-#include "ntfs.h"
-
-/**
- * ntfs_compression_constants - enum of constants used in the compression code
- */
-typedef enum {
- /* Token types and access mask. */
- NTFS_SYMBOL_TOKEN = 0,
- NTFS_PHRASE_TOKEN = 1,
- NTFS_TOKEN_MASK = 1,
-
- /* Compression sub-block constants. */
- NTFS_SB_SIZE_MASK = 0x0fff,
- NTFS_SB_SIZE = 0x1000,
- NTFS_SB_IS_COMPRESSED = 0x8000,
-
- /*
- * The maximum compression block size is by definition 16 * the cluster
- * size, with the maximum supported cluster size being 4kiB. Thus the
- * maximum compression buffer size is 64kiB, so we use this when
- * initializing the compression buffer.
- */
- NTFS_MAX_CB_SIZE = 64 * 1024,
-} ntfs_compression_constants;
-
-/**
- * ntfs_compression_buffer - one buffer for the decompression engine
- */
-static u8 *ntfs_compression_buffer;
-
-/**
- * ntfs_cb_lock - spinlock which protects ntfs_compression_buffer
- */
-static DEFINE_SPINLOCK(ntfs_cb_lock);
-
-/**
- * allocate_compression_buffers - allocate the decompression buffers
- *
- * Caller has to hold the ntfs_lock mutex.
- *
- * Return 0 on success or -ENOMEM if the allocations failed.
- */
-int allocate_compression_buffers(void)
-{
- BUG_ON(ntfs_compression_buffer);
-
- ntfs_compression_buffer = vmalloc(NTFS_MAX_CB_SIZE);
- if (!ntfs_compression_buffer)
- return -ENOMEM;
- return 0;
-}
-
-/**
- * free_compression_buffers - free the decompression buffers
- *
- * Caller has to hold the ntfs_lock mutex.
- */
-void free_compression_buffers(void)
-{
- BUG_ON(!ntfs_compression_buffer);
- vfree(ntfs_compression_buffer);
- ntfs_compression_buffer = NULL;
-}
-
-/**
- * zero_partial_compressed_page - zero out of bounds compressed page region
- */
-static void zero_partial_compressed_page(struct page *page,
- const s64 initialized_size)
-{
- u8 *kp = page_address(page);
- unsigned int kp_ofs;
-
- ntfs_debug("Zeroing page region outside initialized size.");
- if (((s64)page->index << PAGE_SHIFT) >= initialized_size) {
- clear_page(kp);
- return;
- }
- kp_ofs = initialized_size & ~PAGE_MASK;
- memset(kp + kp_ofs, 0, PAGE_SIZE - kp_ofs);
- return;
-}
-
-/**
- * handle_bounds_compressed_page - test for&handle out of bounds compressed page
- */
-static inline void handle_bounds_compressed_page(struct page *page,
- const loff_t i_size, const s64 initialized_size)
-{
- if ((page->index >= (initialized_size >> PAGE_SHIFT)) &&
- (initialized_size < i_size))
- zero_partial_compressed_page(page, initialized_size);
- return;
-}
-
-/**
- * ntfs_decompress - decompress a compression block into an array of pages
- * @dest_pages: destination array of pages
- * @completed_pages: scratch space to track completed pages
- * @dest_index: current index into @dest_pages (IN/OUT)
- * @dest_ofs: current offset within @dest_pages[@dest_index] (IN/OUT)
- * @dest_max_index: maximum index into @dest_pages (IN)
- * @dest_max_ofs: maximum offset within @dest_pages[@dest_max_index] (IN)
- * @xpage: the target page (-1 if none) (IN)
- * @xpage_done: set to 1 if xpage was completed successfully (IN/OUT)
- * @cb_start: compression block to decompress (IN)
- * @cb_size: size of compression block @cb_start in bytes (IN)
- * @i_size: file size when we started the read (IN)
- * @initialized_size: initialized file size when we started the read (IN)
- *
- * The caller must have disabled preemption. ntfs_decompress() reenables it when
- * the critical section is finished.
- *
- * This decompresses the compression block @cb_start into the array of
- * destination pages @dest_pages starting at index @dest_index into @dest_pages
- * and at offset @dest_pos into the page @dest_pages[@dest_index].
- *
- * When the page @dest_pages[@xpage] is completed, @xpage_done is set to 1.
- * If xpage is -1 or @xpage has not been completed, @xpage_done is not modified.
- *
- * @cb_start is a pointer to the compression block which needs decompressing
- * and @cb_size is the size of @cb_start in bytes (8-64kiB).
- *
- * Return 0 if success or -EOVERFLOW on error in the compressed stream.
- * @xpage_done indicates whether the target page (@dest_pages[@xpage]) was
- * completed during the decompression of the compression block (@cb_start).
- *
- * Warning: This function *REQUIRES* PAGE_SIZE >= 4096 or it will blow up
- * unpredicatbly! You have been warned!
- *
- * Note to hackers: This function may not sleep until it has finished accessing
- * the compression block @cb_start as it is a per-CPU buffer.
- */
-static int ntfs_decompress(struct page *dest_pages[], int completed_pages[],
- int *dest_index, int *dest_ofs, const int dest_max_index,
- const int dest_max_ofs, const int xpage, char *xpage_done,
- u8 *const cb_start, const u32 cb_size, const loff_t i_size,
- const s64 initialized_size)
-{
- /*
- * Pointers into the compressed data, i.e. the compression block (cb),
- * and the therein contained sub-blocks (sb).
- */
- u8 *cb_end = cb_start + cb_size; /* End of cb. */
- u8 *cb = cb_start; /* Current position in cb. */
- u8 *cb_sb_start = cb; /* Beginning of the current sb in the cb. */
- u8 *cb_sb_end; /* End of current sb / beginning of next sb. */
-
- /* Variables for uncompressed data / destination. */
- struct page *dp; /* Current destination page being worked on. */
- u8 *dp_addr; /* Current pointer into dp. */
- u8 *dp_sb_start; /* Start of current sub-block in dp. */
- u8 *dp_sb_end; /* End of current sb in dp (dp_sb_start +
- NTFS_SB_SIZE). */
- u16 do_sb_start; /* @dest_ofs when starting this sub-block. */
- u16 do_sb_end; /* @dest_ofs of end of this sb (do_sb_start +
- NTFS_SB_SIZE). */
-
- /* Variables for tag and token parsing. */
- u8 tag; /* Current tag. */
- int token; /* Loop counter for the eight tokens in tag. */
- int nr_completed_pages = 0;
-
- /* Default error code. */
- int err = -EOVERFLOW;
-
- ntfs_debug("Entering, cb_size = 0x%x.", cb_size);
-do_next_sb:
- ntfs_debug("Beginning sub-block at offset = 0x%zx in the cb.",
- cb - cb_start);
- /*
- * Have we reached the end of the compression block or the end of the
- * decompressed data? The latter can happen for example if the current
- * position in the compression block is one byte before its end so the
- * first two checks do not detect it.
- */
- if (cb == cb_end || !le16_to_cpup((le16*)cb) ||
- (*dest_index == dest_max_index &&
- *dest_ofs == dest_max_ofs)) {
- int i;
-
- ntfs_debug("Completed. Returning success (0).");
- err = 0;
-return_error:
- /* We can sleep from now on, so we drop lock. */
- spin_unlock(&ntfs_cb_lock);
- /* Second stage: finalize completed pages. */
- if (nr_completed_pages > 0) {
- for (i = 0; i < nr_completed_pages; i++) {
- int di = completed_pages[i];
-
- dp = dest_pages[di];
- /*
- * If we are outside the initialized size, zero
- * the out of bounds page range.
- */
- handle_bounds_compressed_page(dp, i_size,
- initialized_size);
- flush_dcache_page(dp);
- kunmap(dp);
- SetPageUptodate(dp);
- unlock_page(dp);
- if (di == xpage)
- *xpage_done = 1;
- else
- put_page(dp);
- dest_pages[di] = NULL;
- }
- }
- return err;
- }
-
- /* Setup offsets for the current sub-block destination. */
- do_sb_start = *dest_ofs;
- do_sb_end = do_sb_start + NTFS_SB_SIZE;
-
- /* Check that we are still within allowed boundaries. */
- if (*dest_index == dest_max_index && do_sb_end > dest_max_ofs)
- goto return_overflow;
-
- /* Does the minimum size of a compressed sb overflow valid range? */
- if (cb + 6 > cb_end)
- goto return_overflow;
-
- /* Setup the current sub-block source pointers and validate range. */
- cb_sb_start = cb;
- cb_sb_end = cb_sb_start + (le16_to_cpup((le16*)cb) & NTFS_SB_SIZE_MASK)
- + 3;
- if (cb_sb_end > cb_end)
- goto return_overflow;
-
- /* Get the current destination page. */
- dp = dest_pages[*dest_index];
- if (!dp) {
- /* No page present. Skip decompression of this sub-block. */
- cb = cb_sb_end;
-
- /* Advance destination position to next sub-block. */
- *dest_ofs = (*dest_ofs + NTFS_SB_SIZE) & ~PAGE_MASK;
- if (!*dest_ofs && (++*dest_index > dest_max_index))
- goto return_overflow;
- goto do_next_sb;
- }
-
- /* We have a valid destination page. Setup the destination pointers. */
- dp_addr = (u8*)page_address(dp) + do_sb_start;
-
- /* Now, we are ready to process the current sub-block (sb). */
- if (!(le16_to_cpup((le16*)cb) & NTFS_SB_IS_COMPRESSED)) {
- ntfs_debug("Found uncompressed sub-block.");
- /* This sb is not compressed, just copy it into destination. */
-
- /* Advance source position to first data byte. */
- cb += 2;
-
- /* An uncompressed sb must be full size. */
- if (cb_sb_end - cb != NTFS_SB_SIZE)
- goto return_overflow;
-
- /* Copy the block and advance the source position. */
- memcpy(dp_addr, cb, NTFS_SB_SIZE);
- cb += NTFS_SB_SIZE;
-
- /* Advance destination position to next sub-block. */
- *dest_ofs += NTFS_SB_SIZE;
- if (!(*dest_ofs &= ~PAGE_MASK)) {
-finalize_page:
- /*
- * First stage: add current page index to array of
- * completed pages.
- */
- completed_pages[nr_completed_pages++] = *dest_index;
- if (++*dest_index > dest_max_index)
- goto return_overflow;
- }
- goto do_next_sb;
- }
- ntfs_debug("Found compressed sub-block.");
- /* This sb is compressed, decompress it into destination. */
-
- /* Setup destination pointers. */
- dp_sb_start = dp_addr;
- dp_sb_end = dp_sb_start + NTFS_SB_SIZE;
-
- /* Forward to the first tag in the sub-block. */
- cb += 2;
-do_next_tag:
- if (cb == cb_sb_end) {
- /* Check if the decompressed sub-block was not full-length. */
- if (dp_addr < dp_sb_end) {
- int nr_bytes = do_sb_end - *dest_ofs;
-
- ntfs_debug("Filling incomplete sub-block with "
- "zeroes.");
- /* Zero remainder and update destination position. */
- memset(dp_addr, 0, nr_bytes);
- *dest_ofs += nr_bytes;
- }
- /* We have finished the current sub-block. */
- if (!(*dest_ofs &= ~PAGE_MASK))
- goto finalize_page;
- goto do_next_sb;
- }
-
- /* Check we are still in range. */
- if (cb > cb_sb_end || dp_addr > dp_sb_end)
- goto return_overflow;
-
- /* Get the next tag and advance to first token. */
- tag = *cb++;
-
- /* Parse the eight tokens described by the tag. */
- for (token = 0; token < 8; token++, tag >>= 1) {
- u16 lg, pt, length, max_non_overlap;
- register u16 i;
- u8 *dp_back_addr;
-
- /* Check if we are done / still in range. */
- if (cb >= cb_sb_end || dp_addr > dp_sb_end)
- break;
-
- /* Determine token type and parse appropriately.*/
- if ((tag & NTFS_TOKEN_MASK) == NTFS_SYMBOL_TOKEN) {
- /*
- * We have a symbol token, copy the symbol across, and
- * advance the source and destination positions.
- */
- *dp_addr++ = *cb++;
- ++*dest_ofs;
-
- /* Continue with the next token. */
- continue;
- }
-
- /*
- * We have a phrase token. Make sure it is not the first tag in
- * the sb as this is illegal and would confuse the code below.
- */
- if (dp_addr == dp_sb_start)
- goto return_overflow;
-
- /*
- * Determine the number of bytes to go back (p) and the number
- * of bytes to copy (l). We use an optimized algorithm in which
- * we first calculate log2(current destination position in sb),
- * which allows determination of l and p in O(1) rather than
- * O(n). We just need an arch-optimized log2() function now.
- */
- lg = 0;
- for (i = *dest_ofs - do_sb_start - 1; i >= 0x10; i >>= 1)
- lg++;
-
- /* Get the phrase token into i. */
- pt = le16_to_cpup((le16*)cb);
-
- /*
- * Calculate starting position of the byte sequence in
- * the destination using the fact that p = (pt >> (12 - lg)) + 1
- * and make sure we don't go too far back.
- */
- dp_back_addr = dp_addr - (pt >> (12 - lg)) - 1;
- if (dp_back_addr < dp_sb_start)
- goto return_overflow;
-
- /* Now calculate the length of the byte sequence. */
- length = (pt & (0xfff >> lg)) + 3;
-
- /* Advance destination position and verify it is in range. */
- *dest_ofs += length;
- if (*dest_ofs > do_sb_end)
- goto return_overflow;
-
- /* The number of non-overlapping bytes. */
- max_non_overlap = dp_addr - dp_back_addr;
-
- if (length <= max_non_overlap) {
- /* The byte sequence doesn't overlap, just copy it. */
- memcpy(dp_addr, dp_back_addr, length);
-
- /* Advance destination pointer. */
- dp_addr += length;
- } else {
- /*
- * The byte sequence does overlap, copy non-overlapping
- * part and then do a slow byte by byte copy for the
- * overlapping part. Also, advance the destination
- * pointer.
- */
- memcpy(dp_addr, dp_back_addr, max_non_overlap);
- dp_addr += max_non_overlap;
- dp_back_addr += max_non_overlap;
- length -= max_non_overlap;
- while (length--)
- *dp_addr++ = *dp_back_addr++;
- }
-
- /* Advance source position and continue with the next token. */
- cb += 2;
- }
-
- /* No tokens left in the current tag. Continue with the next tag. */
- goto do_next_tag;
-
-return_overflow:
- ntfs_error(NULL, "Failed. Returning -EOVERFLOW.");
- goto return_error;
-}
-
-/**
- * ntfs_read_compressed_block - read a compressed block into the page cache
- * @page: locked page in the compression block(s) we need to read
- *
- * When we are called the page has already been verified to be locked and the
- * attribute is known to be non-resident, not encrypted, but compressed.
- *
- * 1. Determine which compression block(s) @page is in.
- * 2. Get hold of all pages corresponding to this/these compression block(s).
- * 3. Read the (first) compression block.
- * 4. Decompress it into the corresponding pages.
- * 5. Throw the compressed data away and proceed to 3. for the next compression
- * block or return success if no more compression blocks left.
- *
- * Warning: We have to be careful what we do about existing pages. They might
- * have been written to so that we would lose data if we were to just overwrite
- * them with the out-of-date uncompressed data.
- *
- * FIXME: For PAGE_SIZE > cb_size we are not doing the Right Thing(TM) at
- * the end of the file I think. We need to detect this case and zero the out
- * of bounds remainder of the page in question and mark it as handled. At the
- * moment we would just return -EIO on such a page. This bug will only become
- * apparent if pages are above 8kiB and the NTFS volume only uses 512 byte
- * clusters so is probably not going to be seen by anyone. Still this should
- * be fixed. (AIA)
- *
- * FIXME: Again for PAGE_SIZE > cb_size we are screwing up both in
- * handling sparse and compressed cbs. (AIA)
- *
- * FIXME: At the moment we don't do any zeroing out in the case that
- * initialized_size is less than data_size. This should be safe because of the
- * nature of the compression algorithm used. Just in case we check and output
- * an error message in read inode if the two sizes are not equal for a
- * compressed file. (AIA)
- */
-int ntfs_read_compressed_block(struct page *page)
-{
- loff_t i_size;
- s64 initialized_size;
- struct address_space *mapping = page->mapping;
- ntfs_inode *ni = NTFS_I(mapping->host);
- ntfs_volume *vol = ni->vol;
- struct super_block *sb = vol->sb;
- runlist_element *rl;
- unsigned long flags, block_size = sb->s_blocksize;
- unsigned char block_size_bits = sb->s_blocksize_bits;
- u8 *cb, *cb_pos, *cb_end;
- struct buffer_head **bhs;
- unsigned long offset, index = page->index;
- u32 cb_size = ni->itype.compressed.block_size;
- u64 cb_size_mask = cb_size - 1UL;
- VCN vcn;
- LCN lcn;
- /* The first wanted vcn (minimum alignment is PAGE_SIZE). */
- VCN start_vcn = (((s64)index << PAGE_SHIFT) & ~cb_size_mask) >>
- vol->cluster_size_bits;
- /*
- * The first vcn after the last wanted vcn (minimum alignment is again
- * PAGE_SIZE.
- */
- VCN end_vcn = ((((s64)(index + 1UL) << PAGE_SHIFT) + cb_size - 1)
- & ~cb_size_mask) >> vol->cluster_size_bits;
- /* Number of compression blocks (cbs) in the wanted vcn range. */
- unsigned int nr_cbs = (end_vcn - start_vcn) << vol->cluster_size_bits
- >> ni->itype.compressed.block_size_bits;
- /*
- * Number of pages required to store the uncompressed data from all
- * compression blocks (cbs) overlapping @page. Due to alignment
- * guarantees of start_vcn and end_vcn, no need to round up here.
- */
- unsigned int nr_pages = (end_vcn - start_vcn) <<
- vol->cluster_size_bits >> PAGE_SHIFT;
- unsigned int xpage, max_page, cur_page, cur_ofs, i;
- unsigned int cb_clusters, cb_max_ofs;
- int block, max_block, cb_max_page, bhs_size, nr_bhs, err = 0;
- struct page **pages;
- int *completed_pages;
- unsigned char xpage_done = 0;
-
- ntfs_debug("Entering, page->index = 0x%lx, cb_size = 0x%x, nr_pages = "
- "%i.", index, cb_size, nr_pages);
- /*
- * Bad things happen if we get here for anything that is not an
- * unnamed $DATA attribute.
- */
- BUG_ON(ni->type != AT_DATA);
- BUG_ON(ni->name_len);
-
- pages = kmalloc_array(nr_pages, sizeof(struct page *), GFP_NOFS);
- completed_pages = kmalloc_array(nr_pages + 1, sizeof(int), GFP_NOFS);
-
- /* Allocate memory to store the buffer heads we need. */
- bhs_size = cb_size / block_size * sizeof(struct buffer_head *);
- bhs = kmalloc(bhs_size, GFP_NOFS);
-
- if (unlikely(!pages || !bhs || !completed_pages)) {
- kfree(bhs);
- kfree(pages);
- kfree(completed_pages);
- unlock_page(page);
- ntfs_error(vol->sb, "Failed to allocate internal buffers.");
- return -ENOMEM;
- }
-
- /*
- * We have already been given one page, this is the one we must do.
- * Once again, the alignment guarantees keep it simple.
- */
- offset = start_vcn << vol->cluster_size_bits >> PAGE_SHIFT;
- xpage = index - offset;
- pages[xpage] = page;
- /*
- * The remaining pages need to be allocated and inserted into the page
- * cache, alignment guarantees keep all the below much simpler. (-8
- */
- read_lock_irqsave(&ni->size_lock, flags);
- i_size = i_size_read(VFS_I(ni));
- initialized_size = ni->initialized_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- max_page = ((i_size + PAGE_SIZE - 1) >> PAGE_SHIFT) -
- offset;
- /* Is the page fully outside i_size? (truncate in progress) */
- if (xpage >= max_page) {
- kfree(bhs);
- kfree(pages);
- kfree(completed_pages);
- zero_user(page, 0, PAGE_SIZE);
- ntfs_debug("Compressed read outside i_size - truncated?");
- SetPageUptodate(page);
- unlock_page(page);
- return 0;
- }
- if (nr_pages < max_page)
- max_page = nr_pages;
- for (i = 0; i < max_page; i++, offset++) {
- if (i != xpage)
- pages[i] = grab_cache_page_nowait(mapping, offset);
- page = pages[i];
- if (page) {
- /*
- * We only (re)read the page if it isn't already read
- * in and/or dirty or we would be losing data or at
- * least wasting our time.
- */
- if (!PageDirty(page) && (!PageUptodate(page) ||
- PageError(page))) {
- ClearPageError(page);
- kmap(page);
- continue;
- }
- unlock_page(page);
- put_page(page);
- pages[i] = NULL;
- }
- }
-
- /*
- * We have the runlist, and all the destination pages we need to fill.
- * Now read the first compression block.
- */
- cur_page = 0;
- cur_ofs = 0;
- cb_clusters = ni->itype.compressed.block_clusters;
-do_next_cb:
- nr_cbs--;
- nr_bhs = 0;
-
- /* Read all cb buffer heads one cluster at a time. */
- rl = NULL;
- for (vcn = start_vcn, start_vcn += cb_clusters; vcn < start_vcn;
- vcn++) {
- bool is_retry = false;
-
- if (!rl) {
-lock_retry_remap:
- down_read(&ni->runlist.lock);
- rl = ni->runlist.rl;
- }
- if (likely(rl != NULL)) {
- /* Seek to element containing target vcn. */
- while (rl->length && rl[1].vcn <= vcn)
- rl++;
- lcn = ntfs_rl_vcn_to_lcn(rl, vcn);
- } else
- lcn = LCN_RL_NOT_MAPPED;
- ntfs_debug("Reading vcn = 0x%llx, lcn = 0x%llx.",
- (unsigned long long)vcn,
- (unsigned long long)lcn);
- if (lcn < 0) {
- /*
- * When we reach the first sparse cluster we have
- * finished with the cb.
- */
- if (lcn == LCN_HOLE)
- break;
- if (is_retry || lcn != LCN_RL_NOT_MAPPED)
- goto rl_err;
- is_retry = true;
- /*
- * Attempt to map runlist, dropping lock for the
- * duration.
- */
- up_read(&ni->runlist.lock);
- if (!ntfs_map_runlist(ni, vcn))
- goto lock_retry_remap;
- goto map_rl_err;
- }
- block = lcn << vol->cluster_size_bits >> block_size_bits;
- /* Read the lcn from device in chunks of block_size bytes. */
- max_block = block + (vol->cluster_size >> block_size_bits);
- do {
- ntfs_debug("block = 0x%x.", block);
- if (unlikely(!(bhs[nr_bhs] = sb_getblk(sb, block))))
- goto getblk_err;
- nr_bhs++;
- } while (++block < max_block);
- }
-
- /* Release the lock if we took it. */
- if (rl)
- up_read(&ni->runlist.lock);
-
- /* Setup and initiate io on all buffer heads. */
- for (i = 0; i < nr_bhs; i++) {
- struct buffer_head *tbh = bhs[i];
-
- if (!trylock_buffer(tbh))
- continue;
- if (unlikely(buffer_uptodate(tbh))) {
- unlock_buffer(tbh);
- continue;
- }
- get_bh(tbh);
- tbh->b_end_io = end_buffer_read_sync;
- submit_bh(REQ_OP_READ, tbh);
- }
-
- /* Wait for io completion on all buffer heads. */
- for (i = 0; i < nr_bhs; i++) {
- struct buffer_head *tbh = bhs[i];
-
- if (buffer_uptodate(tbh))
- continue;
- wait_on_buffer(tbh);
- /*
- * We need an optimization barrier here, otherwise we start
- * hitting the below fixup code when accessing a loopback
- * mounted ntfs partition. This indicates either there is a
- * race condition in the loop driver or, more likely, gcc
- * overoptimises the code without the barrier and it doesn't
- * do the Right Thing(TM).
- */
- barrier();
- if (unlikely(!buffer_uptodate(tbh))) {
- ntfs_warning(vol->sb, "Buffer is unlocked but not "
- "uptodate! Unplugging the disk queue "
- "and rescheduling.");
- get_bh(tbh);
- io_schedule();
- put_bh(tbh);
- if (unlikely(!buffer_uptodate(tbh)))
- goto read_err;
- ntfs_warning(vol->sb, "Buffer is now uptodate. Good.");
- }
- }
-
- /*
- * Get the compression buffer. We must not sleep any more
- * until we are finished with it.
- */
- spin_lock(&ntfs_cb_lock);
- cb = ntfs_compression_buffer;
-
- BUG_ON(!cb);
-
- cb_pos = cb;
- cb_end = cb + cb_size;
-
- /* Copy the buffer heads into the contiguous buffer. */
- for (i = 0; i < nr_bhs; i++) {
- memcpy(cb_pos, bhs[i]->b_data, block_size);
- cb_pos += block_size;
- }
-
- /* Just a precaution. */
- if (cb_pos + 2 <= cb + cb_size)
- *(u16*)cb_pos = 0;
-
- /* Reset cb_pos back to the beginning. */
- cb_pos = cb;
-
- /* We now have both source (if present) and destination. */
- ntfs_debug("Successfully read the compression block.");
-
- /* The last page and maximum offset within it for the current cb. */
- cb_max_page = (cur_page << PAGE_SHIFT) + cur_ofs + cb_size;
- cb_max_ofs = cb_max_page & ~PAGE_MASK;
- cb_max_page >>= PAGE_SHIFT;
-
- /* Catch end of file inside a compression block. */
- if (cb_max_page > max_page)
- cb_max_page = max_page;
-
- if (vcn == start_vcn - cb_clusters) {
- /* Sparse cb, zero out page range overlapping the cb. */
- ntfs_debug("Found sparse compression block.");
- /* We can sleep from now on, so we drop lock. */
- spin_unlock(&ntfs_cb_lock);
- if (cb_max_ofs)
- cb_max_page--;
- for (; cur_page < cb_max_page; cur_page++) {
- page = pages[cur_page];
- if (page) {
- if (likely(!cur_ofs))
- clear_page(page_address(page));
- else
- memset(page_address(page) + cur_ofs, 0,
- PAGE_SIZE -
- cur_ofs);
- flush_dcache_page(page);
- kunmap(page);
- SetPageUptodate(page);
- unlock_page(page);
- if (cur_page == xpage)
- xpage_done = 1;
- else
- put_page(page);
- pages[cur_page] = NULL;
- }
- cb_pos += PAGE_SIZE - cur_ofs;
- cur_ofs = 0;
- if (cb_pos >= cb_end)
- break;
- }
- /* If we have a partial final page, deal with it now. */
- if (cb_max_ofs && cb_pos < cb_end) {
- page = pages[cur_page];
- if (page)
- memset(page_address(page) + cur_ofs, 0,
- cb_max_ofs - cur_ofs);
- /*
- * No need to update cb_pos at this stage:
- * cb_pos += cb_max_ofs - cur_ofs;
- */
- cur_ofs = cb_max_ofs;
- }
- } else if (vcn == start_vcn) {
- /* We can't sleep so we need two stages. */
- unsigned int cur2_page = cur_page;
- unsigned int cur_ofs2 = cur_ofs;
- u8 *cb_pos2 = cb_pos;
-
- ntfs_debug("Found uncompressed compression block.");
- /* Uncompressed cb, copy it to the destination pages. */
- /*
- * TODO: As a big optimization, we could detect this case
- * before we read all the pages and use block_read_full_folio()
- * on all full pages instead (we still have to treat partial
- * pages especially but at least we are getting rid of the
- * synchronous io for the majority of pages.
- * Or if we choose not to do the read-ahead/-behind stuff, we
- * could just return block_read_full_folio(pages[xpage]) as long
- * as PAGE_SIZE <= cb_size.
- */
- if (cb_max_ofs)
- cb_max_page--;
- /* First stage: copy data into destination pages. */
- for (; cur_page < cb_max_page; cur_page++) {
- page = pages[cur_page];
- if (page)
- memcpy(page_address(page) + cur_ofs, cb_pos,
- PAGE_SIZE - cur_ofs);
- cb_pos += PAGE_SIZE - cur_ofs;
- cur_ofs = 0;
- if (cb_pos >= cb_end)
- break;
- }
- /* If we have a partial final page, deal with it now. */
- if (cb_max_ofs && cb_pos < cb_end) {
- page = pages[cur_page];
- if (page)
- memcpy(page_address(page) + cur_ofs, cb_pos,
- cb_max_ofs - cur_ofs);
- cb_pos += cb_max_ofs - cur_ofs;
- cur_ofs = cb_max_ofs;
- }
- /* We can sleep from now on, so drop lock. */
- spin_unlock(&ntfs_cb_lock);
- /* Second stage: finalize pages. */
- for (; cur2_page < cb_max_page; cur2_page++) {
- page = pages[cur2_page];
- if (page) {
- /*
- * If we are outside the initialized size, zero
- * the out of bounds page range.
- */
- handle_bounds_compressed_page(page, i_size,
- initialized_size);
- flush_dcache_page(page);
- kunmap(page);
- SetPageUptodate(page);
- unlock_page(page);
- if (cur2_page == xpage)
- xpage_done = 1;
- else
- put_page(page);
- pages[cur2_page] = NULL;
- }
- cb_pos2 += PAGE_SIZE - cur_ofs2;
- cur_ofs2 = 0;
- if (cb_pos2 >= cb_end)
- break;
- }
- } else {
- /* Compressed cb, decompress it into the destination page(s). */
- unsigned int prev_cur_page = cur_page;
-
- ntfs_debug("Found compressed compression block.");
- err = ntfs_decompress(pages, completed_pages, &cur_page,
- &cur_ofs, cb_max_page, cb_max_ofs, xpage,
- &xpage_done, cb_pos, cb_size - (cb_pos - cb),
- i_size, initialized_size);
- /*
- * We can sleep from now on, lock already dropped by
- * ntfs_decompress().
- */
- if (err) {
- ntfs_error(vol->sb, "ntfs_decompress() failed in inode "
- "0x%lx with error code %i. Skipping "
- "this compression block.",
- ni->mft_no, -err);
- /* Release the unfinished pages. */
- for (; prev_cur_page < cur_page; prev_cur_page++) {
- page = pages[prev_cur_page];
- if (page) {
- flush_dcache_page(page);
- kunmap(page);
- unlock_page(page);
- if (prev_cur_page != xpage)
- put_page(page);
- pages[prev_cur_page] = NULL;
- }
- }
- }
- }
-
- /* Release the buffer heads. */
- for (i = 0; i < nr_bhs; i++)
- brelse(bhs[i]);
-
- /* Do we have more work to do? */
- if (nr_cbs)
- goto do_next_cb;
-
- /* We no longer need the list of buffer heads. */
- kfree(bhs);
-
- /* Clean up if we have any pages left. Should never happen. */
- for (cur_page = 0; cur_page < max_page; cur_page++) {
- page = pages[cur_page];
- if (page) {
- ntfs_error(vol->sb, "Still have pages left! "
- "Terminating them with extreme "
- "prejudice. Inode 0x%lx, page index "
- "0x%lx.", ni->mft_no, page->index);
- flush_dcache_page(page);
- kunmap(page);
- unlock_page(page);
- if (cur_page != xpage)
- put_page(page);
- pages[cur_page] = NULL;
- }
- }
-
- /* We no longer need the list of pages. */
- kfree(pages);
- kfree(completed_pages);
-
- /* If we have completed the requested page, we return success. */
- if (likely(xpage_done))
- return 0;
-
- ntfs_debug("Failed. Returning error code %s.", err == -EOVERFLOW ?
- "EOVERFLOW" : (!err ? "EIO" : "unknown error"));
- return err < 0 ? err : -EIO;
-
-read_err:
- ntfs_error(vol->sb, "IO error while reading compressed data.");
- /* Release the buffer heads. */
- for (i = 0; i < nr_bhs; i++)
- brelse(bhs[i]);
- goto err_out;
-
-map_rl_err:
- ntfs_error(vol->sb, "ntfs_map_runlist() failed. Cannot read "
- "compression block.");
- goto err_out;
-
-rl_err:
- up_read(&ni->runlist.lock);
- ntfs_error(vol->sb, "ntfs_rl_vcn_to_lcn() failed. Cannot read "
- "compression block.");
- goto err_out;
-
-getblk_err:
- up_read(&ni->runlist.lock);
- ntfs_error(vol->sb, "getblk() failed. Cannot read compression block.");
-
-err_out:
- kfree(bhs);
- for (i = cur_page; i < max_page; i++) {
- page = pages[i];
- if (page) {
- flush_dcache_page(page);
- kunmap(page);
- unlock_page(page);
- if (i != xpage)
- put_page(page);
- }
- }
- kfree(pages);
- kfree(completed_pages);
- return -EIO;
-}
diff --git a/fs/ntfs/debug.c b/fs/ntfs/debug.c
deleted file mode 100644
index a3c1c5656f8f..000000000000
--- a/fs/ntfs/debug.c
+++ /dev/null
@@ -1,159 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * debug.c - NTFS kernel debug support. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2004 Anton Altaparmakov
- */
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include "debug.h"
-
-/**
- * __ntfs_warning - output a warning to the syslog
- * @function: name of function outputting the warning
- * @sb: super block of mounted ntfs filesystem
- * @fmt: warning string containing format specifications
- * @...: a variable number of arguments specified in @fmt
- *
- * Outputs a warning to the syslog for the mounted ntfs filesystem described
- * by @sb.
- *
- * @fmt and the corresponding @... is printf style format string containing
- * the warning string and the corresponding format arguments, respectively.
- *
- * @function is the name of the function from which __ntfs_warning is being
- * called.
- *
- * Note, you should be using debug.h::ntfs_warning(@sb, @fmt, @...) instead
- * as this provides the @function parameter automatically.
- */
-void __ntfs_warning(const char *function, const struct super_block *sb,
- const char *fmt, ...)
-{
- struct va_format vaf;
- va_list args;
- int flen = 0;
-
-#ifndef DEBUG
- if (!printk_ratelimit())
- return;
-#endif
- if (function)
- flen = strlen(function);
- va_start(args, fmt);
- vaf.fmt = fmt;
- vaf.va = &args;
- if (sb)
- pr_warn("(device %s): %s(): %pV\n",
- sb->s_id, flen ? function : "", &vaf);
- else
- pr_warn("%s(): %pV\n", flen ? function : "", &vaf);
- va_end(args);
-}
-
-/**
- * __ntfs_error - output an error to the syslog
- * @function: name of function outputting the error
- * @sb: super block of mounted ntfs filesystem
- * @fmt: error string containing format specifications
- * @...: a variable number of arguments specified in @fmt
- *
- * Outputs an error to the syslog for the mounted ntfs filesystem described
- * by @sb.
- *
- * @fmt and the corresponding @... is printf style format string containing
- * the error string and the corresponding format arguments, respectively.
- *
- * @function is the name of the function from which __ntfs_error is being
- * called.
- *
- * Note, you should be using debug.h::ntfs_error(@sb, @fmt, @...) instead
- * as this provides the @function parameter automatically.
- */
-void __ntfs_error(const char *function, const struct super_block *sb,
- const char *fmt, ...)
-{
- struct va_format vaf;
- va_list args;
- int flen = 0;
-
-#ifndef DEBUG
- if (!printk_ratelimit())
- return;
-#endif
- if (function)
- flen = strlen(function);
- va_start(args, fmt);
- vaf.fmt = fmt;
- vaf.va = &args;
- if (sb)
- pr_err("(device %s): %s(): %pV\n",
- sb->s_id, flen ? function : "", &vaf);
- else
- pr_err("%s(): %pV\n", flen ? function : "", &vaf);
- va_end(args);
-}
-
-#ifdef DEBUG
-
-/* If 1, output debug messages, and if 0, don't. */
-int debug_msgs = 0;
-
-void __ntfs_debug(const char *file, int line, const char *function,
- const char *fmt, ...)
-{
- struct va_format vaf;
- va_list args;
- int flen = 0;
-
- if (!debug_msgs)
- return;
- if (function)
- flen = strlen(function);
- va_start(args, fmt);
- vaf.fmt = fmt;
- vaf.va = &args;
- pr_debug("(%s, %d): %s(): %pV", file, line, flen ? function : "", &vaf);
- va_end(args);
-}
-
-/* Dump a runlist. Caller has to provide synchronisation for @rl. */
-void ntfs_debug_dump_runlist(const runlist_element *rl)
-{
- int i;
- const char *lcn_str[5] = { "LCN_HOLE ", "LCN_RL_NOT_MAPPED",
- "LCN_ENOENT ", "LCN_unknown " };
-
- if (!debug_msgs)
- return;
- pr_debug("Dumping runlist (values in hex):\n");
- if (!rl) {
- pr_debug("Run list not present.\n");
- return;
- }
- pr_debug("VCN LCN Run length\n");
- for (i = 0; ; i++) {
- LCN lcn = (rl + i)->lcn;
-
- if (lcn < (LCN)0) {
- int index = -lcn - 1;
-
- if (index > -LCN_ENOENT - 1)
- index = 3;
- pr_debug("%-16Lx %s %-16Lx%s\n",
- (long long)(rl + i)->vcn, lcn_str[index],
- (long long)(rl + i)->length,
- (rl + i)->length ? "" :
- " (runlist end)");
- } else
- pr_debug("%-16Lx %-16Lx %-16Lx%s\n",
- (long long)(rl + i)->vcn,
- (long long)(rl + i)->lcn,
- (long long)(rl + i)->length,
- (rl + i)->length ? "" :
- " (runlist end)");
- if (!(rl + i)->length)
- break;
- }
-}
-
-#endif
diff --git a/fs/ntfs/debug.h b/fs/ntfs/debug.h
deleted file mode 100644
index 6fdef388f129..000000000000
--- a/fs/ntfs/debug.h
+++ /dev/null
@@ -1,57 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * debug.h - NTFS kernel debug support. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_DEBUG_H
-#define _LINUX_NTFS_DEBUG_H
-
-#include <linux/fs.h>
-
-#include "runlist.h"
-
-#ifdef DEBUG
-
-extern int debug_msgs;
-
-extern __printf(4, 5)
-void __ntfs_debug(const char *file, int line, const char *function,
- const char *format, ...);
-/**
- * ntfs_debug - write a debug level message to syslog
- * @f: a printf format string containing the message
- * @...: the variables to substitute into @f
- *
- * ntfs_debug() writes a DEBUG level message to the syslog but only if the
- * driver was compiled with -DDEBUG. Otherwise, the call turns into a NOP.
- */
-#define ntfs_debug(f, a...) \
- __ntfs_debug(__FILE__, __LINE__, __func__, f, ##a)
-
-extern void ntfs_debug_dump_runlist(const runlist_element *rl);
-
-#else /* !DEBUG */
-
-#define ntfs_debug(fmt, ...) \
-do { \
- if (0) \
- no_printk(fmt, ##__VA_ARGS__); \
-} while (0)
-
-#define ntfs_debug_dump_runlist(rl) do {} while (0)
-
-#endif /* !DEBUG */
-
-extern __printf(3, 4)
-void __ntfs_warning(const char *function, const struct super_block *sb,
- const char *fmt, ...);
-#define ntfs_warning(sb, f, a...) __ntfs_warning(__func__, sb, f, ##a)
-
-extern __printf(3, 4)
-void __ntfs_error(const char *function, const struct super_block *sb,
- const char *fmt, ...);
-#define ntfs_error(sb, f, a...) __ntfs_error(__func__, sb, f, ##a)
-
-#endif /* _LINUX_NTFS_DEBUG_H */
diff --git a/fs/ntfs/dir.c b/fs/ntfs/dir.c
deleted file mode 100644
index cd96083a12c8..000000000000
--- a/fs/ntfs/dir.c
+++ /dev/null
@@ -1,1538 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/**
- * dir.c - NTFS kernel directory operations. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2007 Anton Altaparmakov
- * Copyright (c) 2002 Richard Russon
- */
-
-#include <linux/buffer_head.h>
-#include <linux/slab.h>
-#include <linux/blkdev.h>
-
-#include "dir.h"
-#include "aops.h"
-#include "attrib.h"
-#include "mft.h"
-#include "debug.h"
-#include "ntfs.h"
-
-/**
- * The little endian Unicode string $I30 as a global constant.
- */
-ntfschar I30[5] = { cpu_to_le16('$'), cpu_to_le16('I'),
- cpu_to_le16('3'), cpu_to_le16('0'), 0 };
-
-/**
- * ntfs_lookup_inode_by_name - find an inode in a directory given its name
- * @dir_ni: ntfs inode of the directory in which to search for the name
- * @uname: Unicode name for which to search in the directory
- * @uname_len: length of the name @uname in Unicode characters
- * @res: return the found file name if necessary (see below)
- *
- * Look for an inode with name @uname in the directory with inode @dir_ni.
- * ntfs_lookup_inode_by_name() walks the contents of the directory looking for
- * the Unicode name. If the name is found in the directory, the corresponding
- * inode number (>= 0) is returned as a mft reference in cpu format, i.e. it
- * is a 64-bit number containing the sequence number.
- *
- * On error, a negative value is returned corresponding to the error code. In
- * particular if the inode is not found -ENOENT is returned. Note that you
- * can't just check the return value for being negative, you have to check the
- * inode number for being negative which you can extract using MREC(return
- * value).
- *
- * Note, @uname_len does not include the (optional) terminating NULL character.
- *
- * Note, we look for a case sensitive match first but we also look for a case
- * insensitive match at the same time. If we find a case insensitive match, we
- * save that for the case that we don't find an exact match, where we return
- * the case insensitive match and setup @res (which we allocate!) with the mft
- * reference, the file name type, length and with a copy of the little endian
- * Unicode file name itself. If we match a file name which is in the DOS name
- * space, we only return the mft reference and file name type in @res.
- * ntfs_lookup() then uses this to find the long file name in the inode itself.
- * This is to avoid polluting the dcache with short file names. We want them to
- * work but we don't care for how quickly one can access them. This also fixes
- * the dcache aliasing issues.
- *
- * Locking: - Caller must hold i_mutex on the directory.
- * - Each page cache page in the index allocation mapping must be
- * locked whilst being accessed otherwise we may find a corrupt
- * page due to it being under ->writepage at the moment which
- * applies the mst protection fixups before writing out and then
- * removes them again after the write is complete after which it
- * unlocks the page.
- */
-MFT_REF ntfs_lookup_inode_by_name(ntfs_inode *dir_ni, const ntfschar *uname,
- const int uname_len, ntfs_name **res)
-{
- ntfs_volume *vol = dir_ni->vol;
- struct super_block *sb = vol->sb;
- MFT_RECORD *m;
- INDEX_ROOT *ir;
- INDEX_ENTRY *ie;
- INDEX_ALLOCATION *ia;
- u8 *index_end;
- u64 mref;
- ntfs_attr_search_ctx *ctx;
- int err, rc;
- VCN vcn, old_vcn;
- struct address_space *ia_mapping;
- struct page *page;
- u8 *kaddr;
- ntfs_name *name = NULL;
-
- BUG_ON(!S_ISDIR(VFS_I(dir_ni)->i_mode));
- BUG_ON(NInoAttr(dir_ni));
- /* Get hold of the mft record for the directory. */
- m = map_mft_record(dir_ni);
- if (IS_ERR(m)) {
- ntfs_error(sb, "map_mft_record() failed with error code %ld.",
- -PTR_ERR(m));
- return ERR_MREF(PTR_ERR(m));
- }
- ctx = ntfs_attr_get_search_ctx(dir_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- /* Find the index root attribute in the mft record. */
- err = ntfs_attr_lookup(AT_INDEX_ROOT, I30, 4, CASE_SENSITIVE, 0, NULL,
- 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT) {
- ntfs_error(sb, "Index root attribute missing in "
- "directory inode 0x%lx.",
- dir_ni->mft_no);
- err = -EIO;
- }
- goto err_out;
- }
- /* Get to the index root value (it's been verified in read_inode). */
- ir = (INDEX_ROOT*)((u8*)ctx->attr +
- le16_to_cpu(ctx->attr->data.resident.value_offset));
- index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length);
- /* The first index entry. */
- ie = (INDEX_ENTRY*)((u8*)&ir->index +
- le32_to_cpu(ir->index.entries_offset));
- /*
- * Loop until we exceed valid memory (corruption case) or until we
- * reach the last entry.
- */
- for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) {
- /* Bounds checks. */
- if ((u8*)ie < (u8*)ctx->mrec || (u8*)ie +
- sizeof(INDEX_ENTRY_HEADER) > index_end ||
- (u8*)ie + le16_to_cpu(ie->key_length) >
- index_end)
- goto dir_err_out;
- /*
- * The last entry cannot contain a name. It can however contain
- * a pointer to a child node in the B+tree so we just break out.
- */
- if (ie->flags & INDEX_ENTRY_END)
- break;
- /*
- * We perform a case sensitive comparison and if that matches
- * we are done and return the mft reference of the inode (i.e.
- * the inode number together with the sequence number for
- * consistency checking). We convert it to cpu format before
- * returning.
- */
- if (ntfs_are_names_equal(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length,
- CASE_SENSITIVE, vol->upcase, vol->upcase_len)) {
-found_it:
- /*
- * We have a perfect match, so we don't need to care
- * about having matched imperfectly before, so we can
- * free name and set *res to NULL.
- * However, if the perfect match is a short file name,
- * we need to signal this through *res, so that
- * ntfs_lookup() can fix dcache aliasing issues.
- * As an optimization we just reuse an existing
- * allocation of *res.
- */
- if (ie->key.file_name.file_name_type == FILE_NAME_DOS) {
- if (!name) {
- name = kmalloc(sizeof(ntfs_name),
- GFP_NOFS);
- if (!name) {
- err = -ENOMEM;
- goto err_out;
- }
- }
- name->mref = le64_to_cpu(
- ie->data.dir.indexed_file);
- name->type = FILE_NAME_DOS;
- name->len = 0;
- *res = name;
- } else {
- kfree(name);
- *res = NULL;
- }
- mref = le64_to_cpu(ie->data.dir.indexed_file);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(dir_ni);
- return mref;
- }
- /*
- * For a case insensitive mount, we also perform a case
- * insensitive comparison (provided the file name is not in the
- * POSIX namespace). If the comparison matches, and the name is
- * in the WIN32 namespace, we cache the filename in *res so
- * that the caller, ntfs_lookup(), can work on it. If the
- * comparison matches, and the name is in the DOS namespace, we
- * only cache the mft reference and the file name type (we set
- * the name length to zero for simplicity).
- */
- if (!NVolCaseSensitive(vol) &&
- ie->key.file_name.file_name_type &&
- ntfs_are_names_equal(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length,
- IGNORE_CASE, vol->upcase, vol->upcase_len)) {
- int name_size = sizeof(ntfs_name);
- u8 type = ie->key.file_name.file_name_type;
- u8 len = ie->key.file_name.file_name_length;
-
- /* Only one case insensitive matching name allowed. */
- if (name) {
- ntfs_error(sb, "Found already allocated name "
- "in phase 1. Please run chkdsk "
- "and if that doesn't find any "
- "errors please report you saw "
- "this message to "
- "linux-ntfs-dev@lists."
- "sourceforge.net.");
- goto dir_err_out;
- }
-
- if (type != FILE_NAME_DOS)
- name_size += len * sizeof(ntfschar);
- name = kmalloc(name_size, GFP_NOFS);
- if (!name) {
- err = -ENOMEM;
- goto err_out;
- }
- name->mref = le64_to_cpu(ie->data.dir.indexed_file);
- name->type = type;
- if (type != FILE_NAME_DOS) {
- name->len = len;
- memcpy(name->name, ie->key.file_name.file_name,
- len * sizeof(ntfschar));
- } else
- name->len = 0;
- *res = name;
- }
- /*
- * Not a perfect match, need to do full blown collation so we
- * know which way in the B+tree we have to go.
- */
- rc = ntfs_collate_names(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, 1,
- IGNORE_CASE, vol->upcase, vol->upcase_len);
- /*
- * If uname collates before the name of the current entry, there
- * is definitely no such name in this index but we might need to
- * descend into the B+tree so we just break out of the loop.
- */
- if (rc == -1)
- break;
- /* The names are not equal, continue the search. */
- if (rc)
- continue;
- /*
- * Names match with case insensitive comparison, now try the
- * case sensitive comparison, which is required for proper
- * collation.
- */
- rc = ntfs_collate_names(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, 1,
- CASE_SENSITIVE, vol->upcase, vol->upcase_len);
- if (rc == -1)
- break;
- if (rc)
- continue;
- /*
- * Perfect match, this will never happen as the
- * ntfs_are_names_equal() call will have gotten a match but we
- * still treat it correctly.
- */
- goto found_it;
- }
- /*
- * We have finished with this index without success. Check for the
- * presence of a child node and if not present return -ENOENT, unless
- * we have got a matching name cached in name in which case return the
- * mft reference associated with it.
- */
- if (!(ie->flags & INDEX_ENTRY_NODE)) {
- if (name) {
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(dir_ni);
- return name->mref;
- }
- ntfs_debug("Entry not found.");
- err = -ENOENT;
- goto err_out;
- } /* Child node present, descend into it. */
- /* Consistency check: Verify that an index allocation exists. */
- if (!NInoIndexAllocPresent(dir_ni)) {
- ntfs_error(sb, "No index allocation attribute but index entry "
- "requires one. Directory inode 0x%lx is "
- "corrupt or driver bug.", dir_ni->mft_no);
- goto err_out;
- }
- /* Get the starting vcn of the index_block holding the child node. */
- vcn = sle64_to_cpup((sle64*)((u8*)ie + le16_to_cpu(ie->length) - 8));
- ia_mapping = VFS_I(dir_ni)->i_mapping;
- /*
- * We are done with the index root and the mft record. Release them,
- * otherwise we deadlock with ntfs_map_page().
- */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(dir_ni);
- m = NULL;
- ctx = NULL;
-descend_into_child_node:
- /*
- * Convert vcn to index into the index allocation attribute in units
- * of PAGE_SIZE and map the page cache page, reading it from
- * disk if necessary.
- */
- page = ntfs_map_page(ia_mapping, vcn <<
- dir_ni->itype.index.vcn_size_bits >> PAGE_SHIFT);
- if (IS_ERR(page)) {
- ntfs_error(sb, "Failed to map directory index page, error %ld.",
- -PTR_ERR(page));
- err = PTR_ERR(page);
- goto err_out;
- }
- lock_page(page);
- kaddr = (u8*)page_address(page);
-fast_descend_into_child_node:
- /* Get to the index allocation block. */
- ia = (INDEX_ALLOCATION*)(kaddr + ((vcn <<
- dir_ni->itype.index.vcn_size_bits) & ~PAGE_MASK));
- /* Bounds checks. */
- if ((u8*)ia < kaddr || (u8*)ia > kaddr + PAGE_SIZE) {
- ntfs_error(sb, "Out of bounds check failed. Corrupt directory "
- "inode 0x%lx or driver bug.", dir_ni->mft_no);
- goto unm_err_out;
- }
- /* Catch multi sector transfer fixup errors. */
- if (unlikely(!ntfs_is_indx_record(ia->magic))) {
- ntfs_error(sb, "Directory index record with vcn 0x%llx is "
- "corrupt. Corrupt inode 0x%lx. Run chkdsk.",
- (unsigned long long)vcn, dir_ni->mft_no);
- goto unm_err_out;
- }
- if (sle64_to_cpu(ia->index_block_vcn) != vcn) {
- ntfs_error(sb, "Actual VCN (0x%llx) of index buffer is "
- "different from expected VCN (0x%llx). "
- "Directory inode 0x%lx is corrupt or driver "
- "bug.", (unsigned long long)
- sle64_to_cpu(ia->index_block_vcn),
- (unsigned long long)vcn, dir_ni->mft_no);
- goto unm_err_out;
- }
- if (le32_to_cpu(ia->index.allocated_size) + 0x18 !=
- dir_ni->itype.index.block_size) {
- ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode "
- "0x%lx has a size (%u) differing from the "
- "directory specified size (%u). Directory "
- "inode is corrupt or driver bug.",
- (unsigned long long)vcn, dir_ni->mft_no,
- le32_to_cpu(ia->index.allocated_size) + 0x18,
- dir_ni->itype.index.block_size);
- goto unm_err_out;
- }
- index_end = (u8*)ia + dir_ni->itype.index.block_size;
- if (index_end > kaddr + PAGE_SIZE) {
- ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode "
- "0x%lx crosses page boundary. Impossible! "
- "Cannot access! This is probably a bug in the "
- "driver.", (unsigned long long)vcn,
- dir_ni->mft_no);
- goto unm_err_out;
- }
- index_end = (u8*)&ia->index + le32_to_cpu(ia->index.index_length);
- if (index_end > (u8*)ia + dir_ni->itype.index.block_size) {
- ntfs_error(sb, "Size of index buffer (VCN 0x%llx) of directory "
- "inode 0x%lx exceeds maximum size.",
- (unsigned long long)vcn, dir_ni->mft_no);
- goto unm_err_out;
- }
- /* The first index entry. */
- ie = (INDEX_ENTRY*)((u8*)&ia->index +
- le32_to_cpu(ia->index.entries_offset));
- /*
- * Iterate similar to above big loop but applied to index buffer, thus
- * loop until we exceed valid memory (corruption case) or until we
- * reach the last entry.
- */
- for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) {
- /* Bounds check. */
- if ((u8*)ie < (u8*)ia || (u8*)ie +
- sizeof(INDEX_ENTRY_HEADER) > index_end ||
- (u8*)ie + le16_to_cpu(ie->key_length) >
- index_end) {
- ntfs_error(sb, "Index entry out of bounds in "
- "directory inode 0x%lx.",
- dir_ni->mft_no);
- goto unm_err_out;
- }
- /*
- * The last entry cannot contain a name. It can however contain
- * a pointer to a child node in the B+tree so we just break out.
- */
- if (ie->flags & INDEX_ENTRY_END)
- break;
- /*
- * We perform a case sensitive comparison and if that matches
- * we are done and return the mft reference of the inode (i.e.
- * the inode number together with the sequence number for
- * consistency checking). We convert it to cpu format before
- * returning.
- */
- if (ntfs_are_names_equal(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length,
- CASE_SENSITIVE, vol->upcase, vol->upcase_len)) {
-found_it2:
- /*
- * We have a perfect match, so we don't need to care
- * about having matched imperfectly before, so we can
- * free name and set *res to NULL.
- * However, if the perfect match is a short file name,
- * we need to signal this through *res, so that
- * ntfs_lookup() can fix dcache aliasing issues.
- * As an optimization we just reuse an existing
- * allocation of *res.
- */
- if (ie->key.file_name.file_name_type == FILE_NAME_DOS) {
- if (!name) {
- name = kmalloc(sizeof(ntfs_name),
- GFP_NOFS);
- if (!name) {
- err = -ENOMEM;
- goto unm_err_out;
- }
- }
- name->mref = le64_to_cpu(
- ie->data.dir.indexed_file);
- name->type = FILE_NAME_DOS;
- name->len = 0;
- *res = name;
- } else {
- kfree(name);
- *res = NULL;
- }
- mref = le64_to_cpu(ie->data.dir.indexed_file);
- unlock_page(page);
- ntfs_unmap_page(page);
- return mref;
- }
- /*
- * For a case insensitive mount, we also perform a case
- * insensitive comparison (provided the file name is not in the
- * POSIX namespace). If the comparison matches, and the name is
- * in the WIN32 namespace, we cache the filename in *res so
- * that the caller, ntfs_lookup(), can work on it. If the
- * comparison matches, and the name is in the DOS namespace, we
- * only cache the mft reference and the file name type (we set
- * the name length to zero for simplicity).
- */
- if (!NVolCaseSensitive(vol) &&
- ie->key.file_name.file_name_type &&
- ntfs_are_names_equal(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length,
- IGNORE_CASE, vol->upcase, vol->upcase_len)) {
- int name_size = sizeof(ntfs_name);
- u8 type = ie->key.file_name.file_name_type;
- u8 len = ie->key.file_name.file_name_length;
-
- /* Only one case insensitive matching name allowed. */
- if (name) {
- ntfs_error(sb, "Found already allocated name "
- "in phase 2. Please run chkdsk "
- "and if that doesn't find any "
- "errors please report you saw "
- "this message to "
- "linux-ntfs-dev@lists."
- "sourceforge.net.");
- unlock_page(page);
- ntfs_unmap_page(page);
- goto dir_err_out;
- }
-
- if (type != FILE_NAME_DOS)
- name_size += len * sizeof(ntfschar);
- name = kmalloc(name_size, GFP_NOFS);
- if (!name) {
- err = -ENOMEM;
- goto unm_err_out;
- }
- name->mref = le64_to_cpu(ie->data.dir.indexed_file);
- name->type = type;
- if (type != FILE_NAME_DOS) {
- name->len = len;
- memcpy(name->name, ie->key.file_name.file_name,
- len * sizeof(ntfschar));
- } else
- name->len = 0;
- *res = name;
- }
- /*
- * Not a perfect match, need to do full blown collation so we
- * know which way in the B+tree we have to go.
- */
- rc = ntfs_collate_names(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, 1,
- IGNORE_CASE, vol->upcase, vol->upcase_len);
- /*
- * If uname collates before the name of the current entry, there
- * is definitely no such name in this index but we might need to
- * descend into the B+tree so we just break out of the loop.
- */
- if (rc == -1)
- break;
- /* The names are not equal, continue the search. */
- if (rc)
- continue;
- /*
- * Names match with case insensitive comparison, now try the
- * case sensitive comparison, which is required for proper
- * collation.
- */
- rc = ntfs_collate_names(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, 1,
- CASE_SENSITIVE, vol->upcase, vol->upcase_len);
- if (rc == -1)
- break;
- if (rc)
- continue;
- /*
- * Perfect match, this will never happen as the
- * ntfs_are_names_equal() call will have gotten a match but we
- * still treat it correctly.
- */
- goto found_it2;
- }
- /*
- * We have finished with this index buffer without success. Check for
- * the presence of a child node.
- */
- if (ie->flags & INDEX_ENTRY_NODE) {
- if ((ia->index.flags & NODE_MASK) == LEAF_NODE) {
- ntfs_error(sb, "Index entry with child node found in "
- "a leaf node in directory inode 0x%lx.",
- dir_ni->mft_no);
- goto unm_err_out;
- }
- /* Child node present, descend into it. */
- old_vcn = vcn;
- vcn = sle64_to_cpup((sle64*)((u8*)ie +
- le16_to_cpu(ie->length) - 8));
- if (vcn >= 0) {
- /* If vcn is in the same page cache page as old_vcn we
- * recycle the mapped page. */
- if (old_vcn << vol->cluster_size_bits >>
- PAGE_SHIFT == vcn <<
- vol->cluster_size_bits >>
- PAGE_SHIFT)
- goto fast_descend_into_child_node;
- unlock_page(page);
- ntfs_unmap_page(page);
- goto descend_into_child_node;
- }
- ntfs_error(sb, "Negative child node vcn in directory inode "
- "0x%lx.", dir_ni->mft_no);
- goto unm_err_out;
- }
- /*
- * No child node present, return -ENOENT, unless we have got a matching
- * name cached in name in which case return the mft reference
- * associated with it.
- */
- if (name) {
- unlock_page(page);
- ntfs_unmap_page(page);
- return name->mref;
- }
- ntfs_debug("Entry not found.");
- err = -ENOENT;
-unm_err_out:
- unlock_page(page);
- ntfs_unmap_page(page);
-err_out:
- if (!err)
- err = -EIO;
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(dir_ni);
- if (name) {
- kfree(name);
- *res = NULL;
- }
- return ERR_MREF(err);
-dir_err_out:
- ntfs_error(sb, "Corrupt directory. Aborting lookup.");
- goto err_out;
-}
-
-#if 0
-
-// TODO: (AIA)
-// The algorithm embedded in this code will be required for the time when we
-// want to support adding of entries to directories, where we require correct
-// collation of file names in order not to cause corruption of the filesystem.
-
-/**
- * ntfs_lookup_inode_by_name - find an inode in a directory given its name
- * @dir_ni: ntfs inode of the directory in which to search for the name
- * @uname: Unicode name for which to search in the directory
- * @uname_len: length of the name @uname in Unicode characters
- *
- * Look for an inode with name @uname in the directory with inode @dir_ni.
- * ntfs_lookup_inode_by_name() walks the contents of the directory looking for
- * the Unicode name. If the name is found in the directory, the corresponding
- * inode number (>= 0) is returned as a mft reference in cpu format, i.e. it
- * is a 64-bit number containing the sequence number.
- *
- * On error, a negative value is returned corresponding to the error code. In
- * particular if the inode is not found -ENOENT is returned. Note that you
- * can't just check the return value for being negative, you have to check the
- * inode number for being negative which you can extract using MREC(return
- * value).
- *
- * Note, @uname_len does not include the (optional) terminating NULL character.
- */
-u64 ntfs_lookup_inode_by_name(ntfs_inode *dir_ni, const ntfschar *uname,
- const int uname_len)
-{
- ntfs_volume *vol = dir_ni->vol;
- struct super_block *sb = vol->sb;
- MFT_RECORD *m;
- INDEX_ROOT *ir;
- INDEX_ENTRY *ie;
- INDEX_ALLOCATION *ia;
- u8 *index_end;
- u64 mref;
- ntfs_attr_search_ctx *ctx;
- int err, rc;
- IGNORE_CASE_BOOL ic;
- VCN vcn, old_vcn;
- struct address_space *ia_mapping;
- struct page *page;
- u8 *kaddr;
-
- /* Get hold of the mft record for the directory. */
- m = map_mft_record(dir_ni);
- if (IS_ERR(m)) {
- ntfs_error(sb, "map_mft_record() failed with error code %ld.",
- -PTR_ERR(m));
- return ERR_MREF(PTR_ERR(m));
- }
- ctx = ntfs_attr_get_search_ctx(dir_ni, m);
- if (!ctx) {
- err = -ENOMEM;
- goto err_out;
- }
- /* Find the index root attribute in the mft record. */
- err = ntfs_attr_lookup(AT_INDEX_ROOT, I30, 4, CASE_SENSITIVE, 0, NULL,
- 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT) {
- ntfs_error(sb, "Index root attribute missing in "
- "directory inode 0x%lx.",
- dir_ni->mft_no);
- err = -EIO;
- }
- goto err_out;
- }
- /* Get to the index root value (it's been verified in read_inode). */
- ir = (INDEX_ROOT*)((u8*)ctx->attr +
- le16_to_cpu(ctx->attr->data.resident.value_offset));
- index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length);
- /* The first index entry. */
- ie = (INDEX_ENTRY*)((u8*)&ir->index +
- le32_to_cpu(ir->index.entries_offset));
- /*
- * Loop until we exceed valid memory (corruption case) or until we
- * reach the last entry.
- */
- for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) {
- /* Bounds checks. */
- if ((u8*)ie < (u8*)ctx->mrec || (u8*)ie +
- sizeof(INDEX_ENTRY_HEADER) > index_end ||
- (u8*)ie + le16_to_cpu(ie->key_length) >
- index_end)
- goto dir_err_out;
- /*
- * The last entry cannot contain a name. It can however contain
- * a pointer to a child node in the B+tree so we just break out.
- */
- if (ie->flags & INDEX_ENTRY_END)
- break;
- /*
- * If the current entry has a name type of POSIX, the name is
- * case sensitive and not otherwise. This has the effect of us
- * not being able to access any POSIX file names which collate
- * after the non-POSIX one when they only differ in case, but
- * anyone doing screwy stuff like that deserves to burn in
- * hell... Doing that kind of stuff on NT4 actually causes
- * corruption on the partition even when using SP6a and Linux
- * is not involved at all.
- */
- ic = ie->key.file_name.file_name_type ? IGNORE_CASE :
- CASE_SENSITIVE;
- /*
- * If the names match perfectly, we are done and return the
- * mft reference of the inode (i.e. the inode number together
- * with the sequence number for consistency checking. We
- * convert it to cpu format before returning.
- */
- if (ntfs_are_names_equal(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, ic,
- vol->upcase, vol->upcase_len)) {
-found_it:
- mref = le64_to_cpu(ie->data.dir.indexed_file);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(dir_ni);
- return mref;
- }
- /*
- * Not a perfect match, need to do full blown collation so we
- * know which way in the B+tree we have to go.
- */
- rc = ntfs_collate_names(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, 1,
- IGNORE_CASE, vol->upcase, vol->upcase_len);
- /*
- * If uname collates before the name of the current entry, there
- * is definitely no such name in this index but we might need to
- * descend into the B+tree so we just break out of the loop.
- */
- if (rc == -1)
- break;
- /* The names are not equal, continue the search. */
- if (rc)
- continue;
- /*
- * Names match with case insensitive comparison, now try the
- * case sensitive comparison, which is required for proper
- * collation.
- */
- rc = ntfs_collate_names(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, 1,
- CASE_SENSITIVE, vol->upcase, vol->upcase_len);
- if (rc == -1)
- break;
- if (rc)
- continue;
- /*
- * Perfect match, this will never happen as the
- * ntfs_are_names_equal() call will have gotten a match but we
- * still treat it correctly.
- */
- goto found_it;
- }
- /*
- * We have finished with this index without success. Check for the
- * presence of a child node.
- */
- if (!(ie->flags & INDEX_ENTRY_NODE)) {
- /* No child node, return -ENOENT. */
- err = -ENOENT;
- goto err_out;
- } /* Child node present, descend into it. */
- /* Consistency check: Verify that an index allocation exists. */
- if (!NInoIndexAllocPresent(dir_ni)) {
- ntfs_error(sb, "No index allocation attribute but index entry "
- "requires one. Directory inode 0x%lx is "
- "corrupt or driver bug.", dir_ni->mft_no);
- goto err_out;
- }
- /* Get the starting vcn of the index_block holding the child node. */
- vcn = sle64_to_cpup((u8*)ie + le16_to_cpu(ie->length) - 8);
- ia_mapping = VFS_I(dir_ni)->i_mapping;
- /*
- * We are done with the index root and the mft record. Release them,
- * otherwise we deadlock with ntfs_map_page().
- */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(dir_ni);
- m = NULL;
- ctx = NULL;
-descend_into_child_node:
- /*
- * Convert vcn to index into the index allocation attribute in units
- * of PAGE_SIZE and map the page cache page, reading it from
- * disk if necessary.
- */
- page = ntfs_map_page(ia_mapping, vcn <<
- dir_ni->itype.index.vcn_size_bits >> PAGE_SHIFT);
- if (IS_ERR(page)) {
- ntfs_error(sb, "Failed to map directory index page, error %ld.",
- -PTR_ERR(page));
- err = PTR_ERR(page);
- goto err_out;
- }
- lock_page(page);
- kaddr = (u8*)page_address(page);
-fast_descend_into_child_node:
- /* Get to the index allocation block. */
- ia = (INDEX_ALLOCATION*)(kaddr + ((vcn <<
- dir_ni->itype.index.vcn_size_bits) & ~PAGE_MASK));
- /* Bounds checks. */
- if ((u8*)ia < kaddr || (u8*)ia > kaddr + PAGE_SIZE) {
- ntfs_error(sb, "Out of bounds check failed. Corrupt directory "
- "inode 0x%lx or driver bug.", dir_ni->mft_no);
- goto unm_err_out;
- }
- /* Catch multi sector transfer fixup errors. */
- if (unlikely(!ntfs_is_indx_record(ia->magic))) {
- ntfs_error(sb, "Directory index record with vcn 0x%llx is "
- "corrupt. Corrupt inode 0x%lx. Run chkdsk.",
- (unsigned long long)vcn, dir_ni->mft_no);
- goto unm_err_out;
- }
- if (sle64_to_cpu(ia->index_block_vcn) != vcn) {
- ntfs_error(sb, "Actual VCN (0x%llx) of index buffer is "
- "different from expected VCN (0x%llx). "
- "Directory inode 0x%lx is corrupt or driver "
- "bug.", (unsigned long long)
- sle64_to_cpu(ia->index_block_vcn),
- (unsigned long long)vcn, dir_ni->mft_no);
- goto unm_err_out;
- }
- if (le32_to_cpu(ia->index.allocated_size) + 0x18 !=
- dir_ni->itype.index.block_size) {
- ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode "
- "0x%lx has a size (%u) differing from the "
- "directory specified size (%u). Directory "
- "inode is corrupt or driver bug.",
- (unsigned long long)vcn, dir_ni->mft_no,
- le32_to_cpu(ia->index.allocated_size) + 0x18,
- dir_ni->itype.index.block_size);
- goto unm_err_out;
- }
- index_end = (u8*)ia + dir_ni->itype.index.block_size;
- if (index_end > kaddr + PAGE_SIZE) {
- ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode "
- "0x%lx crosses page boundary. Impossible! "
- "Cannot access! This is probably a bug in the "
- "driver.", (unsigned long long)vcn,
- dir_ni->mft_no);
- goto unm_err_out;
- }
- index_end = (u8*)&ia->index + le32_to_cpu(ia->index.index_length);
- if (index_end > (u8*)ia + dir_ni->itype.index.block_size) {
- ntfs_error(sb, "Size of index buffer (VCN 0x%llx) of directory "
- "inode 0x%lx exceeds maximum size.",
- (unsigned long long)vcn, dir_ni->mft_no);
- goto unm_err_out;
- }
- /* The first index entry. */
- ie = (INDEX_ENTRY*)((u8*)&ia->index +
- le32_to_cpu(ia->index.entries_offset));
- /*
- * Iterate similar to above big loop but applied to index buffer, thus
- * loop until we exceed valid memory (corruption case) or until we
- * reach the last entry.
- */
- for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) {
- /* Bounds check. */
- if ((u8*)ie < (u8*)ia || (u8*)ie +
- sizeof(INDEX_ENTRY_HEADER) > index_end ||
- (u8*)ie + le16_to_cpu(ie->key_length) >
- index_end) {
- ntfs_error(sb, "Index entry out of bounds in "
- "directory inode 0x%lx.",
- dir_ni->mft_no);
- goto unm_err_out;
- }
- /*
- * The last entry cannot contain a name. It can however contain
- * a pointer to a child node in the B+tree so we just break out.
- */
- if (ie->flags & INDEX_ENTRY_END)
- break;
- /*
- * If the current entry has a name type of POSIX, the name is
- * case sensitive and not otherwise. This has the effect of us
- * not being able to access any POSIX file names which collate
- * after the non-POSIX one when they only differ in case, but
- * anyone doing screwy stuff like that deserves to burn in
- * hell... Doing that kind of stuff on NT4 actually causes
- * corruption on the partition even when using SP6a and Linux
- * is not involved at all.
- */
- ic = ie->key.file_name.file_name_type ? IGNORE_CASE :
- CASE_SENSITIVE;
- /*
- * If the names match perfectly, we are done and return the
- * mft reference of the inode (i.e. the inode number together
- * with the sequence number for consistency checking. We
- * convert it to cpu format before returning.
- */
- if (ntfs_are_names_equal(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, ic,
- vol->upcase, vol->upcase_len)) {
-found_it2:
- mref = le64_to_cpu(ie->data.dir.indexed_file);
- unlock_page(page);
- ntfs_unmap_page(page);
- return mref;
- }
- /*
- * Not a perfect match, need to do full blown collation so we
- * know which way in the B+tree we have to go.
- */
- rc = ntfs_collate_names(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, 1,
- IGNORE_CASE, vol->upcase, vol->upcase_len);
- /*
- * If uname collates before the name of the current entry, there
- * is definitely no such name in this index but we might need to
- * descend into the B+tree so we just break out of the loop.
- */
- if (rc == -1)
- break;
- /* The names are not equal, continue the search. */
- if (rc)
- continue;
- /*
- * Names match with case insensitive comparison, now try the
- * case sensitive comparison, which is required for proper
- * collation.
- */
- rc = ntfs_collate_names(uname, uname_len,
- (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, 1,
- CASE_SENSITIVE, vol->upcase, vol->upcase_len);
- if (rc == -1)
- break;
- if (rc)
- continue;
- /*
- * Perfect match, this will never happen as the
- * ntfs_are_names_equal() call will have gotten a match but we
- * still treat it correctly.
- */
- goto found_it2;
- }
- /*
- * We have finished with this index buffer without success. Check for
- * the presence of a child node.
- */
- if (ie->flags & INDEX_ENTRY_NODE) {
- if ((ia->index.flags & NODE_MASK) == LEAF_NODE) {
- ntfs_error(sb, "Index entry with child node found in "
- "a leaf node in directory inode 0x%lx.",
- dir_ni->mft_no);
- goto unm_err_out;
- }
- /* Child node present, descend into it. */
- old_vcn = vcn;
- vcn = sle64_to_cpup((u8*)ie + le16_to_cpu(ie->length) - 8);
- if (vcn >= 0) {
- /* If vcn is in the same page cache page as old_vcn we
- * recycle the mapped page. */
- if (old_vcn << vol->cluster_size_bits >>
- PAGE_SHIFT == vcn <<
- vol->cluster_size_bits >>
- PAGE_SHIFT)
- goto fast_descend_into_child_node;
- unlock_page(page);
- ntfs_unmap_page(page);
- goto descend_into_child_node;
- }
- ntfs_error(sb, "Negative child node vcn in directory inode "
- "0x%lx.", dir_ni->mft_no);
- goto unm_err_out;
- }
- /* No child node, return -ENOENT. */
- ntfs_debug("Entry not found.");
- err = -ENOENT;
-unm_err_out:
- unlock_page(page);
- ntfs_unmap_page(page);
-err_out:
- if (!err)
- err = -EIO;
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(dir_ni);
- return ERR_MREF(err);
-dir_err_out:
- ntfs_error(sb, "Corrupt directory. Aborting lookup.");
- goto err_out;
-}
-
-#endif
-
-/**
- * ntfs_filldir - ntfs specific filldir method
- * @vol: current ntfs volume
- * @ndir: ntfs inode of current directory
- * @ia_page: page in which the index allocation buffer @ie is in resides
- * @ie: current index entry
- * @name: buffer to use for the converted name
- * @actor: what to feed the entries to
- *
- * Convert the Unicode @name to the loaded NLS and pass it to the @filldir
- * callback.
- *
- * If @ia_page is not NULL it is the locked page containing the index
- * allocation block containing the index entry @ie.
- *
- * Note, we drop (and then reacquire) the page lock on @ia_page across the
- * @filldir() call otherwise we would deadlock with NFSd when it calls ->lookup
- * since ntfs_lookup() will lock the same page. As an optimization, we do not
- * retake the lock if we are returning a non-zero value as ntfs_readdir()
- * would need to drop the lock immediately anyway.
- */
-static inline int ntfs_filldir(ntfs_volume *vol,
- ntfs_inode *ndir, struct page *ia_page, INDEX_ENTRY *ie,
- u8 *name, struct dir_context *actor)
-{
- unsigned long mref;
- int name_len;
- unsigned dt_type;
- FILE_NAME_TYPE_FLAGS name_type;
-
- name_type = ie->key.file_name.file_name_type;
- if (name_type == FILE_NAME_DOS) {
- ntfs_debug("Skipping DOS name space entry.");
- return 0;
- }
- if (MREF_LE(ie->data.dir.indexed_file) == FILE_root) {
- ntfs_debug("Skipping root directory self reference entry.");
- return 0;
- }
- if (MREF_LE(ie->data.dir.indexed_file) < FILE_first_user &&
- !NVolShowSystemFiles(vol)) {
- ntfs_debug("Skipping system file.");
- return 0;
- }
- name_len = ntfs_ucstonls(vol, (ntfschar*)&ie->key.file_name.file_name,
- ie->key.file_name.file_name_length, &name,
- NTFS_MAX_NAME_LEN * NLS_MAX_CHARSET_SIZE + 1);
- if (name_len <= 0) {
- ntfs_warning(vol->sb, "Skipping unrepresentable inode 0x%llx.",
- (long long)MREF_LE(ie->data.dir.indexed_file));
- return 0;
- }
- if (ie->key.file_name.file_attributes &
- FILE_ATTR_DUP_FILE_NAME_INDEX_PRESENT)
- dt_type = DT_DIR;
- else
- dt_type = DT_REG;
- mref = MREF_LE(ie->data.dir.indexed_file);
- /*
- * Drop the page lock otherwise we deadlock with NFS when it calls
- * ->lookup since ntfs_lookup() will lock the same page.
- */
- if (ia_page)
- unlock_page(ia_page);
- ntfs_debug("Calling filldir for %s with len %i, fpos 0x%llx, inode "
- "0x%lx, DT_%s.", name, name_len, actor->pos, mref,
- dt_type == DT_DIR ? "DIR" : "REG");
- if (!dir_emit(actor, name, name_len, mref, dt_type))
- return 1;
- /* Relock the page but not if we are aborting ->readdir. */
- if (ia_page)
- lock_page(ia_page);
- return 0;
-}
-
-/*
- * We use the same basic approach as the old NTFS driver, i.e. we parse the
- * index root entries and then the index allocation entries that are marked
- * as in use in the index bitmap.
- *
- * While this will return the names in random order this doesn't matter for
- * ->readdir but OTOH results in a faster ->readdir.
- *
- * VFS calls ->readdir without BKL but with i_mutex held. This protects the VFS
- * parts (e.g. ->f_pos and ->i_size, and it also protects against directory
- * modifications).
- *
- * Locking: - Caller must hold i_mutex on the directory.
- * - Each page cache page in the index allocation mapping must be
- * locked whilst being accessed otherwise we may find a corrupt
- * page due to it being under ->writepage at the moment which
- * applies the mst protection fixups before writing out and then
- * removes them again after the write is complete after which it
- * unlocks the page.
- */
-static int ntfs_readdir(struct file *file, struct dir_context *actor)
-{
- s64 ia_pos, ia_start, prev_ia_pos, bmp_pos;
- loff_t i_size;
- struct inode *bmp_vi, *vdir = file_inode(file);
- struct super_block *sb = vdir->i_sb;
- ntfs_inode *ndir = NTFS_I(vdir);
- ntfs_volume *vol = NTFS_SB(sb);
- MFT_RECORD *m;
- INDEX_ROOT *ir = NULL;
- INDEX_ENTRY *ie;
- INDEX_ALLOCATION *ia;
- u8 *name = NULL;
- int rc, err, ir_pos, cur_bmp_pos;
- struct address_space *ia_mapping, *bmp_mapping;
- struct page *bmp_page = NULL, *ia_page = NULL;
- u8 *kaddr, *bmp, *index_end;
- ntfs_attr_search_ctx *ctx;
-
- ntfs_debug("Entering for inode 0x%lx, fpos 0x%llx.",
- vdir->i_ino, actor->pos);
- rc = err = 0;
- /* Are we at end of dir yet? */
- i_size = i_size_read(vdir);
- if (actor->pos >= i_size + vol->mft_record_size)
- return 0;
- /* Emulate . and .. for all directories. */
- if (!dir_emit_dots(file, actor))
- return 0;
- m = NULL;
- ctx = NULL;
- /*
- * Allocate a buffer to store the current name being processed
- * converted to format determined by current NLS.
- */
- name = kmalloc(NTFS_MAX_NAME_LEN * NLS_MAX_CHARSET_SIZE + 1, GFP_NOFS);
- if (unlikely(!name)) {
- err = -ENOMEM;
- goto err_out;
- }
- /* Are we jumping straight into the index allocation attribute? */
- if (actor->pos >= vol->mft_record_size)
- goto skip_index_root;
- /* Get hold of the mft record for the directory. */
- m = map_mft_record(ndir);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(ndir, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- /* Get the offset into the index root attribute. */
- ir_pos = (s64)actor->pos;
- /* Find the index root attribute in the mft record. */
- err = ntfs_attr_lookup(AT_INDEX_ROOT, I30, 4, CASE_SENSITIVE, 0, NULL,
- 0, ctx);
- if (unlikely(err)) {
- ntfs_error(sb, "Index root attribute missing in directory "
- "inode 0x%lx.", vdir->i_ino);
- goto err_out;
- }
- /*
- * Copy the index root attribute value to a buffer so that we can put
- * the search context and unmap the mft record before calling the
- * filldir() callback. We need to do this because of NFSd which calls
- * ->lookup() from its filldir callback() and this causes NTFS to
- * deadlock as ntfs_lookup() maps the mft record of the directory and
- * we have got it mapped here already. The only solution is for us to
- * unmap the mft record here so that a call to ntfs_lookup() is able to
- * map the mft record without deadlocking.
- */
- rc = le32_to_cpu(ctx->attr->data.resident.value_length);
- ir = kmalloc(rc, GFP_NOFS);
- if (unlikely(!ir)) {
- err = -ENOMEM;
- goto err_out;
- }
- /* Copy the index root value (it has been verified in read_inode). */
- memcpy(ir, (u8*)ctx->attr +
- le16_to_cpu(ctx->attr->data.resident.value_offset), rc);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ndir);
- ctx = NULL;
- m = NULL;
- index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length);
- /* The first index entry. */
- ie = (INDEX_ENTRY*)((u8*)&ir->index +
- le32_to_cpu(ir->index.entries_offset));
- /*
- * Loop until we exceed valid memory (corruption case) or until we
- * reach the last entry or until filldir tells us it has had enough
- * or signals an error (both covered by the rc test).
- */
- for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) {
- ntfs_debug("In index root, offset 0x%zx.", (u8*)ie - (u8*)ir);
- /* Bounds checks. */
- if (unlikely((u8*)ie < (u8*)ir || (u8*)ie +
- sizeof(INDEX_ENTRY_HEADER) > index_end ||
- (u8*)ie + le16_to_cpu(ie->key_length) >
- index_end))
- goto err_out;
- /* The last entry cannot contain a name. */
- if (ie->flags & INDEX_ENTRY_END)
- break;
- /* Skip index root entry if continuing previous readdir. */
- if (ir_pos > (u8*)ie - (u8*)ir)
- continue;
- /* Advance the position even if going to skip the entry. */
- actor->pos = (u8*)ie - (u8*)ir;
- /* Submit the name to the filldir callback. */
- rc = ntfs_filldir(vol, ndir, NULL, ie, name, actor);
- if (rc) {
- kfree(ir);
- goto abort;
- }
- }
- /* We are done with the index root and can free the buffer. */
- kfree(ir);
- ir = NULL;
- /* If there is no index allocation attribute we are finished. */
- if (!NInoIndexAllocPresent(ndir))
- goto EOD;
- /* Advance fpos to the beginning of the index allocation. */
- actor->pos = vol->mft_record_size;
-skip_index_root:
- kaddr = NULL;
- prev_ia_pos = -1LL;
- /* Get the offset into the index allocation attribute. */
- ia_pos = (s64)actor->pos - vol->mft_record_size;
- ia_mapping = vdir->i_mapping;
- ntfs_debug("Inode 0x%lx, getting index bitmap.", vdir->i_ino);
- bmp_vi = ntfs_attr_iget(vdir, AT_BITMAP, I30, 4);
- if (IS_ERR(bmp_vi)) {
- ntfs_error(sb, "Failed to get bitmap attribute.");
- err = PTR_ERR(bmp_vi);
- goto err_out;
- }
- bmp_mapping = bmp_vi->i_mapping;
- /* Get the starting bitmap bit position and sanity check it. */
- bmp_pos = ia_pos >> ndir->itype.index.block_size_bits;
- if (unlikely(bmp_pos >> 3 >= i_size_read(bmp_vi))) {
- ntfs_error(sb, "Current index allocation position exceeds "
- "index bitmap size.");
- goto iput_err_out;
- }
- /* Get the starting bit position in the current bitmap page. */
- cur_bmp_pos = bmp_pos & ((PAGE_SIZE * 8) - 1);
- bmp_pos &= ~(u64)((PAGE_SIZE * 8) - 1);
-get_next_bmp_page:
- ntfs_debug("Reading bitmap with page index 0x%llx, bit ofs 0x%llx",
- (unsigned long long)bmp_pos >> (3 + PAGE_SHIFT),
- (unsigned long long)bmp_pos &
- (unsigned long long)((PAGE_SIZE * 8) - 1));
- bmp_page = ntfs_map_page(bmp_mapping,
- bmp_pos >> (3 + PAGE_SHIFT));
- if (IS_ERR(bmp_page)) {
- ntfs_error(sb, "Reading index bitmap failed.");
- err = PTR_ERR(bmp_page);
- bmp_page = NULL;
- goto iput_err_out;
- }
- bmp = (u8*)page_address(bmp_page);
- /* Find next index block in use. */
- while (!(bmp[cur_bmp_pos >> 3] & (1 << (cur_bmp_pos & 7)))) {
-find_next_index_buffer:
- cur_bmp_pos++;
- /*
- * If we have reached the end of the bitmap page, get the next
- * page, and put away the old one.
- */
- if (unlikely((cur_bmp_pos >> 3) >= PAGE_SIZE)) {
- ntfs_unmap_page(bmp_page);
- bmp_pos += PAGE_SIZE * 8;
- cur_bmp_pos = 0;
- goto get_next_bmp_page;
- }
- /* If we have reached the end of the bitmap, we are done. */
- if (unlikely(((bmp_pos + cur_bmp_pos) >> 3) >= i_size))
- goto unm_EOD;
- ia_pos = (bmp_pos + cur_bmp_pos) <<
- ndir->itype.index.block_size_bits;
- }
- ntfs_debug("Handling index buffer 0x%llx.",
- (unsigned long long)bmp_pos + cur_bmp_pos);
- /* If the current index buffer is in the same page we reuse the page. */
- if ((prev_ia_pos & (s64)PAGE_MASK) !=
- (ia_pos & (s64)PAGE_MASK)) {
- prev_ia_pos = ia_pos;
- if (likely(ia_page != NULL)) {
- unlock_page(ia_page);
- ntfs_unmap_page(ia_page);
- }
- /*
- * Map the page cache page containing the current ia_pos,
- * reading it from disk if necessary.
- */
- ia_page = ntfs_map_page(ia_mapping, ia_pos >> PAGE_SHIFT);
- if (IS_ERR(ia_page)) {
- ntfs_error(sb, "Reading index allocation data failed.");
- err = PTR_ERR(ia_page);
- ia_page = NULL;
- goto err_out;
- }
- lock_page(ia_page);
- kaddr = (u8*)page_address(ia_page);
- }
- /* Get the current index buffer. */
- ia = (INDEX_ALLOCATION*)(kaddr + (ia_pos & ~PAGE_MASK &
- ~(s64)(ndir->itype.index.block_size - 1)));
- /* Bounds checks. */
- if (unlikely((u8*)ia < kaddr || (u8*)ia > kaddr + PAGE_SIZE)) {
- ntfs_error(sb, "Out of bounds check failed. Corrupt directory "
- "inode 0x%lx or driver bug.", vdir->i_ino);
- goto err_out;
- }
- /* Catch multi sector transfer fixup errors. */
- if (unlikely(!ntfs_is_indx_record(ia->magic))) {
- ntfs_error(sb, "Directory index record with vcn 0x%llx is "
- "corrupt. Corrupt inode 0x%lx. Run chkdsk.",
- (unsigned long long)ia_pos >>
- ndir->itype.index.vcn_size_bits, vdir->i_ino);
- goto err_out;
- }
- if (unlikely(sle64_to_cpu(ia->index_block_vcn) != (ia_pos &
- ~(s64)(ndir->itype.index.block_size - 1)) >>
- ndir->itype.index.vcn_size_bits)) {
- ntfs_error(sb, "Actual VCN (0x%llx) of index buffer is "
- "different from expected VCN (0x%llx). "
- "Directory inode 0x%lx is corrupt or driver "
- "bug. ", (unsigned long long)
- sle64_to_cpu(ia->index_block_vcn),
- (unsigned long long)ia_pos >>
- ndir->itype.index.vcn_size_bits, vdir->i_ino);
- goto err_out;
- }
- if (unlikely(le32_to_cpu(ia->index.allocated_size) + 0x18 !=
- ndir->itype.index.block_size)) {
- ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode "
- "0x%lx has a size (%u) differing from the "
- "directory specified size (%u). Directory "
- "inode is corrupt or driver bug.",
- (unsigned long long)ia_pos >>
- ndir->itype.index.vcn_size_bits, vdir->i_ino,
- le32_to_cpu(ia->index.allocated_size) + 0x18,
- ndir->itype.index.block_size);
- goto err_out;
- }
- index_end = (u8*)ia + ndir->itype.index.block_size;
- if (unlikely(index_end > kaddr + PAGE_SIZE)) {
- ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode "
- "0x%lx crosses page boundary. Impossible! "
- "Cannot access! This is probably a bug in the "
- "driver.", (unsigned long long)ia_pos >>
- ndir->itype.index.vcn_size_bits, vdir->i_ino);
- goto err_out;
- }
- ia_start = ia_pos & ~(s64)(ndir->itype.index.block_size - 1);
- index_end = (u8*)&ia->index + le32_to_cpu(ia->index.index_length);
- if (unlikely(index_end > (u8*)ia + ndir->itype.index.block_size)) {
- ntfs_error(sb, "Size of index buffer (VCN 0x%llx) of directory "
- "inode 0x%lx exceeds maximum size.",
- (unsigned long long)ia_pos >>
- ndir->itype.index.vcn_size_bits, vdir->i_ino);
- goto err_out;
- }
- /* The first index entry in this index buffer. */
- ie = (INDEX_ENTRY*)((u8*)&ia->index +
- le32_to_cpu(ia->index.entries_offset));
- /*
- * Loop until we exceed valid memory (corruption case) or until we
- * reach the last entry or until filldir tells us it has had enough
- * or signals an error (both covered by the rc test).
- */
- for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) {
- ntfs_debug("In index allocation, offset 0x%llx.",
- (unsigned long long)ia_start +
- (unsigned long long)((u8*)ie - (u8*)ia));
- /* Bounds checks. */
- if (unlikely((u8*)ie < (u8*)ia || (u8*)ie +
- sizeof(INDEX_ENTRY_HEADER) > index_end ||
- (u8*)ie + le16_to_cpu(ie->key_length) >
- index_end))
- goto err_out;
- /* The last entry cannot contain a name. */
- if (ie->flags & INDEX_ENTRY_END)
- break;
- /* Skip index block entry if continuing previous readdir. */
- if (ia_pos - ia_start > (u8*)ie - (u8*)ia)
- continue;
- /* Advance the position even if going to skip the entry. */
- actor->pos = (u8*)ie - (u8*)ia +
- (sle64_to_cpu(ia->index_block_vcn) <<
- ndir->itype.index.vcn_size_bits) +
- vol->mft_record_size;
- /*
- * Submit the name to the @filldir callback. Note,
- * ntfs_filldir() drops the lock on @ia_page but it retakes it
- * before returning, unless a non-zero value is returned in
- * which case the page is left unlocked.
- */
- rc = ntfs_filldir(vol, ndir, ia_page, ie, name, actor);
- if (rc) {
- /* @ia_page is already unlocked in this case. */
- ntfs_unmap_page(ia_page);
- ntfs_unmap_page(bmp_page);
- iput(bmp_vi);
- goto abort;
- }
- }
- goto find_next_index_buffer;
-unm_EOD:
- if (ia_page) {
- unlock_page(ia_page);
- ntfs_unmap_page(ia_page);
- }
- ntfs_unmap_page(bmp_page);
- iput(bmp_vi);
-EOD:
- /* We are finished, set fpos to EOD. */
- actor->pos = i_size + vol->mft_record_size;
-abort:
- kfree(name);
- return 0;
-err_out:
- if (bmp_page) {
- ntfs_unmap_page(bmp_page);
-iput_err_out:
- iput(bmp_vi);
- }
- if (ia_page) {
- unlock_page(ia_page);
- ntfs_unmap_page(ia_page);
- }
- kfree(ir);
- kfree(name);
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(ndir);
- if (!err)
- err = -EIO;
- ntfs_debug("Failed. Returning error code %i.", -err);
- return err;
-}
-
-/**
- * ntfs_dir_open - called when an inode is about to be opened
- * @vi: inode to be opened
- * @filp: file structure describing the inode
- *
- * Limit directory size to the page cache limit on architectures where unsigned
- * long is 32-bits. This is the most we can do for now without overflowing the
- * page cache page index. Doing it this way means we don't run into problems
- * because of existing too large directories. It would be better to allow the
- * user to read the accessible part of the directory but I doubt very much
- * anyone is going to hit this check on a 32-bit architecture, so there is no
- * point in adding the extra complexity required to support this.
- *
- * On 64-bit architectures, the check is hopefully optimized away by the
- * compiler.
- */
-static int ntfs_dir_open(struct inode *vi, struct file *filp)
-{
- if (sizeof(unsigned long) < 8) {
- if (i_size_read(vi) > MAX_LFS_FILESIZE)
- return -EFBIG;
- }
- return 0;
-}
-
-#ifdef NTFS_RW
-
-/**
- * ntfs_dir_fsync - sync a directory to disk
- * @filp: directory to be synced
- * @dentry: dentry describing the directory to sync
- * @datasync: if non-zero only flush user data and not metadata
- *
- * Data integrity sync of a directory to disk. Used for fsync, fdatasync, and
- * msync system calls. This function is based on file.c::ntfs_file_fsync().
- *
- * Write the mft record and all associated extent mft records as well as the
- * $INDEX_ALLOCATION and $BITMAP attributes and then sync the block device.
- *
- * If @datasync is true, we do not wait on the inode(s) to be written out
- * but we always wait on the page cache pages to be written out.
- *
- * Note: In the past @filp could be NULL so we ignore it as we don't need it
- * anyway.
- *
- * Locking: Caller must hold i_mutex on the inode.
- *
- * TODO: We should probably also write all attribute/index inodes associated
- * with this inode but since we have no simple way of getting to them we ignore
- * this problem for now. We do write the $BITMAP attribute if it is present
- * which is the important one for a directory so things are not too bad.
- */
-static int ntfs_dir_fsync(struct file *filp, loff_t start, loff_t end,
- int datasync)
-{
- struct inode *bmp_vi, *vi = filp->f_mapping->host;
- int err, ret;
- ntfs_attr na;
-
- ntfs_debug("Entering for inode 0x%lx.", vi->i_ino);
-
- err = file_write_and_wait_range(filp, start, end);
- if (err)
- return err;
- inode_lock(vi);
-
- BUG_ON(!S_ISDIR(vi->i_mode));
- /* If the bitmap attribute inode is in memory sync it, too. */
- na.mft_no = vi->i_ino;
- na.type = AT_BITMAP;
- na.name = I30;
- na.name_len = 4;
- bmp_vi = ilookup5(vi->i_sb, vi->i_ino, ntfs_test_inode, &na);
- if (bmp_vi) {
- write_inode_now(bmp_vi, !datasync);
- iput(bmp_vi);
- }
- ret = __ntfs_write_inode(vi, 1);
- write_inode_now(vi, !datasync);
- err = sync_blockdev(vi->i_sb->s_bdev);
- if (unlikely(err && !ret))
- ret = err;
- if (likely(!ret))
- ntfs_debug("Done.");
- else
- ntfs_warning(vi->i_sb, "Failed to f%ssync inode 0x%lx. Error "
- "%u.", datasync ? "data" : "", vi->i_ino, -ret);
- inode_unlock(vi);
- return ret;
-}
-
-#endif /* NTFS_RW */
-
-const struct file_operations ntfs_dir_ops = {
- .llseek = generic_file_llseek, /* Seek inside directory. */
- .read = generic_read_dir, /* Return -EISDIR. */
- .iterate = ntfs_readdir, /* Read directory contents. */
-#ifdef NTFS_RW
- .fsync = ntfs_dir_fsync, /* Sync a directory to disk. */
-#endif /* NTFS_RW */
- /*.ioctl = ,*/ /* Perform function on the
- mounted filesystem. */
- .open = ntfs_dir_open, /* Open directory. */
-};
diff --git a/fs/ntfs/dir.h b/fs/ntfs/dir.h
deleted file mode 100644
index 0e326753df40..000000000000
--- a/fs/ntfs/dir.h
+++ /dev/null
@@ -1,34 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * dir.h - Defines for directory handling in NTFS Linux kernel driver. Part of
- * the Linux-NTFS project.
- *
- * Copyright (c) 2002-2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_DIR_H
-#define _LINUX_NTFS_DIR_H
-
-#include "layout.h"
-#include "inode.h"
-#include "types.h"
-
-/*
- * ntfs_name is used to return the file name to the caller of
- * ntfs_lookup_inode_by_name() in order for the caller (namei.c::ntfs_lookup())
- * to be able to deal with dcache aliasing issues.
- */
-typedef struct {
- MFT_REF mref;
- FILE_NAME_TYPE_FLAGS type;
- u8 len;
- ntfschar name[0];
-} __attribute__ ((__packed__)) ntfs_name;
-
-/* The little endian Unicode string $I30 as a global constant. */
-extern ntfschar I30[5];
-
-extern MFT_REF ntfs_lookup_inode_by_name(ntfs_inode *dir_ni,
- const ntfschar *uname, const int uname_len, ntfs_name **res);
-
-#endif /* _LINUX_NTFS_FS_DIR_H */
diff --git a/fs/ntfs/endian.h b/fs/ntfs/endian.h
deleted file mode 100644
index f30c139bf9ae..000000000000
--- a/fs/ntfs/endian.h
+++ /dev/null
@@ -1,79 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * endian.h - Defines for endianness handling in NTFS Linux kernel driver.
- * Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_ENDIAN_H
-#define _LINUX_NTFS_ENDIAN_H
-
-#include <asm/byteorder.h>
-#include "types.h"
-
-/*
- * Signed endianness conversion functions.
- */
-
-static inline s16 sle16_to_cpu(sle16 x)
-{
- return le16_to_cpu((__force le16)x);
-}
-
-static inline s32 sle32_to_cpu(sle32 x)
-{
- return le32_to_cpu((__force le32)x);
-}
-
-static inline s64 sle64_to_cpu(sle64 x)
-{
- return le64_to_cpu((__force le64)x);
-}
-
-static inline s16 sle16_to_cpup(sle16 *x)
-{
- return le16_to_cpu(*(__force le16*)x);
-}
-
-static inline s32 sle32_to_cpup(sle32 *x)
-{
- return le32_to_cpu(*(__force le32*)x);
-}
-
-static inline s64 sle64_to_cpup(sle64 *x)
-{
- return le64_to_cpu(*(__force le64*)x);
-}
-
-static inline sle16 cpu_to_sle16(s16 x)
-{
- return (__force sle16)cpu_to_le16(x);
-}
-
-static inline sle32 cpu_to_sle32(s32 x)
-{
- return (__force sle32)cpu_to_le32(x);
-}
-
-static inline sle64 cpu_to_sle64(s64 x)
-{
- return (__force sle64)cpu_to_le64(x);
-}
-
-static inline sle16 cpu_to_sle16p(s16 *x)
-{
- return (__force sle16)cpu_to_le16(*x);
-}
-
-static inline sle32 cpu_to_sle32p(s32 *x)
-{
- return (__force sle32)cpu_to_le32(*x);
-}
-
-static inline sle64 cpu_to_sle64p(s64 *x)
-{
- return (__force sle64)cpu_to_le64(*x);
-}
-
-#endif /* _LINUX_NTFS_ENDIAN_H */
diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c
deleted file mode 100644
index c481b14e4fd9..000000000000
--- a/fs/ntfs/file.c
+++ /dev/null
@@ -1,2006 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * file.c - NTFS kernel file operations. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2015 Anton Altaparmakov and Tuxera Inc.
- */
-
-#include <linux/blkdev.h>
-#include <linux/backing-dev.h>
-#include <linux/buffer_head.h>
-#include <linux/gfp.h>
-#include <linux/pagemap.h>
-#include <linux/pagevec.h>
-#include <linux/sched/signal.h>
-#include <linux/swap.h>
-#include <linux/uio.h>
-#include <linux/writeback.h>
-
-#include <asm/page.h>
-#include <linux/uaccess.h>
-
-#include "attrib.h"
-#include "bitmap.h"
-#include "inode.h"
-#include "debug.h"
-#include "lcnalloc.h"
-#include "malloc.h"
-#include "mft.h"
-#include "ntfs.h"
-
-/**
- * ntfs_file_open - called when an inode is about to be opened
- * @vi: inode to be opened
- * @filp: file structure describing the inode
- *
- * Limit file size to the page cache limit on architectures where unsigned long
- * is 32-bits. This is the most we can do for now without overflowing the page
- * cache page index. Doing it this way means we don't run into problems because
- * of existing too large files. It would be better to allow the user to read
- * the beginning of the file but I doubt very much anyone is going to hit this
- * check on a 32-bit architecture, so there is no point in adding the extra
- * complexity required to support this.
- *
- * On 64-bit architectures, the check is hopefully optimized away by the
- * compiler.
- *
- * After the check passes, just call generic_file_open() to do its work.
- */
-static int ntfs_file_open(struct inode *vi, struct file *filp)
-{
- if (sizeof(unsigned long) < 8) {
- if (i_size_read(vi) > MAX_LFS_FILESIZE)
- return -EOVERFLOW;
- }
- return generic_file_open(vi, filp);
-}
-
-#ifdef NTFS_RW
-
-/**
- * ntfs_attr_extend_initialized - extend the initialized size of an attribute
- * @ni: ntfs inode of the attribute to extend
- * @new_init_size: requested new initialized size in bytes
- *
- * Extend the initialized size of an attribute described by the ntfs inode @ni
- * to @new_init_size bytes. This involves zeroing any non-sparse space between
- * the old initialized size and @new_init_size both in the page cache and on
- * disk (if relevant complete pages are already uptodate in the page cache then
- * these are simply marked dirty).
- *
- * As a side-effect, the file size (vfs inode->i_size) may be incremented as,
- * in the resident attribute case, it is tied to the initialized size and, in
- * the non-resident attribute case, it may not fall below the initialized size.
- *
- * Note that if the attribute is resident, we do not need to touch the page
- * cache at all. This is because if the page cache page is not uptodate we
- * bring it uptodate later, when doing the write to the mft record since we
- * then already have the page mapped. And if the page is uptodate, the
- * non-initialized region will already have been zeroed when the page was
- * brought uptodate and the region may in fact already have been overwritten
- * with new data via mmap() based writes, so we cannot just zero it. And since
- * POSIX specifies that the behaviour of resizing a file whilst it is mmap()ped
- * is unspecified, we choose not to do zeroing and thus we do not need to touch
- * the page at all. For a more detailed explanation see ntfs_truncate() in
- * fs/ntfs/inode.c.
- *
- * Return 0 on success and -errno on error. In the case that an error is
- * encountered it is possible that the initialized size will already have been
- * incremented some way towards @new_init_size but it is guaranteed that if
- * this is the case, the necessary zeroing will also have happened and that all
- * metadata is self-consistent.
- *
- * Locking: i_mutex on the vfs inode corrseponsind to the ntfs inode @ni must be
- * held by the caller.
- */
-static int ntfs_attr_extend_initialized(ntfs_inode *ni, const s64 new_init_size)
-{
- s64 old_init_size;
- loff_t old_i_size;
- pgoff_t index, end_index;
- unsigned long flags;
- struct inode *vi = VFS_I(ni);
- ntfs_inode *base_ni;
- MFT_RECORD *m = NULL;
- ATTR_RECORD *a;
- ntfs_attr_search_ctx *ctx = NULL;
- struct address_space *mapping;
- struct page *page = NULL;
- u8 *kattr;
- int err;
- u32 attr_len;
-
- read_lock_irqsave(&ni->size_lock, flags);
- old_init_size = ni->initialized_size;
- old_i_size = i_size_read(vi);
- BUG_ON(new_init_size > ni->allocated_size);
- read_unlock_irqrestore(&ni->size_lock, flags);
- ntfs_debug("Entering for i_ino 0x%lx, attribute type 0x%x, "
- "old_initialized_size 0x%llx, "
- "new_initialized_size 0x%llx, i_size 0x%llx.",
- vi->i_ino, (unsigned)le32_to_cpu(ni->type),
- (unsigned long long)old_init_size,
- (unsigned long long)new_init_size, old_i_size);
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- /* Use goto to reduce indentation and we need the label below anyway. */
- if (NInoNonResident(ni))
- goto do_non_resident_extend;
- BUG_ON(old_init_size != old_i_size);
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- goto err_out;
- }
- m = ctx->mrec;
- a = ctx->attr;
- BUG_ON(a->non_resident);
- /* The total length of the attribute value. */
- attr_len = le32_to_cpu(a->data.resident.value_length);
- BUG_ON(old_i_size != (loff_t)attr_len);
- /*
- * Do the zeroing in the mft record and update the attribute size in
- * the mft record.
- */
- kattr = (u8*)a + le16_to_cpu(a->data.resident.value_offset);
- memset(kattr + attr_len, 0, new_init_size - attr_len);
- a->data.resident.value_length = cpu_to_le32((u32)new_init_size);
- /* Finally, update the sizes in the vfs and ntfs inodes. */
- write_lock_irqsave(&ni->size_lock, flags);
- i_size_write(vi, new_init_size);
- ni->initialized_size = new_init_size;
- write_unlock_irqrestore(&ni->size_lock, flags);
- goto done;
-do_non_resident_extend:
- /*
- * If the new initialized size @new_init_size exceeds the current file
- * size (vfs inode->i_size), we need to extend the file size to the
- * new initialized size.
- */
- if (new_init_size > old_i_size) {
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- goto err_out;
- }
- m = ctx->mrec;
- a = ctx->attr;
- BUG_ON(!a->non_resident);
- BUG_ON(old_i_size != (loff_t)
- sle64_to_cpu(a->data.non_resident.data_size));
- a->data.non_resident.data_size = cpu_to_sle64(new_init_size);
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- /* Update the file size in the vfs inode. */
- i_size_write(vi, new_init_size);
- ntfs_attr_put_search_ctx(ctx);
- ctx = NULL;
- unmap_mft_record(base_ni);
- m = NULL;
- }
- mapping = vi->i_mapping;
- index = old_init_size >> PAGE_SHIFT;
- end_index = (new_init_size + PAGE_SIZE - 1) >> PAGE_SHIFT;
- do {
- /*
- * Read the page. If the page is not present, this will zero
- * the uninitialized regions for us.
- */
- page = read_mapping_page(mapping, index, NULL);
- if (IS_ERR(page)) {
- err = PTR_ERR(page);
- goto init_err_out;
- }
- /*
- * Update the initialized size in the ntfs inode. This is
- * enough to make ntfs_writepage() work.
- */
- write_lock_irqsave(&ni->size_lock, flags);
- ni->initialized_size = (s64)(index + 1) << PAGE_SHIFT;
- if (ni->initialized_size > new_init_size)
- ni->initialized_size = new_init_size;
- write_unlock_irqrestore(&ni->size_lock, flags);
- /* Set the page dirty so it gets written out. */
- set_page_dirty(page);
- put_page(page);
- /*
- * Play nice with the vm and the rest of the system. This is
- * very much needed as we can potentially be modifying the
- * initialised size from a very small value to a really huge
- * value, e.g.
- * f = open(somefile, O_TRUNC);
- * truncate(f, 10GiB);
- * seek(f, 10GiB);
- * write(f, 1);
- * And this would mean we would be marking dirty hundreds of
- * thousands of pages or as in the above example more than
- * two and a half million pages!
- *
- * TODO: For sparse pages could optimize this workload by using
- * the FsMisc / MiscFs page bit as a "PageIsSparse" bit. This
- * would be set in read_folio for sparse pages and here we would
- * not need to mark dirty any pages which have this bit set.
- * The only caveat is that we have to clear the bit everywhere
- * where we allocate any clusters that lie in the page or that
- * contain the page.
- *
- * TODO: An even greater optimization would be for us to only
- * call read_folio() on pages which are not in sparse regions as
- * determined from the runlist. This would greatly reduce the
- * number of pages we read and make dirty in the case of sparse
- * files.
- */
- balance_dirty_pages_ratelimited(mapping);
- cond_resched();
- } while (++index < end_index);
- read_lock_irqsave(&ni->size_lock, flags);
- BUG_ON(ni->initialized_size != new_init_size);
- read_unlock_irqrestore(&ni->size_lock, flags);
- /* Now bring in sync the initialized_size in the mft record. */
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- goto init_err_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto init_err_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- goto init_err_out;
- }
- m = ctx->mrec;
- a = ctx->attr;
- BUG_ON(!a->non_resident);
- a->data.non_resident.initialized_size = cpu_to_sle64(new_init_size);
-done:
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
- ntfs_debug("Done, initialized_size 0x%llx, i_size 0x%llx.",
- (unsigned long long)new_init_size, i_size_read(vi));
- return 0;
-init_err_out:
- write_lock_irqsave(&ni->size_lock, flags);
- ni->initialized_size = old_init_size;
- write_unlock_irqrestore(&ni->size_lock, flags);
-err_out:
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
- ntfs_debug("Failed. Returning error code %i.", err);
- return err;
-}
-
-static ssize_t ntfs_prepare_file_for_write(struct kiocb *iocb,
- struct iov_iter *from)
-{
- loff_t pos;
- s64 end, ll;
- ssize_t err;
- unsigned long flags;
- struct file *file = iocb->ki_filp;
- struct inode *vi = file_inode(file);
- ntfs_inode *ni = NTFS_I(vi);
- ntfs_volume *vol = ni->vol;
-
- ntfs_debug("Entering for i_ino 0x%lx, attribute type 0x%x, pos "
- "0x%llx, count 0x%zx.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type),
- (unsigned long long)iocb->ki_pos,
- iov_iter_count(from));
- err = generic_write_checks(iocb, from);
- if (unlikely(err <= 0))
- goto out;
- /*
- * All checks have passed. Before we start doing any writing we want
- * to abort any totally illegal writes.
- */
- BUG_ON(NInoMstProtected(ni));
- BUG_ON(ni->type != AT_DATA);
- /* If file is encrypted, deny access, just like NT4. */
- if (NInoEncrypted(ni)) {
- /* Only $DATA attributes can be encrypted. */
- /*
- * Reminder for later: Encrypted files are _always_
- * non-resident so that the content can always be encrypted.
- */
- ntfs_debug("Denying write access to encrypted file.");
- err = -EACCES;
- goto out;
- }
- if (NInoCompressed(ni)) {
- /* Only unnamed $DATA attribute can be compressed. */
- BUG_ON(ni->name_len);
- /*
- * Reminder for later: If resident, the data is not actually
- * compressed. Only on the switch to non-resident does
- * compression kick in. This is in contrast to encrypted files
- * (see above).
- */
- ntfs_error(vi->i_sb, "Writing to compressed files is not "
- "implemented yet. Sorry.");
- err = -EOPNOTSUPP;
- goto out;
- }
- err = file_remove_privs(file);
- if (unlikely(err))
- goto out;
- /*
- * Our ->update_time method always succeeds thus file_update_time()
- * cannot fail either so there is no need to check the return code.
- */
- file_update_time(file);
- pos = iocb->ki_pos;
- /* The first byte after the last cluster being written to. */
- end = (pos + iov_iter_count(from) + vol->cluster_size_mask) &
- ~(u64)vol->cluster_size_mask;
- /*
- * If the write goes beyond the allocated size, extend the allocation
- * to cover the whole of the write, rounded up to the nearest cluster.
- */
- read_lock_irqsave(&ni->size_lock, flags);
- ll = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (end > ll) {
- /*
- * Extend the allocation without changing the data size.
- *
- * Note we ensure the allocation is big enough to at least
- * write some data but we do not require the allocation to be
- * complete, i.e. it may be partial.
- */
- ll = ntfs_attr_extend_allocation(ni, end, -1, pos);
- if (likely(ll >= 0)) {
- BUG_ON(pos >= ll);
- /* If the extension was partial truncate the write. */
- if (end > ll) {
- ntfs_debug("Truncating write to inode 0x%lx, "
- "attribute type 0x%x, because "
- "the allocation was only "
- "partially extended.",
- vi->i_ino, (unsigned)
- le32_to_cpu(ni->type));
- iov_iter_truncate(from, ll - pos);
- }
- } else {
- err = ll;
- read_lock_irqsave(&ni->size_lock, flags);
- ll = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- /* Perform a partial write if possible or fail. */
- if (pos < ll) {
- ntfs_debug("Truncating write to inode 0x%lx "
- "attribute type 0x%x, because "
- "extending the allocation "
- "failed (error %d).",
- vi->i_ino, (unsigned)
- le32_to_cpu(ni->type),
- (int)-err);
- iov_iter_truncate(from, ll - pos);
- } else {
- if (err != -ENOSPC)
- ntfs_error(vi->i_sb, "Cannot perform "
- "write to inode "
- "0x%lx, attribute "
- "type 0x%x, because "
- "extending the "
- "allocation failed "
- "(error %ld).",
- vi->i_ino, (unsigned)
- le32_to_cpu(ni->type),
- (long)-err);
- else
- ntfs_debug("Cannot perform write to "
- "inode 0x%lx, "
- "attribute type 0x%x, "
- "because there is not "
- "space left.",
- vi->i_ino, (unsigned)
- le32_to_cpu(ni->type));
- goto out;
- }
- }
- }
- /*
- * If the write starts beyond the initialized size, extend it up to the
- * beginning of the write and initialize all non-sparse space between
- * the old initialized size and the new one. This automatically also
- * increments the vfs inode->i_size to keep it above or equal to the
- * initialized_size.
- */
- read_lock_irqsave(&ni->size_lock, flags);
- ll = ni->initialized_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (pos > ll) {
- /*
- * Wait for ongoing direct i/o to complete before proceeding.
- * New direct i/o cannot start as we hold i_mutex.
- */
- inode_dio_wait(vi);
- err = ntfs_attr_extend_initialized(ni, pos);
- if (unlikely(err < 0))
- ntfs_error(vi->i_sb, "Cannot perform write to inode "
- "0x%lx, attribute type 0x%x, because "
- "extending the initialized size "
- "failed (error %d).", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type),
- (int)-err);
- }
-out:
- return err;
-}
-
-/**
- * __ntfs_grab_cache_pages - obtain a number of locked pages
- * @mapping: address space mapping from which to obtain page cache pages
- * @index: starting index in @mapping at which to begin obtaining pages
- * @nr_pages: number of page cache pages to obtain
- * @pages: array of pages in which to return the obtained page cache pages
- * @cached_page: allocated but as yet unused page
- *
- * Obtain @nr_pages locked page cache pages from the mapping @mapping and
- * starting at index @index.
- *
- * If a page is newly created, add it to lru list
- *
- * Note, the page locks are obtained in ascending page index order.
- */
-static inline int __ntfs_grab_cache_pages(struct address_space *mapping,
- pgoff_t index, const unsigned nr_pages, struct page **pages,
- struct page **cached_page)
-{
- int err, nr;
-
- BUG_ON(!nr_pages);
- err = nr = 0;
- do {
- pages[nr] = find_get_page_flags(mapping, index, FGP_LOCK |
- FGP_ACCESSED);
- if (!pages[nr]) {
- if (!*cached_page) {
- *cached_page = page_cache_alloc(mapping);
- if (unlikely(!*cached_page)) {
- err = -ENOMEM;
- goto err_out;
- }
- }
- err = add_to_page_cache_lru(*cached_page, mapping,
- index,
- mapping_gfp_constraint(mapping, GFP_KERNEL));
- if (unlikely(err)) {
- if (err == -EEXIST)
- continue;
- goto err_out;
- }
- pages[nr] = *cached_page;
- *cached_page = NULL;
- }
- index++;
- nr++;
- } while (nr < nr_pages);
-out:
- return err;
-err_out:
- while (nr > 0) {
- unlock_page(pages[--nr]);
- put_page(pages[nr]);
- }
- goto out;
-}
-
-static inline void ntfs_submit_bh_for_read(struct buffer_head *bh)
-{
- lock_buffer(bh);
- get_bh(bh);
- bh->b_end_io = end_buffer_read_sync;
- submit_bh(REQ_OP_READ, bh);
-}
-
-/**
- * ntfs_prepare_pages_for_non_resident_write - prepare pages for receiving data
- * @pages: array of destination pages
- * @nr_pages: number of pages in @pages
- * @pos: byte position in file at which the write begins
- * @bytes: number of bytes to be written
- *
- * This is called for non-resident attributes from ntfs_file_buffered_write()
- * with i_mutex held on the inode (@pages[0]->mapping->host). There are
- * @nr_pages pages in @pages which are locked but not kmap()ped. The source
- * data has not yet been copied into the @pages.
- *
- * Need to fill any holes with actual clusters, allocate buffers if necessary,
- * ensure all the buffers are mapped, and bring uptodate any buffers that are
- * only partially being written to.
- *
- * If @nr_pages is greater than one, we are guaranteed that the cluster size is
- * greater than PAGE_SIZE, that all pages in @pages are entirely inside
- * the same cluster and that they are the entirety of that cluster, and that
- * the cluster is sparse, i.e. we need to allocate a cluster to fill the hole.
- *
- * i_size is not to be modified yet.
- *
- * Return 0 on success or -errno on error.
- */
-static int ntfs_prepare_pages_for_non_resident_write(struct page **pages,
- unsigned nr_pages, s64 pos, size_t bytes)
-{
- VCN vcn, highest_vcn = 0, cpos, cend, bh_cpos, bh_cend;
- LCN lcn;
- s64 bh_pos, vcn_len, end, initialized_size;
- sector_t lcn_block;
- struct page *page;
- struct inode *vi;
- ntfs_inode *ni, *base_ni = NULL;
- ntfs_volume *vol;
- runlist_element *rl, *rl2;
- struct buffer_head *bh, *head, *wait[2], **wait_bh = wait;
- ntfs_attr_search_ctx *ctx = NULL;
- MFT_RECORD *m = NULL;
- ATTR_RECORD *a = NULL;
- unsigned long flags;
- u32 attr_rec_len = 0;
- unsigned blocksize, u;
- int err, mp_size;
- bool rl_write_locked, was_hole, is_retry;
- unsigned char blocksize_bits;
- struct {
- u8 runlist_merged:1;
- u8 mft_attr_mapped:1;
- u8 mp_rebuilt:1;
- u8 attr_switched:1;
- } status = { 0, 0, 0, 0 };
-
- BUG_ON(!nr_pages);
- BUG_ON(!pages);
- BUG_ON(!*pages);
- vi = pages[0]->mapping->host;
- ni = NTFS_I(vi);
- vol = ni->vol;
- ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, start page "
- "index 0x%lx, nr_pages 0x%x, pos 0x%llx, bytes 0x%zx.",
- vi->i_ino, ni->type, pages[0]->index, nr_pages,
- (long long)pos, bytes);
- blocksize = vol->sb->s_blocksize;
- blocksize_bits = vol->sb->s_blocksize_bits;
- u = 0;
- do {
- page = pages[u];
- BUG_ON(!page);
- /*
- * create_empty_buffers() will create uptodate/dirty buffers if
- * the page is uptodate/dirty.
- */
- if (!page_has_buffers(page)) {
- create_empty_buffers(page, blocksize, 0);
- if (unlikely(!page_has_buffers(page)))
- return -ENOMEM;
- }
- } while (++u < nr_pages);
- rl_write_locked = false;
- rl = NULL;
- err = 0;
- vcn = lcn = -1;
- vcn_len = 0;
- lcn_block = -1;
- was_hole = false;
- cpos = pos >> vol->cluster_size_bits;
- end = pos + bytes;
- cend = (end + vol->cluster_size - 1) >> vol->cluster_size_bits;
- /*
- * Loop over each page and for each page over each buffer. Use goto to
- * reduce indentation.
- */
- u = 0;
-do_next_page:
- page = pages[u];
- bh_pos = (s64)page->index << PAGE_SHIFT;
- bh = head = page_buffers(page);
- do {
- VCN cdelta;
- s64 bh_end;
- unsigned bh_cofs;
-
- /* Clear buffer_new on all buffers to reinitialise state. */
- if (buffer_new(bh))
- clear_buffer_new(bh);
- bh_end = bh_pos + blocksize;
- bh_cpos = bh_pos >> vol->cluster_size_bits;
- bh_cofs = bh_pos & vol->cluster_size_mask;
- if (buffer_mapped(bh)) {
- /*
- * The buffer is already mapped. If it is uptodate,
- * ignore it.
- */
- if (buffer_uptodate(bh))
- continue;
- /*
- * The buffer is not uptodate. If the page is uptodate
- * set the buffer uptodate and otherwise ignore it.
- */
- if (PageUptodate(page)) {
- set_buffer_uptodate(bh);
- continue;
- }
- /*
- * Neither the page nor the buffer are uptodate. If
- * the buffer is only partially being written to, we
- * need to read it in before the write, i.e. now.
- */
- if ((bh_pos < pos && bh_end > pos) ||
- (bh_pos < end && bh_end > end)) {
- /*
- * If the buffer is fully or partially within
- * the initialized size, do an actual read.
- * Otherwise, simply zero the buffer.
- */
- read_lock_irqsave(&ni->size_lock, flags);
- initialized_size = ni->initialized_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (bh_pos < initialized_size) {
- ntfs_submit_bh_for_read(bh);
- *wait_bh++ = bh;
- } else {
- zero_user(page, bh_offset(bh),
- blocksize);
- set_buffer_uptodate(bh);
- }
- }
- continue;
- }
- /* Unmapped buffer. Need to map it. */
- bh->b_bdev = vol->sb->s_bdev;
- /*
- * If the current buffer is in the same clusters as the map
- * cache, there is no need to check the runlist again. The
- * map cache is made up of @vcn, which is the first cached file
- * cluster, @vcn_len which is the number of cached file
- * clusters, @lcn is the device cluster corresponding to @vcn,
- * and @lcn_block is the block number corresponding to @lcn.
- */
- cdelta = bh_cpos - vcn;
- if (likely(!cdelta || (cdelta > 0 && cdelta < vcn_len))) {
-map_buffer_cached:
- BUG_ON(lcn < 0);
- bh->b_blocknr = lcn_block +
- (cdelta << (vol->cluster_size_bits -
- blocksize_bits)) +
- (bh_cofs >> blocksize_bits);
- set_buffer_mapped(bh);
- /*
- * If the page is uptodate so is the buffer. If the
- * buffer is fully outside the write, we ignore it if
- * it was already allocated and we mark it dirty so it
- * gets written out if we allocated it. On the other
- * hand, if we allocated the buffer but we are not
- * marking it dirty we set buffer_new so we can do
- * error recovery.
- */
- if (PageUptodate(page)) {
- if (!buffer_uptodate(bh))
- set_buffer_uptodate(bh);
- if (unlikely(was_hole)) {
- /* We allocated the buffer. */
- clean_bdev_bh_alias(bh);
- if (bh_end <= pos || bh_pos >= end)
- mark_buffer_dirty(bh);
- else
- set_buffer_new(bh);
- }
- continue;
- }
- /* Page is _not_ uptodate. */
- if (likely(!was_hole)) {
- /*
- * Buffer was already allocated. If it is not
- * uptodate and is only partially being written
- * to, we need to read it in before the write,
- * i.e. now.
- */
- if (!buffer_uptodate(bh) && bh_pos < end &&
- bh_end > pos &&
- (bh_pos < pos ||
- bh_end > end)) {
- /*
- * If the buffer is fully or partially
- * within the initialized size, do an
- * actual read. Otherwise, simply zero
- * the buffer.
- */
- read_lock_irqsave(&ni->size_lock,
- flags);
- initialized_size = ni->initialized_size;
- read_unlock_irqrestore(&ni->size_lock,
- flags);
- if (bh_pos < initialized_size) {
- ntfs_submit_bh_for_read(bh);
- *wait_bh++ = bh;
- } else {
- zero_user(page, bh_offset(bh),
- blocksize);
- set_buffer_uptodate(bh);
- }
- }
- continue;
- }
- /* We allocated the buffer. */
- clean_bdev_bh_alias(bh);
- /*
- * If the buffer is fully outside the write, zero it,
- * set it uptodate, and mark it dirty so it gets
- * written out. If it is partially being written to,
- * zero region surrounding the write but leave it to
- * commit write to do anything else. Finally, if the
- * buffer is fully being overwritten, do nothing.
- */
- if (bh_end <= pos || bh_pos >= end) {
- if (!buffer_uptodate(bh)) {
- zero_user(page, bh_offset(bh),
- blocksize);
- set_buffer_uptodate(bh);
- }
- mark_buffer_dirty(bh);
- continue;
- }
- set_buffer_new(bh);
- if (!buffer_uptodate(bh) &&
- (bh_pos < pos || bh_end > end)) {
- u8 *kaddr;
- unsigned pofs;
-
- kaddr = kmap_atomic(page);
- if (bh_pos < pos) {
- pofs = bh_pos & ~PAGE_MASK;
- memset(kaddr + pofs, 0, pos - bh_pos);
- }
- if (bh_end > end) {
- pofs = end & ~PAGE_MASK;
- memset(kaddr + pofs, 0, bh_end - end);
- }
- kunmap_atomic(kaddr);
- flush_dcache_page(page);
- }
- continue;
- }
- /*
- * Slow path: this is the first buffer in the cluster. If it
- * is outside allocated size and is not uptodate, zero it and
- * set it uptodate.
- */
- read_lock_irqsave(&ni->size_lock, flags);
- initialized_size = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (bh_pos > initialized_size) {
- if (PageUptodate(page)) {
- if (!buffer_uptodate(bh))
- set_buffer_uptodate(bh);
- } else if (!buffer_uptodate(bh)) {
- zero_user(page, bh_offset(bh), blocksize);
- set_buffer_uptodate(bh);
- }
- continue;
- }
- is_retry = false;
- if (!rl) {
- down_read(&ni->runlist.lock);
-retry_remap:
- rl = ni->runlist.rl;
- }
- if (likely(rl != NULL)) {
- /* Seek to element containing target cluster. */
- while (rl->length && rl[1].vcn <= bh_cpos)
- rl++;
- lcn = ntfs_rl_vcn_to_lcn(rl, bh_cpos);
- if (likely(lcn >= 0)) {
- /*
- * Successful remap, setup the map cache and
- * use that to deal with the buffer.
- */
- was_hole = false;
- vcn = bh_cpos;
- vcn_len = rl[1].vcn - vcn;
- lcn_block = lcn << (vol->cluster_size_bits -
- blocksize_bits);
- cdelta = 0;
- /*
- * If the number of remaining clusters touched
- * by the write is smaller or equal to the
- * number of cached clusters, unlock the
- * runlist as the map cache will be used from
- * now on.
- */
- if (likely(vcn + vcn_len >= cend)) {
- if (rl_write_locked) {
- up_write(&ni->runlist.lock);
- rl_write_locked = false;
- } else
- up_read(&ni->runlist.lock);
- rl = NULL;
- }
- goto map_buffer_cached;
- }
- } else
- lcn = LCN_RL_NOT_MAPPED;
- /*
- * If it is not a hole and not out of bounds, the runlist is
- * probably unmapped so try to map it now.
- */
- if (unlikely(lcn != LCN_HOLE && lcn != LCN_ENOENT)) {
- if (likely(!is_retry && lcn == LCN_RL_NOT_MAPPED)) {
- /* Attempt to map runlist. */
- if (!rl_write_locked) {
- /*
- * We need the runlist locked for
- * writing, so if it is locked for
- * reading relock it now and retry in
- * case it changed whilst we dropped
- * the lock.
- */
- up_read(&ni->runlist.lock);
- down_write(&ni->runlist.lock);
- rl_write_locked = true;
- goto retry_remap;
- }
- err = ntfs_map_runlist_nolock(ni, bh_cpos,
- NULL);
- if (likely(!err)) {
- is_retry = true;
- goto retry_remap;
- }
- /*
- * If @vcn is out of bounds, pretend @lcn is
- * LCN_ENOENT. As long as the buffer is out
- * of bounds this will work fine.
- */
- if (err == -ENOENT) {
- lcn = LCN_ENOENT;
- err = 0;
- goto rl_not_mapped_enoent;
- }
- } else
- err = -EIO;
- /* Failed to map the buffer, even after retrying. */
- bh->b_blocknr = -1;
- ntfs_error(vol->sb, "Failed to write to inode 0x%lx, "
- "attribute type 0x%x, vcn 0x%llx, "
- "vcn offset 0x%x, because its "
- "location on disk could not be "
- "determined%s (error code %i).",
- ni->mft_no, ni->type,
- (unsigned long long)bh_cpos,
- (unsigned)bh_pos &
- vol->cluster_size_mask,
- is_retry ? " even after retrying" : "",
- err);
- break;
- }
-rl_not_mapped_enoent:
- /*
- * The buffer is in a hole or out of bounds. We need to fill
- * the hole, unless the buffer is in a cluster which is not
- * touched by the write, in which case we just leave the buffer
- * unmapped. This can only happen when the cluster size is
- * less than the page cache size.
- */
- if (unlikely(vol->cluster_size < PAGE_SIZE)) {
- bh_cend = (bh_end + vol->cluster_size - 1) >>
- vol->cluster_size_bits;
- if ((bh_cend <= cpos || bh_cpos >= cend)) {
- bh->b_blocknr = -1;
- /*
- * If the buffer is uptodate we skip it. If it
- * is not but the page is uptodate, we can set
- * the buffer uptodate. If the page is not
- * uptodate, we can clear the buffer and set it
- * uptodate. Whether this is worthwhile is
- * debatable and this could be removed.
- */
- if (PageUptodate(page)) {
- if (!buffer_uptodate(bh))
- set_buffer_uptodate(bh);
- } else if (!buffer_uptodate(bh)) {
- zero_user(page, bh_offset(bh),
- blocksize);
- set_buffer_uptodate(bh);
- }
- continue;
- }
- }
- /*
- * Out of bounds buffer is invalid if it was not really out of
- * bounds.
- */
- BUG_ON(lcn != LCN_HOLE);
- /*
- * We need the runlist locked for writing, so if it is locked
- * for reading relock it now and retry in case it changed
- * whilst we dropped the lock.
- */
- BUG_ON(!rl);
- if (!rl_write_locked) {
- up_read(&ni->runlist.lock);
- down_write(&ni->runlist.lock);
- rl_write_locked = true;
- goto retry_remap;
- }
- /* Find the previous last allocated cluster. */
- BUG_ON(rl->lcn != LCN_HOLE);
- lcn = -1;
- rl2 = rl;
- while (--rl2 >= ni->runlist.rl) {
- if (rl2->lcn >= 0) {
- lcn = rl2->lcn + rl2->length;
- break;
- }
- }
- rl2 = ntfs_cluster_alloc(vol, bh_cpos, 1, lcn, DATA_ZONE,
- false);
- if (IS_ERR(rl2)) {
- err = PTR_ERR(rl2);
- ntfs_debug("Failed to allocate cluster, error code %i.",
- err);
- break;
- }
- lcn = rl2->lcn;
- rl = ntfs_runlists_merge(ni->runlist.rl, rl2);
- if (IS_ERR(rl)) {
- err = PTR_ERR(rl);
- if (err != -ENOMEM)
- err = -EIO;
- if (ntfs_cluster_free_from_rl(vol, rl2)) {
- ntfs_error(vol->sb, "Failed to release "
- "allocated cluster in error "
- "code path. Run chkdsk to "
- "recover the lost cluster.");
- NVolSetErrors(vol);
- }
- ntfs_free(rl2);
- break;
- }
- ni->runlist.rl = rl;
- status.runlist_merged = 1;
- ntfs_debug("Allocated cluster, lcn 0x%llx.",
- (unsigned long long)lcn);
- /* Map and lock the mft record and get the attribute record. */
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- break;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- unmap_mft_record(base_ni);
- break;
- }
- status.mft_attr_mapped = 1;
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, bh_cpos, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- break;
- }
- m = ctx->mrec;
- a = ctx->attr;
- /*
- * Find the runlist element with which the attribute extent
- * starts. Note, we cannot use the _attr_ version because we
- * have mapped the mft record. That is ok because we know the
- * runlist fragment must be mapped already to have ever gotten
- * here, so we can just use the _rl_ version.
- */
- vcn = sle64_to_cpu(a->data.non_resident.lowest_vcn);
- rl2 = ntfs_rl_find_vcn_nolock(rl, vcn);
- BUG_ON(!rl2);
- BUG_ON(!rl2->length);
- BUG_ON(rl2->lcn < LCN_HOLE);
- highest_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn);
- /*
- * If @highest_vcn is zero, calculate the real highest_vcn
- * (which can really be zero).
- */
- if (!highest_vcn)
- highest_vcn = (sle64_to_cpu(
- a->data.non_resident.allocated_size) >>
- vol->cluster_size_bits) - 1;
- /*
- * Determine the size of the mapping pairs array for the new
- * extent, i.e. the old extent with the hole filled.
- */
- mp_size = ntfs_get_size_for_mapping_pairs(vol, rl2, vcn,
- highest_vcn);
- if (unlikely(mp_size <= 0)) {
- if (!(err = mp_size))
- err = -EIO;
- ntfs_debug("Failed to get size for mapping pairs "
- "array, error code %i.", err);
- break;
- }
- /*
- * Resize the attribute record to fit the new mapping pairs
- * array.
- */
- attr_rec_len = le32_to_cpu(a->length);
- err = ntfs_attr_record_resize(m, a, mp_size + le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset));
- if (unlikely(err)) {
- BUG_ON(err != -ENOSPC);
- // TODO: Deal with this by using the current attribute
- // and fill it with as much of the mapping pairs
- // array as possible. Then loop over each attribute
- // extent rewriting the mapping pairs arrays as we go
- // along and if when we reach the end we have not
- // enough space, try to resize the last attribute
- // extent and if even that fails, add a new attribute
- // extent.
- // We could also try to resize at each step in the hope
- // that we will not need to rewrite every single extent.
- // Note, we may need to decompress some extents to fill
- // the runlist as we are walking the extents...
- ntfs_error(vol->sb, "Not enough space in the mft "
- "record for the extended attribute "
- "record. This case is not "
- "implemented yet.");
- err = -EOPNOTSUPP;
- break ;
- }
- status.mp_rebuilt = 1;
- /*
- * Generate the mapping pairs array directly into the attribute
- * record.
- */
- err = ntfs_mapping_pairs_build(vol, (u8*)a + le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset),
- mp_size, rl2, vcn, highest_vcn, NULL);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Cannot fill hole in inode 0x%lx, "
- "attribute type 0x%x, because building "
- "the mapping pairs failed with error "
- "code %i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- err = -EIO;
- break;
- }
- /* Update the highest_vcn but only if it was not set. */
- if (unlikely(!a->data.non_resident.highest_vcn))
- a->data.non_resident.highest_vcn =
- cpu_to_sle64(highest_vcn);
- /*
- * If the attribute is sparse/compressed, update the compressed
- * size in the ntfs_inode structure and the attribute record.
- */
- if (likely(NInoSparse(ni) || NInoCompressed(ni))) {
- /*
- * If we are not in the first attribute extent, switch
- * to it, but first ensure the changes will make it to
- * disk later.
- */
- if (a->data.non_resident.lowest_vcn) {
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_reinit_search_ctx(ctx);
- err = ntfs_attr_lookup(ni->type, ni->name,
- ni->name_len, CASE_SENSITIVE,
- 0, NULL, 0, ctx);
- if (unlikely(err)) {
- status.attr_switched = 1;
- break;
- }
- /* @m is not used any more so do not set it. */
- a = ctx->attr;
- }
- write_lock_irqsave(&ni->size_lock, flags);
- ni->itype.compressed.size += vol->cluster_size;
- a->data.non_resident.compressed_size =
- cpu_to_sle64(ni->itype.compressed.size);
- write_unlock_irqrestore(&ni->size_lock, flags);
- }
- /* Ensure the changes make it to disk. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- /* Successfully filled the hole. */
- status.runlist_merged = 0;
- status.mft_attr_mapped = 0;
- status.mp_rebuilt = 0;
- /* Setup the map cache and use that to deal with the buffer. */
- was_hole = true;
- vcn = bh_cpos;
- vcn_len = 1;
- lcn_block = lcn << (vol->cluster_size_bits - blocksize_bits);
- cdelta = 0;
- /*
- * If the number of remaining clusters in the @pages is smaller
- * or equal to the number of cached clusters, unlock the
- * runlist as the map cache will be used from now on.
- */
- if (likely(vcn + vcn_len >= cend)) {
- up_write(&ni->runlist.lock);
- rl_write_locked = false;
- rl = NULL;
- }
- goto map_buffer_cached;
- } while (bh_pos += blocksize, (bh = bh->b_this_page) != head);
- /* If there are no errors, do the next page. */
- if (likely(!err && ++u < nr_pages))
- goto do_next_page;
- /* If there are no errors, release the runlist lock if we took it. */
- if (likely(!err)) {
- if (unlikely(rl_write_locked)) {
- up_write(&ni->runlist.lock);
- rl_write_locked = false;
- } else if (unlikely(rl))
- up_read(&ni->runlist.lock);
- rl = NULL;
- }
- /* If we issued read requests, let them complete. */
- read_lock_irqsave(&ni->size_lock, flags);
- initialized_size = ni->initialized_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- while (wait_bh > wait) {
- bh = *--wait_bh;
- wait_on_buffer(bh);
- if (likely(buffer_uptodate(bh))) {
- page = bh->b_page;
- bh_pos = ((s64)page->index << PAGE_SHIFT) +
- bh_offset(bh);
- /*
- * If the buffer overflows the initialized size, need
- * to zero the overflowing region.
- */
- if (unlikely(bh_pos + blocksize > initialized_size)) {
- int ofs = 0;
-
- if (likely(bh_pos < initialized_size))
- ofs = initialized_size - bh_pos;
- zero_user_segment(page, bh_offset(bh) + ofs,
- blocksize);
- }
- } else /* if (unlikely(!buffer_uptodate(bh))) */
- err = -EIO;
- }
- if (likely(!err)) {
- /* Clear buffer_new on all buffers. */
- u = 0;
- do {
- bh = head = page_buffers(pages[u]);
- do {
- if (buffer_new(bh))
- clear_buffer_new(bh);
- } while ((bh = bh->b_this_page) != head);
- } while (++u < nr_pages);
- ntfs_debug("Done.");
- return err;
- }
- if (status.attr_switched) {
- /* Get back to the attribute extent we modified. */
- ntfs_attr_reinit_search_ctx(ctx);
- if (ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, bh_cpos, NULL, 0, ctx)) {
- ntfs_error(vol->sb, "Failed to find required "
- "attribute extent of attribute in "
- "error code path. Run chkdsk to "
- "recover.");
- write_lock_irqsave(&ni->size_lock, flags);
- ni->itype.compressed.size += vol->cluster_size;
- write_unlock_irqrestore(&ni->size_lock, flags);
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- /*
- * The only thing that is now wrong is the compressed
- * size of the base attribute extent which chkdsk
- * should be able to fix.
- */
- NVolSetErrors(vol);
- } else {
- m = ctx->mrec;
- a = ctx->attr;
- status.attr_switched = 0;
- }
- }
- /*
- * If the runlist has been modified, need to restore it by punching a
- * hole into it and we then need to deallocate the on-disk cluster as
- * well. Note, we only modify the runlist if we are able to generate a
- * new mapping pairs array, i.e. only when the mapped attribute extent
- * is not switched.
- */
- if (status.runlist_merged && !status.attr_switched) {
- BUG_ON(!rl_write_locked);
- /* Make the file cluster we allocated sparse in the runlist. */
- if (ntfs_rl_punch_nolock(vol, &ni->runlist, bh_cpos, 1)) {
- ntfs_error(vol->sb, "Failed to punch hole into "
- "attribute runlist in error code "
- "path. Run chkdsk to recover the "
- "lost cluster.");
- NVolSetErrors(vol);
- } else /* if (success) */ {
- status.runlist_merged = 0;
- /*
- * Deallocate the on-disk cluster we allocated but only
- * if we succeeded in punching its vcn out of the
- * runlist.
- */
- down_write(&vol->lcnbmp_lock);
- if (ntfs_bitmap_clear_bit(vol->lcnbmp_ino, lcn)) {
- ntfs_error(vol->sb, "Failed to release "
- "allocated cluster in error "
- "code path. Run chkdsk to "
- "recover the lost cluster.");
- NVolSetErrors(vol);
- }
- up_write(&vol->lcnbmp_lock);
- }
- }
- /*
- * Resize the attribute record to its old size and rebuild the mapping
- * pairs array. Note, we only can do this if the runlist has been
- * restored to its old state which also implies that the mapped
- * attribute extent is not switched.
- */
- if (status.mp_rebuilt && !status.runlist_merged) {
- if (ntfs_attr_record_resize(m, a, attr_rec_len)) {
- ntfs_error(vol->sb, "Failed to restore attribute "
- "record in error code path. Run "
- "chkdsk to recover.");
- NVolSetErrors(vol);
- } else /* if (success) */ {
- if (ntfs_mapping_pairs_build(vol, (u8*)a +
- le16_to_cpu(a->data.non_resident.
- mapping_pairs_offset), attr_rec_len -
- le16_to_cpu(a->data.non_resident.
- mapping_pairs_offset), ni->runlist.rl,
- vcn, highest_vcn, NULL)) {
- ntfs_error(vol->sb, "Failed to restore "
- "mapping pairs array in error "
- "code path. Run chkdsk to "
- "recover.");
- NVolSetErrors(vol);
- }
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- }
- }
- /* Release the mft record and the attribute. */
- if (status.mft_attr_mapped) {
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- }
- /* Release the runlist lock. */
- if (rl_write_locked)
- up_write(&ni->runlist.lock);
- else if (rl)
- up_read(&ni->runlist.lock);
- /*
- * Zero out any newly allocated blocks to avoid exposing stale data.
- * If BH_New is set, we know that the block was newly allocated above
- * and that it has not been fully zeroed and marked dirty yet.
- */
- nr_pages = u;
- u = 0;
- end = bh_cpos << vol->cluster_size_bits;
- do {
- page = pages[u];
- bh = head = page_buffers(page);
- do {
- if (u == nr_pages &&
- ((s64)page->index << PAGE_SHIFT) +
- bh_offset(bh) >= end)
- break;
- if (!buffer_new(bh))
- continue;
- clear_buffer_new(bh);
- if (!buffer_uptodate(bh)) {
- if (PageUptodate(page))
- set_buffer_uptodate(bh);
- else {
- zero_user(page, bh_offset(bh),
- blocksize);
- set_buffer_uptodate(bh);
- }
- }
- mark_buffer_dirty(bh);
- } while ((bh = bh->b_this_page) != head);
- } while (++u <= nr_pages);
- ntfs_error(vol->sb, "Failed. Returning error code %i.", err);
- return err;
-}
-
-static inline void ntfs_flush_dcache_pages(struct page **pages,
- unsigned nr_pages)
-{
- BUG_ON(!nr_pages);
- /*
- * Warning: Do not do the decrement at the same time as the call to
- * flush_dcache_page() because it is a NULL macro on i386 and hence the
- * decrement never happens so the loop never terminates.
- */
- do {
- --nr_pages;
- flush_dcache_page(pages[nr_pages]);
- } while (nr_pages > 0);
-}
-
-/**
- * ntfs_commit_pages_after_non_resident_write - commit the received data
- * @pages: array of destination pages
- * @nr_pages: number of pages in @pages
- * @pos: byte position in file at which the write begins
- * @bytes: number of bytes to be written
- *
- * See description of ntfs_commit_pages_after_write(), below.
- */
-static inline int ntfs_commit_pages_after_non_resident_write(
- struct page **pages, const unsigned nr_pages,
- s64 pos, size_t bytes)
-{
- s64 end, initialized_size;
- struct inode *vi;
- ntfs_inode *ni, *base_ni;
- struct buffer_head *bh, *head;
- ntfs_attr_search_ctx *ctx;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- unsigned long flags;
- unsigned blocksize, u;
- int err;
-
- vi = pages[0]->mapping->host;
- ni = NTFS_I(vi);
- blocksize = vi->i_sb->s_blocksize;
- end = pos + bytes;
- u = 0;
- do {
- s64 bh_pos;
- struct page *page;
- bool partial;
-
- page = pages[u];
- bh_pos = (s64)page->index << PAGE_SHIFT;
- bh = head = page_buffers(page);
- partial = false;
- do {
- s64 bh_end;
-
- bh_end = bh_pos + blocksize;
- if (bh_end <= pos || bh_pos >= end) {
- if (!buffer_uptodate(bh))
- partial = true;
- } else {
- set_buffer_uptodate(bh);
- mark_buffer_dirty(bh);
- }
- } while (bh_pos += blocksize, (bh = bh->b_this_page) != head);
- /*
- * If all buffers are now uptodate but the page is not, set the
- * page uptodate.
- */
- if (!partial && !PageUptodate(page))
- SetPageUptodate(page);
- } while (++u < nr_pages);
- /*
- * Finally, if we do not need to update initialized_size or i_size we
- * are finished.
- */
- read_lock_irqsave(&ni->size_lock, flags);
- initialized_size = ni->initialized_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- if (end <= initialized_size) {
- ntfs_debug("Done.");
- return 0;
- }
- /*
- * Update initialized_size/i_size as appropriate, both in the inode and
- * the mft record.
- */
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- /* Map, pin, and lock the mft record. */
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- ctx = NULL;
- goto err_out;
- }
- BUG_ON(!NInoNonResident(ni));
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- goto err_out;
- }
- a = ctx->attr;
- BUG_ON(!a->non_resident);
- write_lock_irqsave(&ni->size_lock, flags);
- BUG_ON(end > ni->allocated_size);
- ni->initialized_size = end;
- a->data.non_resident.initialized_size = cpu_to_sle64(end);
- if (end > i_size_read(vi)) {
- i_size_write(vi, end);
- a->data.non_resident.data_size =
- a->data.non_resident.initialized_size;
- }
- write_unlock_irqrestore(&ni->size_lock, flags);
- /* Mark the mft record dirty, so it gets written back. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- ntfs_debug("Done.");
- return 0;
-err_out:
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
- ntfs_error(vi->i_sb, "Failed to update initialized_size/i_size (error "
- "code %i).", err);
- if (err != -ENOMEM)
- NVolSetErrors(ni->vol);
- return err;
-}
-
-/**
- * ntfs_commit_pages_after_write - commit the received data
- * @pages: array of destination pages
- * @nr_pages: number of pages in @pages
- * @pos: byte position in file at which the write begins
- * @bytes: number of bytes to be written
- *
- * This is called from ntfs_file_buffered_write() with i_mutex held on the inode
- * (@pages[0]->mapping->host). There are @nr_pages pages in @pages which are
- * locked but not kmap()ped. The source data has already been copied into the
- * @page. ntfs_prepare_pages_for_non_resident_write() has been called before
- * the data was copied (for non-resident attributes only) and it returned
- * success.
- *
- * Need to set uptodate and mark dirty all buffers within the boundary of the
- * write. If all buffers in a page are uptodate we set the page uptodate, too.
- *
- * Setting the buffers dirty ensures that they get written out later when
- * ntfs_writepage() is invoked by the VM.
- *
- * Finally, we need to update i_size and initialized_size as appropriate both
- * in the inode and the mft record.
- *
- * This is modelled after fs/buffer.c::generic_commit_write(), which marks
- * buffers uptodate and dirty, sets the page uptodate if all buffers in the
- * page are uptodate, and updates i_size if the end of io is beyond i_size. In
- * that case, it also marks the inode dirty.
- *
- * If things have gone as outlined in
- * ntfs_prepare_pages_for_non_resident_write(), we do not need to do any page
- * content modifications here for non-resident attributes. For resident
- * attributes we need to do the uptodate bringing here which we combine with
- * the copying into the mft record which means we save one atomic kmap.
- *
- * Return 0 on success or -errno on error.
- */
-static int ntfs_commit_pages_after_write(struct page **pages,
- const unsigned nr_pages, s64 pos, size_t bytes)
-{
- s64 end, initialized_size;
- loff_t i_size;
- struct inode *vi;
- ntfs_inode *ni, *base_ni;
- struct page *page;
- ntfs_attr_search_ctx *ctx;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- char *kattr, *kaddr;
- unsigned long flags;
- u32 attr_len;
- int err;
-
- BUG_ON(!nr_pages);
- BUG_ON(!pages);
- page = pages[0];
- BUG_ON(!page);
- vi = page->mapping->host;
- ni = NTFS_I(vi);
- ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, start page "
- "index 0x%lx, nr_pages 0x%x, pos 0x%llx, bytes 0x%zx.",
- vi->i_ino, ni->type, page->index, nr_pages,
- (long long)pos, bytes);
- if (NInoNonResident(ni))
- return ntfs_commit_pages_after_non_resident_write(pages,
- nr_pages, pos, bytes);
- BUG_ON(nr_pages > 1);
- /*
- * Attribute is resident, implying it is not compressed, encrypted, or
- * sparse.
- */
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- BUG_ON(NInoNonResident(ni));
- /* Map, pin, and lock the mft record. */
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- ctx = NULL;
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- err = -EIO;
- goto err_out;
- }
- a = ctx->attr;
- BUG_ON(a->non_resident);
- /* The total length of the attribute value. */
- attr_len = le32_to_cpu(a->data.resident.value_length);
- i_size = i_size_read(vi);
- BUG_ON(attr_len != i_size);
- BUG_ON(pos > attr_len);
- end = pos + bytes;
- BUG_ON(end > le32_to_cpu(a->length) -
- le16_to_cpu(a->data.resident.value_offset));
- kattr = (u8*)a + le16_to_cpu(a->data.resident.value_offset);
- kaddr = kmap_atomic(page);
- /* Copy the received data from the page to the mft record. */
- memcpy(kattr + pos, kaddr + pos, bytes);
- /* Update the attribute length if necessary. */
- if (end > attr_len) {
- attr_len = end;
- a->data.resident.value_length = cpu_to_le32(attr_len);
- }
- /*
- * If the page is not uptodate, bring the out of bounds area(s)
- * uptodate by copying data from the mft record to the page.
- */
- if (!PageUptodate(page)) {
- if (pos > 0)
- memcpy(kaddr, kattr, pos);
- if (end < attr_len)
- memcpy(kaddr + end, kattr + end, attr_len - end);
- /* Zero the region outside the end of the attribute value. */
- memset(kaddr + attr_len, 0, PAGE_SIZE - attr_len);
- flush_dcache_page(page);
- SetPageUptodate(page);
- }
- kunmap_atomic(kaddr);
- /* Update initialized_size/i_size if necessary. */
- read_lock_irqsave(&ni->size_lock, flags);
- initialized_size = ni->initialized_size;
- BUG_ON(end > ni->allocated_size);
- read_unlock_irqrestore(&ni->size_lock, flags);
- BUG_ON(initialized_size != i_size);
- if (end > initialized_size) {
- write_lock_irqsave(&ni->size_lock, flags);
- ni->initialized_size = end;
- i_size_write(vi, end);
- write_unlock_irqrestore(&ni->size_lock, flags);
- }
- /* Mark the mft record dirty, so it gets written back. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- ntfs_debug("Done.");
- return 0;
-err_out:
- if (err == -ENOMEM) {
- ntfs_warning(vi->i_sb, "Error allocating memory required to "
- "commit the write.");
- if (PageUptodate(page)) {
- ntfs_warning(vi->i_sb, "Page is uptodate, setting "
- "dirty so the write will be retried "
- "later on by the VM.");
- /*
- * Put the page on mapping->dirty_pages, but leave its
- * buffers' dirty state as-is.
- */
- __set_page_dirty_nobuffers(page);
- err = 0;
- } else
- ntfs_error(vi->i_sb, "Page is not uptodate. Written "
- "data has been lost.");
- } else {
- ntfs_error(vi->i_sb, "Resident attribute commit write failed "
- "with error %i.", err);
- NVolSetErrors(ni->vol);
- }
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
- return err;
-}
-
-/*
- * Copy as much as we can into the pages and return the number of bytes which
- * were successfully copied. If a fault is encountered then clear the pages
- * out to (ofs + bytes) and return the number of bytes which were copied.
- */
-static size_t ntfs_copy_from_user_iter(struct page **pages, unsigned nr_pages,
- unsigned ofs, struct iov_iter *i, size_t bytes)
-{
- struct page **last_page = pages + nr_pages;
- size_t total = 0;
- unsigned len, copied;
-
- do {
- len = PAGE_SIZE - ofs;
- if (len > bytes)
- len = bytes;
- copied = copy_page_from_iter_atomic(*pages, ofs, len, i);
- total += copied;
- bytes -= copied;
- if (!bytes)
- break;
- if (copied < len)
- goto err;
- ofs = 0;
- } while (++pages < last_page);
-out:
- return total;
-err:
- /* Zero the rest of the target like __copy_from_user(). */
- len = PAGE_SIZE - copied;
- do {
- if (len > bytes)
- len = bytes;
- zero_user(*pages, copied, len);
- bytes -= len;
- copied = 0;
- len = PAGE_SIZE;
- } while (++pages < last_page);
- goto out;
-}
-
-/**
- * ntfs_perform_write - perform buffered write to a file
- * @file: file to write to
- * @i: iov_iter with data to write
- * @pos: byte offset in file at which to begin writing to
- */
-static ssize_t ntfs_perform_write(struct file *file, struct iov_iter *i,
- loff_t pos)
-{
- struct address_space *mapping = file->f_mapping;
- struct inode *vi = mapping->host;
- ntfs_inode *ni = NTFS_I(vi);
- ntfs_volume *vol = ni->vol;
- struct page *pages[NTFS_MAX_PAGES_PER_CLUSTER];
- struct page *cached_page = NULL;
- VCN last_vcn;
- LCN lcn;
- size_t bytes;
- ssize_t status, written = 0;
- unsigned nr_pages;
-
- ntfs_debug("Entering for i_ino 0x%lx, attribute type 0x%x, pos "
- "0x%llx, count 0x%lx.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type),
- (unsigned long long)pos,
- (unsigned long)iov_iter_count(i));
- /*
- * If a previous ntfs_truncate() failed, repeat it and abort if it
- * fails again.
- */
- if (unlikely(NInoTruncateFailed(ni))) {
- int err;
-
- inode_dio_wait(vi);
- err = ntfs_truncate(vi);
- if (err || NInoTruncateFailed(ni)) {
- if (!err)
- err = -EIO;
- ntfs_error(vol->sb, "Cannot perform write to inode "
- "0x%lx, attribute type 0x%x, because "
- "ntfs_truncate() failed (error code "
- "%i).", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- return err;
- }
- }
- /*
- * Determine the number of pages per cluster for non-resident
- * attributes.
- */
- nr_pages = 1;
- if (vol->cluster_size > PAGE_SIZE && NInoNonResident(ni))
- nr_pages = vol->cluster_size >> PAGE_SHIFT;
- last_vcn = -1;
- do {
- VCN vcn;
- pgoff_t start_idx;
- unsigned ofs, do_pages, u;
- size_t copied;
-
- start_idx = pos >> PAGE_SHIFT;
- ofs = pos & ~PAGE_MASK;
- bytes = PAGE_SIZE - ofs;
- do_pages = 1;
- if (nr_pages > 1) {
- vcn = pos >> vol->cluster_size_bits;
- if (vcn != last_vcn) {
- last_vcn = vcn;
- /*
- * Get the lcn of the vcn the write is in. If
- * it is a hole, need to lock down all pages in
- * the cluster.
- */
- down_read(&ni->runlist.lock);
- lcn = ntfs_attr_vcn_to_lcn_nolock(ni, pos >>
- vol->cluster_size_bits, false);
- up_read(&ni->runlist.lock);
- if (unlikely(lcn < LCN_HOLE)) {
- if (lcn == LCN_ENOMEM)
- status = -ENOMEM;
- else {
- status = -EIO;
- ntfs_error(vol->sb, "Cannot "
- "perform write to "
- "inode 0x%lx, "
- "attribute type 0x%x, "
- "because the attribute "
- "is corrupt.",
- vi->i_ino, (unsigned)
- le32_to_cpu(ni->type));
- }
- break;
- }
- if (lcn == LCN_HOLE) {
- start_idx = (pos & ~(s64)
- vol->cluster_size_mask)
- >> PAGE_SHIFT;
- bytes = vol->cluster_size - (pos &
- vol->cluster_size_mask);
- do_pages = nr_pages;
- }
- }
- }
- if (bytes > iov_iter_count(i))
- bytes = iov_iter_count(i);
-again:
- /*
- * Bring in the user page(s) that we will copy from _first_.
- * Otherwise there is a nasty deadlock on copying from the same
- * page(s) as we are writing to, without it/them being marked
- * up-to-date. Note, at present there is nothing to stop the
- * pages being swapped out between us bringing them into memory
- * and doing the actual copying.
- */
- if (unlikely(fault_in_iov_iter_readable(i, bytes))) {
- status = -EFAULT;
- break;
- }
- /* Get and lock @do_pages starting at index @start_idx. */
- status = __ntfs_grab_cache_pages(mapping, start_idx, do_pages,
- pages, &cached_page);
- if (unlikely(status))
- break;
- /*
- * For non-resident attributes, we need to fill any holes with
- * actual clusters and ensure all bufferes are mapped. We also
- * need to bring uptodate any buffers that are only partially
- * being written to.
- */
- if (NInoNonResident(ni)) {
- status = ntfs_prepare_pages_for_non_resident_write(
- pages, do_pages, pos, bytes);
- if (unlikely(status)) {
- do {
- unlock_page(pages[--do_pages]);
- put_page(pages[do_pages]);
- } while (do_pages);
- break;
- }
- }
- u = (pos >> PAGE_SHIFT) - pages[0]->index;
- copied = ntfs_copy_from_user_iter(pages + u, do_pages - u, ofs,
- i, bytes);
- ntfs_flush_dcache_pages(pages + u, do_pages - u);
- status = 0;
- if (likely(copied == bytes)) {
- status = ntfs_commit_pages_after_write(pages, do_pages,
- pos, bytes);
- }
- do {
- unlock_page(pages[--do_pages]);
- put_page(pages[do_pages]);
- } while (do_pages);
- if (unlikely(status < 0)) {
- iov_iter_revert(i, copied);
- break;
- }
- cond_resched();
- if (unlikely(copied < bytes)) {
- iov_iter_revert(i, copied);
- if (copied)
- bytes = copied;
- else if (bytes > PAGE_SIZE - ofs)
- bytes = PAGE_SIZE - ofs;
- goto again;
- }
- pos += copied;
- written += copied;
- balance_dirty_pages_ratelimited(mapping);
- if (fatal_signal_pending(current)) {
- status = -EINTR;
- break;
- }
- } while (iov_iter_count(i));
- if (cached_page)
- put_page(cached_page);
- ntfs_debug("Done. Returning %s (written 0x%lx, status %li).",
- written ? "written" : "status", (unsigned long)written,
- (long)status);
- return written ? written : status;
-}
-
-/**
- * ntfs_file_write_iter - simple wrapper for ntfs_file_write_iter_nolock()
- * @iocb: IO state structure
- * @from: iov_iter with data to write
- *
- * Basically the same as generic_file_write_iter() except that it ends up
- * up calling ntfs_perform_write() instead of generic_perform_write() and that
- * O_DIRECT is not implemented.
- */
-static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
-{
- struct file *file = iocb->ki_filp;
- struct inode *vi = file_inode(file);
- ssize_t written = 0;
- ssize_t err;
-
- inode_lock(vi);
- /* We can write back this queue in page reclaim. */
- current->backing_dev_info = inode_to_bdi(vi);
- err = ntfs_prepare_file_for_write(iocb, from);
- if (iov_iter_count(from) && !err)
- written = ntfs_perform_write(file, from, iocb->ki_pos);
- current->backing_dev_info = NULL;
- inode_unlock(vi);
- iocb->ki_pos += written;
- if (likely(written > 0))
- written = generic_write_sync(iocb, written);
- return written ? written : err;
-}
-
-/**
- * ntfs_file_fsync - sync a file to disk
- * @filp: file to be synced
- * @datasync: if non-zero only flush user data and not metadata
- *
- * Data integrity sync of a file to disk. Used for fsync, fdatasync, and msync
- * system calls. This function is inspired by fs/buffer.c::file_fsync().
- *
- * If @datasync is false, write the mft record and all associated extent mft
- * records as well as the $DATA attribute and then sync the block device.
- *
- * If @datasync is true and the attribute is non-resident, we skip the writing
- * of the mft record and all associated extent mft records (this might still
- * happen due to the write_inode_now() call).
- *
- * Also, if @datasync is true, we do not wait on the inode to be written out
- * but we always wait on the page cache pages to be written out.
- *
- * Locking: Caller must hold i_mutex on the inode.
- *
- * TODO: We should probably also write all attribute/index inodes associated
- * with this inode but since we have no simple way of getting to them we ignore
- * this problem for now.
- */
-static int ntfs_file_fsync(struct file *filp, loff_t start, loff_t end,
- int datasync)
-{
- struct inode *vi = filp->f_mapping->host;
- int err, ret = 0;
-
- ntfs_debug("Entering for inode 0x%lx.", vi->i_ino);
-
- err = file_write_and_wait_range(filp, start, end);
- if (err)
- return err;
- inode_lock(vi);
-
- BUG_ON(S_ISDIR(vi->i_mode));
- if (!datasync || !NInoNonResident(NTFS_I(vi)))
- ret = __ntfs_write_inode(vi, 1);
- write_inode_now(vi, !datasync);
- /*
- * NOTE: If we were to use mapping->private_list (see ext2 and
- * fs/buffer.c) for dirty blocks then we could optimize the below to be
- * sync_mapping_buffers(vi->i_mapping).
- */
- err = sync_blockdev(vi->i_sb->s_bdev);
- if (unlikely(err && !ret))
- ret = err;
- if (likely(!ret))
- ntfs_debug("Done.");
- else
- ntfs_warning(vi->i_sb, "Failed to f%ssync inode 0x%lx. Error "
- "%u.", datasync ? "data" : "", vi->i_ino, -ret);
- inode_unlock(vi);
- return ret;
-}
-
-#endif /* NTFS_RW */
-
-const struct file_operations ntfs_file_ops = {
- .llseek = generic_file_llseek,
- .read_iter = generic_file_read_iter,
-#ifdef NTFS_RW
- .write_iter = ntfs_file_write_iter,
- .fsync = ntfs_file_fsync,
-#endif /* NTFS_RW */
- .mmap = generic_file_mmap,
- .open = ntfs_file_open,
- .splice_read = generic_file_splice_read,
-};
-
-const struct inode_operations ntfs_file_inode_ops = {
-#ifdef NTFS_RW
- .setattr = ntfs_setattr,
-#endif /* NTFS_RW */
-};
-
-const struct file_operations ntfs_empty_file_ops = {};
-
-const struct inode_operations ntfs_empty_inode_ops = {};
diff --git a/fs/ntfs/index.c b/fs/ntfs/index.c
deleted file mode 100644
index d46c2c03a032..000000000000
--- a/fs/ntfs/index.c
+++ /dev/null
@@ -1,440 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * index.c - NTFS kernel index handling. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2004-2005 Anton Altaparmakov
- */
-
-#include <linux/slab.h>
-
-#include "aops.h"
-#include "collate.h"
-#include "debug.h"
-#include "index.h"
-#include "ntfs.h"
-
-/**
- * ntfs_index_ctx_get - allocate and initialize a new index context
- * @idx_ni: ntfs index inode with which to initialize the context
- *
- * Allocate a new index context, initialize it with @idx_ni and return it.
- * Return NULL if allocation failed.
- *
- * Locking: Caller must hold i_mutex on the index inode.
- */
-ntfs_index_context *ntfs_index_ctx_get(ntfs_inode *idx_ni)
-{
- ntfs_index_context *ictx;
-
- ictx = kmem_cache_alloc(ntfs_index_ctx_cache, GFP_NOFS);
- if (ictx)
- *ictx = (ntfs_index_context){ .idx_ni = idx_ni };
- return ictx;
-}
-
-/**
- * ntfs_index_ctx_put - release an index context
- * @ictx: index context to free
- *
- * Release the index context @ictx, releasing all associated resources.
- *
- * Locking: Caller must hold i_mutex on the index inode.
- */
-void ntfs_index_ctx_put(ntfs_index_context *ictx)
-{
- if (ictx->entry) {
- if (ictx->is_in_root) {
- if (ictx->actx)
- ntfs_attr_put_search_ctx(ictx->actx);
- if (ictx->base_ni)
- unmap_mft_record(ictx->base_ni);
- } else {
- struct page *page = ictx->page;
- if (page) {
- BUG_ON(!PageLocked(page));
- unlock_page(page);
- ntfs_unmap_page(page);
- }
- }
- }
- kmem_cache_free(ntfs_index_ctx_cache, ictx);
- return;
-}
-
-/**
- * ntfs_index_lookup - find a key in an index and return its index entry
- * @key: [IN] key for which to search in the index
- * @key_len: [IN] length of @key in bytes
- * @ictx: [IN/OUT] context describing the index and the returned entry
- *
- * Before calling ntfs_index_lookup(), @ictx must have been obtained from a
- * call to ntfs_index_ctx_get().
- *
- * Look for the @key in the index specified by the index lookup context @ictx.
- * ntfs_index_lookup() walks the contents of the index looking for the @key.
- *
- * If the @key is found in the index, 0 is returned and @ictx is setup to
- * describe the index entry containing the matching @key. @ictx->entry is the
- * index entry and @ictx->data and @ictx->data_len are the index entry data and
- * its length in bytes, respectively.
- *
- * If the @key is not found in the index, -ENOENT is returned and @ictx is
- * setup to describe the index entry whose key collates immediately after the
- * search @key, i.e. this is the position in the index at which an index entry
- * with a key of @key would need to be inserted.
- *
- * If an error occurs return the negative error code and @ictx is left
- * untouched.
- *
- * When finished with the entry and its data, call ntfs_index_ctx_put() to free
- * the context and other associated resources.
- *
- * If the index entry was modified, call flush_dcache_index_entry_page()
- * immediately after the modification and either ntfs_index_entry_mark_dirty()
- * or ntfs_index_entry_write() before the call to ntfs_index_ctx_put() to
- * ensure that the changes are written to disk.
- *
- * Locking: - Caller must hold i_mutex on the index inode.
- * - Each page cache page in the index allocation mapping must be
- * locked whilst being accessed otherwise we may find a corrupt
- * page due to it being under ->writepage at the moment which
- * applies the mst protection fixups before writing out and then
- * removes them again after the write is complete after which it
- * unlocks the page.
- */
-int ntfs_index_lookup(const void *key, const int key_len,
- ntfs_index_context *ictx)
-{
- VCN vcn, old_vcn;
- ntfs_inode *idx_ni = ictx->idx_ni;
- ntfs_volume *vol = idx_ni->vol;
- struct super_block *sb = vol->sb;
- ntfs_inode *base_ni = idx_ni->ext.base_ntfs_ino;
- MFT_RECORD *m;
- INDEX_ROOT *ir;
- INDEX_ENTRY *ie;
- INDEX_ALLOCATION *ia;
- u8 *index_end, *kaddr;
- ntfs_attr_search_ctx *actx;
- struct address_space *ia_mapping;
- struct page *page;
- int rc, err = 0;
-
- ntfs_debug("Entering.");
- BUG_ON(!NInoAttr(idx_ni));
- BUG_ON(idx_ni->type != AT_INDEX_ALLOCATION);
- BUG_ON(idx_ni->nr_extents != -1);
- BUG_ON(!base_ni);
- BUG_ON(!key);
- BUG_ON(key_len <= 0);
- if (!ntfs_is_collation_rule_supported(
- idx_ni->itype.index.collation_rule)) {
- ntfs_error(sb, "Index uses unsupported collation rule 0x%x. "
- "Aborting lookup.", le32_to_cpu(
- idx_ni->itype.index.collation_rule));
- return -EOPNOTSUPP;
- }
- /* Get hold of the mft record for the index inode. */
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- ntfs_error(sb, "map_mft_record() failed with error code %ld.",
- -PTR_ERR(m));
- return PTR_ERR(m);
- }
- actx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!actx)) {
- err = -ENOMEM;
- goto err_out;
- }
- /* Find the index root attribute in the mft record. */
- err = ntfs_attr_lookup(AT_INDEX_ROOT, idx_ni->name, idx_ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, actx);
- if (unlikely(err)) {
- if (err == -ENOENT) {
- ntfs_error(sb, "Index root attribute missing in inode "
- "0x%lx.", idx_ni->mft_no);
- err = -EIO;
- }
- goto err_out;
- }
- /* Get to the index root value (it has been verified in read_inode). */
- ir = (INDEX_ROOT*)((u8*)actx->attr +
- le16_to_cpu(actx->attr->data.resident.value_offset));
- index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length);
- /* The first index entry. */
- ie = (INDEX_ENTRY*)((u8*)&ir->index +
- le32_to_cpu(ir->index.entries_offset));
- /*
- * Loop until we exceed valid memory (corruption case) or until we
- * reach the last entry.
- */
- for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) {
- /* Bounds checks. */
- if ((u8*)ie < (u8*)actx->mrec || (u8*)ie +
- sizeof(INDEX_ENTRY_HEADER) > index_end ||
- (u8*)ie + le16_to_cpu(ie->length) > index_end)
- goto idx_err_out;
- /*
- * The last entry cannot contain a key. It can however contain
- * a pointer to a child node in the B+tree so we just break out.
- */
- if (ie->flags & INDEX_ENTRY_END)
- break;
- /* Further bounds checks. */
- if ((u32)sizeof(INDEX_ENTRY_HEADER) +
- le16_to_cpu(ie->key_length) >
- le16_to_cpu(ie->data.vi.data_offset) ||
- (u32)le16_to_cpu(ie->data.vi.data_offset) +
- le16_to_cpu(ie->data.vi.data_length) >
- le16_to_cpu(ie->length))
- goto idx_err_out;
- /* If the keys match perfectly, we setup @ictx and return 0. */
- if ((key_len == le16_to_cpu(ie->key_length)) && !memcmp(key,
- &ie->key, key_len)) {
-ir_done:
- ictx->is_in_root = true;
- ictx->ir = ir;
- ictx->actx = actx;
- ictx->base_ni = base_ni;
- ictx->ia = NULL;
- ictx->page = NULL;
-done:
- ictx->entry = ie;
- ictx->data = (u8*)ie +
- le16_to_cpu(ie->data.vi.data_offset);
- ictx->data_len = le16_to_cpu(ie->data.vi.data_length);
- ntfs_debug("Done.");
- return err;
- }
- /*
- * Not a perfect match, need to do full blown collation so we
- * know which way in the B+tree we have to go.
- */
- rc = ntfs_collate(vol, idx_ni->itype.index.collation_rule, key,
- key_len, &ie->key, le16_to_cpu(ie->key_length));
- /*
- * If @key collates before the key of the current entry, there
- * is definitely no such key in this index but we might need to
- * descend into the B+tree so we just break out of the loop.
- */
- if (rc == -1)
- break;
- /*
- * A match should never happen as the memcmp() call should have
- * cought it, but we still treat it correctly.
- */
- if (!rc)
- goto ir_done;
- /* The keys are not equal, continue the search. */
- }
- /*
- * We have finished with this index without success. Check for the
- * presence of a child node and if not present setup @ictx and return
- * -ENOENT.
- */
- if (!(ie->flags & INDEX_ENTRY_NODE)) {
- ntfs_debug("Entry not found.");
- err = -ENOENT;
- goto ir_done;
- } /* Child node present, descend into it. */
- /* Consistency check: Verify that an index allocation exists. */
- if (!NInoIndexAllocPresent(idx_ni)) {
- ntfs_error(sb, "No index allocation attribute but index entry "
- "requires one. Inode 0x%lx is corrupt or "
- "driver bug.", idx_ni->mft_no);
- goto err_out;
- }
- /* Get the starting vcn of the index_block holding the child node. */
- vcn = sle64_to_cpup((sle64*)((u8*)ie + le16_to_cpu(ie->length) - 8));
- ia_mapping = VFS_I(idx_ni)->i_mapping;
- /*
- * We are done with the index root and the mft record. Release them,
- * otherwise we deadlock with ntfs_map_page().
- */
- ntfs_attr_put_search_ctx(actx);
- unmap_mft_record(base_ni);
- m = NULL;
- actx = NULL;
-descend_into_child_node:
- /*
- * Convert vcn to index into the index allocation attribute in units
- * of PAGE_SIZE and map the page cache page, reading it from
- * disk if necessary.
- */
- page = ntfs_map_page(ia_mapping, vcn <<
- idx_ni->itype.index.vcn_size_bits >> PAGE_SHIFT);
- if (IS_ERR(page)) {
- ntfs_error(sb, "Failed to map index page, error %ld.",
- -PTR_ERR(page));
- err = PTR_ERR(page);
- goto err_out;
- }
- lock_page(page);
- kaddr = (u8*)page_address(page);
-fast_descend_into_child_node:
- /* Get to the index allocation block. */
- ia = (INDEX_ALLOCATION*)(kaddr + ((vcn <<
- idx_ni->itype.index.vcn_size_bits) & ~PAGE_MASK));
- /* Bounds checks. */
- if ((u8*)ia < kaddr || (u8*)ia > kaddr + PAGE_SIZE) {
- ntfs_error(sb, "Out of bounds check failed. Corrupt inode "
- "0x%lx or driver bug.", idx_ni->mft_no);
- goto unm_err_out;
- }
- /* Catch multi sector transfer fixup errors. */
- if (unlikely(!ntfs_is_indx_record(ia->magic))) {
- ntfs_error(sb, "Index record with vcn 0x%llx is corrupt. "
- "Corrupt inode 0x%lx. Run chkdsk.",
- (long long)vcn, idx_ni->mft_no);
- goto unm_err_out;
- }
- if (sle64_to_cpu(ia->index_block_vcn) != vcn) {
- ntfs_error(sb, "Actual VCN (0x%llx) of index buffer is "
- "different from expected VCN (0x%llx). Inode "
- "0x%lx is corrupt or driver bug.",
- (unsigned long long)
- sle64_to_cpu(ia->index_block_vcn),
- (unsigned long long)vcn, idx_ni->mft_no);
- goto unm_err_out;
- }
- if (le32_to_cpu(ia->index.allocated_size) + 0x18 !=
- idx_ni->itype.index.block_size) {
- ntfs_error(sb, "Index buffer (VCN 0x%llx) of inode 0x%lx has "
- "a size (%u) differing from the index "
- "specified size (%u). Inode is corrupt or "
- "driver bug.", (unsigned long long)vcn,
- idx_ni->mft_no,
- le32_to_cpu(ia->index.allocated_size) + 0x18,
- idx_ni->itype.index.block_size);
- goto unm_err_out;
- }
- index_end = (u8*)ia + idx_ni->itype.index.block_size;
- if (index_end > kaddr + PAGE_SIZE) {
- ntfs_error(sb, "Index buffer (VCN 0x%llx) of inode 0x%lx "
- "crosses page boundary. Impossible! Cannot "
- "access! This is probably a bug in the "
- "driver.", (unsigned long long)vcn,
- idx_ni->mft_no);
- goto unm_err_out;
- }
- index_end = (u8*)&ia->index + le32_to_cpu(ia->index.index_length);
- if (index_end > (u8*)ia + idx_ni->itype.index.block_size) {
- ntfs_error(sb, "Size of index buffer (VCN 0x%llx) of inode "
- "0x%lx exceeds maximum size.",
- (unsigned long long)vcn, idx_ni->mft_no);
- goto unm_err_out;
- }
- /* The first index entry. */
- ie = (INDEX_ENTRY*)((u8*)&ia->index +
- le32_to_cpu(ia->index.entries_offset));
- /*
- * Iterate similar to above big loop but applied to index buffer, thus
- * loop until we exceed valid memory (corruption case) or until we
- * reach the last entry.
- */
- for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) {
- /* Bounds checks. */
- if ((u8*)ie < (u8*)ia || (u8*)ie +
- sizeof(INDEX_ENTRY_HEADER) > index_end ||
- (u8*)ie + le16_to_cpu(ie->length) > index_end) {
- ntfs_error(sb, "Index entry out of bounds in inode "
- "0x%lx.", idx_ni->mft_no);
- goto unm_err_out;
- }
- /*
- * The last entry cannot contain a key. It can however contain
- * a pointer to a child node in the B+tree so we just break out.
- */
- if (ie->flags & INDEX_ENTRY_END)
- break;
- /* Further bounds checks. */
- if ((u32)sizeof(INDEX_ENTRY_HEADER) +
- le16_to_cpu(ie->key_length) >
- le16_to_cpu(ie->data.vi.data_offset) ||
- (u32)le16_to_cpu(ie->data.vi.data_offset) +
- le16_to_cpu(ie->data.vi.data_length) >
- le16_to_cpu(ie->length)) {
- ntfs_error(sb, "Index entry out of bounds in inode "
- "0x%lx.", idx_ni->mft_no);
- goto unm_err_out;
- }
- /* If the keys match perfectly, we setup @ictx and return 0. */
- if ((key_len == le16_to_cpu(ie->key_length)) && !memcmp(key,
- &ie->key, key_len)) {
-ia_done:
- ictx->is_in_root = false;
- ictx->actx = NULL;
- ictx->base_ni = NULL;
- ictx->ia = ia;
- ictx->page = page;
- goto done;
- }
- /*
- * Not a perfect match, need to do full blown collation so we
- * know which way in the B+tree we have to go.
- */
- rc = ntfs_collate(vol, idx_ni->itype.index.collation_rule, key,
- key_len, &ie->key, le16_to_cpu(ie->key_length));
- /*
- * If @key collates before the key of the current entry, there
- * is definitely no such key in this index but we might need to
- * descend into the B+tree so we just break out of the loop.
- */
- if (rc == -1)
- break;
- /*
- * A match should never happen as the memcmp() call should have
- * cought it, but we still treat it correctly.
- */
- if (!rc)
- goto ia_done;
- /* The keys are not equal, continue the search. */
- }
- /*
- * We have finished with this index buffer without success. Check for
- * the presence of a child node and if not present return -ENOENT.
- */
- if (!(ie->flags & INDEX_ENTRY_NODE)) {
- ntfs_debug("Entry not found.");
- err = -ENOENT;
- goto ia_done;
- }
- if ((ia->index.flags & NODE_MASK) == LEAF_NODE) {
- ntfs_error(sb, "Index entry with child node found in a leaf "
- "node in inode 0x%lx.", idx_ni->mft_no);
- goto unm_err_out;
- }
- /* Child node present, descend into it. */
- old_vcn = vcn;
- vcn = sle64_to_cpup((sle64*)((u8*)ie + le16_to_cpu(ie->length) - 8));
- if (vcn >= 0) {
- /*
- * If vcn is in the same page cache page as old_vcn we recycle
- * the mapped page.
- */
- if (old_vcn << vol->cluster_size_bits >>
- PAGE_SHIFT == vcn <<
- vol->cluster_size_bits >>
- PAGE_SHIFT)
- goto fast_descend_into_child_node;
- unlock_page(page);
- ntfs_unmap_page(page);
- goto descend_into_child_node;
- }
- ntfs_error(sb, "Negative child node vcn in inode 0x%lx.",
- idx_ni->mft_no);
-unm_err_out:
- unlock_page(page);
- ntfs_unmap_page(page);
-err_out:
- if (!err)
- err = -EIO;
- if (actx)
- ntfs_attr_put_search_ctx(actx);
- if (m)
- unmap_mft_record(base_ni);
- return err;
-idx_err_out:
- ntfs_error(sb, "Corrupt index. Aborting lookup.");
- goto err_out;
-}
diff --git a/fs/ntfs/index.h b/fs/ntfs/index.h
deleted file mode 100644
index bb3c3ae55138..000000000000
--- a/fs/ntfs/index.h
+++ /dev/null
@@ -1,134 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * index.h - Defines for NTFS kernel index handling. Part of the Linux-NTFS
- * project.
- *
- * Copyright (c) 2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_INDEX_H
-#define _LINUX_NTFS_INDEX_H
-
-#include <linux/fs.h>
-
-#include "types.h"
-#include "layout.h"
-#include "inode.h"
-#include "attrib.h"
-#include "mft.h"
-#include "aops.h"
-
-/**
- * @idx_ni: index inode containing the @entry described by this context
- * @entry: index entry (points into @ir or @ia)
- * @data: index entry data (points into @entry)
- * @data_len: length in bytes of @data
- * @is_in_root: 'true' if @entry is in @ir and 'false' if it is in @ia
- * @ir: index root if @is_in_root and NULL otherwise
- * @actx: attribute search context if @is_in_root and NULL otherwise
- * @base_ni: base inode if @is_in_root and NULL otherwise
- * @ia: index block if @is_in_root is 'false' and NULL otherwise
- * @page: page if @is_in_root is 'false' and NULL otherwise
- *
- * @idx_ni is the index inode this context belongs to.
- *
- * @entry is the index entry described by this context. @data and @data_len
- * are the index entry data and its length in bytes, respectively. @data
- * simply points into @entry. This is probably what the user is interested in.
- *
- * If @is_in_root is 'true', @entry is in the index root attribute @ir described
- * by the attribute search context @actx and the base inode @base_ni. @ia and
- * @page are NULL in this case.
- *
- * If @is_in_root is 'false', @entry is in the index allocation attribute and @ia
- * and @page point to the index allocation block and the mapped, locked page it
- * is in, respectively. @ir, @actx and @base_ni are NULL in this case.
- *
- * To obtain a context call ntfs_index_ctx_get().
- *
- * We use this context to allow ntfs_index_lookup() to return the found index
- * @entry and its @data without having to allocate a buffer and copy the @entry
- * and/or its @data into it.
- *
- * When finished with the @entry and its @data, call ntfs_index_ctx_put() to
- * free the context and other associated resources.
- *
- * If the index entry was modified, call flush_dcache_index_entry_page()
- * immediately after the modification and either ntfs_index_entry_mark_dirty()
- * or ntfs_index_entry_write() before the call to ntfs_index_ctx_put() to
- * ensure that the changes are written to disk.
- */
-typedef struct {
- ntfs_inode *idx_ni;
- INDEX_ENTRY *entry;
- void *data;
- u16 data_len;
- bool is_in_root;
- INDEX_ROOT *ir;
- ntfs_attr_search_ctx *actx;
- ntfs_inode *base_ni;
- INDEX_ALLOCATION *ia;
- struct page *page;
-} ntfs_index_context;
-
-extern ntfs_index_context *ntfs_index_ctx_get(ntfs_inode *idx_ni);
-extern void ntfs_index_ctx_put(ntfs_index_context *ictx);
-
-extern int ntfs_index_lookup(const void *key, const int key_len,
- ntfs_index_context *ictx);
-
-#ifdef NTFS_RW
-
-/**
- * ntfs_index_entry_flush_dcache_page - flush_dcache_page() for index entries
- * @ictx: ntfs index context describing the index entry
- *
- * Call flush_dcache_page() for the page in which an index entry resides.
- *
- * This must be called every time an index entry is modified, just after the
- * modification.
- *
- * If the index entry is in the index root attribute, simply flush the page
- * containing the mft record containing the index root attribute.
- *
- * If the index entry is in an index block belonging to the index allocation
- * attribute, simply flush the page cache page containing the index block.
- */
-static inline void ntfs_index_entry_flush_dcache_page(ntfs_index_context *ictx)
-{
- if (ictx->is_in_root)
- flush_dcache_mft_record_page(ictx->actx->ntfs_ino);
- else
- flush_dcache_page(ictx->page);
-}
-
-/**
- * ntfs_index_entry_mark_dirty - mark an index entry dirty
- * @ictx: ntfs index context describing the index entry
- *
- * Mark the index entry described by the index entry context @ictx dirty.
- *
- * If the index entry is in the index root attribute, simply mark the mft
- * record containing the index root attribute dirty. This ensures the mft
- * record, and hence the index root attribute, will be written out to disk
- * later.
- *
- * If the index entry is in an index block belonging to the index allocation
- * attribute, mark the buffers belonging to the index record as well as the
- * page cache page the index block is in dirty. This automatically marks the
- * VFS inode of the ntfs index inode to which the index entry belongs dirty,
- * too (I_DIRTY_PAGES) and this in turn ensures the page buffers, and hence the
- * dirty index block, will be written out to disk later.
- */
-static inline void ntfs_index_entry_mark_dirty(ntfs_index_context *ictx)
-{
- if (ictx->is_in_root)
- mark_mft_record_dirty(ictx->actx->ntfs_ino);
- else
- mark_ntfs_record_dirty(ictx->page,
- (u8*)ictx->ia - (u8*)page_address(ictx->page));
-}
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_INDEX_H */
diff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c
deleted file mode 100644
index 08c659332e26..000000000000
--- a/fs/ntfs/inode.c
+++ /dev/null
@@ -1,3100 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/**
- * inode.c - NTFS kernel inode handling.
- *
- * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc.
- */
-
-#include <linux/buffer_head.h>
-#include <linux/fs.h>
-#include <linux/mm.h>
-#include <linux/mount.h>
-#include <linux/mutex.h>
-#include <linux/pagemap.h>
-#include <linux/quotaops.h>
-#include <linux/slab.h>
-#include <linux/log2.h>
-
-#include "aops.h"
-#include "attrib.h"
-#include "bitmap.h"
-#include "dir.h"
-#include "debug.h"
-#include "inode.h"
-#include "lcnalloc.h"
-#include "malloc.h"
-#include "mft.h"
-#include "time.h"
-#include "ntfs.h"
-
-/**
- * ntfs_test_inode - compare two (possibly fake) inodes for equality
- * @vi: vfs inode which to test
- * @data: data which is being tested with
- *
- * Compare the ntfs attribute embedded in the ntfs specific part of the vfs
- * inode @vi for equality with the ntfs attribute @data.
- *
- * If searching for the normal file/directory inode, set @na->type to AT_UNUSED.
- * @na->name and @na->name_len are then ignored.
- *
- * Return 1 if the attributes match and 0 if not.
- *
- * NOTE: This function runs with the inode_hash_lock spin lock held so it is not
- * allowed to sleep.
- */
-int ntfs_test_inode(struct inode *vi, void *data)
-{
- ntfs_attr *na = (ntfs_attr *)data;
- ntfs_inode *ni;
-
- if (vi->i_ino != na->mft_no)
- return 0;
- ni = NTFS_I(vi);
- /* If !NInoAttr(ni), @vi is a normal file or directory inode. */
- if (likely(!NInoAttr(ni))) {
- /* If not looking for a normal inode this is a mismatch. */
- if (unlikely(na->type != AT_UNUSED))
- return 0;
- } else {
- /* A fake inode describing an attribute. */
- if (ni->type != na->type)
- return 0;
- if (ni->name_len != na->name_len)
- return 0;
- if (na->name_len && memcmp(ni->name, na->name,
- na->name_len * sizeof(ntfschar)))
- return 0;
- }
- /* Match! */
- return 1;
-}
-
-/**
- * ntfs_init_locked_inode - initialize an inode
- * @vi: vfs inode to initialize
- * @data: data which to initialize @vi to
- *
- * Initialize the vfs inode @vi with the values from the ntfs attribute @data in
- * order to enable ntfs_test_inode() to do its work.
- *
- * If initializing the normal file/directory inode, set @na->type to AT_UNUSED.
- * In that case, @na->name and @na->name_len should be set to NULL and 0,
- * respectively. Although that is not strictly necessary as
- * ntfs_read_locked_inode() will fill them in later.
- *
- * Return 0 on success and -errno on error.
- *
- * NOTE: This function runs with the inode->i_lock spin lock held so it is not
- * allowed to sleep. (Hence the GFP_ATOMIC allocation.)
- */
-static int ntfs_init_locked_inode(struct inode *vi, void *data)
-{
- ntfs_attr *na = (ntfs_attr *)data;
- ntfs_inode *ni = NTFS_I(vi);
-
- vi->i_ino = na->mft_no;
-
- ni->type = na->type;
- if (na->type == AT_INDEX_ALLOCATION)
- NInoSetMstProtected(ni);
-
- ni->name = na->name;
- ni->name_len = na->name_len;
-
- /* If initializing a normal inode, we are done. */
- if (likely(na->type == AT_UNUSED)) {
- BUG_ON(na->name);
- BUG_ON(na->name_len);
- return 0;
- }
-
- /* It is a fake inode. */
- NInoSetAttr(ni);
-
- /*
- * We have I30 global constant as an optimization as it is the name
- * in >99.9% of named attributes! The other <0.1% incur a GFP_ATOMIC
- * allocation but that is ok. And most attributes are unnamed anyway,
- * thus the fraction of named attributes with name != I30 is actually
- * absolutely tiny.
- */
- if (na->name_len && na->name != I30) {
- unsigned int i;
-
- BUG_ON(!na->name);
- i = na->name_len * sizeof(ntfschar);
- ni->name = kmalloc(i + sizeof(ntfschar), GFP_ATOMIC);
- if (!ni->name)
- return -ENOMEM;
- memcpy(ni->name, na->name, i);
- ni->name[na->name_len] = 0;
- }
- return 0;
-}
-
-static int ntfs_read_locked_inode(struct inode *vi);
-static int ntfs_read_locked_attr_inode(struct inode *base_vi, struct inode *vi);
-static int ntfs_read_locked_index_inode(struct inode *base_vi,
- struct inode *vi);
-
-/**
- * ntfs_iget - obtain a struct inode corresponding to a specific normal inode
- * @sb: super block of mounted volume
- * @mft_no: mft record number / inode number to obtain
- *
- * Obtain the struct inode corresponding to a specific normal inode (i.e. a
- * file or directory).
- *
- * If the inode is in the cache, it is just returned with an increased
- * reference count. Otherwise, a new struct inode is allocated and initialized,
- * and finally ntfs_read_locked_inode() is called to read in the inode and
- * fill in the remainder of the inode structure.
- *
- * Return the struct inode on success. Check the return value with IS_ERR() and
- * if true, the function failed and the error code is obtained from PTR_ERR().
- */
-struct inode *ntfs_iget(struct super_block *sb, unsigned long mft_no)
-{
- struct inode *vi;
- int err;
- ntfs_attr na;
-
- na.mft_no = mft_no;
- na.type = AT_UNUSED;
- na.name = NULL;
- na.name_len = 0;
-
- vi = iget5_locked(sb, mft_no, ntfs_test_inode,
- ntfs_init_locked_inode, &na);
- if (unlikely(!vi))
- return ERR_PTR(-ENOMEM);
-
- err = 0;
-
- /* If this is a freshly allocated inode, need to read it now. */
- if (vi->i_state & I_NEW) {
- err = ntfs_read_locked_inode(vi);
- unlock_new_inode(vi);
- }
- /*
- * There is no point in keeping bad inodes around if the failure was
- * due to ENOMEM. We want to be able to retry again later.
- */
- if (unlikely(err == -ENOMEM)) {
- iput(vi);
- vi = ERR_PTR(err);
- }
- return vi;
-}
-
-/**
- * ntfs_attr_iget - obtain a struct inode corresponding to an attribute
- * @base_vi: vfs base inode containing the attribute
- * @type: attribute type
- * @name: Unicode name of the attribute (NULL if unnamed)
- * @name_len: length of @name in Unicode characters (0 if unnamed)
- *
- * Obtain the (fake) struct inode corresponding to the attribute specified by
- * @type, @name, and @name_len, which is present in the base mft record
- * specified by the vfs inode @base_vi.
- *
- * If the attribute inode is in the cache, it is just returned with an
- * increased reference count. Otherwise, a new struct inode is allocated and
- * initialized, and finally ntfs_read_locked_attr_inode() is called to read the
- * attribute and fill in the inode structure.
- *
- * Note, for index allocation attributes, you need to use ntfs_index_iget()
- * instead of ntfs_attr_iget() as working with indices is a lot more complex.
- *
- * Return the struct inode of the attribute inode on success. Check the return
- * value with IS_ERR() and if true, the function failed and the error code is
- * obtained from PTR_ERR().
- */
-struct inode *ntfs_attr_iget(struct inode *base_vi, ATTR_TYPE type,
- ntfschar *name, u32 name_len)
-{
- struct inode *vi;
- int err;
- ntfs_attr na;
-
- /* Make sure no one calls ntfs_attr_iget() for indices. */
- BUG_ON(type == AT_INDEX_ALLOCATION);
-
- na.mft_no = base_vi->i_ino;
- na.type = type;
- na.name = name;
- na.name_len = name_len;
-
- vi = iget5_locked(base_vi->i_sb, na.mft_no, ntfs_test_inode,
- ntfs_init_locked_inode, &na);
- if (unlikely(!vi))
- return ERR_PTR(-ENOMEM);
-
- err = 0;
-
- /* If this is a freshly allocated inode, need to read it now. */
- if (vi->i_state & I_NEW) {
- err = ntfs_read_locked_attr_inode(base_vi, vi);
- unlock_new_inode(vi);
- }
- /*
- * There is no point in keeping bad attribute inodes around. This also
- * simplifies things in that we never need to check for bad attribute
- * inodes elsewhere.
- */
- if (unlikely(err)) {
- iput(vi);
- vi = ERR_PTR(err);
- }
- return vi;
-}
-
-/**
- * ntfs_index_iget - obtain a struct inode corresponding to an index
- * @base_vi: vfs base inode containing the index related attributes
- * @name: Unicode name of the index
- * @name_len: length of @name in Unicode characters
- *
- * Obtain the (fake) struct inode corresponding to the index specified by @name
- * and @name_len, which is present in the base mft record specified by the vfs
- * inode @base_vi.
- *
- * If the index inode is in the cache, it is just returned with an increased
- * reference count. Otherwise, a new struct inode is allocated and
- * initialized, and finally ntfs_read_locked_index_inode() is called to read
- * the index related attributes and fill in the inode structure.
- *
- * Return the struct inode of the index inode on success. Check the return
- * value with IS_ERR() and if true, the function failed and the error code is
- * obtained from PTR_ERR().
- */
-struct inode *ntfs_index_iget(struct inode *base_vi, ntfschar *name,
- u32 name_len)
-{
- struct inode *vi;
- int err;
- ntfs_attr na;
-
- na.mft_no = base_vi->i_ino;
- na.type = AT_INDEX_ALLOCATION;
- na.name = name;
- na.name_len = name_len;
-
- vi = iget5_locked(base_vi->i_sb, na.mft_no, ntfs_test_inode,
- ntfs_init_locked_inode, &na);
- if (unlikely(!vi))
- return ERR_PTR(-ENOMEM);
-
- err = 0;
-
- /* If this is a freshly allocated inode, need to read it now. */
- if (vi->i_state & I_NEW) {
- err = ntfs_read_locked_index_inode(base_vi, vi);
- unlock_new_inode(vi);
- }
- /*
- * There is no point in keeping bad index inodes around. This also
- * simplifies things in that we never need to check for bad index
- * inodes elsewhere.
- */
- if (unlikely(err)) {
- iput(vi);
- vi = ERR_PTR(err);
- }
- return vi;
-}
-
-struct inode *ntfs_alloc_big_inode(struct super_block *sb)
-{
- ntfs_inode *ni;
-
- ntfs_debug("Entering.");
- ni = alloc_inode_sb(sb, ntfs_big_inode_cache, GFP_NOFS);
- if (likely(ni != NULL)) {
- ni->state = 0;
- return VFS_I(ni);
- }
- ntfs_error(sb, "Allocation of NTFS big inode structure failed.");
- return NULL;
-}
-
-void ntfs_free_big_inode(struct inode *inode)
-{
- kmem_cache_free(ntfs_big_inode_cache, NTFS_I(inode));
-}
-
-static inline ntfs_inode *ntfs_alloc_extent_inode(void)
-{
- ntfs_inode *ni;
-
- ntfs_debug("Entering.");
- ni = kmem_cache_alloc(ntfs_inode_cache, GFP_NOFS);
- if (likely(ni != NULL)) {
- ni->state = 0;
- return ni;
- }
- ntfs_error(NULL, "Allocation of NTFS inode structure failed.");
- return NULL;
-}
-
-static void ntfs_destroy_extent_inode(ntfs_inode *ni)
-{
- ntfs_debug("Entering.");
- BUG_ON(ni->page);
- if (!atomic_dec_and_test(&ni->count))
- BUG();
- kmem_cache_free(ntfs_inode_cache, ni);
-}
-
-/*
- * The attribute runlist lock has separate locking rules from the
- * normal runlist lock, so split the two lock-classes:
- */
-static struct lock_class_key attr_list_rl_lock_class;
-
-/**
- * __ntfs_init_inode - initialize ntfs specific part of an inode
- * @sb: super block of mounted volume
- * @ni: freshly allocated ntfs inode which to initialize
- *
- * Initialize an ntfs inode to defaults.
- *
- * NOTE: ni->mft_no, ni->state, ni->type, ni->name, and ni->name_len are left
- * untouched. Make sure to initialize them elsewhere.
- *
- * Return zero on success and -ENOMEM on error.
- */
-void __ntfs_init_inode(struct super_block *sb, ntfs_inode *ni)
-{
- ntfs_debug("Entering.");
- rwlock_init(&ni->size_lock);
- ni->initialized_size = ni->allocated_size = 0;
- ni->seq_no = 0;
- atomic_set(&ni->count, 1);
- ni->vol = NTFS_SB(sb);
- ntfs_init_runlist(&ni->runlist);
- mutex_init(&ni->mrec_lock);
- ni->page = NULL;
- ni->page_ofs = 0;
- ni->attr_list_size = 0;
- ni->attr_list = NULL;
- ntfs_init_runlist(&ni->attr_list_rl);
- lockdep_set_class(&ni->attr_list_rl.lock,
- &attr_list_rl_lock_class);
- ni->itype.index.block_size = 0;
- ni->itype.index.vcn_size = 0;
- ni->itype.index.collation_rule = 0;
- ni->itype.index.block_size_bits = 0;
- ni->itype.index.vcn_size_bits = 0;
- mutex_init(&ni->extent_lock);
- ni->nr_extents = 0;
- ni->ext.base_ntfs_ino = NULL;
-}
-
-/*
- * Extent inodes get MFT-mapped in a nested way, while the base inode
- * is still mapped. Teach this nesting to the lock validator by creating
- * a separate class for nested inode's mrec_lock's:
- */
-static struct lock_class_key extent_inode_mrec_lock_key;
-
-inline ntfs_inode *ntfs_new_extent_inode(struct super_block *sb,
- unsigned long mft_no)
-{
- ntfs_inode *ni = ntfs_alloc_extent_inode();
-
- ntfs_debug("Entering.");
- if (likely(ni != NULL)) {
- __ntfs_init_inode(sb, ni);
- lockdep_set_class(&ni->mrec_lock, &extent_inode_mrec_lock_key);
- ni->mft_no = mft_no;
- ni->type = AT_UNUSED;
- ni->name = NULL;
- ni->name_len = 0;
- }
- return ni;
-}
-
-/**
- * ntfs_is_extended_system_file - check if a file is in the $Extend directory
- * @ctx: initialized attribute search context
- *
- * Search all file name attributes in the inode described by the attribute
- * search context @ctx and check if any of the names are in the $Extend system
- * directory.
- *
- * Return values:
- * 1: file is in $Extend directory
- * 0: file is not in $Extend directory
- * -errno: failed to determine if the file is in the $Extend directory
- */
-static int ntfs_is_extended_system_file(ntfs_attr_search_ctx *ctx)
-{
- int nr_links, err;
-
- /* Restart search. */
- ntfs_attr_reinit_search_ctx(ctx);
-
- /* Get number of hard links. */
- nr_links = le16_to_cpu(ctx->mrec->link_count);
-
- /* Loop through all hard links. */
- while (!(err = ntfs_attr_lookup(AT_FILE_NAME, NULL, 0, 0, 0, NULL, 0,
- ctx))) {
- FILE_NAME_ATTR *file_name_attr;
- ATTR_RECORD *attr = ctx->attr;
- u8 *p, *p2;
-
- nr_links--;
- /*
- * Maximum sanity checking as we are called on an inode that
- * we suspect might be corrupt.
- */
- p = (u8*)attr + le32_to_cpu(attr->length);
- if (p < (u8*)ctx->mrec || (u8*)p > (u8*)ctx->mrec +
- le32_to_cpu(ctx->mrec->bytes_in_use)) {
-err_corrupt_attr:
- ntfs_error(ctx->ntfs_ino->vol->sb, "Corrupt file name "
- "attribute. You should run chkdsk.");
- return -EIO;
- }
- if (attr->non_resident) {
- ntfs_error(ctx->ntfs_ino->vol->sb, "Non-resident file "
- "name. You should run chkdsk.");
- return -EIO;
- }
- if (attr->flags) {
- ntfs_error(ctx->ntfs_ino->vol->sb, "File name with "
- "invalid flags. You should run "
- "chkdsk.");
- return -EIO;
- }
- if (!(attr->data.resident.flags & RESIDENT_ATTR_IS_INDEXED)) {
- ntfs_error(ctx->ntfs_ino->vol->sb, "Unindexed file "
- "name. You should run chkdsk.");
- return -EIO;
- }
- file_name_attr = (FILE_NAME_ATTR*)((u8*)attr +
- le16_to_cpu(attr->data.resident.value_offset));
- p2 = (u8 *)file_name_attr + le32_to_cpu(attr->data.resident.value_length);
- if (p2 < (u8*)attr || p2 > p)
- goto err_corrupt_attr;
- /* This attribute is ok, but is it in the $Extend directory? */
- if (MREF_LE(file_name_attr->parent_directory) == FILE_Extend)
- return 1; /* YES, it's an extended system file. */
- }
- if (unlikely(err != -ENOENT))
- return err;
- if (unlikely(nr_links)) {
- ntfs_error(ctx->ntfs_ino->vol->sb, "Inode hard link count "
- "doesn't match number of name attributes. You "
- "should run chkdsk.");
- return -EIO;
- }
- return 0; /* NO, it is not an extended system file. */
-}
-
-/**
- * ntfs_read_locked_inode - read an inode from its device
- * @vi: inode to read
- *
- * ntfs_read_locked_inode() is called from ntfs_iget() to read the inode
- * described by @vi into memory from the device.
- *
- * The only fields in @vi that we need to/can look at when the function is
- * called are i_sb, pointing to the mounted device's super block, and i_ino,
- * the number of the inode to load.
- *
- * ntfs_read_locked_inode() maps, pins and locks the mft record number i_ino
- * for reading and sets up the necessary @vi fields as well as initializing
- * the ntfs inode.
- *
- * Q: What locks are held when the function is called?
- * A: i_state has I_NEW set, hence the inode is locked, also
- * i_count is set to 1, so it is not going to go away
- * i_flags is set to 0 and we have no business touching it. Only an ioctl()
- * is allowed to write to them. We should of course be honouring them but
- * we need to do that using the IS_* macros defined in include/linux/fs.h.
- * In any case ntfs_read_locked_inode() has nothing to do with i_flags.
- *
- * Return 0 on success and -errno on error. In the error case, the inode will
- * have had make_bad_inode() executed on it.
- */
-static int ntfs_read_locked_inode(struct inode *vi)
-{
- ntfs_volume *vol = NTFS_SB(vi->i_sb);
- ntfs_inode *ni;
- struct inode *bvi;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- STANDARD_INFORMATION *si;
- ntfs_attr_search_ctx *ctx;
- int err = 0;
-
- ntfs_debug("Entering for i_ino 0x%lx.", vi->i_ino);
-
- /* Setup the generic vfs inode parts now. */
- vi->i_uid = vol->uid;
- vi->i_gid = vol->gid;
- vi->i_mode = 0;
-
- /*
- * Initialize the ntfs specific part of @vi special casing
- * FILE_MFT which we need to do at mount time.
- */
- if (vi->i_ino != FILE_MFT)
- ntfs_init_big_inode(vi);
- ni = NTFS_I(vi);
-
- m = map_mft_record(ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(ni, m);
- if (!ctx) {
- err = -ENOMEM;
- goto unm_err_out;
- }
-
- if (!(m->flags & MFT_RECORD_IN_USE)) {
- ntfs_error(vi->i_sb, "Inode is not in use!");
- goto unm_err_out;
- }
- if (m->base_mft_record) {
- ntfs_error(vi->i_sb, "Inode is an extent inode!");
- goto unm_err_out;
- }
-
- /* Transfer information from mft record into vfs and ntfs inodes. */
- vi->i_generation = ni->seq_no = le16_to_cpu(m->sequence_number);
-
- /*
- * FIXME: Keep in mind that link_count is two for files which have both
- * a long file name and a short file name as separate entries, so if
- * we are hiding short file names this will be too high. Either we need
- * to account for the short file names by subtracting them or we need
- * to make sure we delete files even though i_nlink is not zero which
- * might be tricky due to vfs interactions. Need to think about this
- * some more when implementing the unlink command.
- */
- set_nlink(vi, le16_to_cpu(m->link_count));
- /*
- * FIXME: Reparse points can have the directory bit set even though
- * they would be S_IFLNK. Need to deal with this further below when we
- * implement reparse points / symbolic links but it will do for now.
- * Also if not a directory, it could be something else, rather than
- * a regular file. But again, will do for now.
- */
- /* Everyone gets all permissions. */
- vi->i_mode |= S_IRWXUGO;
- /* If read-only, no one gets write permissions. */
- if (IS_RDONLY(vi))
- vi->i_mode &= ~S_IWUGO;
- if (m->flags & MFT_RECORD_IS_DIRECTORY) {
- vi->i_mode |= S_IFDIR;
- /*
- * Apply the directory permissions mask set in the mount
- * options.
- */
- vi->i_mode &= ~vol->dmask;
- /* Things break without this kludge! */
- if (vi->i_nlink > 1)
- set_nlink(vi, 1);
- } else {
- vi->i_mode |= S_IFREG;
- /* Apply the file permissions mask set in the mount options. */
- vi->i_mode &= ~vol->fmask;
- }
- /*
- * Find the standard information attribute in the mft record. At this
- * stage we haven't setup the attribute list stuff yet, so this could
- * in fact fail if the standard information is in an extent record, but
- * I don't think this actually ever happens.
- */
- err = ntfs_attr_lookup(AT_STANDARD_INFORMATION, NULL, 0, 0, 0, NULL, 0,
- ctx);
- if (unlikely(err)) {
- if (err == -ENOENT) {
- /*
- * TODO: We should be performing a hot fix here (if the
- * recover mount option is set) by creating a new
- * attribute.
- */
- ntfs_error(vi->i_sb, "$STANDARD_INFORMATION attribute "
- "is missing.");
- }
- goto unm_err_out;
- }
- a = ctx->attr;
- /* Get the standard information attribute value. */
- if ((u8 *)a + le16_to_cpu(a->data.resident.value_offset)
- + le32_to_cpu(a->data.resident.value_length) >
- (u8 *)ctx->mrec + vol->mft_record_size) {
- ntfs_error(vi->i_sb, "Corrupt standard information attribute in inode.");
- goto unm_err_out;
- }
- si = (STANDARD_INFORMATION*)((u8*)a +
- le16_to_cpu(a->data.resident.value_offset));
-
- /* Transfer information from the standard information into vi. */
- /*
- * Note: The i_?times do not quite map perfectly onto the NTFS times,
- * but they are close enough, and in the end it doesn't really matter
- * that much...
- */
- /*
- * mtime is the last change of the data within the file. Not changed
- * when only metadata is changed, e.g. a rename doesn't affect mtime.
- */
- vi->i_mtime = ntfs2utc(si->last_data_change_time);
- /*
- * ctime is the last change of the metadata of the file. This obviously
- * always changes, when mtime is changed. ctime can be changed on its
- * own, mtime is then not changed, e.g. when a file is renamed.
- */
- vi->i_ctime = ntfs2utc(si->last_mft_change_time);
- /*
- * Last access to the data within the file. Not changed during a rename
- * for example but changed whenever the file is written to.
- */
- vi->i_atime = ntfs2utc(si->last_access_time);
-
- /* Find the attribute list attribute if present. */
- ntfs_attr_reinit_search_ctx(ctx);
- err = ntfs_attr_lookup(AT_ATTRIBUTE_LIST, NULL, 0, 0, 0, NULL, 0, ctx);
- if (err) {
- if (unlikely(err != -ENOENT)) {
- ntfs_error(vi->i_sb, "Failed to lookup attribute list "
- "attribute.");
- goto unm_err_out;
- }
- } else /* if (!err) */ {
- if (vi->i_ino == FILE_MFT)
- goto skip_attr_list_load;
- ntfs_debug("Attribute list found in inode 0x%lx.", vi->i_ino);
- NInoSetAttrList(ni);
- a = ctx->attr;
- if (a->flags & ATTR_COMPRESSION_MASK) {
- ntfs_error(vi->i_sb, "Attribute list attribute is "
- "compressed.");
- goto unm_err_out;
- }
- if (a->flags & ATTR_IS_ENCRYPTED ||
- a->flags & ATTR_IS_SPARSE) {
- if (a->non_resident) {
- ntfs_error(vi->i_sb, "Non-resident attribute "
- "list attribute is encrypted/"
- "sparse.");
- goto unm_err_out;
- }
- ntfs_warning(vi->i_sb, "Resident attribute list "
- "attribute in inode 0x%lx is marked "
- "encrypted/sparse which is not true. "
- "However, Windows allows this and "
- "chkdsk does not detect or correct it "
- "so we will just ignore the invalid "
- "flags and pretend they are not set.",
- vi->i_ino);
- }
- /* Now allocate memory for the attribute list. */
- ni->attr_list_size = (u32)ntfs_attr_size(a);
- ni->attr_list = ntfs_malloc_nofs(ni->attr_list_size);
- if (!ni->attr_list) {
- ntfs_error(vi->i_sb, "Not enough memory to allocate "
- "buffer for attribute list.");
- err = -ENOMEM;
- goto unm_err_out;
- }
- if (a->non_resident) {
- NInoSetAttrListNonResident(ni);
- if (a->data.non_resident.lowest_vcn) {
- ntfs_error(vi->i_sb, "Attribute list has non "
- "zero lowest_vcn.");
- goto unm_err_out;
- }
- /*
- * Setup the runlist. No need for locking as we have
- * exclusive access to the inode at this time.
- */
- ni->attr_list_rl.rl = ntfs_mapping_pairs_decompress(vol,
- a, NULL);
- if (IS_ERR(ni->attr_list_rl.rl)) {
- err = PTR_ERR(ni->attr_list_rl.rl);
- ni->attr_list_rl.rl = NULL;
- ntfs_error(vi->i_sb, "Mapping pairs "
- "decompression failed.");
- goto unm_err_out;
- }
- /* Now load the attribute list. */
- if ((err = load_attribute_list(vol, &ni->attr_list_rl,
- ni->attr_list, ni->attr_list_size,
- sle64_to_cpu(a->data.non_resident.
- initialized_size)))) {
- ntfs_error(vi->i_sb, "Failed to load "
- "attribute list attribute.");
- goto unm_err_out;
- }
- } else /* if (!a->non_resident) */ {
- if ((u8*)a + le16_to_cpu(a->data.resident.value_offset)
- + le32_to_cpu(
- a->data.resident.value_length) >
- (u8*)ctx->mrec + vol->mft_record_size) {
- ntfs_error(vi->i_sb, "Corrupt attribute list "
- "in inode.");
- goto unm_err_out;
- }
- /* Now copy the attribute list. */
- memcpy(ni->attr_list, (u8*)a + le16_to_cpu(
- a->data.resident.value_offset),
- le32_to_cpu(
- a->data.resident.value_length));
- }
- }
-skip_attr_list_load:
- /*
- * If an attribute list is present we now have the attribute list value
- * in ntfs_ino->attr_list and it is ntfs_ino->attr_list_size bytes.
- */
- if (S_ISDIR(vi->i_mode)) {
- loff_t bvi_size;
- ntfs_inode *bni;
- INDEX_ROOT *ir;
- u8 *ir_end, *index_end;
-
- /* It is a directory, find index root attribute. */
- ntfs_attr_reinit_search_ctx(ctx);
- err = ntfs_attr_lookup(AT_INDEX_ROOT, I30, 4, CASE_SENSITIVE,
- 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT) {
- // FIXME: File is corrupt! Hot-fix with empty
- // index root attribute if recovery option is
- // set.
- ntfs_error(vi->i_sb, "$INDEX_ROOT attribute "
- "is missing.");
- }
- goto unm_err_out;
- }
- a = ctx->attr;
- /* Set up the state. */
- if (unlikely(a->non_resident)) {
- ntfs_error(vol->sb, "$INDEX_ROOT attribute is not "
- "resident.");
- goto unm_err_out;
- }
- /* Ensure the attribute name is placed before the value. */
- if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >=
- le16_to_cpu(a->data.resident.value_offset)))) {
- ntfs_error(vol->sb, "$INDEX_ROOT attribute name is "
- "placed after the attribute value.");
- goto unm_err_out;
- }
- /*
- * Compressed/encrypted index root just means that the newly
- * created files in that directory should be created compressed/
- * encrypted. However index root cannot be both compressed and
- * encrypted.
- */
- if (a->flags & ATTR_COMPRESSION_MASK)
- NInoSetCompressed(ni);
- if (a->flags & ATTR_IS_ENCRYPTED) {
- if (a->flags & ATTR_COMPRESSION_MASK) {
- ntfs_error(vi->i_sb, "Found encrypted and "
- "compressed attribute.");
- goto unm_err_out;
- }
- NInoSetEncrypted(ni);
- }
- if (a->flags & ATTR_IS_SPARSE)
- NInoSetSparse(ni);
- ir = (INDEX_ROOT*)((u8*)a +
- le16_to_cpu(a->data.resident.value_offset));
- ir_end = (u8*)ir + le32_to_cpu(a->data.resident.value_length);
- if (ir_end > (u8*)ctx->mrec + vol->mft_record_size) {
- ntfs_error(vi->i_sb, "$INDEX_ROOT attribute is "
- "corrupt.");
- goto unm_err_out;
- }
- index_end = (u8*)&ir->index +
- le32_to_cpu(ir->index.index_length);
- if (index_end > ir_end) {
- ntfs_error(vi->i_sb, "Directory index is corrupt.");
- goto unm_err_out;
- }
- if (ir->type != AT_FILE_NAME) {
- ntfs_error(vi->i_sb, "Indexed attribute is not "
- "$FILE_NAME.");
- goto unm_err_out;
- }
- if (ir->collation_rule != COLLATION_FILE_NAME) {
- ntfs_error(vi->i_sb, "Index collation rule is not "
- "COLLATION_FILE_NAME.");
- goto unm_err_out;
- }
- ni->itype.index.collation_rule = ir->collation_rule;
- ni->itype.index.block_size = le32_to_cpu(ir->index_block_size);
- if (ni->itype.index.block_size &
- (ni->itype.index.block_size - 1)) {
- ntfs_error(vi->i_sb, "Index block size (%u) is not a "
- "power of two.",
- ni->itype.index.block_size);
- goto unm_err_out;
- }
- if (ni->itype.index.block_size > PAGE_SIZE) {
- ntfs_error(vi->i_sb, "Index block size (%u) > "
- "PAGE_SIZE (%ld) is not "
- "supported. Sorry.",
- ni->itype.index.block_size,
- PAGE_SIZE);
- err = -EOPNOTSUPP;
- goto unm_err_out;
- }
- if (ni->itype.index.block_size < NTFS_BLOCK_SIZE) {
- ntfs_error(vi->i_sb, "Index block size (%u) < "
- "NTFS_BLOCK_SIZE (%i) is not "
- "supported. Sorry.",
- ni->itype.index.block_size,
- NTFS_BLOCK_SIZE);
- err = -EOPNOTSUPP;
- goto unm_err_out;
- }
- ni->itype.index.block_size_bits =
- ffs(ni->itype.index.block_size) - 1;
- /* Determine the size of a vcn in the directory index. */
- if (vol->cluster_size <= ni->itype.index.block_size) {
- ni->itype.index.vcn_size = vol->cluster_size;
- ni->itype.index.vcn_size_bits = vol->cluster_size_bits;
- } else {
- ni->itype.index.vcn_size = vol->sector_size;
- ni->itype.index.vcn_size_bits = vol->sector_size_bits;
- }
-
- /* Setup the index allocation attribute, even if not present. */
- NInoSetMstProtected(ni);
- ni->type = AT_INDEX_ALLOCATION;
- ni->name = I30;
- ni->name_len = 4;
-
- if (!(ir->index.flags & LARGE_INDEX)) {
- /* No index allocation. */
- vi->i_size = ni->initialized_size =
- ni->allocated_size = 0;
- /* We are done with the mft record, so we release it. */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ni);
- m = NULL;
- ctx = NULL;
- goto skip_large_dir_stuff;
- } /* LARGE_INDEX: Index allocation present. Setup state. */
- NInoSetIndexAllocPresent(ni);
- /* Find index allocation attribute. */
- ntfs_attr_reinit_search_ctx(ctx);
- err = ntfs_attr_lookup(AT_INDEX_ALLOCATION, I30, 4,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION "
- "attribute is not present but "
- "$INDEX_ROOT indicated it is.");
- else
- ntfs_error(vi->i_sb, "Failed to lookup "
- "$INDEX_ALLOCATION "
- "attribute.");
- goto unm_err_out;
- }
- a = ctx->attr;
- if (!a->non_resident) {
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute "
- "is resident.");
- goto unm_err_out;
- }
- /*
- * Ensure the attribute name is placed before the mapping pairs
- * array.
- */
- if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >=
- le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset)))) {
- ntfs_error(vol->sb, "$INDEX_ALLOCATION attribute name "
- "is placed after the mapping pairs "
- "array.");
- goto unm_err_out;
- }
- if (a->flags & ATTR_IS_ENCRYPTED) {
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute "
- "is encrypted.");
- goto unm_err_out;
- }
- if (a->flags & ATTR_IS_SPARSE) {
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute "
- "is sparse.");
- goto unm_err_out;
- }
- if (a->flags & ATTR_COMPRESSION_MASK) {
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute "
- "is compressed.");
- goto unm_err_out;
- }
- if (a->data.non_resident.lowest_vcn) {
- ntfs_error(vi->i_sb, "First extent of "
- "$INDEX_ALLOCATION attribute has non "
- "zero lowest_vcn.");
- goto unm_err_out;
- }
- vi->i_size = sle64_to_cpu(a->data.non_resident.data_size);
- ni->initialized_size = sle64_to_cpu(
- a->data.non_resident.initialized_size);
- ni->allocated_size = sle64_to_cpu(
- a->data.non_resident.allocated_size);
- /*
- * We are done with the mft record, so we release it. Otherwise
- * we would deadlock in ntfs_attr_iget().
- */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ni);
- m = NULL;
- ctx = NULL;
- /* Get the index bitmap attribute inode. */
- bvi = ntfs_attr_iget(vi, AT_BITMAP, I30, 4);
- if (IS_ERR(bvi)) {
- ntfs_error(vi->i_sb, "Failed to get bitmap attribute.");
- err = PTR_ERR(bvi);
- goto unm_err_out;
- }
- bni = NTFS_I(bvi);
- if (NInoCompressed(bni) || NInoEncrypted(bni) ||
- NInoSparse(bni)) {
- ntfs_error(vi->i_sb, "$BITMAP attribute is compressed "
- "and/or encrypted and/or sparse.");
- goto iput_unm_err_out;
- }
- /* Consistency check bitmap size vs. index allocation size. */
- bvi_size = i_size_read(bvi);
- if ((bvi_size << 3) < (vi->i_size >>
- ni->itype.index.block_size_bits)) {
- ntfs_error(vi->i_sb, "Index bitmap too small (0x%llx) "
- "for index allocation (0x%llx).",
- bvi_size << 3, vi->i_size);
- goto iput_unm_err_out;
- }
- /* No longer need the bitmap attribute inode. */
- iput(bvi);
-skip_large_dir_stuff:
- /* Setup the operations for this inode. */
- vi->i_op = &ntfs_dir_inode_ops;
- vi->i_fop = &ntfs_dir_ops;
- vi->i_mapping->a_ops = &ntfs_mst_aops;
- } else {
- /* It is a file. */
- ntfs_attr_reinit_search_ctx(ctx);
-
- /* Setup the data attribute, even if not present. */
- ni->type = AT_DATA;
- ni->name = NULL;
- ni->name_len = 0;
-
- /* Find first extent of the unnamed data attribute. */
- err = ntfs_attr_lookup(AT_DATA, NULL, 0, 0, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- vi->i_size = ni->initialized_size =
- ni->allocated_size = 0;
- if (err != -ENOENT) {
- ntfs_error(vi->i_sb, "Failed to lookup $DATA "
- "attribute.");
- goto unm_err_out;
- }
- /*
- * FILE_Secure does not have an unnamed $DATA
- * attribute, so we special case it here.
- */
- if (vi->i_ino == FILE_Secure)
- goto no_data_attr_special_case;
- /*
- * Most if not all the system files in the $Extend
- * system directory do not have unnamed data
- * attributes so we need to check if the parent
- * directory of the file is FILE_Extend and if it is
- * ignore this error. To do this we need to get the
- * name of this inode from the mft record as the name
- * contains the back reference to the parent directory.
- */
- if (ntfs_is_extended_system_file(ctx) > 0)
- goto no_data_attr_special_case;
- // FIXME: File is corrupt! Hot-fix with empty data
- // attribute if recovery option is set.
- ntfs_error(vi->i_sb, "$DATA attribute is missing.");
- goto unm_err_out;
- }
- a = ctx->attr;
- /* Setup the state. */
- if (a->flags & (ATTR_COMPRESSION_MASK | ATTR_IS_SPARSE)) {
- if (a->flags & ATTR_COMPRESSION_MASK) {
- NInoSetCompressed(ni);
- if (vol->cluster_size > 4096) {
- ntfs_error(vi->i_sb, "Found "
- "compressed data but "
- "compression is "
- "disabled due to "
- "cluster size (%i) > "
- "4kiB.",
- vol->cluster_size);
- goto unm_err_out;
- }
- if ((a->flags & ATTR_COMPRESSION_MASK)
- != ATTR_IS_COMPRESSED) {
- ntfs_error(vi->i_sb, "Found unknown "
- "compression method "
- "or corrupt file.");
- goto unm_err_out;
- }
- }
- if (a->flags & ATTR_IS_SPARSE)
- NInoSetSparse(ni);
- }
- if (a->flags & ATTR_IS_ENCRYPTED) {
- if (NInoCompressed(ni)) {
- ntfs_error(vi->i_sb, "Found encrypted and "
- "compressed data.");
- goto unm_err_out;
- }
- NInoSetEncrypted(ni);
- }
- if (a->non_resident) {
- NInoSetNonResident(ni);
- if (NInoCompressed(ni) || NInoSparse(ni)) {
- if (NInoCompressed(ni) && a->data.non_resident.
- compression_unit != 4) {
- ntfs_error(vi->i_sb, "Found "
- "non-standard "
- "compression unit (%u "
- "instead of 4). "
- "Cannot handle this.",
- a->data.non_resident.
- compression_unit);
- err = -EOPNOTSUPP;
- goto unm_err_out;
- }
- if (a->data.non_resident.compression_unit) {
- ni->itype.compressed.block_size = 1U <<
- (a->data.non_resident.
- compression_unit +
- vol->cluster_size_bits);
- ni->itype.compressed.block_size_bits =
- ffs(ni->itype.
- compressed.
- block_size) - 1;
- ni->itype.compressed.block_clusters =
- 1U << a->data.
- non_resident.
- compression_unit;
- } else {
- ni->itype.compressed.block_size = 0;
- ni->itype.compressed.block_size_bits =
- 0;
- ni->itype.compressed.block_clusters =
- 0;
- }
- ni->itype.compressed.size = sle64_to_cpu(
- a->data.non_resident.
- compressed_size);
- }
- if (a->data.non_resident.lowest_vcn) {
- ntfs_error(vi->i_sb, "First extent of $DATA "
- "attribute has non zero "
- "lowest_vcn.");
- goto unm_err_out;
- }
- vi->i_size = sle64_to_cpu(
- a->data.non_resident.data_size);
- ni->initialized_size = sle64_to_cpu(
- a->data.non_resident.initialized_size);
- ni->allocated_size = sle64_to_cpu(
- a->data.non_resident.allocated_size);
- } else { /* Resident attribute. */
- vi->i_size = ni->initialized_size = le32_to_cpu(
- a->data.resident.value_length);
- ni->allocated_size = le32_to_cpu(a->length) -
- le16_to_cpu(
- a->data.resident.value_offset);
- if (vi->i_size > ni->allocated_size) {
- ntfs_error(vi->i_sb, "Resident data attribute "
- "is corrupt (size exceeds "
- "allocation).");
- goto unm_err_out;
- }
- }
-no_data_attr_special_case:
- /* We are done with the mft record, so we release it. */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ni);
- m = NULL;
- ctx = NULL;
- /* Setup the operations for this inode. */
- vi->i_op = &ntfs_file_inode_ops;
- vi->i_fop = &ntfs_file_ops;
- vi->i_mapping->a_ops = &ntfs_normal_aops;
- if (NInoMstProtected(ni))
- vi->i_mapping->a_ops = &ntfs_mst_aops;
- else if (NInoCompressed(ni))
- vi->i_mapping->a_ops = &ntfs_compressed_aops;
- }
- /*
- * The number of 512-byte blocks used on disk (for stat). This is in so
- * far inaccurate as it doesn't account for any named streams or other
- * special non-resident attributes, but that is how Windows works, too,
- * so we are at least consistent with Windows, if not entirely
- * consistent with the Linux Way. Doing it the Linux Way would cause a
- * significant slowdown as it would involve iterating over all
- * attributes in the mft record and adding the allocated/compressed
- * sizes of all non-resident attributes present to give us the Linux
- * correct size that should go into i_blocks (after division by 512).
- */
- if (S_ISREG(vi->i_mode) && (NInoCompressed(ni) || NInoSparse(ni)))
- vi->i_blocks = ni->itype.compressed.size >> 9;
- else
- vi->i_blocks = ni->allocated_size >> 9;
- ntfs_debug("Done.");
- return 0;
-iput_unm_err_out:
- iput(bvi);
-unm_err_out:
- if (!err)
- err = -EIO;
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(ni);
-err_out:
- ntfs_error(vol->sb, "Failed with error code %i. Marking corrupt "
- "inode 0x%lx as bad. Run chkdsk.", err, vi->i_ino);
- make_bad_inode(vi);
- if (err != -EOPNOTSUPP && err != -ENOMEM)
- NVolSetErrors(vol);
- return err;
-}
-
-/**
- * ntfs_read_locked_attr_inode - read an attribute inode from its base inode
- * @base_vi: base inode
- * @vi: attribute inode to read
- *
- * ntfs_read_locked_attr_inode() is called from ntfs_attr_iget() to read the
- * attribute inode described by @vi into memory from the base mft record
- * described by @base_ni.
- *
- * ntfs_read_locked_attr_inode() maps, pins and locks the base inode for
- * reading and looks up the attribute described by @vi before setting up the
- * necessary fields in @vi as well as initializing the ntfs inode.
- *
- * Q: What locks are held when the function is called?
- * A: i_state has I_NEW set, hence the inode is locked, also
- * i_count is set to 1, so it is not going to go away
- *
- * Return 0 on success and -errno on error. In the error case, the inode will
- * have had make_bad_inode() executed on it.
- *
- * Note this cannot be called for AT_INDEX_ALLOCATION.
- */
-static int ntfs_read_locked_attr_inode(struct inode *base_vi, struct inode *vi)
-{
- ntfs_volume *vol = NTFS_SB(vi->i_sb);
- ntfs_inode *ni, *base_ni;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- ntfs_attr_search_ctx *ctx;
- int err = 0;
-
- ntfs_debug("Entering for i_ino 0x%lx.", vi->i_ino);
-
- ntfs_init_big_inode(vi);
-
- ni = NTFS_I(vi);
- base_ni = NTFS_I(base_vi);
-
- /* Just mirror the values from the base inode. */
- vi->i_uid = base_vi->i_uid;
- vi->i_gid = base_vi->i_gid;
- set_nlink(vi, base_vi->i_nlink);
- vi->i_mtime = base_vi->i_mtime;
- vi->i_ctime = base_vi->i_ctime;
- vi->i_atime = base_vi->i_atime;
- vi->i_generation = ni->seq_no = base_ni->seq_no;
-
- /* Set inode type to zero but preserve permissions. */
- vi->i_mode = base_vi->i_mode & ~S_IFMT;
-
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (!ctx) {
- err = -ENOMEM;
- goto unm_err_out;
- }
- /* Find the attribute. */
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err))
- goto unm_err_out;
- a = ctx->attr;
- if (a->flags & (ATTR_COMPRESSION_MASK | ATTR_IS_SPARSE)) {
- if (a->flags & ATTR_COMPRESSION_MASK) {
- NInoSetCompressed(ni);
- if ((ni->type != AT_DATA) || (ni->type == AT_DATA &&
- ni->name_len)) {
- ntfs_error(vi->i_sb, "Found compressed "
- "non-data or named data "
- "attribute. Please report "
- "you saw this message to "
- "linux-ntfs-dev@lists."
- "sourceforge.net");
- goto unm_err_out;
- }
- if (vol->cluster_size > 4096) {
- ntfs_error(vi->i_sb, "Found compressed "
- "attribute but compression is "
- "disabled due to cluster size "
- "(%i) > 4kiB.",
- vol->cluster_size);
- goto unm_err_out;
- }
- if ((a->flags & ATTR_COMPRESSION_MASK) !=
- ATTR_IS_COMPRESSED) {
- ntfs_error(vi->i_sb, "Found unknown "
- "compression method.");
- goto unm_err_out;
- }
- }
- /*
- * The compressed/sparse flag set in an index root just means
- * to compress all files.
- */
- if (NInoMstProtected(ni) && ni->type != AT_INDEX_ROOT) {
- ntfs_error(vi->i_sb, "Found mst protected attribute "
- "but the attribute is %s. Please "
- "report you saw this message to "
- "linux-ntfs-dev@lists.sourceforge.net",
- NInoCompressed(ni) ? "compressed" :
- "sparse");
- goto unm_err_out;
- }
- if (a->flags & ATTR_IS_SPARSE)
- NInoSetSparse(ni);
- }
- if (a->flags & ATTR_IS_ENCRYPTED) {
- if (NInoCompressed(ni)) {
- ntfs_error(vi->i_sb, "Found encrypted and compressed "
- "data.");
- goto unm_err_out;
- }
- /*
- * The encryption flag set in an index root just means to
- * encrypt all files.
- */
- if (NInoMstProtected(ni) && ni->type != AT_INDEX_ROOT) {
- ntfs_error(vi->i_sb, "Found mst protected attribute "
- "but the attribute is encrypted. "
- "Please report you saw this message "
- "to linux-ntfs-dev@lists.sourceforge."
- "net");
- goto unm_err_out;
- }
- if (ni->type != AT_DATA) {
- ntfs_error(vi->i_sb, "Found encrypted non-data "
- "attribute.");
- goto unm_err_out;
- }
- NInoSetEncrypted(ni);
- }
- if (!a->non_resident) {
- /* Ensure the attribute name is placed before the value. */
- if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >=
- le16_to_cpu(a->data.resident.value_offset)))) {
- ntfs_error(vol->sb, "Attribute name is placed after "
- "the attribute value.");
- goto unm_err_out;
- }
- if (NInoMstProtected(ni)) {
- ntfs_error(vi->i_sb, "Found mst protected attribute "
- "but the attribute is resident. "
- "Please report you saw this message to "
- "linux-ntfs-dev@lists.sourceforge.net");
- goto unm_err_out;
- }
- vi->i_size = ni->initialized_size = le32_to_cpu(
- a->data.resident.value_length);
- ni->allocated_size = le32_to_cpu(a->length) -
- le16_to_cpu(a->data.resident.value_offset);
- if (vi->i_size > ni->allocated_size) {
- ntfs_error(vi->i_sb, "Resident attribute is corrupt "
- "(size exceeds allocation).");
- goto unm_err_out;
- }
- } else {
- NInoSetNonResident(ni);
- /*
- * Ensure the attribute name is placed before the mapping pairs
- * array.
- */
- if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >=
- le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset)))) {
- ntfs_error(vol->sb, "Attribute name is placed after "
- "the mapping pairs array.");
- goto unm_err_out;
- }
- if (NInoCompressed(ni) || NInoSparse(ni)) {
- if (NInoCompressed(ni) && a->data.non_resident.
- compression_unit != 4) {
- ntfs_error(vi->i_sb, "Found non-standard "
- "compression unit (%u instead "
- "of 4). Cannot handle this.",
- a->data.non_resident.
- compression_unit);
- err = -EOPNOTSUPP;
- goto unm_err_out;
- }
- if (a->data.non_resident.compression_unit) {
- ni->itype.compressed.block_size = 1U <<
- (a->data.non_resident.
- compression_unit +
- vol->cluster_size_bits);
- ni->itype.compressed.block_size_bits =
- ffs(ni->itype.compressed.
- block_size) - 1;
- ni->itype.compressed.block_clusters = 1U <<
- a->data.non_resident.
- compression_unit;
- } else {
- ni->itype.compressed.block_size = 0;
- ni->itype.compressed.block_size_bits = 0;
- ni->itype.compressed.block_clusters = 0;
- }
- ni->itype.compressed.size = sle64_to_cpu(
- a->data.non_resident.compressed_size);
- }
- if (a->data.non_resident.lowest_vcn) {
- ntfs_error(vi->i_sb, "First extent of attribute has "
- "non-zero lowest_vcn.");
- goto unm_err_out;
- }
- vi->i_size = sle64_to_cpu(a->data.non_resident.data_size);
- ni->initialized_size = sle64_to_cpu(
- a->data.non_resident.initialized_size);
- ni->allocated_size = sle64_to_cpu(
- a->data.non_resident.allocated_size);
- }
- vi->i_mapping->a_ops = &ntfs_normal_aops;
- if (NInoMstProtected(ni))
- vi->i_mapping->a_ops = &ntfs_mst_aops;
- else if (NInoCompressed(ni))
- vi->i_mapping->a_ops = &ntfs_compressed_aops;
- if ((NInoCompressed(ni) || NInoSparse(ni)) && ni->type != AT_INDEX_ROOT)
- vi->i_blocks = ni->itype.compressed.size >> 9;
- else
- vi->i_blocks = ni->allocated_size >> 9;
- /*
- * Make sure the base inode does not go away and attach it to the
- * attribute inode.
- */
- igrab(base_vi);
- ni->ext.base_ntfs_ino = base_ni;
- ni->nr_extents = -1;
-
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
-
- ntfs_debug("Done.");
- return 0;
-
-unm_err_out:
- if (!err)
- err = -EIO;
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
-err_out:
- ntfs_error(vol->sb, "Failed with error code %i while reading attribute "
- "inode (mft_no 0x%lx, type 0x%x, name_len %i). "
- "Marking corrupt inode and base inode 0x%lx as bad. "
- "Run chkdsk.", err, vi->i_ino, ni->type, ni->name_len,
- base_vi->i_ino);
- make_bad_inode(vi);
- if (err != -ENOMEM)
- NVolSetErrors(vol);
- return err;
-}
-
-/**
- * ntfs_read_locked_index_inode - read an index inode from its base inode
- * @base_vi: base inode
- * @vi: index inode to read
- *
- * ntfs_read_locked_index_inode() is called from ntfs_index_iget() to read the
- * index inode described by @vi into memory from the base mft record described
- * by @base_ni.
- *
- * ntfs_read_locked_index_inode() maps, pins and locks the base inode for
- * reading and looks up the attributes relating to the index described by @vi
- * before setting up the necessary fields in @vi as well as initializing the
- * ntfs inode.
- *
- * Note, index inodes are essentially attribute inodes (NInoAttr() is true)
- * with the attribute type set to AT_INDEX_ALLOCATION. Apart from that, they
- * are setup like directory inodes since directories are a special case of
- * indices ao they need to be treated in much the same way. Most importantly,
- * for small indices the index allocation attribute might not actually exist.
- * However, the index root attribute always exists but this does not need to
- * have an inode associated with it and this is why we define a new inode type
- * index. Also, like for directories, we need to have an attribute inode for
- * the bitmap attribute corresponding to the index allocation attribute and we
- * can store this in the appropriate field of the inode, just like we do for
- * normal directory inodes.
- *
- * Q: What locks are held when the function is called?
- * A: i_state has I_NEW set, hence the inode is locked, also
- * i_count is set to 1, so it is not going to go away
- *
- * Return 0 on success and -errno on error. In the error case, the inode will
- * have had make_bad_inode() executed on it.
- */
-static int ntfs_read_locked_index_inode(struct inode *base_vi, struct inode *vi)
-{
- loff_t bvi_size;
- ntfs_volume *vol = NTFS_SB(vi->i_sb);
- ntfs_inode *ni, *base_ni, *bni;
- struct inode *bvi;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- ntfs_attr_search_ctx *ctx;
- INDEX_ROOT *ir;
- u8 *ir_end, *index_end;
- int err = 0;
-
- ntfs_debug("Entering for i_ino 0x%lx.", vi->i_ino);
- ntfs_init_big_inode(vi);
- ni = NTFS_I(vi);
- base_ni = NTFS_I(base_vi);
- /* Just mirror the values from the base inode. */
- vi->i_uid = base_vi->i_uid;
- vi->i_gid = base_vi->i_gid;
- set_nlink(vi, base_vi->i_nlink);
- vi->i_mtime = base_vi->i_mtime;
- vi->i_ctime = base_vi->i_ctime;
- vi->i_atime = base_vi->i_atime;
- vi->i_generation = ni->seq_no = base_ni->seq_no;
- /* Set inode type to zero but preserve permissions. */
- vi->i_mode = base_vi->i_mode & ~S_IFMT;
- /* Map the mft record for the base inode. */
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (!ctx) {
- err = -ENOMEM;
- goto unm_err_out;
- }
- /* Find the index root attribute. */
- err = ntfs_attr_lookup(AT_INDEX_ROOT, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- ntfs_error(vi->i_sb, "$INDEX_ROOT attribute is "
- "missing.");
- goto unm_err_out;
- }
- a = ctx->attr;
- /* Set up the state. */
- if (unlikely(a->non_resident)) {
- ntfs_error(vol->sb, "$INDEX_ROOT attribute is not resident.");
- goto unm_err_out;
- }
- /* Ensure the attribute name is placed before the value. */
- if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >=
- le16_to_cpu(a->data.resident.value_offset)))) {
- ntfs_error(vol->sb, "$INDEX_ROOT attribute name is placed "
- "after the attribute value.");
- goto unm_err_out;
- }
- /*
- * Compressed/encrypted/sparse index root is not allowed, except for
- * directories of course but those are not dealt with here.
- */
- if (a->flags & (ATTR_COMPRESSION_MASK | ATTR_IS_ENCRYPTED |
- ATTR_IS_SPARSE)) {
- ntfs_error(vi->i_sb, "Found compressed/encrypted/sparse index "
- "root attribute.");
- goto unm_err_out;
- }
- ir = (INDEX_ROOT*)((u8*)a + le16_to_cpu(a->data.resident.value_offset));
- ir_end = (u8*)ir + le32_to_cpu(a->data.resident.value_length);
- if (ir_end > (u8*)ctx->mrec + vol->mft_record_size) {
- ntfs_error(vi->i_sb, "$INDEX_ROOT attribute is corrupt.");
- goto unm_err_out;
- }
- index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length);
- if (index_end > ir_end) {
- ntfs_error(vi->i_sb, "Index is corrupt.");
- goto unm_err_out;
- }
- if (ir->type) {
- ntfs_error(vi->i_sb, "Index type is not 0 (type is 0x%x).",
- le32_to_cpu(ir->type));
- goto unm_err_out;
- }
- ni->itype.index.collation_rule = ir->collation_rule;
- ntfs_debug("Index collation rule is 0x%x.",
- le32_to_cpu(ir->collation_rule));
- ni->itype.index.block_size = le32_to_cpu(ir->index_block_size);
- if (!is_power_of_2(ni->itype.index.block_size)) {
- ntfs_error(vi->i_sb, "Index block size (%u) is not a power of "
- "two.", ni->itype.index.block_size);
- goto unm_err_out;
- }
- if (ni->itype.index.block_size > PAGE_SIZE) {
- ntfs_error(vi->i_sb, "Index block size (%u) > PAGE_SIZE "
- "(%ld) is not supported. Sorry.",
- ni->itype.index.block_size, PAGE_SIZE);
- err = -EOPNOTSUPP;
- goto unm_err_out;
- }
- if (ni->itype.index.block_size < NTFS_BLOCK_SIZE) {
- ntfs_error(vi->i_sb, "Index block size (%u) < NTFS_BLOCK_SIZE "
- "(%i) is not supported. Sorry.",
- ni->itype.index.block_size, NTFS_BLOCK_SIZE);
- err = -EOPNOTSUPP;
- goto unm_err_out;
- }
- ni->itype.index.block_size_bits = ffs(ni->itype.index.block_size) - 1;
- /* Determine the size of a vcn in the index. */
- if (vol->cluster_size <= ni->itype.index.block_size) {
- ni->itype.index.vcn_size = vol->cluster_size;
- ni->itype.index.vcn_size_bits = vol->cluster_size_bits;
- } else {
- ni->itype.index.vcn_size = vol->sector_size;
- ni->itype.index.vcn_size_bits = vol->sector_size_bits;
- }
- /* Check for presence of index allocation attribute. */
- if (!(ir->index.flags & LARGE_INDEX)) {
- /* No index allocation. */
- vi->i_size = ni->initialized_size = ni->allocated_size = 0;
- /* We are done with the mft record, so we release it. */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- m = NULL;
- ctx = NULL;
- goto skip_large_index_stuff;
- } /* LARGE_INDEX: Index allocation present. Setup state. */
- NInoSetIndexAllocPresent(ni);
- /* Find index allocation attribute. */
- ntfs_attr_reinit_search_ctx(ctx);
- err = ntfs_attr_lookup(AT_INDEX_ALLOCATION, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT)
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is "
- "not present but $INDEX_ROOT "
- "indicated it is.");
- else
- ntfs_error(vi->i_sb, "Failed to lookup "
- "$INDEX_ALLOCATION attribute.");
- goto unm_err_out;
- }
- a = ctx->attr;
- if (!a->non_resident) {
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is "
- "resident.");
- goto unm_err_out;
- }
- /*
- * Ensure the attribute name is placed before the mapping pairs array.
- */
- if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >=
- le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset)))) {
- ntfs_error(vol->sb, "$INDEX_ALLOCATION attribute name is "
- "placed after the mapping pairs array.");
- goto unm_err_out;
- }
- if (a->flags & ATTR_IS_ENCRYPTED) {
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is "
- "encrypted.");
- goto unm_err_out;
- }
- if (a->flags & ATTR_IS_SPARSE) {
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is sparse.");
- goto unm_err_out;
- }
- if (a->flags & ATTR_COMPRESSION_MASK) {
- ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is "
- "compressed.");
- goto unm_err_out;
- }
- if (a->data.non_resident.lowest_vcn) {
- ntfs_error(vi->i_sb, "First extent of $INDEX_ALLOCATION "
- "attribute has non zero lowest_vcn.");
- goto unm_err_out;
- }
- vi->i_size = sle64_to_cpu(a->data.non_resident.data_size);
- ni->initialized_size = sle64_to_cpu(
- a->data.non_resident.initialized_size);
- ni->allocated_size = sle64_to_cpu(a->data.non_resident.allocated_size);
- /*
- * We are done with the mft record, so we release it. Otherwise
- * we would deadlock in ntfs_attr_iget().
- */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- m = NULL;
- ctx = NULL;
- /* Get the index bitmap attribute inode. */
- bvi = ntfs_attr_iget(base_vi, AT_BITMAP, ni->name, ni->name_len);
- if (IS_ERR(bvi)) {
- ntfs_error(vi->i_sb, "Failed to get bitmap attribute.");
- err = PTR_ERR(bvi);
- goto unm_err_out;
- }
- bni = NTFS_I(bvi);
- if (NInoCompressed(bni) || NInoEncrypted(bni) ||
- NInoSparse(bni)) {
- ntfs_error(vi->i_sb, "$BITMAP attribute is compressed and/or "
- "encrypted and/or sparse.");
- goto iput_unm_err_out;
- }
- /* Consistency check bitmap size vs. index allocation size. */
- bvi_size = i_size_read(bvi);
- if ((bvi_size << 3) < (vi->i_size >> ni->itype.index.block_size_bits)) {
- ntfs_error(vi->i_sb, "Index bitmap too small (0x%llx) for "
- "index allocation (0x%llx).", bvi_size << 3,
- vi->i_size);
- goto iput_unm_err_out;
- }
- iput(bvi);
-skip_large_index_stuff:
- /* Setup the operations for this index inode. */
- vi->i_mapping->a_ops = &ntfs_mst_aops;
- vi->i_blocks = ni->allocated_size >> 9;
- /*
- * Make sure the base inode doesn't go away and attach it to the
- * index inode.
- */
- igrab(base_vi);
- ni->ext.base_ntfs_ino = base_ni;
- ni->nr_extents = -1;
-
- ntfs_debug("Done.");
- return 0;
-iput_unm_err_out:
- iput(bvi);
-unm_err_out:
- if (!err)
- err = -EIO;
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
-err_out:
- ntfs_error(vi->i_sb, "Failed with error code %i while reading index "
- "inode (mft_no 0x%lx, name_len %i.", err, vi->i_ino,
- ni->name_len);
- make_bad_inode(vi);
- if (err != -EOPNOTSUPP && err != -ENOMEM)
- NVolSetErrors(vol);
- return err;
-}
-
-/*
- * The MFT inode has special locking, so teach the lock validator
- * about this by splitting off the locking rules of the MFT from
- * the locking rules of other inodes. The MFT inode can never be
- * accessed from the VFS side (or even internally), only by the
- * map_mft functions.
- */
-static struct lock_class_key mft_ni_runlist_lock_key, mft_ni_mrec_lock_key;
-
-/**
- * ntfs_read_inode_mount - special read_inode for mount time use only
- * @vi: inode to read
- *
- * Read inode FILE_MFT at mount time, only called with super_block lock
- * held from within the read_super() code path.
- *
- * This function exists because when it is called the page cache for $MFT/$DATA
- * is not initialized and hence we cannot get at the contents of mft records
- * by calling map_mft_record*().
- *
- * Further it needs to cope with the circular references problem, i.e. cannot
- * load any attributes other than $ATTRIBUTE_LIST until $DATA is loaded, because
- * we do not know where the other extent mft records are yet and again, because
- * we cannot call map_mft_record*() yet. Obviously this applies only when an
- * attribute list is actually present in $MFT inode.
- *
- * We solve these problems by starting with the $DATA attribute before anything
- * else and iterating using ntfs_attr_lookup($DATA) over all extents. As each
- * extent is found, we ntfs_mapping_pairs_decompress() including the implied
- * ntfs_runlists_merge(). Each step of the iteration necessarily provides
- * sufficient information for the next step to complete.
- *
- * This should work but there are two possible pit falls (see inline comments
- * below), but only time will tell if they are real pits or just smoke...
- */
-int ntfs_read_inode_mount(struct inode *vi)
-{
- VCN next_vcn, last_vcn, highest_vcn;
- s64 block;
- struct super_block *sb = vi->i_sb;
- ntfs_volume *vol = NTFS_SB(sb);
- struct buffer_head *bh;
- ntfs_inode *ni;
- MFT_RECORD *m = NULL;
- ATTR_RECORD *a;
- ntfs_attr_search_ctx *ctx;
- unsigned int i, nr_blocks;
- int err;
-
- ntfs_debug("Entering.");
-
- /* Initialize the ntfs specific part of @vi. */
- ntfs_init_big_inode(vi);
-
- ni = NTFS_I(vi);
-
- /* Setup the data attribute. It is special as it is mst protected. */
- NInoSetNonResident(ni);
- NInoSetMstProtected(ni);
- NInoSetSparseDisabled(ni);
- ni->type = AT_DATA;
- ni->name = NULL;
- ni->name_len = 0;
- /*
- * This sets up our little cheat allowing us to reuse the async read io
- * completion handler for directories.
- */
- ni->itype.index.block_size = vol->mft_record_size;
- ni->itype.index.block_size_bits = vol->mft_record_size_bits;
-
- /* Very important! Needed to be able to call map_mft_record*(). */
- vol->mft_ino = vi;
-
- /* Allocate enough memory to read the first mft record. */
- if (vol->mft_record_size > 64 * 1024) {
- ntfs_error(sb, "Unsupported mft record size %i (max 64kiB).",
- vol->mft_record_size);
- goto err_out;
- }
- i = vol->mft_record_size;
- if (i < sb->s_blocksize)
- i = sb->s_blocksize;
- m = (MFT_RECORD*)ntfs_malloc_nofs(i);
- if (!m) {
- ntfs_error(sb, "Failed to allocate buffer for $MFT record 0.");
- goto err_out;
- }
-
- /* Determine the first block of the $MFT/$DATA attribute. */
- block = vol->mft_lcn << vol->cluster_size_bits >>
- sb->s_blocksize_bits;
- nr_blocks = vol->mft_record_size >> sb->s_blocksize_bits;
- if (!nr_blocks)
- nr_blocks = 1;
-
- /* Load $MFT/$DATA's first mft record. */
- for (i = 0; i < nr_blocks; i++) {
- bh = sb_bread(sb, block++);
- if (!bh) {
- ntfs_error(sb, "Device read failed.");
- goto err_out;
- }
- memcpy((char*)m + (i << sb->s_blocksize_bits), bh->b_data,
- sb->s_blocksize);
- brelse(bh);
- }
-
- if (le32_to_cpu(m->bytes_allocated) != vol->mft_record_size) {
- ntfs_error(sb, "Incorrect mft record size %u in superblock, should be %u.",
- le32_to_cpu(m->bytes_allocated), vol->mft_record_size);
- goto err_out;
- }
-
- /* Apply the mst fixups. */
- if (post_read_mst_fixup((NTFS_RECORD*)m, vol->mft_record_size)) {
- /* FIXME: Try to use the $MFTMirr now. */
- ntfs_error(sb, "MST fixup failed. $MFT is corrupt.");
- goto err_out;
- }
-
- /* Sanity check offset to the first attribute */
- if (le16_to_cpu(m->attrs_offset) >= le32_to_cpu(m->bytes_allocated)) {
- ntfs_error(sb, "Incorrect mft offset to the first attribute %u in superblock.",
- le16_to_cpu(m->attrs_offset));
- goto err_out;
- }
-
- /* Need this to sanity check attribute list references to $MFT. */
- vi->i_generation = ni->seq_no = le16_to_cpu(m->sequence_number);
-
- /* Provides read_folio() for map_mft_record(). */
- vi->i_mapping->a_ops = &ntfs_mst_aops;
-
- ctx = ntfs_attr_get_search_ctx(ni, m);
- if (!ctx) {
- err = -ENOMEM;
- goto err_out;
- }
-
- /* Find the attribute list attribute if present. */
- err = ntfs_attr_lookup(AT_ATTRIBUTE_LIST, NULL, 0, 0, 0, NULL, 0, ctx);
- if (err) {
- if (unlikely(err != -ENOENT)) {
- ntfs_error(sb, "Failed to lookup attribute list "
- "attribute. You should run chkdsk.");
- goto put_err_out;
- }
- } else /* if (!err) */ {
- ATTR_LIST_ENTRY *al_entry, *next_al_entry;
- u8 *al_end;
- static const char *es = " Not allowed. $MFT is corrupt. "
- "You should run chkdsk.";
-
- ntfs_debug("Attribute list attribute found in $MFT.");
- NInoSetAttrList(ni);
- a = ctx->attr;
- if (a->flags & ATTR_COMPRESSION_MASK) {
- ntfs_error(sb, "Attribute list attribute is "
- "compressed.%s", es);
- goto put_err_out;
- }
- if (a->flags & ATTR_IS_ENCRYPTED ||
- a->flags & ATTR_IS_SPARSE) {
- if (a->non_resident) {
- ntfs_error(sb, "Non-resident attribute list "
- "attribute is encrypted/"
- "sparse.%s", es);
- goto put_err_out;
- }
- ntfs_warning(sb, "Resident attribute list attribute "
- "in $MFT system file is marked "
- "encrypted/sparse which is not true. "
- "However, Windows allows this and "
- "chkdsk does not detect or correct it "
- "so we will just ignore the invalid "
- "flags and pretend they are not set.");
- }
- /* Now allocate memory for the attribute list. */
- ni->attr_list_size = (u32)ntfs_attr_size(a);
- if (!ni->attr_list_size) {
- ntfs_error(sb, "Attr_list_size is zero");
- goto put_err_out;
- }
- ni->attr_list = ntfs_malloc_nofs(ni->attr_list_size);
- if (!ni->attr_list) {
- ntfs_error(sb, "Not enough memory to allocate buffer "
- "for attribute list.");
- goto put_err_out;
- }
- if (a->non_resident) {
- NInoSetAttrListNonResident(ni);
- if (a->data.non_resident.lowest_vcn) {
- ntfs_error(sb, "Attribute list has non zero "
- "lowest_vcn. $MFT is corrupt. "
- "You should run chkdsk.");
- goto put_err_out;
- }
- /* Setup the runlist. */
- ni->attr_list_rl.rl = ntfs_mapping_pairs_decompress(vol,
- a, NULL);
- if (IS_ERR(ni->attr_list_rl.rl)) {
- err = PTR_ERR(ni->attr_list_rl.rl);
- ni->attr_list_rl.rl = NULL;
- ntfs_error(sb, "Mapping pairs decompression "
- "failed with error code %i.",
- -err);
- goto put_err_out;
- }
- /* Now load the attribute list. */
- if ((err = load_attribute_list(vol, &ni->attr_list_rl,
- ni->attr_list, ni->attr_list_size,
- sle64_to_cpu(a->data.
- non_resident.initialized_size)))) {
- ntfs_error(sb, "Failed to load attribute list "
- "attribute with error code %i.",
- -err);
- goto put_err_out;
- }
- } else /* if (!ctx.attr->non_resident) */ {
- if ((u8*)a + le16_to_cpu(
- a->data.resident.value_offset) +
- le32_to_cpu(
- a->data.resident.value_length) >
- (u8*)ctx->mrec + vol->mft_record_size) {
- ntfs_error(sb, "Corrupt attribute list "
- "attribute.");
- goto put_err_out;
- }
- /* Now copy the attribute list. */
- memcpy(ni->attr_list, (u8*)a + le16_to_cpu(
- a->data.resident.value_offset),
- le32_to_cpu(
- a->data.resident.value_length));
- }
- /* The attribute list is now setup in memory. */
- /*
- * FIXME: I don't know if this case is actually possible.
- * According to logic it is not possible but I have seen too
- * many weird things in MS software to rely on logic... Thus we
- * perform a manual search and make sure the first $MFT/$DATA
- * extent is in the base inode. If it is not we abort with an
- * error and if we ever see a report of this error we will need
- * to do some magic in order to have the necessary mft record
- * loaded and in the right place in the page cache. But
- * hopefully logic will prevail and this never happens...
- */
- al_entry = (ATTR_LIST_ENTRY*)ni->attr_list;
- al_end = (u8*)al_entry + ni->attr_list_size;
- for (;; al_entry = next_al_entry) {
- /* Out of bounds check. */
- if ((u8*)al_entry < ni->attr_list ||
- (u8*)al_entry > al_end)
- goto em_put_err_out;
- /* Catch the end of the attribute list. */
- if ((u8*)al_entry == al_end)
- goto em_put_err_out;
- if (!al_entry->length)
- goto em_put_err_out;
- if ((u8*)al_entry + 6 > al_end || (u8*)al_entry +
- le16_to_cpu(al_entry->length) > al_end)
- goto em_put_err_out;
- next_al_entry = (ATTR_LIST_ENTRY*)((u8*)al_entry +
- le16_to_cpu(al_entry->length));
- if (le32_to_cpu(al_entry->type) > le32_to_cpu(AT_DATA))
- goto em_put_err_out;
- if (AT_DATA != al_entry->type)
- continue;
- /* We want an unnamed attribute. */
- if (al_entry->name_length)
- goto em_put_err_out;
- /* Want the first entry, i.e. lowest_vcn == 0. */
- if (al_entry->lowest_vcn)
- goto em_put_err_out;
- /* First entry has to be in the base mft record. */
- if (MREF_LE(al_entry->mft_reference) != vi->i_ino) {
- /* MFT references do not match, logic fails. */
- ntfs_error(sb, "BUG: The first $DATA extent "
- "of $MFT is not in the base "
- "mft record. Please report "
- "you saw this message to "
- "linux-ntfs-dev@lists."
- "sourceforge.net");
- goto put_err_out;
- } else {
- /* Sequence numbers must match. */
- if (MSEQNO_LE(al_entry->mft_reference) !=
- ni->seq_no)
- goto em_put_err_out;
- /* Got it. All is ok. We can stop now. */
- break;
- }
- }
- }
-
- ntfs_attr_reinit_search_ctx(ctx);
-
- /* Now load all attribute extents. */
- a = NULL;
- next_vcn = last_vcn = highest_vcn = 0;
- while (!(err = ntfs_attr_lookup(AT_DATA, NULL, 0, 0, next_vcn, NULL, 0,
- ctx))) {
- runlist_element *nrl;
-
- /* Cache the current attribute. */
- a = ctx->attr;
- /* $MFT must be non-resident. */
- if (!a->non_resident) {
- ntfs_error(sb, "$MFT must be non-resident but a "
- "resident extent was found. $MFT is "
- "corrupt. Run chkdsk.");
- goto put_err_out;
- }
- /* $MFT must be uncompressed and unencrypted. */
- if (a->flags & ATTR_COMPRESSION_MASK ||
- a->flags & ATTR_IS_ENCRYPTED ||
- a->flags & ATTR_IS_SPARSE) {
- ntfs_error(sb, "$MFT must be uncompressed, "
- "non-sparse, and unencrypted but a "
- "compressed/sparse/encrypted extent "
- "was found. $MFT is corrupt. Run "
- "chkdsk.");
- goto put_err_out;
- }
- /*
- * Decompress the mapping pairs array of this extent and merge
- * the result into the existing runlist. No need for locking
- * as we have exclusive access to the inode at this time and we
- * are a mount in progress task, too.
- */
- nrl = ntfs_mapping_pairs_decompress(vol, a, ni->runlist.rl);
- if (IS_ERR(nrl)) {
- ntfs_error(sb, "ntfs_mapping_pairs_decompress() "
- "failed with error code %ld. $MFT is "
- "corrupt.", PTR_ERR(nrl));
- goto put_err_out;
- }
- ni->runlist.rl = nrl;
-
- /* Are we in the first extent? */
- if (!next_vcn) {
- if (a->data.non_resident.lowest_vcn) {
- ntfs_error(sb, "First extent of $DATA "
- "attribute has non zero "
- "lowest_vcn. $MFT is corrupt. "
- "You should run chkdsk.");
- goto put_err_out;
- }
- /* Get the last vcn in the $DATA attribute. */
- last_vcn = sle64_to_cpu(
- a->data.non_resident.allocated_size)
- >> vol->cluster_size_bits;
- /* Fill in the inode size. */
- vi->i_size = sle64_to_cpu(
- a->data.non_resident.data_size);
- ni->initialized_size = sle64_to_cpu(
- a->data.non_resident.initialized_size);
- ni->allocated_size = sle64_to_cpu(
- a->data.non_resident.allocated_size);
- /*
- * Verify the number of mft records does not exceed
- * 2^32 - 1.
- */
- if ((vi->i_size >> vol->mft_record_size_bits) >=
- (1ULL << 32)) {
- ntfs_error(sb, "$MFT is too big! Aborting.");
- goto put_err_out;
- }
- /*
- * We have got the first extent of the runlist for
- * $MFT which means it is now relatively safe to call
- * the normal ntfs_read_inode() function.
- * Complete reading the inode, this will actually
- * re-read the mft record for $MFT, this time entering
- * it into the page cache with which we complete the
- * kick start of the volume. It should be safe to do
- * this now as the first extent of $MFT/$DATA is
- * already known and we would hope that we don't need
- * further extents in order to find the other
- * attributes belonging to $MFT. Only time will tell if
- * this is really the case. If not we will have to play
- * magic at this point, possibly duplicating a lot of
- * ntfs_read_inode() at this point. We will need to
- * ensure we do enough of its work to be able to call
- * ntfs_read_inode() on extents of $MFT/$DATA. But lets
- * hope this never happens...
- */
- ntfs_read_locked_inode(vi);
- if (is_bad_inode(vi)) {
- ntfs_error(sb, "ntfs_read_inode() of $MFT "
- "failed. BUG or corrupt $MFT. "
- "Run chkdsk and if no errors "
- "are found, please report you "
- "saw this message to "
- "linux-ntfs-dev@lists."
- "sourceforge.net");
- ntfs_attr_put_search_ctx(ctx);
- /* Revert to the safe super operations. */
- ntfs_free(m);
- return -1;
- }
- /*
- * Re-initialize some specifics about $MFT's inode as
- * ntfs_read_inode() will have set up the default ones.
- */
- /* Set uid and gid to root. */
- vi->i_uid = GLOBAL_ROOT_UID;
- vi->i_gid = GLOBAL_ROOT_GID;
- /* Regular file. No access for anyone. */
- vi->i_mode = S_IFREG;
- /* No VFS initiated operations allowed for $MFT. */
- vi->i_op = &ntfs_empty_inode_ops;
- vi->i_fop = &ntfs_empty_file_ops;
- }
-
- /* Get the lowest vcn for the next extent. */
- highest_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn);
- next_vcn = highest_vcn + 1;
-
- /* Only one extent or error, which we catch below. */
- if (next_vcn <= 0)
- break;
-
- /* Avoid endless loops due to corruption. */
- if (next_vcn < sle64_to_cpu(
- a->data.non_resident.lowest_vcn)) {
- ntfs_error(sb, "$MFT has corrupt attribute list "
- "attribute. Run chkdsk.");
- goto put_err_out;
- }
- }
- if (err != -ENOENT) {
- ntfs_error(sb, "Failed to lookup $MFT/$DATA attribute extent. "
- "$MFT is corrupt. Run chkdsk.");
- goto put_err_out;
- }
- if (!a) {
- ntfs_error(sb, "$MFT/$DATA attribute not found. $MFT is "
- "corrupt. Run chkdsk.");
- goto put_err_out;
- }
- if (highest_vcn && highest_vcn != last_vcn - 1) {
- ntfs_error(sb, "Failed to load the complete runlist for "
- "$MFT/$DATA. Driver bug or corrupt $MFT. "
- "Run chkdsk.");
- ntfs_debug("highest_vcn = 0x%llx, last_vcn - 1 = 0x%llx",
- (unsigned long long)highest_vcn,
- (unsigned long long)last_vcn - 1);
- goto put_err_out;
- }
- ntfs_attr_put_search_ctx(ctx);
- ntfs_debug("Done.");
- ntfs_free(m);
-
- /*
- * Split the locking rules of the MFT inode from the
- * locking rules of other inodes:
- */
- lockdep_set_class(&ni->runlist.lock, &mft_ni_runlist_lock_key);
- lockdep_set_class(&ni->mrec_lock, &mft_ni_mrec_lock_key);
-
- return 0;
-
-em_put_err_out:
- ntfs_error(sb, "Couldn't find first extent of $DATA attribute in "
- "attribute list. $MFT is corrupt. Run chkdsk.");
-put_err_out:
- ntfs_attr_put_search_ctx(ctx);
-err_out:
- ntfs_error(sb, "Failed. Marking inode as bad.");
- make_bad_inode(vi);
- ntfs_free(m);
- return -1;
-}
-
-static void __ntfs_clear_inode(ntfs_inode *ni)
-{
- /* Free all alocated memory. */
- down_write(&ni->runlist.lock);
- if (ni->runlist.rl) {
- ntfs_free(ni->runlist.rl);
- ni->runlist.rl = NULL;
- }
- up_write(&ni->runlist.lock);
-
- if (ni->attr_list) {
- ntfs_free(ni->attr_list);
- ni->attr_list = NULL;
- }
-
- down_write(&ni->attr_list_rl.lock);
- if (ni->attr_list_rl.rl) {
- ntfs_free(ni->attr_list_rl.rl);
- ni->attr_list_rl.rl = NULL;
- }
- up_write(&ni->attr_list_rl.lock);
-
- if (ni->name_len && ni->name != I30) {
- /* Catch bugs... */
- BUG_ON(!ni->name);
- kfree(ni->name);
- }
-}
-
-void ntfs_clear_extent_inode(ntfs_inode *ni)
-{
- ntfs_debug("Entering for inode 0x%lx.", ni->mft_no);
-
- BUG_ON(NInoAttr(ni));
- BUG_ON(ni->nr_extents != -1);
-
-#ifdef NTFS_RW
- if (NInoDirty(ni)) {
- if (!is_bad_inode(VFS_I(ni->ext.base_ntfs_ino)))
- ntfs_error(ni->vol->sb, "Clearing dirty extent inode! "
- "Losing data! This is a BUG!!!");
- // FIXME: Do something!!!
- }
-#endif /* NTFS_RW */
-
- __ntfs_clear_inode(ni);
-
- /* Bye, bye... */
- ntfs_destroy_extent_inode(ni);
-}
-
-/**
- * ntfs_evict_big_inode - clean up the ntfs specific part of an inode
- * @vi: vfs inode pending annihilation
- *
- * When the VFS is going to remove an inode from memory, ntfs_clear_big_inode()
- * is called, which deallocates all memory belonging to the NTFS specific part
- * of the inode and returns.
- *
- * If the MFT record is dirty, we commit it before doing anything else.
- */
-void ntfs_evict_big_inode(struct inode *vi)
-{
- ntfs_inode *ni = NTFS_I(vi);
-
- truncate_inode_pages_final(&vi->i_data);
- clear_inode(vi);
-
-#ifdef NTFS_RW
- if (NInoDirty(ni)) {
- bool was_bad = (is_bad_inode(vi));
-
- /* Committing the inode also commits all extent inodes. */
- ntfs_commit_inode(vi);
-
- if (!was_bad && (is_bad_inode(vi) || NInoDirty(ni))) {
- ntfs_error(vi->i_sb, "Failed to commit dirty inode "
- "0x%lx. Losing data!", vi->i_ino);
- // FIXME: Do something!!!
- }
- }
-#endif /* NTFS_RW */
-
- /* No need to lock at this stage as no one else has a reference. */
- if (ni->nr_extents > 0) {
- int i;
-
- for (i = 0; i < ni->nr_extents; i++)
- ntfs_clear_extent_inode(ni->ext.extent_ntfs_inos[i]);
- kfree(ni->ext.extent_ntfs_inos);
- }
-
- __ntfs_clear_inode(ni);
-
- if (NInoAttr(ni)) {
- /* Release the base inode if we are holding it. */
- if (ni->nr_extents == -1) {
- iput(VFS_I(ni->ext.base_ntfs_ino));
- ni->nr_extents = 0;
- ni->ext.base_ntfs_ino = NULL;
- }
- }
- BUG_ON(ni->page);
- if (!atomic_dec_and_test(&ni->count))
- BUG();
- return;
-}
-
-/**
- * ntfs_show_options - show mount options in /proc/mounts
- * @sf: seq_file in which to write our mount options
- * @root: root of the mounted tree whose mount options to display
- *
- * Called by the VFS once for each mounted ntfs volume when someone reads
- * /proc/mounts in order to display the NTFS specific mount options of each
- * mount. The mount options of fs specified by @root are written to the seq file
- * @sf and success is returned.
- */
-int ntfs_show_options(struct seq_file *sf, struct dentry *root)
-{
- ntfs_volume *vol = NTFS_SB(root->d_sb);
- int i;
-
- seq_printf(sf, ",uid=%i", from_kuid_munged(&init_user_ns, vol->uid));
- seq_printf(sf, ",gid=%i", from_kgid_munged(&init_user_ns, vol->gid));
- if (vol->fmask == vol->dmask)
- seq_printf(sf, ",umask=0%o", vol->fmask);
- else {
- seq_printf(sf, ",fmask=0%o", vol->fmask);
- seq_printf(sf, ",dmask=0%o", vol->dmask);
- }
- seq_printf(sf, ",nls=%s", vol->nls_map->charset);
- if (NVolCaseSensitive(vol))
- seq_printf(sf, ",case_sensitive");
- if (NVolShowSystemFiles(vol))
- seq_printf(sf, ",show_sys_files");
- if (!NVolSparseEnabled(vol))
- seq_printf(sf, ",disable_sparse");
- for (i = 0; on_errors_arr[i].val; i++) {
- if (on_errors_arr[i].val & vol->on_errors)
- seq_printf(sf, ",errors=%s", on_errors_arr[i].str);
- }
- seq_printf(sf, ",mft_zone_multiplier=%i", vol->mft_zone_multiplier);
- return 0;
-}
-
-#ifdef NTFS_RW
-
-static const char *es = " Leaving inconsistent metadata. Unmount and run "
- "chkdsk.";
-
-/**
- * ntfs_truncate - called when the i_size of an ntfs inode is changed
- * @vi: inode for which the i_size was changed
- *
- * We only support i_size changes for normal files at present, i.e. not
- * compressed and not encrypted. This is enforced in ntfs_setattr(), see
- * below.
- *
- * The kernel guarantees that @vi is a regular file (S_ISREG() is true) and
- * that the change is allowed.
- *
- * This implies for us that @vi is a file inode rather than a directory, index,
- * or attribute inode as well as that @vi is a base inode.
- *
- * Returns 0 on success or -errno on error.
- *
- * Called with ->i_mutex held.
- */
-int ntfs_truncate(struct inode *vi)
-{
- s64 new_size, old_size, nr_freed, new_alloc_size, old_alloc_size;
- VCN highest_vcn;
- unsigned long flags;
- ntfs_inode *base_ni, *ni = NTFS_I(vi);
- ntfs_volume *vol = ni->vol;
- ntfs_attr_search_ctx *ctx;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- const char *te = " Leaving file length out of sync with i_size.";
- int err, mp_size, size_change, alloc_change;
-
- ntfs_debug("Entering for inode 0x%lx.", vi->i_ino);
- BUG_ON(NInoAttr(ni));
- BUG_ON(S_ISDIR(vi->i_mode));
- BUG_ON(NInoMstProtected(ni));
- BUG_ON(ni->nr_extents < 0);
-retry_truncate:
- /*
- * Lock the runlist for writing and map the mft record to ensure it is
- * safe to mess with the attribute runlist and sizes.
- */
- down_write(&ni->runlist.lock);
- if (!NInoAttr(ni))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- m = map_mft_record(base_ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- ntfs_error(vi->i_sb, "Failed to map mft record for inode 0x%lx "
- "(error code %d).%s", vi->i_ino, err, te);
- ctx = NULL;
- m = NULL;
- goto old_bad_out;
- }
- ctx = ntfs_attr_get_search_ctx(base_ni, m);
- if (unlikely(!ctx)) {
- ntfs_error(vi->i_sb, "Failed to allocate a search context for "
- "inode 0x%lx (not enough memory).%s",
- vi->i_ino, te);
- err = -ENOMEM;
- goto old_bad_out;
- }
- err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- if (err == -ENOENT) {
- ntfs_error(vi->i_sb, "Open attribute is missing from "
- "mft record. Inode 0x%lx is corrupt. "
- "Run chkdsk.%s", vi->i_ino, te);
- err = -EIO;
- } else
- ntfs_error(vi->i_sb, "Failed to lookup attribute in "
- "inode 0x%lx (error code %d).%s",
- vi->i_ino, err, te);
- goto old_bad_out;
- }
- m = ctx->mrec;
- a = ctx->attr;
- /*
- * The i_size of the vfs inode is the new size for the attribute value.
- */
- new_size = i_size_read(vi);
- /* The current size of the attribute value is the old size. */
- old_size = ntfs_attr_size(a);
- /* Calculate the new allocated size. */
- if (NInoNonResident(ni))
- new_alloc_size = (new_size + vol->cluster_size - 1) &
- ~(s64)vol->cluster_size_mask;
- else
- new_alloc_size = (new_size + 7) & ~7;
- /* The current allocated size is the old allocated size. */
- read_lock_irqsave(&ni->size_lock, flags);
- old_alloc_size = ni->allocated_size;
- read_unlock_irqrestore(&ni->size_lock, flags);
- /*
- * The change in the file size. This will be 0 if no change, >0 if the
- * size is growing, and <0 if the size is shrinking.
- */
- size_change = -1;
- if (new_size - old_size >= 0) {
- size_change = 1;
- if (new_size == old_size)
- size_change = 0;
- }
- /* As above for the allocated size. */
- alloc_change = -1;
- if (new_alloc_size - old_alloc_size >= 0) {
- alloc_change = 1;
- if (new_alloc_size == old_alloc_size)
- alloc_change = 0;
- }
- /*
- * If neither the size nor the allocation are being changed there is
- * nothing to do.
- */
- if (!size_change && !alloc_change)
- goto unm_done;
- /* If the size is changing, check if new size is allowed in $AttrDef. */
- if (size_change) {
- err = ntfs_attr_size_bounds_check(vol, ni->type, new_size);
- if (unlikely(err)) {
- if (err == -ERANGE) {
- ntfs_error(vol->sb, "Truncate would cause the "
- "inode 0x%lx to %simum size "
- "for its attribute type "
- "(0x%x). Aborting truncate.",
- vi->i_ino,
- new_size > old_size ? "exceed "
- "the max" : "go under the min",
- le32_to_cpu(ni->type));
- err = -EFBIG;
- } else {
- ntfs_error(vol->sb, "Inode 0x%lx has unknown "
- "attribute type 0x%x. "
- "Aborting truncate.",
- vi->i_ino,
- le32_to_cpu(ni->type));
- err = -EIO;
- }
- /* Reset the vfs inode size to the old size. */
- i_size_write(vi, old_size);
- goto err_out;
- }
- }
- if (NInoCompressed(ni) || NInoEncrypted(ni)) {
- ntfs_warning(vi->i_sb, "Changes in inode size are not "
- "supported yet for %s files, ignoring.",
- NInoCompressed(ni) ? "compressed" :
- "encrypted");
- err = -EOPNOTSUPP;
- goto bad_out;
- }
- if (a->non_resident)
- goto do_non_resident_truncate;
- BUG_ON(NInoNonResident(ni));
- /* Resize the attribute record to best fit the new attribute size. */
- if (new_size < vol->mft_record_size &&
- !ntfs_resident_attr_value_resize(m, a, new_size)) {
- /* The resize succeeded! */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- write_lock_irqsave(&ni->size_lock, flags);
- /* Update the sizes in the ntfs inode and all is done. */
- ni->allocated_size = le32_to_cpu(a->length) -
- le16_to_cpu(a->data.resident.value_offset);
- /*
- * Note ntfs_resident_attr_value_resize() has already done any
- * necessary data clearing in the attribute record. When the
- * file is being shrunk vmtruncate() will already have cleared
- * the top part of the last partial page, i.e. since this is
- * the resident case this is the page with index 0. However,
- * when the file is being expanded, the page cache page data
- * between the old data_size, i.e. old_size, and the new_size
- * has not been zeroed. Fortunately, we do not need to zero it
- * either since on one hand it will either already be zero due
- * to both read_folio and writepage clearing partial page data
- * beyond i_size in which case there is nothing to do or in the
- * case of the file being mmap()ped at the same time, POSIX
- * specifies that the behaviour is unspecified thus we do not
- * have to do anything. This means that in our implementation
- * in the rare case that the file is mmap()ped and a write
- * occurred into the mmap()ped region just beyond the file size
- * and writepage has not yet been called to write out the page
- * (which would clear the area beyond the file size) and we now
- * extend the file size to incorporate this dirty region
- * outside the file size, a write of the page would result in
- * this data being written to disk instead of being cleared.
- * Given both POSIX and the Linux mmap(2) man page specify that
- * this corner case is undefined, we choose to leave it like
- * that as this is much simpler for us as we cannot lock the
- * relevant page now since we are holding too many ntfs locks
- * which would result in a lock reversal deadlock.
- */
- ni->initialized_size = new_size;
- write_unlock_irqrestore(&ni->size_lock, flags);
- goto unm_done;
- }
- /* If the above resize failed, this must be an attribute extension. */
- BUG_ON(size_change < 0);
- /*
- * We have to drop all the locks so we can call
- * ntfs_attr_make_non_resident(). This could be optimised by try-
- * locking the first page cache page and only if that fails dropping
- * the locks, locking the page, and redoing all the locking and
- * lookups. While this would be a huge optimisation, it is not worth
- * it as this is definitely a slow code path as it only ever can happen
- * once for any given file.
- */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
- /*
- * Not enough space in the mft record, try to make the attribute
- * non-resident and if successful restart the truncation process.
- */
- err = ntfs_attr_make_non_resident(ni, old_size);
- if (likely(!err))
- goto retry_truncate;
- /*
- * Could not make non-resident. If this is due to this not being
- * permitted for this attribute type or there not being enough space,
- * try to make other attributes non-resident. Otherwise fail.
- */
- if (unlikely(err != -EPERM && err != -ENOSPC)) {
- ntfs_error(vol->sb, "Cannot truncate inode 0x%lx, attribute "
- "type 0x%x, because the conversion from "
- "resident to non-resident attribute failed "
- "with error code %i.", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), err);
- if (err != -ENOMEM)
- err = -EIO;
- goto conv_err_out;
- }
- /* TODO: Not implemented from here, abort. */
- if (err == -ENOSPC)
- ntfs_error(vol->sb, "Not enough space in the mft record/on "
- "disk for the non-resident attribute value. "
- "This case is not implemented yet.");
- else /* if (err == -EPERM) */
- ntfs_error(vol->sb, "This attribute type may not be "
- "non-resident. This case is not implemented "
- "yet.");
- err = -EOPNOTSUPP;
- goto conv_err_out;
-#if 0
- // TODO: Attempt to make other attributes non-resident.
- if (!err)
- goto do_resident_extend;
- /*
- * Both the attribute list attribute and the standard information
- * attribute must remain in the base inode. Thus, if this is one of
- * these attributes, we have to try to move other attributes out into
- * extent mft records instead.
- */
- if (ni->type == AT_ATTRIBUTE_LIST ||
- ni->type == AT_STANDARD_INFORMATION) {
- // TODO: Attempt to move other attributes into extent mft
- // records.
- err = -EOPNOTSUPP;
- if (!err)
- goto do_resident_extend;
- goto err_out;
- }
- // TODO: Attempt to move this attribute to an extent mft record, but
- // only if it is not already the only attribute in an mft record in
- // which case there would be nothing to gain.
- err = -EOPNOTSUPP;
- if (!err)
- goto do_resident_extend;
- /* There is nothing we can do to make enough space. )-: */
- goto err_out;
-#endif
-do_non_resident_truncate:
- BUG_ON(!NInoNonResident(ni));
- if (alloc_change < 0) {
- highest_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn);
- if (highest_vcn > 0 &&
- old_alloc_size >> vol->cluster_size_bits >
- highest_vcn + 1) {
- /*
- * This attribute has multiple extents. Not yet
- * supported.
- */
- ntfs_error(vol->sb, "Cannot truncate inode 0x%lx, "
- "attribute type 0x%x, because the "
- "attribute is highly fragmented (it "
- "consists of multiple extents) and "
- "this case is not implemented yet.",
- vi->i_ino,
- (unsigned)le32_to_cpu(ni->type));
- err = -EOPNOTSUPP;
- goto bad_out;
- }
- }
- /*
- * If the size is shrinking, need to reduce the initialized_size and
- * the data_size before reducing the allocation.
- */
- if (size_change < 0) {
- /*
- * Make the valid size smaller (i_size is already up-to-date).
- */
- write_lock_irqsave(&ni->size_lock, flags);
- if (new_size < ni->initialized_size) {
- ni->initialized_size = new_size;
- a->data.non_resident.initialized_size =
- cpu_to_sle64(new_size);
- }
- a->data.non_resident.data_size = cpu_to_sle64(new_size);
- write_unlock_irqrestore(&ni->size_lock, flags);
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- /* If the allocated size is not changing, we are done. */
- if (!alloc_change)
- goto unm_done;
- /*
- * If the size is shrinking it makes no sense for the
- * allocation to be growing.
- */
- BUG_ON(alloc_change > 0);
- } else /* if (size_change >= 0) */ {
- /*
- * The file size is growing or staying the same but the
- * allocation can be shrinking, growing or staying the same.
- */
- if (alloc_change > 0) {
- /*
- * We need to extend the allocation and possibly update
- * the data size. If we are updating the data size,
- * since we are not touching the initialized_size we do
- * not need to worry about the actual data on disk.
- * And as far as the page cache is concerned, there
- * will be no pages beyond the old data size and any
- * partial region in the last page between the old and
- * new data size (or the end of the page if the new
- * data size is outside the page) does not need to be
- * modified as explained above for the resident
- * attribute truncate case. To do this, we simply drop
- * the locks we hold and leave all the work to our
- * friendly helper ntfs_attr_extend_allocation().
- */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
- err = ntfs_attr_extend_allocation(ni, new_size,
- size_change > 0 ? new_size : -1, -1);
- /*
- * ntfs_attr_extend_allocation() will have done error
- * output already.
- */
- goto done;
- }
- if (!alloc_change)
- goto alloc_done;
- }
- /* alloc_change < 0 */
- /* Free the clusters. */
- nr_freed = ntfs_cluster_free(ni, new_alloc_size >>
- vol->cluster_size_bits, -1, ctx);
- m = ctx->mrec;
- a = ctx->attr;
- if (unlikely(nr_freed < 0)) {
- ntfs_error(vol->sb, "Failed to release cluster(s) (error code "
- "%lli). Unmount and run chkdsk to recover "
- "the lost cluster(s).", (long long)nr_freed);
- NVolSetErrors(vol);
- nr_freed = 0;
- }
- /* Truncate the runlist. */
- err = ntfs_rl_truncate_nolock(vol, &ni->runlist,
- new_alloc_size >> vol->cluster_size_bits);
- /*
- * If the runlist truncation failed and/or the search context is no
- * longer valid, we cannot resize the attribute record or build the
- * mapping pairs array thus we mark the inode bad so that no access to
- * the freed clusters can happen.
- */
- if (unlikely(err || IS_ERR(m))) {
- ntfs_error(vol->sb, "Failed to %s (error code %li).%s",
- IS_ERR(m) ?
- "restore attribute search context" :
- "truncate attribute runlist",
- IS_ERR(m) ? PTR_ERR(m) : err, es);
- err = -EIO;
- goto bad_out;
- }
- /* Get the size for the shrunk mapping pairs array for the runlist. */
- mp_size = ntfs_get_size_for_mapping_pairs(vol, ni->runlist.rl, 0, -1);
- if (unlikely(mp_size <= 0)) {
- ntfs_error(vol->sb, "Cannot shrink allocation of inode 0x%lx, "
- "attribute type 0x%x, because determining the "
- "size for the mapping pairs failed with error "
- "code %i.%s", vi->i_ino,
- (unsigned)le32_to_cpu(ni->type), mp_size, es);
- err = -EIO;
- goto bad_out;
- }
- /*
- * Shrink the attribute record for the new mapping pairs array. Note,
- * this cannot fail since we are making the attribute smaller thus by
- * definition there is enough space to do so.
- */
- err = ntfs_attr_record_resize(m, a, mp_size +
- le16_to_cpu(a->data.non_resident.mapping_pairs_offset));
- BUG_ON(err);
- /*
- * Generate the mapping pairs array directly into the attribute record.
- */
- err = ntfs_mapping_pairs_build(vol, (u8*)a +
- le16_to_cpu(a->data.non_resident.mapping_pairs_offset),
- mp_size, ni->runlist.rl, 0, -1, NULL);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Cannot shrink allocation of inode 0x%lx, "
- "attribute type 0x%x, because building the "
- "mapping pairs failed with error code %i.%s",
- vi->i_ino, (unsigned)le32_to_cpu(ni->type),
- err, es);
- err = -EIO;
- goto bad_out;
- }
- /* Update the allocated/compressed size as well as the highest vcn. */
- a->data.non_resident.highest_vcn = cpu_to_sle64((new_alloc_size >>
- vol->cluster_size_bits) - 1);
- write_lock_irqsave(&ni->size_lock, flags);
- ni->allocated_size = new_alloc_size;
- a->data.non_resident.allocated_size = cpu_to_sle64(new_alloc_size);
- if (NInoSparse(ni) || NInoCompressed(ni)) {
- if (nr_freed) {
- ni->itype.compressed.size -= nr_freed <<
- vol->cluster_size_bits;
- BUG_ON(ni->itype.compressed.size < 0);
- a->data.non_resident.compressed_size = cpu_to_sle64(
- ni->itype.compressed.size);
- vi->i_blocks = ni->itype.compressed.size >> 9;
- }
- } else
- vi->i_blocks = new_alloc_size >> 9;
- write_unlock_irqrestore(&ni->size_lock, flags);
- /*
- * We have shrunk the allocation. If this is a shrinking truncate we
- * have already dealt with the initialized_size and the data_size above
- * and we are done. If the truncate is only changing the allocation
- * and not the data_size, we are also done. If this is an extending
- * truncate, need to extend the data_size now which is ensured by the
- * fact that @size_change is positive.
- */
-alloc_done:
- /*
- * If the size is growing, need to update it now. If it is shrinking,
- * we have already updated it above (before the allocation change).
- */
- if (size_change > 0)
- a->data.non_resident.data_size = cpu_to_sle64(new_size);
- /* Ensure the modified mft record is written out. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
-unm_done:
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
-done:
- /* Update the mtime and ctime on the base inode. */
- /* normally ->truncate shouldn't update ctime or mtime,
- * but ntfs did before so it got a copy & paste version
- * of file_update_time. one day someone should fix this
- * for real.
- */
- if (!IS_NOCMTIME(VFS_I(base_ni)) && !IS_RDONLY(VFS_I(base_ni))) {
- struct timespec64 now = current_time(VFS_I(base_ni));
- int sync_it = 0;
-
- if (!timespec64_equal(&VFS_I(base_ni)->i_mtime, &now) ||
- !timespec64_equal(&VFS_I(base_ni)->i_ctime, &now))
- sync_it = 1;
- VFS_I(base_ni)->i_mtime = now;
- VFS_I(base_ni)->i_ctime = now;
-
- if (sync_it)
- mark_inode_dirty_sync(VFS_I(base_ni));
- }
-
- if (likely(!err)) {
- NInoClearTruncateFailed(ni);
- ntfs_debug("Done.");
- }
- return err;
-old_bad_out:
- old_size = -1;
-bad_out:
- if (err != -ENOMEM && err != -EOPNOTSUPP)
- NVolSetErrors(vol);
- if (err != -EOPNOTSUPP)
- NInoSetTruncateFailed(ni);
- else if (old_size >= 0)
- i_size_write(vi, old_size);
-err_out:
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(base_ni);
- up_write(&ni->runlist.lock);
-out:
- ntfs_debug("Failed. Returning error code %i.", err);
- return err;
-conv_err_out:
- if (err != -ENOMEM && err != -EOPNOTSUPP)
- NVolSetErrors(vol);
- if (err != -EOPNOTSUPP)
- NInoSetTruncateFailed(ni);
- else
- i_size_write(vi, old_size);
- goto out;
-}
-
-/**
- * ntfs_truncate_vfs - wrapper for ntfs_truncate() that has no return value
- * @vi: inode for which the i_size was changed
- *
- * Wrapper for ntfs_truncate() that has no return value.
- *
- * See ntfs_truncate() description above for details.
- */
-#ifdef NTFS_RW
-void ntfs_truncate_vfs(struct inode *vi) {
- ntfs_truncate(vi);
-}
-#endif
-
-/**
- * ntfs_setattr - called from notify_change() when an attribute is being changed
- * @mnt_userns: user namespace of the mount the inode was found from
- * @dentry: dentry whose attributes to change
- * @attr: structure describing the attributes and the changes
- *
- * We have to trap VFS attempts to truncate the file described by @dentry as
- * soon as possible, because we do not implement changes in i_size yet. So we
- * abort all i_size changes here.
- *
- * We also abort all changes of user, group, and mode as we do not implement
- * the NTFS ACLs yet.
- *
- * Called with ->i_mutex held.
- */
-int ntfs_setattr(struct user_namespace *mnt_userns, struct dentry *dentry,
- struct iattr *attr)
-{
- struct inode *vi = d_inode(dentry);
- int err;
- unsigned int ia_valid = attr->ia_valid;
-
- err = setattr_prepare(&init_user_ns, dentry, attr);
- if (err)
- goto out;
- /* We do not support NTFS ACLs yet. */
- if (ia_valid & (ATTR_UID | ATTR_GID | ATTR_MODE)) {
- ntfs_warning(vi->i_sb, "Changes in user/group/mode are not "
- "supported yet, ignoring.");
- err = -EOPNOTSUPP;
- goto out;
- }
- if (ia_valid & ATTR_SIZE) {
- if (attr->ia_size != i_size_read(vi)) {
- ntfs_inode *ni = NTFS_I(vi);
- /*
- * FIXME: For now we do not support resizing of
- * compressed or encrypted files yet.
- */
- if (NInoCompressed(ni) || NInoEncrypted(ni)) {
- ntfs_warning(vi->i_sb, "Changes in inode size "
- "are not supported yet for "
- "%s files, ignoring.",
- NInoCompressed(ni) ?
- "compressed" : "encrypted");
- err = -EOPNOTSUPP;
- } else {
- truncate_setsize(vi, attr->ia_size);
- ntfs_truncate_vfs(vi);
- }
- if (err || ia_valid == ATTR_SIZE)
- goto out;
- } else {
- /*
- * We skipped the truncate but must still update
- * timestamps.
- */
- ia_valid |= ATTR_MTIME | ATTR_CTIME;
- }
- }
- if (ia_valid & ATTR_ATIME)
- vi->i_atime = attr->ia_atime;
- if (ia_valid & ATTR_MTIME)
- vi->i_mtime = attr->ia_mtime;
- if (ia_valid & ATTR_CTIME)
- vi->i_ctime = attr->ia_ctime;
- mark_inode_dirty(vi);
-out:
- return err;
-}
-
-/**
- * ntfs_write_inode - write out a dirty inode
- * @vi: inode to write out
- * @sync: if true, write out synchronously
- *
- * Write out a dirty inode to disk including any extent inodes if present.
- *
- * If @sync is true, commit the inode to disk and wait for io completion. This
- * is done using write_mft_record().
- *
- * If @sync is false, just schedule the write to happen but do not wait for i/o
- * completion. In 2.6 kernels, scheduling usually happens just by virtue of
- * marking the page (and in this case mft record) dirty but we do not implement
- * this yet as write_mft_record() largely ignores the @sync parameter and
- * always performs synchronous writes.
- *
- * Return 0 on success and -errno on error.
- */
-int __ntfs_write_inode(struct inode *vi, int sync)
-{
- sle64 nt;
- ntfs_inode *ni = NTFS_I(vi);
- ntfs_attr_search_ctx *ctx;
- MFT_RECORD *m;
- STANDARD_INFORMATION *si;
- int err = 0;
- bool modified = false;
-
- ntfs_debug("Entering for %sinode 0x%lx.", NInoAttr(ni) ? "attr " : "",
- vi->i_ino);
- /*
- * Dirty attribute inodes are written via their real inodes so just
- * clean them here. Access time updates are taken care off when the
- * real inode is written.
- */
- if (NInoAttr(ni)) {
- NInoClearDirty(ni);
- ntfs_debug("Done.");
- return 0;
- }
- /* Map, pin, and lock the mft record belonging to the inode. */
- m = map_mft_record(ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- goto err_out;
- }
- /* Update the access times in the standard information attribute. */
- ctx = ntfs_attr_get_search_ctx(ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto unm_err_out;
- }
- err = ntfs_attr_lookup(AT_STANDARD_INFORMATION, NULL, 0,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- ntfs_attr_put_search_ctx(ctx);
- goto unm_err_out;
- }
- si = (STANDARD_INFORMATION*)((u8*)ctx->attr +
- le16_to_cpu(ctx->attr->data.resident.value_offset));
- /* Update the access times if they have changed. */
- nt = utc2ntfs(vi->i_mtime);
- if (si->last_data_change_time != nt) {
- ntfs_debug("Updating mtime for inode 0x%lx: old = 0x%llx, "
- "new = 0x%llx", vi->i_ino, (long long)
- sle64_to_cpu(si->last_data_change_time),
- (long long)sle64_to_cpu(nt));
- si->last_data_change_time = nt;
- modified = true;
- }
- nt = utc2ntfs(vi->i_ctime);
- if (si->last_mft_change_time != nt) {
- ntfs_debug("Updating ctime for inode 0x%lx: old = 0x%llx, "
- "new = 0x%llx", vi->i_ino, (long long)
- sle64_to_cpu(si->last_mft_change_time),
- (long long)sle64_to_cpu(nt));
- si->last_mft_change_time = nt;
- modified = true;
- }
- nt = utc2ntfs(vi->i_atime);
- if (si->last_access_time != nt) {
- ntfs_debug("Updating atime for inode 0x%lx: old = 0x%llx, "
- "new = 0x%llx", vi->i_ino,
- (long long)sle64_to_cpu(si->last_access_time),
- (long long)sle64_to_cpu(nt));
- si->last_access_time = nt;
- modified = true;
- }
- /*
- * If we just modified the standard information attribute we need to
- * mark the mft record it is in dirty. We do this manually so that
- * mark_inode_dirty() is not called which would redirty the inode and
- * hence result in an infinite loop of trying to write the inode.
- * There is no need to mark the base inode nor the base mft record
- * dirty, since we are going to write this mft record below in any case
- * and the base mft record may actually not have been modified so it
- * might not need to be written out.
- * NOTE: It is not a problem when the inode for $MFT itself is being
- * written out as mark_ntfs_record_dirty() will only set I_DIRTY_PAGES
- * on the $MFT inode and hence ntfs_write_inode() will not be
- * re-invoked because of it which in turn is ok since the dirtied mft
- * record will be cleaned and written out to disk below, i.e. before
- * this function returns.
- */
- if (modified) {
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- if (!NInoTestSetDirty(ctx->ntfs_ino))
- mark_ntfs_record_dirty(ctx->ntfs_ino->page,
- ctx->ntfs_ino->page_ofs);
- }
- ntfs_attr_put_search_ctx(ctx);
- /* Now the access times are updated, write the base mft record. */
- if (NInoDirty(ni))
- err = write_mft_record(ni, m, sync);
- /* Write all attached extent mft records. */
- mutex_lock(&ni->extent_lock);
- if (ni->nr_extents > 0) {
- ntfs_inode **extent_nis = ni->ext.extent_ntfs_inos;
- int i;
-
- ntfs_debug("Writing %i extent inodes.", ni->nr_extents);
- for (i = 0; i < ni->nr_extents; i++) {
- ntfs_inode *tni = extent_nis[i];
-
- if (NInoDirty(tni)) {
- MFT_RECORD *tm = map_mft_record(tni);
- int ret;
-
- if (IS_ERR(tm)) {
- if (!err || err == -ENOMEM)
- err = PTR_ERR(tm);
- continue;
- }
- ret = write_mft_record(tni, tm, sync);
- unmap_mft_record(tni);
- if (unlikely(ret)) {
- if (!err || err == -ENOMEM)
- err = ret;
- }
- }
- }
- }
- mutex_unlock(&ni->extent_lock);
- unmap_mft_record(ni);
- if (unlikely(err))
- goto err_out;
- ntfs_debug("Done.");
- return 0;
-unm_err_out:
- unmap_mft_record(ni);
-err_out:
- if (err == -ENOMEM) {
- ntfs_warning(vi->i_sb, "Not enough memory to write inode. "
- "Marking the inode dirty again, so the VFS "
- "retries later.");
- mark_inode_dirty(vi);
- } else {
- ntfs_error(vi->i_sb, "Failed (error %i): Run chkdsk.", -err);
- NVolSetErrors(ni->vol);
- }
- return err;
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/inode.h b/fs/ntfs/inode.h
deleted file mode 100644
index 6f78ee00f57f..000000000000
--- a/fs/ntfs/inode.h
+++ /dev/null
@@ -1,310 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * inode.h - Defines for inode structures NTFS Linux kernel driver. Part of
- * the Linux-NTFS project.
- *
- * Copyright (c) 2001-2007 Anton Altaparmakov
- * Copyright (c) 2002 Richard Russon
- */
-
-#ifndef _LINUX_NTFS_INODE_H
-#define _LINUX_NTFS_INODE_H
-
-#include <linux/atomic.h>
-
-#include <linux/fs.h>
-#include <linux/list.h>
-#include <linux/mm.h>
-#include <linux/mutex.h>
-#include <linux/seq_file.h>
-
-#include "layout.h"
-#include "volume.h"
-#include "types.h"
-#include "runlist.h"
-#include "debug.h"
-
-typedef struct _ntfs_inode ntfs_inode;
-
-/*
- * The NTFS in-memory inode structure. It is just used as an extension to the
- * fields already provided in the VFS inode.
- */
-struct _ntfs_inode {
- rwlock_t size_lock; /* Lock serializing access to inode sizes. */
- s64 initialized_size; /* Copy from the attribute record. */
- s64 allocated_size; /* Copy from the attribute record. */
- unsigned long state; /* NTFS specific flags describing this inode.
- See ntfs_inode_state_bits below. */
- unsigned long mft_no; /* Number of the mft record / inode. */
- u16 seq_no; /* Sequence number of the mft record. */
- atomic_t count; /* Inode reference count for book keeping. */
- ntfs_volume *vol; /* Pointer to the ntfs volume of this inode. */
- /*
- * If NInoAttr() is true, the below fields describe the attribute which
- * this fake inode belongs to. The actual inode of this attribute is
- * pointed to by base_ntfs_ino and nr_extents is always set to -1 (see
- * below). For real inodes, we also set the type (AT_DATA for files and
- * AT_INDEX_ALLOCATION for directories), with the name = NULL and
- * name_len = 0 for files and name = I30 (global constant) and
- * name_len = 4 for directories.
- */
- ATTR_TYPE type; /* Attribute type of this fake inode. */
- ntfschar *name; /* Attribute name of this fake inode. */
- u32 name_len; /* Attribute name length of this fake inode. */
- runlist runlist; /* If state has the NI_NonResident bit set,
- the runlist of the unnamed data attribute
- (if a file) or of the index allocation
- attribute (directory) or of the attribute
- described by the fake inode (if NInoAttr()).
- If runlist.rl is NULL, the runlist has not
- been read in yet or has been unmapped. If
- NI_NonResident is clear, the attribute is
- resident (file and fake inode) or there is
- no $I30 index allocation attribute
- (small directory). In the latter case
- runlist.rl is always NULL.*/
- /*
- * The following fields are only valid for real inodes and extent
- * inodes.
- */
- struct mutex mrec_lock; /* Lock for serializing access to the
- mft record belonging to this inode. */
- struct page *page; /* The page containing the mft record of the
- inode. This should only be touched by the
- (un)map_mft_record*() functions. */
- int page_ofs; /* Offset into the page at which the mft record
- begins. This should only be touched by the
- (un)map_mft_record*() functions. */
- /*
- * Attribute list support (only for use by the attribute lookup
- * functions). Setup during read_inode for all inodes with attribute
- * lists. Only valid if NI_AttrList is set in state, and attr_list_rl is
- * further only valid if NI_AttrListNonResident is set.
- */
- u32 attr_list_size; /* Length of attribute list value in bytes. */
- u8 *attr_list; /* Attribute list value itself. */
- runlist attr_list_rl; /* Run list for the attribute list value. */
- union {
- struct { /* It is a directory, $MFT, or an index inode. */
- u32 block_size; /* Size of an index block. */
- u32 vcn_size; /* Size of a vcn in this
- index. */
- COLLATION_RULE collation_rule; /* The collation rule
- for the index. */
- u8 block_size_bits; /* Log2 of the above. */
- u8 vcn_size_bits; /* Log2 of the above. */
- } index;
- struct { /* It is a compressed/sparse file/attribute inode. */
- s64 size; /* Copy of compressed_size from
- $DATA. */
- u32 block_size; /* Size of a compression block
- (cb). */
- u8 block_size_bits; /* Log2 of the size of a cb. */
- u8 block_clusters; /* Number of clusters per cb. */
- } compressed;
- } itype;
- struct mutex extent_lock; /* Lock for accessing/modifying the
- below . */
- s32 nr_extents; /* For a base mft record, the number of attached extent
- inodes (0 if none), for extent records and for fake
- inodes describing an attribute this is -1. */
- union { /* This union is only used if nr_extents != 0. */
- ntfs_inode **extent_ntfs_inos; /* For nr_extents > 0, array of
- the ntfs inodes of the extent
- mft records belonging to
- this base inode which have
- been loaded. */
- ntfs_inode *base_ntfs_ino; /* For nr_extents == -1, the
- ntfs inode of the base mft
- record. For fake inodes, the
- real (base) inode to which
- the attribute belongs. */
- } ext;
-};
-
-/*
- * Defined bits for the state field in the ntfs_inode structure.
- * (f) = files only, (d) = directories only, (a) = attributes/fake inodes only
- */
-typedef enum {
- NI_Dirty, /* 1: Mft record needs to be written to disk. */
- NI_AttrList, /* 1: Mft record contains an attribute list. */
- NI_AttrListNonResident, /* 1: Attribute list is non-resident. Implies
- NI_AttrList is set. */
-
- NI_Attr, /* 1: Fake inode for attribute i/o.
- 0: Real inode or extent inode. */
-
- NI_MstProtected, /* 1: Attribute is protected by MST fixups.
- 0: Attribute is not protected by fixups. */
- NI_NonResident, /* 1: Unnamed data attr is non-resident (f).
- 1: Attribute is non-resident (a). */
- NI_IndexAllocPresent = NI_NonResident, /* 1: $I30 index alloc attr is
- present (d). */
- NI_Compressed, /* 1: Unnamed data attr is compressed (f).
- 1: Create compressed files by default (d).
- 1: Attribute is compressed (a). */
- NI_Encrypted, /* 1: Unnamed data attr is encrypted (f).
- 1: Create encrypted files by default (d).
- 1: Attribute is encrypted (a). */
- NI_Sparse, /* 1: Unnamed data attr is sparse (f).
- 1: Create sparse files by default (d).
- 1: Attribute is sparse (a). */
- NI_SparseDisabled, /* 1: May not create sparse regions. */
- NI_TruncateFailed, /* 1: Last ntfs_truncate() call failed. */
-} ntfs_inode_state_bits;
-
-/*
- * NOTE: We should be adding dirty mft records to a list somewhere and they
- * should be independent of the (ntfs/vfs) inode structure so that an inode can
- * be removed but the record can be left dirty for syncing later.
- */
-
-/*
- * Macro tricks to expand the NInoFoo(), NInoSetFoo(), and NInoClearFoo()
- * functions.
- */
-#define NINO_FNS(flag) \
-static inline int NIno##flag(ntfs_inode *ni) \
-{ \
- return test_bit(NI_##flag, &(ni)->state); \
-} \
-static inline void NInoSet##flag(ntfs_inode *ni) \
-{ \
- set_bit(NI_##flag, &(ni)->state); \
-} \
-static inline void NInoClear##flag(ntfs_inode *ni) \
-{ \
- clear_bit(NI_##flag, &(ni)->state); \
-}
-
-/*
- * As above for NInoTestSetFoo() and NInoTestClearFoo().
- */
-#define TAS_NINO_FNS(flag) \
-static inline int NInoTestSet##flag(ntfs_inode *ni) \
-{ \
- return test_and_set_bit(NI_##flag, &(ni)->state); \
-} \
-static inline int NInoTestClear##flag(ntfs_inode *ni) \
-{ \
- return test_and_clear_bit(NI_##flag, &(ni)->state); \
-}
-
-/* Emit the ntfs inode bitops functions. */
-NINO_FNS(Dirty)
-TAS_NINO_FNS(Dirty)
-NINO_FNS(AttrList)
-NINO_FNS(AttrListNonResident)
-NINO_FNS(Attr)
-NINO_FNS(MstProtected)
-NINO_FNS(NonResident)
-NINO_FNS(IndexAllocPresent)
-NINO_FNS(Compressed)
-NINO_FNS(Encrypted)
-NINO_FNS(Sparse)
-NINO_FNS(SparseDisabled)
-NINO_FNS(TruncateFailed)
-
-/*
- * The full structure containing a ntfs_inode and a vfs struct inode. Used for
- * all real and fake inodes but not for extent inodes which lack the vfs struct
- * inode.
- */
-typedef struct {
- ntfs_inode ntfs_inode;
- struct inode vfs_inode; /* The vfs inode structure. */
-} big_ntfs_inode;
-
-/**
- * NTFS_I - return the ntfs inode given a vfs inode
- * @inode: VFS inode
- *
- * NTFS_I() returns the ntfs inode associated with the VFS @inode.
- */
-static inline ntfs_inode *NTFS_I(struct inode *inode)
-{
- return (ntfs_inode *)container_of(inode, big_ntfs_inode, vfs_inode);
-}
-
-static inline struct inode *VFS_I(ntfs_inode *ni)
-{
- return &((big_ntfs_inode *)ni)->vfs_inode;
-}
-
-/**
- * ntfs_attr - ntfs in memory attribute structure
- * @mft_no: mft record number of the base mft record of this attribute
- * @name: Unicode name of the attribute (NULL if unnamed)
- * @name_len: length of @name in Unicode characters (0 if unnamed)
- * @type: attribute type (see layout.h)
- *
- * This structure exists only to provide a small structure for the
- * ntfs_{attr_}iget()/ntfs_test_inode()/ntfs_init_locked_inode() mechanism.
- *
- * NOTE: Elements are ordered by size to make the structure as compact as
- * possible on all architectures.
- */
-typedef struct {
- unsigned long mft_no;
- ntfschar *name;
- u32 name_len;
- ATTR_TYPE type;
-} ntfs_attr;
-
-extern int ntfs_test_inode(struct inode *vi, void *data);
-
-extern struct inode *ntfs_iget(struct super_block *sb, unsigned long mft_no);
-extern struct inode *ntfs_attr_iget(struct inode *base_vi, ATTR_TYPE type,
- ntfschar *name, u32 name_len);
-extern struct inode *ntfs_index_iget(struct inode *base_vi, ntfschar *name,
- u32 name_len);
-
-extern struct inode *ntfs_alloc_big_inode(struct super_block *sb);
-extern void ntfs_free_big_inode(struct inode *inode);
-extern void ntfs_evict_big_inode(struct inode *vi);
-
-extern void __ntfs_init_inode(struct super_block *sb, ntfs_inode *ni);
-
-static inline void ntfs_init_big_inode(struct inode *vi)
-{
- ntfs_inode *ni = NTFS_I(vi);
-
- ntfs_debug("Entering.");
- __ntfs_init_inode(vi->i_sb, ni);
- ni->mft_no = vi->i_ino;
-}
-
-extern ntfs_inode *ntfs_new_extent_inode(struct super_block *sb,
- unsigned long mft_no);
-extern void ntfs_clear_extent_inode(ntfs_inode *ni);
-
-extern int ntfs_read_inode_mount(struct inode *vi);
-
-extern int ntfs_show_options(struct seq_file *sf, struct dentry *root);
-
-#ifdef NTFS_RW
-
-extern int ntfs_truncate(struct inode *vi);
-extern void ntfs_truncate_vfs(struct inode *vi);
-
-extern int ntfs_setattr(struct user_namespace *mnt_userns,
- struct dentry *dentry, struct iattr *attr);
-
-extern int __ntfs_write_inode(struct inode *vi, int sync);
-
-static inline void ntfs_commit_inode(struct inode *vi)
-{
- if (!is_bad_inode(vi))
- __ntfs_write_inode(vi, 1);
- return;
-}
-
-#else
-
-static inline void ntfs_truncate_vfs(struct inode *vi) {}
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_INODE_H */
diff --git a/fs/ntfs/layout.h b/fs/ntfs/layout.h
deleted file mode 100644
index 5d4bf7a3259f..000000000000
--- a/fs/ntfs/layout.h
+++ /dev/null
@@ -1,2421 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * layout.h - All NTFS associated on-disk structures. Part of the Linux-NTFS
- * project.
- *
- * Copyright (c) 2001-2005 Anton Altaparmakov
- * Copyright (c) 2002 Richard Russon
- */
-
-#ifndef _LINUX_NTFS_LAYOUT_H
-#define _LINUX_NTFS_LAYOUT_H
-
-#include <linux/types.h>
-#include <linux/bitops.h>
-#include <linux/list.h>
-#include <asm/byteorder.h>
-
-#include "types.h"
-
-/* The NTFS oem_id "NTFS " */
-#define magicNTFS cpu_to_le64(0x202020205346544eULL)
-
-/*
- * Location of bootsector on partition:
- * The standard NTFS_BOOT_SECTOR is on sector 0 of the partition.
- * On NT4 and above there is one backup copy of the boot sector to
- * be found on the last sector of the partition (not normally accessible
- * from within Windows as the bootsector contained number of sectors
- * value is one less than the actual value!).
- * On versions of NT 3.51 and earlier, the backup copy was located at
- * number of sectors/2 (integer divide), i.e. in the middle of the volume.
- */
-
-/*
- * BIOS parameter block (bpb) structure.
- */
-typedef struct {
- le16 bytes_per_sector; /* Size of a sector in bytes. */
- u8 sectors_per_cluster; /* Size of a cluster in sectors. */
- le16 reserved_sectors; /* zero */
- u8 fats; /* zero */
- le16 root_entries; /* zero */
- le16 sectors; /* zero */
- u8 media_type; /* 0xf8 = hard disk */
- le16 sectors_per_fat; /* zero */
- le16 sectors_per_track; /* irrelevant */
- le16 heads; /* irrelevant */
- le32 hidden_sectors; /* zero */
- le32 large_sectors; /* zero */
-} __attribute__ ((__packed__)) BIOS_PARAMETER_BLOCK;
-
-/*
- * NTFS boot sector structure.
- */
-typedef struct {
- u8 jump[3]; /* Irrelevant (jump to boot up code).*/
- le64 oem_id; /* Magic "NTFS ". */
- BIOS_PARAMETER_BLOCK bpb; /* See BIOS_PARAMETER_BLOCK. */
- u8 unused[4]; /* zero, NTFS diskedit.exe states that
- this is actually:
- __u8 physical_drive; // 0x80
- __u8 current_head; // zero
- __u8 extended_boot_signature;
- // 0x80
- __u8 unused; // zero
- */
-/*0x28*/sle64 number_of_sectors; /* Number of sectors in volume. Gives
- maximum volume size of 2^63 sectors.
- Assuming standard sector size of 512
- bytes, the maximum byte size is
- approx. 4.7x10^21 bytes. (-; */
- sle64 mft_lcn; /* Cluster location of mft data. */
- sle64 mftmirr_lcn; /* Cluster location of copy of mft. */
- s8 clusters_per_mft_record; /* Mft record size in clusters. */
- u8 reserved0[3]; /* zero */
- s8 clusters_per_index_record; /* Index block size in clusters. */
- u8 reserved1[3]; /* zero */
- le64 volume_serial_number; /* Irrelevant (serial number). */
- le32 checksum; /* Boot sector checksum. */
-/*0x54*/u8 bootstrap[426]; /* Irrelevant (boot up code). */
- le16 end_of_sector_marker; /* End of bootsector magic. Always is
- 0xaa55 in little endian. */
-/* sizeof() = 512 (0x200) bytes */
-} __attribute__ ((__packed__)) NTFS_BOOT_SECTOR;
-
-/*
- * Magic identifiers present at the beginning of all ntfs record containing
- * records (like mft records for example).
- */
-enum {
- /* Found in $MFT/$DATA. */
- magic_FILE = cpu_to_le32(0x454c4946), /* Mft entry. */
- magic_INDX = cpu_to_le32(0x58444e49), /* Index buffer. */
- magic_HOLE = cpu_to_le32(0x454c4f48), /* ? (NTFS 3.0+?) */
-
- /* Found in $LogFile/$DATA. */
- magic_RSTR = cpu_to_le32(0x52545352), /* Restart page. */
- magic_RCRD = cpu_to_le32(0x44524352), /* Log record page. */
-
- /* Found in $LogFile/$DATA. (May be found in $MFT/$DATA, also?) */
- magic_CHKD = cpu_to_le32(0x444b4843), /* Modified by chkdsk. */
-
- /* Found in all ntfs record containing records. */
- magic_BAAD = cpu_to_le32(0x44414142), /* Failed multi sector
- transfer was detected. */
- /*
- * Found in $LogFile/$DATA when a page is full of 0xff bytes and is
- * thus not initialized. Page must be initialized before using it.
- */
- magic_empty = cpu_to_le32(0xffffffff) /* Record is empty. */
-};
-
-typedef le32 NTFS_RECORD_TYPE;
-
-/*
- * Generic magic comparison macros. Finally found a use for the ## preprocessor
- * operator! (-8
- */
-
-static inline bool __ntfs_is_magic(le32 x, NTFS_RECORD_TYPE r)
-{
- return (x == r);
-}
-#define ntfs_is_magic(x, m) __ntfs_is_magic(x, magic_##m)
-
-static inline bool __ntfs_is_magicp(le32 *p, NTFS_RECORD_TYPE r)
-{
- return (*p == r);
-}
-#define ntfs_is_magicp(p, m) __ntfs_is_magicp(p, magic_##m)
-
-/*
- * Specialised magic comparison macros for the NTFS_RECORD_TYPEs defined above.
- */
-#define ntfs_is_file_record(x) ( ntfs_is_magic (x, FILE) )
-#define ntfs_is_file_recordp(p) ( ntfs_is_magicp(p, FILE) )
-#define ntfs_is_mft_record(x) ( ntfs_is_file_record (x) )
-#define ntfs_is_mft_recordp(p) ( ntfs_is_file_recordp(p) )
-#define ntfs_is_indx_record(x) ( ntfs_is_magic (x, INDX) )
-#define ntfs_is_indx_recordp(p) ( ntfs_is_magicp(p, INDX) )
-#define ntfs_is_hole_record(x) ( ntfs_is_magic (x, HOLE) )
-#define ntfs_is_hole_recordp(p) ( ntfs_is_magicp(p, HOLE) )
-
-#define ntfs_is_rstr_record(x) ( ntfs_is_magic (x, RSTR) )
-#define ntfs_is_rstr_recordp(p) ( ntfs_is_magicp(p, RSTR) )
-#define ntfs_is_rcrd_record(x) ( ntfs_is_magic (x, RCRD) )
-#define ntfs_is_rcrd_recordp(p) ( ntfs_is_magicp(p, RCRD) )
-
-#define ntfs_is_chkd_record(x) ( ntfs_is_magic (x, CHKD) )
-#define ntfs_is_chkd_recordp(p) ( ntfs_is_magicp(p, CHKD) )
-
-#define ntfs_is_baad_record(x) ( ntfs_is_magic (x, BAAD) )
-#define ntfs_is_baad_recordp(p) ( ntfs_is_magicp(p, BAAD) )
-
-#define ntfs_is_empty_record(x) ( ntfs_is_magic (x, empty) )
-#define ntfs_is_empty_recordp(p) ( ntfs_is_magicp(p, empty) )
-
-/*
- * The Update Sequence Array (usa) is an array of the le16 values which belong
- * to the end of each sector protected by the update sequence record in which
- * this array is contained. Note that the first entry is the Update Sequence
- * Number (usn), a cyclic counter of how many times the protected record has
- * been written to disk. The values 0 and -1 (ie. 0xffff) are not used. All
- * last le16's of each sector have to be equal to the usn (during reading) or
- * are set to it (during writing). If they are not, an incomplete multi sector
- * transfer has occurred when the data was written.
- * The maximum size for the update sequence array is fixed to:
- * maximum size = usa_ofs + (usa_count * 2) = 510 bytes
- * The 510 bytes comes from the fact that the last le16 in the array has to
- * (obviously) finish before the last le16 of the first 512-byte sector.
- * This formula can be used as a consistency check in that usa_ofs +
- * (usa_count * 2) has to be less than or equal to 510.
- */
-typedef struct {
- NTFS_RECORD_TYPE magic; /* A four-byte magic identifying the record
- type and/or status. */
- le16 usa_ofs; /* Offset to the Update Sequence Array (usa)
- from the start of the ntfs record. */
- le16 usa_count; /* Number of le16 sized entries in the usa
- including the Update Sequence Number (usn),
- thus the number of fixups is the usa_count
- minus 1. */
-} __attribute__ ((__packed__)) NTFS_RECORD;
-
-/*
- * System files mft record numbers. All these files are always marked as used
- * in the bitmap attribute of the mft; presumably in order to avoid accidental
- * allocation for random other mft records. Also, the sequence number for each
- * of the system files is always equal to their mft record number and it is
- * never modified.
- */
-typedef enum {
- FILE_MFT = 0, /* Master file table (mft). Data attribute
- contains the entries and bitmap attribute
- records which ones are in use (bit==1). */
- FILE_MFTMirr = 1, /* Mft mirror: copy of first four mft records
- in data attribute. If cluster size > 4kiB,
- copy of first N mft records, with
- N = cluster_size / mft_record_size. */
- FILE_LogFile = 2, /* Journalling log in data attribute. */
- FILE_Volume = 3, /* Volume name attribute and volume information
- attribute (flags and ntfs version). Windows
- refers to this file as volume DASD (Direct
- Access Storage Device). */
- FILE_AttrDef = 4, /* Array of attribute definitions in data
- attribute. */
- FILE_root = 5, /* Root directory. */
- FILE_Bitmap = 6, /* Allocation bitmap of all clusters (lcns) in
- data attribute. */
- FILE_Boot = 7, /* Boot sector (always at cluster 0) in data
- attribute. */
- FILE_BadClus = 8, /* Contains all bad clusters in the non-resident
- data attribute. */
- FILE_Secure = 9, /* Shared security descriptors in data attribute
- and two indexes into the descriptors.
- Appeared in Windows 2000. Before that, this
- file was named $Quota but was unused. */
- FILE_UpCase = 10, /* Uppercase equivalents of all 65536 Unicode
- characters in data attribute. */
- FILE_Extend = 11, /* Directory containing other system files (eg.
- $ObjId, $Quota, $Reparse and $UsnJrnl). This
- is new to NTFS3.0. */
- FILE_reserved12 = 12, /* Reserved for future use (records 12-15). */
- FILE_reserved13 = 13,
- FILE_reserved14 = 14,
- FILE_reserved15 = 15,
- FILE_first_user = 16, /* First user file, used as test limit for
- whether to allow opening a file or not. */
-} NTFS_SYSTEM_FILES;
-
-/*
- * These are the so far known MFT_RECORD_* flags (16-bit) which contain
- * information about the mft record in which they are present.
- */
-enum {
- MFT_RECORD_IN_USE = cpu_to_le16(0x0001),
- MFT_RECORD_IS_DIRECTORY = cpu_to_le16(0x0002),
-} __attribute__ ((__packed__));
-
-typedef le16 MFT_RECORD_FLAGS;
-
-/*
- * mft references (aka file references or file record segment references) are
- * used whenever a structure needs to refer to a record in the mft.
- *
- * A reference consists of a 48-bit index into the mft and a 16-bit sequence
- * number used to detect stale references.
- *
- * For error reporting purposes we treat the 48-bit index as a signed quantity.
- *
- * The sequence number is a circular counter (skipping 0) describing how many
- * times the referenced mft record has been (re)used. This has to match the
- * sequence number of the mft record being referenced, otherwise the reference
- * is considered stale and removed (FIXME: only ntfsck or the driver itself?).
- *
- * If the sequence number is zero it is assumed that no sequence number
- * consistency checking should be performed.
- *
- * FIXME: Since inodes are 32-bit as of now, the driver needs to always check
- * for high_part being 0 and if not either BUG(), cause a panic() or handle
- * the situation in some other way. This shouldn't be a problem as a volume has
- * to become HUGE in order to need more than 32-bits worth of mft records.
- * Assuming the standard mft record size of 1kb only the records (never mind
- * the non-resident attributes, etc.) would require 4Tb of space on their own
- * for the first 32 bits worth of records. This is only if some strange person
- * doesn't decide to foul play and make the mft sparse which would be a really
- * horrible thing to do as it would trash our current driver implementation. )-:
- * Do I hear screams "we want 64-bit inodes!" ?!? (-;
- *
- * FIXME: The mft zone is defined as the first 12% of the volume. This space is
- * reserved so that the mft can grow contiguously and hence doesn't become
- * fragmented. Volume free space includes the empty part of the mft zone and
- * when the volume's free 88% are used up, the mft zone is shrunk by a factor
- * of 2, thus making more space available for more files/data. This process is
- * repeated every time there is no more free space except for the mft zone until
- * there really is no more free space.
- */
-
-/*
- * Typedef the MFT_REF as a 64-bit value for easier handling.
- * Also define two unpacking macros to get to the reference (MREF) and
- * sequence number (MSEQNO) respectively.
- * The _LE versions are to be applied on little endian MFT_REFs.
- * Note: The _LE versions will return a CPU endian formatted value!
- */
-#define MFT_REF_MASK_CPU 0x0000ffffffffffffULL
-#define MFT_REF_MASK_LE cpu_to_le64(MFT_REF_MASK_CPU)
-
-typedef u64 MFT_REF;
-typedef le64 leMFT_REF;
-
-#define MK_MREF(m, s) ((MFT_REF)(((MFT_REF)(s) << 48) | \
- ((MFT_REF)(m) & MFT_REF_MASK_CPU)))
-#define MK_LE_MREF(m, s) cpu_to_le64(MK_MREF(m, s))
-
-#define MREF(x) ((unsigned long)((x) & MFT_REF_MASK_CPU))
-#define MSEQNO(x) ((u16)(((x) >> 48) & 0xffff))
-#define MREF_LE(x) ((unsigned long)(le64_to_cpu(x) & MFT_REF_MASK_CPU))
-#define MSEQNO_LE(x) ((u16)((le64_to_cpu(x) >> 48) & 0xffff))
-
-#define IS_ERR_MREF(x) (((x) & 0x0000800000000000ULL) ? true : false)
-#define ERR_MREF(x) ((u64)((s64)(x)))
-#define MREF_ERR(x) ((int)((s64)(x)))
-
-/*
- * The mft record header present at the beginning of every record in the mft.
- * This is followed by a sequence of variable length attribute records which
- * is terminated by an attribute of type AT_END which is a truncated attribute
- * in that it only consists of the attribute type code AT_END and none of the
- * other members of the attribute structure are present.
- */
-typedef struct {
-/*Ofs*/
-/* 0 NTFS_RECORD; -- Unfolded here as gcc doesn't like unnamed structs. */
- NTFS_RECORD_TYPE magic; /* Usually the magic is "FILE". */
- le16 usa_ofs; /* See NTFS_RECORD definition above. */
- le16 usa_count; /* See NTFS_RECORD definition above. */
-
-/* 8*/ le64 lsn; /* $LogFile sequence number for this record.
- Changed every time the record is modified. */
-/* 16*/ le16 sequence_number; /* Number of times this mft record has been
- reused. (See description for MFT_REF
- above.) NOTE: The increment (skipping zero)
- is done when the file is deleted. NOTE: If
- this is zero it is left zero. */
-/* 18*/ le16 link_count; /* Number of hard links, i.e. the number of
- directory entries referencing this record.
- NOTE: Only used in mft base records.
- NOTE: When deleting a directory entry we
- check the link_count and if it is 1 we
- delete the file. Otherwise we delete the
- FILE_NAME_ATTR being referenced by the
- directory entry from the mft record and
- decrement the link_count.
- FIXME: Careful with Win32 + DOS names! */
-/* 20*/ le16 attrs_offset; /* Byte offset to the first attribute in this
- mft record from the start of the mft record.
- NOTE: Must be aligned to 8-byte boundary. */
-/* 22*/ MFT_RECORD_FLAGS flags; /* Bit array of MFT_RECORD_FLAGS. When a file
- is deleted, the MFT_RECORD_IN_USE flag is
- set to zero. */
-/* 24*/ le32 bytes_in_use; /* Number of bytes used in this mft record.
- NOTE: Must be aligned to 8-byte boundary. */
-/* 28*/ le32 bytes_allocated; /* Number of bytes allocated for this mft
- record. This should be equal to the mft
- record size. */
-/* 32*/ leMFT_REF base_mft_record;/* This is zero for base mft records.
- When it is not zero it is a mft reference
- pointing to the base mft record to which
- this record belongs (this is then used to
- locate the attribute list attribute present
- in the base record which describes this
- extension record and hence might need
- modification when the extension record
- itself is modified, also locating the
- attribute list also means finding the other
- potential extents, belonging to the non-base
- mft record). */
-/* 40*/ le16 next_attr_instance;/* The instance number that will be assigned to
- the next attribute added to this mft record.
- NOTE: Incremented each time after it is used.
- NOTE: Every time the mft record is reused
- this number is set to zero. NOTE: The first
- instance number is always 0. */
-/* The below fields are specific to NTFS 3.1+ (Windows XP and above): */
-/* 42*/ le16 reserved; /* Reserved/alignment. */
-/* 44*/ le32 mft_record_number; /* Number of this mft record. */
-/* sizeof() = 48 bytes */
-/*
- * When (re)using the mft record, we place the update sequence array at this
- * offset, i.e. before we start with the attributes. This also makes sense,
- * otherwise we could run into problems with the update sequence array
- * containing in itself the last two bytes of a sector which would mean that
- * multi sector transfer protection wouldn't work. As you can't protect data
- * by overwriting it since you then can't get it back...
- * When reading we obviously use the data from the ntfs record header.
- */
-} __attribute__ ((__packed__)) MFT_RECORD;
-
-/* This is the version without the NTFS 3.1+ specific fields. */
-typedef struct {
-/*Ofs*/
-/* 0 NTFS_RECORD; -- Unfolded here as gcc doesn't like unnamed structs. */
- NTFS_RECORD_TYPE magic; /* Usually the magic is "FILE". */
- le16 usa_ofs; /* See NTFS_RECORD definition above. */
- le16 usa_count; /* See NTFS_RECORD definition above. */
-
-/* 8*/ le64 lsn; /* $LogFile sequence number for this record.
- Changed every time the record is modified. */
-/* 16*/ le16 sequence_number; /* Number of times this mft record has been
- reused. (See description for MFT_REF
- above.) NOTE: The increment (skipping zero)
- is done when the file is deleted. NOTE: If
- this is zero it is left zero. */
-/* 18*/ le16 link_count; /* Number of hard links, i.e. the number of
- directory entries referencing this record.
- NOTE: Only used in mft base records.
- NOTE: When deleting a directory entry we
- check the link_count and if it is 1 we
- delete the file. Otherwise we delete the
- FILE_NAME_ATTR being referenced by the
- directory entry from the mft record and
- decrement the link_count.
- FIXME: Careful with Win32 + DOS names! */
-/* 20*/ le16 attrs_offset; /* Byte offset to the first attribute in this
- mft record from the start of the mft record.
- NOTE: Must be aligned to 8-byte boundary. */
-/* 22*/ MFT_RECORD_FLAGS flags; /* Bit array of MFT_RECORD_FLAGS. When a file
- is deleted, the MFT_RECORD_IN_USE flag is
- set to zero. */
-/* 24*/ le32 bytes_in_use; /* Number of bytes used in this mft record.
- NOTE: Must be aligned to 8-byte boundary. */
-/* 28*/ le32 bytes_allocated; /* Number of bytes allocated for this mft
- record. This should be equal to the mft
- record size. */
-/* 32*/ leMFT_REF base_mft_record;/* This is zero for base mft records.
- When it is not zero it is a mft reference
- pointing to the base mft record to which
- this record belongs (this is then used to
- locate the attribute list attribute present
- in the base record which describes this
- extension record and hence might need
- modification when the extension record
- itself is modified, also locating the
- attribute list also means finding the other
- potential extents, belonging to the non-base
- mft record). */
-/* 40*/ le16 next_attr_instance;/* The instance number that will be assigned to
- the next attribute added to this mft record.
- NOTE: Incremented each time after it is used.
- NOTE: Every time the mft record is reused
- this number is set to zero. NOTE: The first
- instance number is always 0. */
-/* sizeof() = 42 bytes */
-/*
- * When (re)using the mft record, we place the update sequence array at this
- * offset, i.e. before we start with the attributes. This also makes sense,
- * otherwise we could run into problems with the update sequence array
- * containing in itself the last two bytes of a sector which would mean that
- * multi sector transfer protection wouldn't work. As you can't protect data
- * by overwriting it since you then can't get it back...
- * When reading we obviously use the data from the ntfs record header.
- */
-} __attribute__ ((__packed__)) MFT_RECORD_OLD;
-
-/*
- * System defined attributes (32-bit). Each attribute type has a corresponding
- * attribute name (Unicode string of maximum 64 character length) as described
- * by the attribute definitions present in the data attribute of the $AttrDef
- * system file. On NTFS 3.0 volumes the names are just as the types are named
- * in the below defines exchanging AT_ for the dollar sign ($). If that is not
- * a revealing choice of symbol I do not know what is... (-;
- */
-enum {
- AT_UNUSED = cpu_to_le32( 0),
- AT_STANDARD_INFORMATION = cpu_to_le32( 0x10),
- AT_ATTRIBUTE_LIST = cpu_to_le32( 0x20),
- AT_FILE_NAME = cpu_to_le32( 0x30),
- AT_OBJECT_ID = cpu_to_le32( 0x40),
- AT_SECURITY_DESCRIPTOR = cpu_to_le32( 0x50),
- AT_VOLUME_NAME = cpu_to_le32( 0x60),
- AT_VOLUME_INFORMATION = cpu_to_le32( 0x70),
- AT_DATA = cpu_to_le32( 0x80),
- AT_INDEX_ROOT = cpu_to_le32( 0x90),
- AT_INDEX_ALLOCATION = cpu_to_le32( 0xa0),
- AT_BITMAP = cpu_to_le32( 0xb0),
- AT_REPARSE_POINT = cpu_to_le32( 0xc0),
- AT_EA_INFORMATION = cpu_to_le32( 0xd0),
- AT_EA = cpu_to_le32( 0xe0),
- AT_PROPERTY_SET = cpu_to_le32( 0xf0),
- AT_LOGGED_UTILITY_STREAM = cpu_to_le32( 0x100),
- AT_FIRST_USER_DEFINED_ATTRIBUTE = cpu_to_le32( 0x1000),
- AT_END = cpu_to_le32(0xffffffff)
-};
-
-typedef le32 ATTR_TYPE;
-
-/*
- * The collation rules for sorting views/indexes/etc (32-bit).
- *
- * COLLATION_BINARY - Collate by binary compare where the first byte is most
- * significant.
- * COLLATION_UNICODE_STRING - Collate Unicode strings by comparing their binary
- * Unicode values, except that when a character can be uppercased, the
- * upper case value collates before the lower case one.
- * COLLATION_FILE_NAME - Collate file names as Unicode strings. The collation
- * is done very much like COLLATION_UNICODE_STRING. In fact I have no idea
- * what the difference is. Perhaps the difference is that file names
- * would treat some special characters in an odd way (see
- * unistr.c::ntfs_collate_names() and unistr.c::legal_ansi_char_array[]
- * for what I mean but COLLATION_UNICODE_STRING would not give any special
- * treatment to any characters at all, but this is speculation.
- * COLLATION_NTOFS_ULONG - Sorting is done according to ascending le32 key
- * values. E.g. used for $SII index in FILE_Secure, which sorts by
- * security_id (le32).
- * COLLATION_NTOFS_SID - Sorting is done according to ascending SID values.
- * E.g. used for $O index in FILE_Extend/$Quota.
- * COLLATION_NTOFS_SECURITY_HASH - Sorting is done first by ascending hash
- * values and second by ascending security_id values. E.g. used for $SDH
- * index in FILE_Secure.
- * COLLATION_NTOFS_ULONGS - Sorting is done according to a sequence of ascending
- * le32 key values. E.g. used for $O index in FILE_Extend/$ObjId, which
- * sorts by object_id (16-byte), by splitting up the object_id in four
- * le32 values and using them as individual keys. E.g. take the following
- * two security_ids, stored as follows on disk:
- * 1st: a1 61 65 b7 65 7b d4 11 9e 3d 00 e0 81 10 42 59
- * 2nd: 38 14 37 d2 d2 f3 d4 11 a5 21 c8 6b 79 b1 97 45
- * To compare them, they are split into four le32 values each, like so:
- * 1st: 0xb76561a1 0x11d47b65 0xe0003d9e 0x59421081
- * 2nd: 0xd2371438 0x11d4f3d2 0x6bc821a5 0x4597b179
- * Now, it is apparent why the 2nd object_id collates after the 1st: the
- * first le32 value of the 1st object_id is less than the first le32 of
- * the 2nd object_id. If the first le32 values of both object_ids were
- * equal then the second le32 values would be compared, etc.
- */
-enum {
- COLLATION_BINARY = cpu_to_le32(0x00),
- COLLATION_FILE_NAME = cpu_to_le32(0x01),
- COLLATION_UNICODE_STRING = cpu_to_le32(0x02),
- COLLATION_NTOFS_ULONG = cpu_to_le32(0x10),
- COLLATION_NTOFS_SID = cpu_to_le32(0x11),
- COLLATION_NTOFS_SECURITY_HASH = cpu_to_le32(0x12),
- COLLATION_NTOFS_ULONGS = cpu_to_le32(0x13),
-};
-
-typedef le32 COLLATION_RULE;
-
-/*
- * The flags (32-bit) describing attribute properties in the attribute
- * definition structure. FIXME: This information is based on Regis's
- * information and, according to him, it is not certain and probably
- * incomplete. The INDEXABLE flag is fairly certainly correct as only the file
- * name attribute has this flag set and this is the only attribute indexed in
- * NT4.
- */
-enum {
- ATTR_DEF_INDEXABLE = cpu_to_le32(0x02), /* Attribute can be
- indexed. */
- ATTR_DEF_MULTIPLE = cpu_to_le32(0x04), /* Attribute type
- can be present multiple times in the
- mft records of an inode. */
- ATTR_DEF_NOT_ZERO = cpu_to_le32(0x08), /* Attribute value
- must contain at least one non-zero
- byte. */
- ATTR_DEF_INDEXED_UNIQUE = cpu_to_le32(0x10), /* Attribute must be
- indexed and the attribute value must be
- unique for the attribute type in all of
- the mft records of an inode. */
- ATTR_DEF_NAMED_UNIQUE = cpu_to_le32(0x20), /* Attribute must be
- named and the name must be unique for
- the attribute type in all of the mft
- records of an inode. */
- ATTR_DEF_RESIDENT = cpu_to_le32(0x40), /* Attribute must be
- resident. */
- ATTR_DEF_ALWAYS_LOG = cpu_to_le32(0x80), /* Always log
- modifications to this attribute,
- regardless of whether it is resident or
- non-resident. Without this, only log
- modifications if the attribute is
- resident. */
-};
-
-typedef le32 ATTR_DEF_FLAGS;
-
-/*
- * The data attribute of FILE_AttrDef contains a sequence of attribute
- * definitions for the NTFS volume. With this, it is supposed to be safe for an
- * older NTFS driver to mount a volume containing a newer NTFS version without
- * damaging it (that's the theory. In practice it's: not damaging it too much).
- * Entries are sorted by attribute type. The flags describe whether the
- * attribute can be resident/non-resident and possibly other things, but the
- * actual bits are unknown.
- */
-typedef struct {
-/*hex ofs*/
-/* 0*/ ntfschar name[0x40]; /* Unicode name of the attribute. Zero
- terminated. */
-/* 80*/ ATTR_TYPE type; /* Type of the attribute. */
-/* 84*/ le32 display_rule; /* Default display rule.
- FIXME: What does it mean? (AIA) */
-/* 88*/ COLLATION_RULE collation_rule; /* Default collation rule. */
-/* 8c*/ ATTR_DEF_FLAGS flags; /* Flags describing the attribute. */
-/* 90*/ sle64 min_size; /* Optional minimum attribute size. */
-/* 98*/ sle64 max_size; /* Maximum size of attribute. */
-/* sizeof() = 0xa0 or 160 bytes */
-} __attribute__ ((__packed__)) ATTR_DEF;
-
-/*
- * Attribute flags (16-bit).
- */
-enum {
- ATTR_IS_COMPRESSED = cpu_to_le16(0x0001),
- ATTR_COMPRESSION_MASK = cpu_to_le16(0x00ff), /* Compression method
- mask. Also, first
- illegal value. */
- ATTR_IS_ENCRYPTED = cpu_to_le16(0x4000),
- ATTR_IS_SPARSE = cpu_to_le16(0x8000),
-} __attribute__ ((__packed__));
-
-typedef le16 ATTR_FLAGS;
-
-/*
- * Attribute compression.
- *
- * Only the data attribute is ever compressed in the current ntfs driver in
- * Windows. Further, compression is only applied when the data attribute is
- * non-resident. Finally, to use compression, the maximum allowed cluster size
- * on a volume is 4kib.
- *
- * The compression method is based on independently compressing blocks of X
- * clusters, where X is determined from the compression_unit value found in the
- * non-resident attribute record header (more precisely: X = 2^compression_unit
- * clusters). On Windows NT/2k, X always is 16 clusters (compression_unit = 4).
- *
- * There are three different cases of how a compression block of X clusters
- * can be stored:
- *
- * 1) The data in the block is all zero (a sparse block):
- * This is stored as a sparse block in the runlist, i.e. the runlist
- * entry has length = X and lcn = -1. The mapping pairs array actually
- * uses a delta_lcn value length of 0, i.e. delta_lcn is not present at
- * all, which is then interpreted by the driver as lcn = -1.
- * NOTE: Even uncompressed files can be sparse on NTFS 3.0 volumes, then
- * the same principles apply as above, except that the length is not
- * restricted to being any particular value.
- *
- * 2) The data in the block is not compressed:
- * This happens when compression doesn't reduce the size of the block
- * in clusters. I.e. if compression has a small effect so that the
- * compressed data still occupies X clusters, then the uncompressed data
- * is stored in the block.
- * This case is recognised by the fact that the runlist entry has
- * length = X and lcn >= 0. The mapping pairs array stores this as
- * normal with a run length of X and some specific delta_lcn, i.e.
- * delta_lcn has to be present.
- *
- * 3) The data in the block is compressed:
- * The common case. This case is recognised by the fact that the run
- * list entry has length L < X and lcn >= 0. The mapping pairs array
- * stores this as normal with a run length of X and some specific
- * delta_lcn, i.e. delta_lcn has to be present. This runlist entry is
- * immediately followed by a sparse entry with length = X - L and
- * lcn = -1. The latter entry is to make up the vcn counting to the
- * full compression block size X.
- *
- * In fact, life is more complicated because adjacent entries of the same type
- * can be coalesced. This means that one has to keep track of the number of
- * clusters handled and work on a basis of X clusters at a time being one
- * block. An example: if length L > X this means that this particular runlist
- * entry contains a block of length X and part of one or more blocks of length
- * L - X. Another example: if length L < X, this does not necessarily mean that
- * the block is compressed as it might be that the lcn changes inside the block
- * and hence the following runlist entry describes the continuation of the
- * potentially compressed block. The block would be compressed if the
- * following runlist entry describes at least X - L sparse clusters, thus
- * making up the compression block length as described in point 3 above. (Of
- * course, there can be several runlist entries with small lengths so that the
- * sparse entry does not follow the first data containing entry with
- * length < X.)
- *
- * NOTE: At the end of the compressed attribute value, there most likely is not
- * just the right amount of data to make up a compression block, thus this data
- * is not even attempted to be compressed. It is just stored as is, unless
- * the number of clusters it occupies is reduced when compressed in which case
- * it is stored as a compressed compression block, complete with sparse
- * clusters at the end.
- */
-
-/*
- * Flags of resident attributes (8-bit).
- */
-enum {
- RESIDENT_ATTR_IS_INDEXED = 0x01, /* Attribute is referenced in an index
- (has implications for deleting and
- modifying the attribute). */
-} __attribute__ ((__packed__));
-
-typedef u8 RESIDENT_ATTR_FLAGS;
-
-/*
- * Attribute record header. Always aligned to 8-byte boundary.
- */
-typedef struct {
-/*Ofs*/
-/* 0*/ ATTR_TYPE type; /* The (32-bit) type of the attribute. */
-/* 4*/ le32 length; /* Byte size of the resident part of the
- attribute (aligned to 8-byte boundary).
- Used to get to the next attribute. */
-/* 8*/ u8 non_resident; /* If 0, attribute is resident.
- If 1, attribute is non-resident. */
-/* 9*/ u8 name_length; /* Unicode character size of name of attribute.
- 0 if unnamed. */
-/* 10*/ le16 name_offset; /* If name_length != 0, the byte offset to the
- beginning of the name from the attribute
- record. Note that the name is stored as a
- Unicode string. When creating, place offset
- just at the end of the record header. Then,
- follow with attribute value or mapping pairs
- array, resident and non-resident attributes
- respectively, aligning to an 8-byte
- boundary. */
-/* 12*/ ATTR_FLAGS flags; /* Flags describing the attribute. */
-/* 14*/ le16 instance; /* The instance of this attribute record. This
- number is unique within this mft record (see
- MFT_RECORD/next_attribute_instance notes in
- mft.h for more details). */
-/* 16*/ union {
- /* Resident attributes. */
- struct {
-/* 16 */ le32 value_length;/* Byte size of attribute value. */
-/* 20 */ le16 value_offset;/* Byte offset of the attribute
- value from the start of the
- attribute record. When creating,
- align to 8-byte boundary if we
- have a name present as this might
- not have a length of a multiple
- of 8-bytes. */
-/* 22 */ RESIDENT_ATTR_FLAGS flags; /* See above. */
-/* 23 */ s8 reserved; /* Reserved/alignment to 8-byte
- boundary. */
- } __attribute__ ((__packed__)) resident;
- /* Non-resident attributes. */
- struct {
-/* 16*/ leVCN lowest_vcn;/* Lowest valid virtual cluster number
- for this portion of the attribute value or
- 0 if this is the only extent (usually the
- case). - Only when an attribute list is used
- does lowest_vcn != 0 ever occur. */
-/* 24*/ leVCN highest_vcn;/* Highest valid vcn of this extent of
- the attribute value. - Usually there is only one
- portion, so this usually equals the attribute
- value size in clusters minus 1. Can be -1 for
- zero length files. Can be 0 for "single extent"
- attributes. */
-/* 32*/ le16 mapping_pairs_offset; /* Byte offset from the
- beginning of the structure to the mapping pairs
- array which contains the mappings between the
- vcns and the logical cluster numbers (lcns).
- When creating, place this at the end of this
- record header aligned to 8-byte boundary. */
-/* 34*/ u8 compression_unit; /* The compression unit expressed
- as the log to the base 2 of the number of
- clusters in a compression unit. 0 means not
- compressed. (This effectively limits the
- compression unit size to be a power of two
- clusters.) WinNT4 only uses a value of 4.
- Sparse files have this set to 0 on XPSP2. */
-/* 35*/ u8 reserved[5]; /* Align to 8-byte boundary. */
-/* The sizes below are only used when lowest_vcn is zero, as otherwise it would
- be difficult to keep them up-to-date.*/
-/* 40*/ sle64 allocated_size; /* Byte size of disk space
- allocated to hold the attribute value. Always
- is a multiple of the cluster size. When a file
- is compressed, this field is a multiple of the
- compression block size (2^compression_unit) and
- it represents the logically allocated space
- rather than the actual on disk usage. For this
- use the compressed_size (see below). */
-/* 48*/ sle64 data_size; /* Byte size of the attribute
- value. Can be larger than allocated_size if
- attribute value is compressed or sparse. */
-/* 56*/ sle64 initialized_size; /* Byte size of initialized
- portion of the attribute value. Usually equals
- data_size. */
-/* sizeof(uncompressed attr) = 64*/
-/* 64*/ sle64 compressed_size; /* Byte size of the attribute
- value after compression. Only present when
- compressed or sparse. Always is a multiple of
- the cluster size. Represents the actual amount
- of disk space being used on the disk. */
-/* sizeof(compressed attr) = 72*/
- } __attribute__ ((__packed__)) non_resident;
- } __attribute__ ((__packed__)) data;
-} __attribute__ ((__packed__)) ATTR_RECORD;
-
-typedef ATTR_RECORD ATTR_REC;
-
-/*
- * File attribute flags (32-bit) appearing in the file_attributes fields of the
- * STANDARD_INFORMATION attribute of MFT_RECORDs and the FILENAME_ATTR
- * attributes of MFT_RECORDs and directory index entries.
- *
- * All of the below flags appear in the directory index entries but only some
- * appear in the STANDARD_INFORMATION attribute whilst only some others appear
- * in the FILENAME_ATTR attribute of MFT_RECORDs. Unless otherwise stated the
- * flags appear in all of the above.
- */
-enum {
- FILE_ATTR_READONLY = cpu_to_le32(0x00000001),
- FILE_ATTR_HIDDEN = cpu_to_le32(0x00000002),
- FILE_ATTR_SYSTEM = cpu_to_le32(0x00000004),
- /* Old DOS volid. Unused in NT. = cpu_to_le32(0x00000008), */
-
- FILE_ATTR_DIRECTORY = cpu_to_le32(0x00000010),
- /* Note, FILE_ATTR_DIRECTORY is not considered valid in NT. It is
- reserved for the DOS SUBDIRECTORY flag. */
- FILE_ATTR_ARCHIVE = cpu_to_le32(0x00000020),
- FILE_ATTR_DEVICE = cpu_to_le32(0x00000040),
- FILE_ATTR_NORMAL = cpu_to_le32(0x00000080),
-
- FILE_ATTR_TEMPORARY = cpu_to_le32(0x00000100),
- FILE_ATTR_SPARSE_FILE = cpu_to_le32(0x00000200),
- FILE_ATTR_REPARSE_POINT = cpu_to_le32(0x00000400),
- FILE_ATTR_COMPRESSED = cpu_to_le32(0x00000800),
-
- FILE_ATTR_OFFLINE = cpu_to_le32(0x00001000),
- FILE_ATTR_NOT_CONTENT_INDEXED = cpu_to_le32(0x00002000),
- FILE_ATTR_ENCRYPTED = cpu_to_le32(0x00004000),
-
- FILE_ATTR_VALID_FLAGS = cpu_to_le32(0x00007fb7),
- /* Note, FILE_ATTR_VALID_FLAGS masks out the old DOS VolId and the
- FILE_ATTR_DEVICE and preserves everything else. This mask is used
- to obtain all flags that are valid for reading. */
- FILE_ATTR_VALID_SET_FLAGS = cpu_to_le32(0x000031a7),
- /* Note, FILE_ATTR_VALID_SET_FLAGS masks out the old DOS VolId, the
- F_A_DEVICE, F_A_DIRECTORY, F_A_SPARSE_FILE, F_A_REPARSE_POINT,
- F_A_COMPRESSED, and F_A_ENCRYPTED and preserves the rest. This mask
- is used to obtain all flags that are valid for setting. */
- /*
- * The flag FILE_ATTR_DUP_FILENAME_INDEX_PRESENT is present in all
- * FILENAME_ATTR attributes but not in the STANDARD_INFORMATION
- * attribute of an mft record.
- */
- FILE_ATTR_DUP_FILE_NAME_INDEX_PRESENT = cpu_to_le32(0x10000000),
- /* Note, this is a copy of the corresponding bit from the mft record,
- telling us whether this is a directory or not, i.e. whether it has
- an index root attribute or not. */
- FILE_ATTR_DUP_VIEW_INDEX_PRESENT = cpu_to_le32(0x20000000),
- /* Note, this is a copy of the corresponding bit from the mft record,
- telling us whether this file has a view index present (eg. object id
- index, quota index, one of the security indexes or the encrypting
- filesystem related indexes). */
-};
-
-typedef le32 FILE_ATTR_FLAGS;
-
-/*
- * NOTE on times in NTFS: All times are in MS standard time format, i.e. they
- * are the number of 100-nanosecond intervals since 1st January 1601, 00:00:00
- * universal coordinated time (UTC). (In Linux time starts 1st January 1970,
- * 00:00:00 UTC and is stored as the number of 1-second intervals since then.)
- */
-
-/*
- * Attribute: Standard information (0x10).
- *
- * NOTE: Always resident.
- * NOTE: Present in all base file records on a volume.
- * NOTE: There is conflicting information about the meaning of each of the time
- * fields but the meaning as defined below has been verified to be
- * correct by practical experimentation on Windows NT4 SP6a and is hence
- * assumed to be the one and only correct interpretation.
- */
-typedef struct {
-/*Ofs*/
-/* 0*/ sle64 creation_time; /* Time file was created. Updated when
- a filename is changed(?). */
-/* 8*/ sle64 last_data_change_time; /* Time the data attribute was last
- modified. */
-/* 16*/ sle64 last_mft_change_time; /* Time this mft record was last
- modified. */
-/* 24*/ sle64 last_access_time; /* Approximate time when the file was
- last accessed (obviously this is not
- updated on read-only volumes). In
- Windows this is only updated when
- accessed if some time delta has
- passed since the last update. Also,
- last access time updates can be
- disabled altogether for speed. */
-/* 32*/ FILE_ATTR_FLAGS file_attributes; /* Flags describing the file. */
-/* 36*/ union {
- /* NTFS 1.2 */
- struct {
- /* 36*/ u8 reserved12[12]; /* Reserved/alignment to 8-byte
- boundary. */
- } __attribute__ ((__packed__)) v1;
- /* sizeof() = 48 bytes */
- /* NTFS 3.x */
- struct {
-/*
- * If a volume has been upgraded from a previous NTFS version, then these
- * fields are present only if the file has been accessed since the upgrade.
- * Recognize the difference by comparing the length of the resident attribute
- * value. If it is 48, then the following fields are missing. If it is 72 then
- * the fields are present. Maybe just check like this:
- * if (resident.ValueLength < sizeof(STANDARD_INFORMATION)) {
- * Assume NTFS 1.2- format.
- * If (volume version is 3.x)
- * Upgrade attribute to NTFS 3.x format.
- * else
- * Use NTFS 1.2- format for access.
- * } else
- * Use NTFS 3.x format for access.
- * Only problem is that it might be legal to set the length of the value to
- * arbitrarily large values thus spoiling this check. - But chkdsk probably
- * views that as a corruption, assuming that it behaves like this for all
- * attributes.
- */
- /* 36*/ le32 maximum_versions; /* Maximum allowed versions for
- file. Zero if version numbering is disabled. */
- /* 40*/ le32 version_number; /* This file's version (if any).
- Set to zero if maximum_versions is zero. */
- /* 44*/ le32 class_id; /* Class id from bidirectional
- class id index (?). */
- /* 48*/ le32 owner_id; /* Owner_id of the user owning
- the file. Translate via $Q index in FILE_Extend
- /$Quota to the quota control entry for the user
- owning the file. Zero if quotas are disabled. */
- /* 52*/ le32 security_id; /* Security_id for the file.
- Translate via $SII index and $SDS data stream
- in FILE_Secure to the security descriptor. */
- /* 56*/ le64 quota_charged; /* Byte size of the charge to
- the quota for all streams of the file. Note: Is
- zero if quotas are disabled. */
- /* 64*/ leUSN usn; /* Last update sequence number
- of the file. This is a direct index into the
- transaction log file ($UsnJrnl). It is zero if
- the usn journal is disabled or this file has
- not been subject to logging yet. See usnjrnl.h
- for details. */
- } __attribute__ ((__packed__)) v3;
- /* sizeof() = 72 bytes (NTFS 3.x) */
- } __attribute__ ((__packed__)) ver;
-} __attribute__ ((__packed__)) STANDARD_INFORMATION;
-
-/*
- * Attribute: Attribute list (0x20).
- *
- * - Can be either resident or non-resident.
- * - Value consists of a sequence of variable length, 8-byte aligned,
- * ATTR_LIST_ENTRY records.
- * - The list is not terminated by anything at all! The only way to know when
- * the end is reached is to keep track of the current offset and compare it to
- * the attribute value size.
- * - The attribute list attribute contains one entry for each attribute of
- * the file in which the list is located, except for the list attribute
- * itself. The list is sorted: first by attribute type, second by attribute
- * name (if present), third by instance number. The extents of one
- * non-resident attribute (if present) immediately follow after the initial
- * extent. They are ordered by lowest_vcn and have their instace set to zero.
- * It is not allowed to have two attributes with all sorting keys equal.
- * - Further restrictions:
- * - If not resident, the vcn to lcn mapping array has to fit inside the
- * base mft record.
- * - The attribute list attribute value has a maximum size of 256kb. This
- * is imposed by the Windows cache manager.
- * - Attribute lists are only used when the attributes of mft record do not
- * fit inside the mft record despite all attributes (that can be made
- * non-resident) having been made non-resident. This can happen e.g. when:
- * - File has a large number of hard links (lots of file name
- * attributes present).
- * - The mapping pairs array of some non-resident attribute becomes so
- * large due to fragmentation that it overflows the mft record.
- * - The security descriptor is very complex (not applicable to
- * NTFS 3.0 volumes).
- * - There are many named streams.
- */
-typedef struct {
-/*Ofs*/
-/* 0*/ ATTR_TYPE type; /* Type of referenced attribute. */
-/* 4*/ le16 length; /* Byte size of this entry (8-byte aligned). */
-/* 6*/ u8 name_length; /* Size in Unicode chars of the name of the
- attribute or 0 if unnamed. */
-/* 7*/ u8 name_offset; /* Byte offset to beginning of attribute name
- (always set this to where the name would
- start even if unnamed). */
-/* 8*/ leVCN lowest_vcn; /* Lowest virtual cluster number of this portion
- of the attribute value. This is usually 0. It
- is non-zero for the case where one attribute
- does not fit into one mft record and thus
- several mft records are allocated to hold
- this attribute. In the latter case, each mft
- record holds one extent of the attribute and
- there is one attribute list entry for each
- extent. NOTE: This is DEFINITELY a signed
- value! The windows driver uses cmp, followed
- by jg when comparing this, thus it treats it
- as signed. */
-/* 16*/ leMFT_REF mft_reference;/* The reference of the mft record holding
- the ATTR_RECORD for this portion of the
- attribute value. */
-/* 24*/ le16 instance; /* If lowest_vcn = 0, the instance of the
- attribute being referenced; otherwise 0. */
-/* 26*/ ntfschar name[0]; /* Use when creating only. When reading use
- name_offset to determine the location of the
- name. */
-/* sizeof() = 26 + (attribute_name_length * 2) bytes */
-} __attribute__ ((__packed__)) ATTR_LIST_ENTRY;
-
-/*
- * The maximum allowed length for a file name.
- */
-#define MAXIMUM_FILE_NAME_LENGTH 255
-
-/*
- * Possible namespaces for filenames in ntfs (8-bit).
- */
-enum {
- FILE_NAME_POSIX = 0x00,
- /* This is the largest namespace. It is case sensitive and allows all
- Unicode characters except for: '\0' and '/'. Beware that in
- WinNT/2k/2003 by default files which eg have the same name except
- for their case will not be distinguished by the standard utilities
- and thus a "del filename" will delete both "filename" and "fileName"
- without warning. However if for example Services For Unix (SFU) are
- installed and the case sensitive option was enabled at installation
- time, then you can create/access/delete such files.
- Note that even SFU places restrictions on the filenames beyond the
- '\0' and '/' and in particular the following set of characters is
- not allowed: '"', '/', '<', '>', '\'. All other characters,
- including the ones no allowed in WIN32 namespace are allowed.
- Tested with SFU 3.5 (this is now free) running on Windows XP. */
- FILE_NAME_WIN32 = 0x01,
- /* The standard WinNT/2k NTFS long filenames. Case insensitive. All
- Unicode chars except: '\0', '"', '*', '/', ':', '<', '>', '?', '\',
- and '|'. Further, names cannot end with a '.' or a space. */
- FILE_NAME_DOS = 0x02,
- /* The standard DOS filenames (8.3 format). Uppercase only. All 8-bit
- characters greater space, except: '"', '*', '+', ',', '/', ':', ';',
- '<', '=', '>', '?', and '\'. */
- FILE_NAME_WIN32_AND_DOS = 0x03,
- /* 3 means that both the Win32 and the DOS filenames are identical and
- hence have been saved in this single filename record. */
-} __attribute__ ((__packed__));
-
-typedef u8 FILE_NAME_TYPE_FLAGS;
-
-/*
- * Attribute: Filename (0x30).
- *
- * NOTE: Always resident.
- * NOTE: All fields, except the parent_directory, are only updated when the
- * filename is changed. Until then, they just become out of sync with
- * reality and the more up to date values are present in the standard
- * information attribute.
- * NOTE: There is conflicting information about the meaning of each of the time
- * fields but the meaning as defined below has been verified to be
- * correct by practical experimentation on Windows NT4 SP6a and is hence
- * assumed to be the one and only correct interpretation.
- */
-typedef struct {
-/*hex ofs*/
-/* 0*/ leMFT_REF parent_directory; /* Directory this filename is
- referenced from. */
-/* 8*/ sle64 creation_time; /* Time file was created. */
-/* 10*/ sle64 last_data_change_time; /* Time the data attribute was last
- modified. */
-/* 18*/ sle64 last_mft_change_time; /* Time this mft record was last
- modified. */
-/* 20*/ sle64 last_access_time; /* Time this mft record was last
- accessed. */
-/* 28*/ sle64 allocated_size; /* Byte size of on-disk allocated space
- for the unnamed data attribute. So
- for normal $DATA, this is the
- allocated_size from the unnamed
- $DATA attribute and for compressed
- and/or sparse $DATA, this is the
- compressed_size from the unnamed
- $DATA attribute. For a directory or
- other inode without an unnamed $DATA
- attribute, this is always 0. NOTE:
- This is a multiple of the cluster
- size. */
-/* 30*/ sle64 data_size; /* Byte size of actual data in unnamed
- data attribute. For a directory or
- other inode without an unnamed $DATA
- attribute, this is always 0. */
-/* 38*/ FILE_ATTR_FLAGS file_attributes; /* Flags describing the file. */
-/* 3c*/ union {
- /* 3c*/ struct {
- /* 3c*/ le16 packed_ea_size; /* Size of the buffer needed to
- pack the extended attributes
- (EAs), if such are present.*/
- /* 3e*/ le16 reserved; /* Reserved for alignment. */
- } __attribute__ ((__packed__)) ea;
- /* 3c*/ struct {
- /* 3c*/ le32 reparse_point_tag; /* Type of reparse point,
- present only in reparse
- points and only if there are
- no EAs. */
- } __attribute__ ((__packed__)) rp;
- } __attribute__ ((__packed__)) type;
-/* 40*/ u8 file_name_length; /* Length of file name in
- (Unicode) characters. */
-/* 41*/ FILE_NAME_TYPE_FLAGS file_name_type; /* Namespace of the file name.*/
-/* 42*/ ntfschar file_name[0]; /* File name in Unicode. */
-} __attribute__ ((__packed__)) FILE_NAME_ATTR;
-
-/*
- * GUID structures store globally unique identifiers (GUID). A GUID is a
- * 128-bit value consisting of one group of eight hexadecimal digits, followed
- * by three groups of four hexadecimal digits each, followed by one group of
- * twelve hexadecimal digits. GUIDs are Microsoft's implementation of the
- * distributed computing environment (DCE) universally unique identifier (UUID).
- * Example of a GUID:
- * 1F010768-5A73-BC91-0010A52216A7
- */
-typedef struct {
- le32 data1; /* The first eight hexadecimal digits of the GUID. */
- le16 data2; /* The first group of four hexadecimal digits. */
- le16 data3; /* The second group of four hexadecimal digits. */
- u8 data4[8]; /* The first two bytes are the third group of four
- hexadecimal digits. The remaining six bytes are the
- final 12 hexadecimal digits. */
-} __attribute__ ((__packed__)) GUID;
-
-/*
- * FILE_Extend/$ObjId contains an index named $O. This index contains all
- * object_ids present on the volume as the index keys and the corresponding
- * mft_record numbers as the index entry data parts. The data part (defined
- * below) also contains three other object_ids:
- * birth_volume_id - object_id of FILE_Volume on which the file was first
- * created. Optional (i.e. can be zero).
- * birth_object_id - object_id of file when it was first created. Usually
- * equals the object_id. Optional (i.e. can be zero).
- * domain_id - Reserved (always zero).
- */
-typedef struct {
- leMFT_REF mft_reference;/* Mft record containing the object_id in
- the index entry key. */
- union {
- struct {
- GUID birth_volume_id;
- GUID birth_object_id;
- GUID domain_id;
- } __attribute__ ((__packed__)) origin;
- u8 extended_info[48];
- } __attribute__ ((__packed__)) opt;
-} __attribute__ ((__packed__)) OBJ_ID_INDEX_DATA;
-
-/*
- * Attribute: Object id (NTFS 3.0+) (0x40).
- *
- * NOTE: Always resident.
- */
-typedef struct {
- GUID object_id; /* Unique id assigned to the
- file.*/
- /* The following fields are optional. The attribute value size is 16
- bytes, i.e. sizeof(GUID), if these are not present at all. Note,
- the entries can be present but one or more (or all) can be zero
- meaning that that particular value(s) is(are) not defined. */
- union {
- struct {
- GUID birth_volume_id; /* Unique id of volume on which
- the file was first created.*/
- GUID birth_object_id; /* Unique id of file when it was
- first created. */
- GUID domain_id; /* Reserved, zero. */
- } __attribute__ ((__packed__)) origin;
- u8 extended_info[48];
- } __attribute__ ((__packed__)) opt;
-} __attribute__ ((__packed__)) OBJECT_ID_ATTR;
-
-/*
- * The pre-defined IDENTIFIER_AUTHORITIES used as SID_IDENTIFIER_AUTHORITY in
- * the SID structure (see below).
- */
-//typedef enum { /* SID string prefix. */
-// SECURITY_NULL_SID_AUTHORITY = {0, 0, 0, 0, 0, 0}, /* S-1-0 */
-// SECURITY_WORLD_SID_AUTHORITY = {0, 0, 0, 0, 0, 1}, /* S-1-1 */
-// SECURITY_LOCAL_SID_AUTHORITY = {0, 0, 0, 0, 0, 2}, /* S-1-2 */
-// SECURITY_CREATOR_SID_AUTHORITY = {0, 0, 0, 0, 0, 3}, /* S-1-3 */
-// SECURITY_NON_UNIQUE_AUTHORITY = {0, 0, 0, 0, 0, 4}, /* S-1-4 */
-// SECURITY_NT_SID_AUTHORITY = {0, 0, 0, 0, 0, 5}, /* S-1-5 */
-//} IDENTIFIER_AUTHORITIES;
-
-/*
- * These relative identifiers (RIDs) are used with the above identifier
- * authorities to make up universal well-known SIDs.
- *
- * Note: The relative identifier (RID) refers to the portion of a SID, which
- * identifies a user or group in relation to the authority that issued the SID.
- * For example, the universal well-known SID Creator Owner ID (S-1-3-0) is
- * made up of the identifier authority SECURITY_CREATOR_SID_AUTHORITY (3) and
- * the relative identifier SECURITY_CREATOR_OWNER_RID (0).
- */
-typedef enum { /* Identifier authority. */
- SECURITY_NULL_RID = 0, /* S-1-0 */
- SECURITY_WORLD_RID = 0, /* S-1-1 */
- SECURITY_LOCAL_RID = 0, /* S-1-2 */
-
- SECURITY_CREATOR_OWNER_RID = 0, /* S-1-3 */
- SECURITY_CREATOR_GROUP_RID = 1, /* S-1-3 */
-
- SECURITY_CREATOR_OWNER_SERVER_RID = 2, /* S-1-3 */
- SECURITY_CREATOR_GROUP_SERVER_RID = 3, /* S-1-3 */
-
- SECURITY_DIALUP_RID = 1,
- SECURITY_NETWORK_RID = 2,
- SECURITY_BATCH_RID = 3,
- SECURITY_INTERACTIVE_RID = 4,
- SECURITY_SERVICE_RID = 6,
- SECURITY_ANONYMOUS_LOGON_RID = 7,
- SECURITY_PROXY_RID = 8,
- SECURITY_ENTERPRISE_CONTROLLERS_RID=9,
- SECURITY_SERVER_LOGON_RID = 9,
- SECURITY_PRINCIPAL_SELF_RID = 0xa,
- SECURITY_AUTHENTICATED_USER_RID = 0xb,
- SECURITY_RESTRICTED_CODE_RID = 0xc,
- SECURITY_TERMINAL_SERVER_RID = 0xd,
-
- SECURITY_LOGON_IDS_RID = 5,
- SECURITY_LOGON_IDS_RID_COUNT = 3,
-
- SECURITY_LOCAL_SYSTEM_RID = 0x12,
-
- SECURITY_NT_NON_UNIQUE = 0x15,
-
- SECURITY_BUILTIN_DOMAIN_RID = 0x20,
-
- /*
- * Well-known domain relative sub-authority values (RIDs).
- */
-
- /* Users. */
- DOMAIN_USER_RID_ADMIN = 0x1f4,
- DOMAIN_USER_RID_GUEST = 0x1f5,
- DOMAIN_USER_RID_KRBTGT = 0x1f6,
-
- /* Groups. */
- DOMAIN_GROUP_RID_ADMINS = 0x200,
- DOMAIN_GROUP_RID_USERS = 0x201,
- DOMAIN_GROUP_RID_GUESTS = 0x202,
- DOMAIN_GROUP_RID_COMPUTERS = 0x203,
- DOMAIN_GROUP_RID_CONTROLLERS = 0x204,
- DOMAIN_GROUP_RID_CERT_ADMINS = 0x205,
- DOMAIN_GROUP_RID_SCHEMA_ADMINS = 0x206,
- DOMAIN_GROUP_RID_ENTERPRISE_ADMINS= 0x207,
- DOMAIN_GROUP_RID_POLICY_ADMINS = 0x208,
-
- /* Aliases. */
- DOMAIN_ALIAS_RID_ADMINS = 0x220,
- DOMAIN_ALIAS_RID_USERS = 0x221,
- DOMAIN_ALIAS_RID_GUESTS = 0x222,
- DOMAIN_ALIAS_RID_POWER_USERS = 0x223,
-
- DOMAIN_ALIAS_RID_ACCOUNT_OPS = 0x224,
- DOMAIN_ALIAS_RID_SYSTEM_OPS = 0x225,
- DOMAIN_ALIAS_RID_PRINT_OPS = 0x226,
- DOMAIN_ALIAS_RID_BACKUP_OPS = 0x227,
-
- DOMAIN_ALIAS_RID_REPLICATOR = 0x228,
- DOMAIN_ALIAS_RID_RAS_SERVERS = 0x229,
- DOMAIN_ALIAS_RID_PREW2KCOMPACCESS = 0x22a,
-} RELATIVE_IDENTIFIERS;
-
-/*
- * The universal well-known SIDs:
- *
- * NULL_SID S-1-0-0
- * WORLD_SID S-1-1-0
- * LOCAL_SID S-1-2-0
- * CREATOR_OWNER_SID S-1-3-0
- * CREATOR_GROUP_SID S-1-3-1
- * CREATOR_OWNER_SERVER_SID S-1-3-2
- * CREATOR_GROUP_SERVER_SID S-1-3-3
- *
- * (Non-unique IDs) S-1-4
- *
- * NT well-known SIDs:
- *
- * NT_AUTHORITY_SID S-1-5
- * DIALUP_SID S-1-5-1
- *
- * NETWORD_SID S-1-5-2
- * BATCH_SID S-1-5-3
- * INTERACTIVE_SID S-1-5-4
- * SERVICE_SID S-1-5-6
- * ANONYMOUS_LOGON_SID S-1-5-7 (aka null logon session)
- * PROXY_SID S-1-5-8
- * SERVER_LOGON_SID S-1-5-9 (aka domain controller account)
- * SELF_SID S-1-5-10 (self RID)
- * AUTHENTICATED_USER_SID S-1-5-11
- * RESTRICTED_CODE_SID S-1-5-12 (running restricted code)
- * TERMINAL_SERVER_SID S-1-5-13 (running on terminal server)
- *
- * (Logon IDs) S-1-5-5-X-Y
- *
- * (NT non-unique IDs) S-1-5-0x15-...
- *
- * (Built-in domain) S-1-5-0x20
- */
-
-/*
- * The SID_IDENTIFIER_AUTHORITY is a 48-bit value used in the SID structure.
- *
- * NOTE: This is stored as a big endian number, hence the high_part comes
- * before the low_part.
- */
-typedef union {
- struct {
- u16 high_part; /* High 16-bits. */
- u32 low_part; /* Low 32-bits. */
- } __attribute__ ((__packed__)) parts;
- u8 value[6]; /* Value as individual bytes. */
-} __attribute__ ((__packed__)) SID_IDENTIFIER_AUTHORITY;
-
-/*
- * The SID structure is a variable-length structure used to uniquely identify
- * users or groups. SID stands for security identifier.
- *
- * The standard textual representation of the SID is of the form:
- * S-R-I-S-S...
- * Where:
- * - The first "S" is the literal character 'S' identifying the following
- * digits as a SID.
- * - R is the revision level of the SID expressed as a sequence of digits
- * either in decimal or hexadecimal (if the later, prefixed by "0x").
- * - I is the 48-bit identifier_authority, expressed as digits as R above.
- * - S... is one or more sub_authority values, expressed as digits as above.
- *
- * Example SID; the domain-relative SID of the local Administrators group on
- * Windows NT/2k:
- * S-1-5-32-544
- * This translates to a SID with:
- * revision = 1,
- * sub_authority_count = 2,
- * identifier_authority = {0,0,0,0,0,5}, // SECURITY_NT_AUTHORITY
- * sub_authority[0] = 32, // SECURITY_BUILTIN_DOMAIN_RID
- * sub_authority[1] = 544 // DOMAIN_ALIAS_RID_ADMINS
- */
-typedef struct {
- u8 revision;
- u8 sub_authority_count;
- SID_IDENTIFIER_AUTHORITY identifier_authority;
- le32 sub_authority[1]; /* At least one sub_authority. */
-} __attribute__ ((__packed__)) SID;
-
-/*
- * Current constants for SIDs.
- */
-typedef enum {
- SID_REVISION = 1, /* Current revision level. */
- SID_MAX_SUB_AUTHORITIES = 15, /* Maximum number of those. */
- SID_RECOMMENDED_SUB_AUTHORITIES = 1, /* Will change to around 6 in
- a future revision. */
-} SID_CONSTANTS;
-
-/*
- * The predefined ACE types (8-bit, see below).
- */
-enum {
- ACCESS_MIN_MS_ACE_TYPE = 0,
- ACCESS_ALLOWED_ACE_TYPE = 0,
- ACCESS_DENIED_ACE_TYPE = 1,
- SYSTEM_AUDIT_ACE_TYPE = 2,
- SYSTEM_ALARM_ACE_TYPE = 3, /* Not implemented as of Win2k. */
- ACCESS_MAX_MS_V2_ACE_TYPE = 3,
-
- ACCESS_ALLOWED_COMPOUND_ACE_TYPE= 4,
- ACCESS_MAX_MS_V3_ACE_TYPE = 4,
-
- /* The following are Win2k only. */
- ACCESS_MIN_MS_OBJECT_ACE_TYPE = 5,
- ACCESS_ALLOWED_OBJECT_ACE_TYPE = 5,
- ACCESS_DENIED_OBJECT_ACE_TYPE = 6,
- SYSTEM_AUDIT_OBJECT_ACE_TYPE = 7,
- SYSTEM_ALARM_OBJECT_ACE_TYPE = 8,
- ACCESS_MAX_MS_OBJECT_ACE_TYPE = 8,
-
- ACCESS_MAX_MS_V4_ACE_TYPE = 8,
-
- /* This one is for WinNT/2k. */
- ACCESS_MAX_MS_ACE_TYPE = 8,
-} __attribute__ ((__packed__));
-
-typedef u8 ACE_TYPES;
-
-/*
- * The ACE flags (8-bit) for audit and inheritance (see below).
- *
- * SUCCESSFUL_ACCESS_ACE_FLAG is only used with system audit and alarm ACE
- * types to indicate that a message is generated (in Windows!) for successful
- * accesses.
- *
- * FAILED_ACCESS_ACE_FLAG is only used with system audit and alarm ACE types
- * to indicate that a message is generated (in Windows!) for failed accesses.
- */
-enum {
- /* The inheritance flags. */
- OBJECT_INHERIT_ACE = 0x01,
- CONTAINER_INHERIT_ACE = 0x02,
- NO_PROPAGATE_INHERIT_ACE = 0x04,
- INHERIT_ONLY_ACE = 0x08,
- INHERITED_ACE = 0x10, /* Win2k only. */
- VALID_INHERIT_FLAGS = 0x1f,
-
- /* The audit flags. */
- SUCCESSFUL_ACCESS_ACE_FLAG = 0x40,
- FAILED_ACCESS_ACE_FLAG = 0x80,
-} __attribute__ ((__packed__));
-
-typedef u8 ACE_FLAGS;
-
-/*
- * An ACE is an access-control entry in an access-control list (ACL).
- * An ACE defines access to an object for a specific user or group or defines
- * the types of access that generate system-administration messages or alarms
- * for a specific user or group. The user or group is identified by a security
- * identifier (SID).
- *
- * Each ACE starts with an ACE_HEADER structure (aligned on 4-byte boundary),
- * which specifies the type and size of the ACE. The format of the subsequent
- * data depends on the ACE type.
- */
-typedef struct {
-/*Ofs*/
-/* 0*/ ACE_TYPES type; /* Type of the ACE. */
-/* 1*/ ACE_FLAGS flags; /* Flags describing the ACE. */
-/* 2*/ le16 size; /* Size in bytes of the ACE. */
-} __attribute__ ((__packed__)) ACE_HEADER;
-
-/*
- * The access mask (32-bit). Defines the access rights.
- *
- * The specific rights (bits 0 to 15). These depend on the type of the object
- * being secured by the ACE.
- */
-enum {
- /* Specific rights for files and directories are as follows: */
-
- /* Right to read data from the file. (FILE) */
- FILE_READ_DATA = cpu_to_le32(0x00000001),
- /* Right to list contents of a directory. (DIRECTORY) */
- FILE_LIST_DIRECTORY = cpu_to_le32(0x00000001),
-
- /* Right to write data to the file. (FILE) */
- FILE_WRITE_DATA = cpu_to_le32(0x00000002),
- /* Right to create a file in the directory. (DIRECTORY) */
- FILE_ADD_FILE = cpu_to_le32(0x00000002),
-
- /* Right to append data to the file. (FILE) */
- FILE_APPEND_DATA = cpu_to_le32(0x00000004),
- /* Right to create a subdirectory. (DIRECTORY) */
- FILE_ADD_SUBDIRECTORY = cpu_to_le32(0x00000004),
-
- /* Right to read extended attributes. (FILE/DIRECTORY) */
- FILE_READ_EA = cpu_to_le32(0x00000008),
-
- /* Right to write extended attributes. (FILE/DIRECTORY) */
- FILE_WRITE_EA = cpu_to_le32(0x00000010),
-
- /* Right to execute a file. (FILE) */
- FILE_EXECUTE = cpu_to_le32(0x00000020),
- /* Right to traverse the directory. (DIRECTORY) */
- FILE_TRAVERSE = cpu_to_le32(0x00000020),
-
- /*
- * Right to delete a directory and all the files it contains (its
- * children), even if the files are read-only. (DIRECTORY)
- */
- FILE_DELETE_CHILD = cpu_to_le32(0x00000040),
-
- /* Right to read file attributes. (FILE/DIRECTORY) */
- FILE_READ_ATTRIBUTES = cpu_to_le32(0x00000080),
-
- /* Right to change file attributes. (FILE/DIRECTORY) */
- FILE_WRITE_ATTRIBUTES = cpu_to_le32(0x00000100),
-
- /*
- * The standard rights (bits 16 to 23). These are independent of the
- * type of object being secured.
- */
-
- /* Right to delete the object. */
- DELETE = cpu_to_le32(0x00010000),
-
- /*
- * Right to read the information in the object's security descriptor,
- * not including the information in the SACL, i.e. right to read the
- * security descriptor and owner.
- */
- READ_CONTROL = cpu_to_le32(0x00020000),
-
- /* Right to modify the DACL in the object's security descriptor. */
- WRITE_DAC = cpu_to_le32(0x00040000),
-
- /* Right to change the owner in the object's security descriptor. */
- WRITE_OWNER = cpu_to_le32(0x00080000),
-
- /*
- * Right to use the object for synchronization. Enables a process to
- * wait until the object is in the signalled state. Some object types
- * do not support this access right.
- */
- SYNCHRONIZE = cpu_to_le32(0x00100000),
-
- /*
- * The following STANDARD_RIGHTS_* are combinations of the above for
- * convenience and are defined by the Win32 API.
- */
-
- /* These are currently defined to READ_CONTROL. */
- STANDARD_RIGHTS_READ = cpu_to_le32(0x00020000),
- STANDARD_RIGHTS_WRITE = cpu_to_le32(0x00020000),
- STANDARD_RIGHTS_EXECUTE = cpu_to_le32(0x00020000),
-
- /* Combines DELETE, READ_CONTROL, WRITE_DAC, and WRITE_OWNER access. */
- STANDARD_RIGHTS_REQUIRED = cpu_to_le32(0x000f0000),
-
- /*
- * Combines DELETE, READ_CONTROL, WRITE_DAC, WRITE_OWNER, and
- * SYNCHRONIZE access.
- */
- STANDARD_RIGHTS_ALL = cpu_to_le32(0x001f0000),
-
- /*
- * The access system ACL and maximum allowed access types (bits 24 to
- * 25, bits 26 to 27 are reserved).
- */
- ACCESS_SYSTEM_SECURITY = cpu_to_le32(0x01000000),
- MAXIMUM_ALLOWED = cpu_to_le32(0x02000000),
-
- /*
- * The generic rights (bits 28 to 31). These map onto the standard and
- * specific rights.
- */
-
- /* Read, write, and execute access. */
- GENERIC_ALL = cpu_to_le32(0x10000000),
-
- /* Execute access. */
- GENERIC_EXECUTE = cpu_to_le32(0x20000000),
-
- /*
- * Write access. For files, this maps onto:
- * FILE_APPEND_DATA | FILE_WRITE_ATTRIBUTES | FILE_WRITE_DATA |
- * FILE_WRITE_EA | STANDARD_RIGHTS_WRITE | SYNCHRONIZE
- * For directories, the mapping has the same numerical value. See
- * above for the descriptions of the rights granted.
- */
- GENERIC_WRITE = cpu_to_le32(0x40000000),
-
- /*
- * Read access. For files, this maps onto:
- * FILE_READ_ATTRIBUTES | FILE_READ_DATA | FILE_READ_EA |
- * STANDARD_RIGHTS_READ | SYNCHRONIZE
- * For directories, the mapping has the same numberical value. See
- * above for the descriptions of the rights granted.
- */
- GENERIC_READ = cpu_to_le32(0x80000000),
-};
-
-typedef le32 ACCESS_MASK;
-
-/*
- * The generic mapping array. Used to denote the mapping of each generic
- * access right to a specific access mask.
- *
- * FIXME: What exactly is this and what is it for? (AIA)
- */
-typedef struct {
- ACCESS_MASK generic_read;
- ACCESS_MASK generic_write;
- ACCESS_MASK generic_execute;
- ACCESS_MASK generic_all;
-} __attribute__ ((__packed__)) GENERIC_MAPPING;
-
-/*
- * The predefined ACE type structures are as defined below.
- */
-
-/*
- * ACCESS_ALLOWED_ACE, ACCESS_DENIED_ACE, SYSTEM_AUDIT_ACE, SYSTEM_ALARM_ACE
- */
-typedef struct {
-/* 0 ACE_HEADER; -- Unfolded here as gcc doesn't like unnamed structs. */
- ACE_TYPES type; /* Type of the ACE. */
- ACE_FLAGS flags; /* Flags describing the ACE. */
- le16 size; /* Size in bytes of the ACE. */
-/* 4*/ ACCESS_MASK mask; /* Access mask associated with the ACE. */
-
-/* 8*/ SID sid; /* The SID associated with the ACE. */
-} __attribute__ ((__packed__)) ACCESS_ALLOWED_ACE, ACCESS_DENIED_ACE,
- SYSTEM_AUDIT_ACE, SYSTEM_ALARM_ACE;
-
-/*
- * The object ACE flags (32-bit).
- */
-enum {
- ACE_OBJECT_TYPE_PRESENT = cpu_to_le32(1),
- ACE_INHERITED_OBJECT_TYPE_PRESENT = cpu_to_le32(2),
-};
-
-typedef le32 OBJECT_ACE_FLAGS;
-
-typedef struct {
-/* 0 ACE_HEADER; -- Unfolded here as gcc doesn't like unnamed structs. */
- ACE_TYPES type; /* Type of the ACE. */
- ACE_FLAGS flags; /* Flags describing the ACE. */
- le16 size; /* Size in bytes of the ACE. */
-/* 4*/ ACCESS_MASK mask; /* Access mask associated with the ACE. */
-
-/* 8*/ OBJECT_ACE_FLAGS object_flags; /* Flags describing the object ACE. */
-/* 12*/ GUID object_type;
-/* 28*/ GUID inherited_object_type;
-
-/* 44*/ SID sid; /* The SID associated with the ACE. */
-} __attribute__ ((__packed__)) ACCESS_ALLOWED_OBJECT_ACE,
- ACCESS_DENIED_OBJECT_ACE,
- SYSTEM_AUDIT_OBJECT_ACE,
- SYSTEM_ALARM_OBJECT_ACE;
-
-/*
- * An ACL is an access-control list (ACL).
- * An ACL starts with an ACL header structure, which specifies the size of
- * the ACL and the number of ACEs it contains. The ACL header is followed by
- * zero or more access control entries (ACEs). The ACL as well as each ACE
- * are aligned on 4-byte boundaries.
- */
-typedef struct {
- u8 revision; /* Revision of this ACL. */
- u8 alignment1;
- le16 size; /* Allocated space in bytes for ACL. Includes this
- header, the ACEs and the remaining free space. */
- le16 ace_count; /* Number of ACEs in the ACL. */
- le16 alignment2;
-/* sizeof() = 8 bytes */
-} __attribute__ ((__packed__)) ACL;
-
-/*
- * Current constants for ACLs.
- */
-typedef enum {
- /* Current revision. */
- ACL_REVISION = 2,
- ACL_REVISION_DS = 4,
-
- /* History of revisions. */
- ACL_REVISION1 = 1,
- MIN_ACL_REVISION = 2,
- ACL_REVISION2 = 2,
- ACL_REVISION3 = 3,
- ACL_REVISION4 = 4,
- MAX_ACL_REVISION = 4,
-} ACL_CONSTANTS;
-
-/*
- * The security descriptor control flags (16-bit).
- *
- * SE_OWNER_DEFAULTED - This boolean flag, when set, indicates that the SID
- * pointed to by the Owner field was provided by a defaulting mechanism
- * rather than explicitly provided by the original provider of the
- * security descriptor. This may affect the treatment of the SID with
- * respect to inheritance of an owner.
- *
- * SE_GROUP_DEFAULTED - This boolean flag, when set, indicates that the SID in
- * the Group field was provided by a defaulting mechanism rather than
- * explicitly provided by the original provider of the security
- * descriptor. This may affect the treatment of the SID with respect to
- * inheritance of a primary group.
- *
- * SE_DACL_PRESENT - This boolean flag, when set, indicates that the security
- * descriptor contains a discretionary ACL. If this flag is set and the
- * Dacl field of the SECURITY_DESCRIPTOR is null, then a null ACL is
- * explicitly being specified.
- *
- * SE_DACL_DEFAULTED - This boolean flag, when set, indicates that the ACL
- * pointed to by the Dacl field was provided by a defaulting mechanism
- * rather than explicitly provided by the original provider of the
- * security descriptor. This may affect the treatment of the ACL with
- * respect to inheritance of an ACL. This flag is ignored if the
- * DaclPresent flag is not set.
- *
- * SE_SACL_PRESENT - This boolean flag, when set, indicates that the security
- * descriptor contains a system ACL pointed to by the Sacl field. If this
- * flag is set and the Sacl field of the SECURITY_DESCRIPTOR is null, then
- * an empty (but present) ACL is being specified.
- *
- * SE_SACL_DEFAULTED - This boolean flag, when set, indicates that the ACL
- * pointed to by the Sacl field was provided by a defaulting mechanism
- * rather than explicitly provided by the original provider of the
- * security descriptor. This may affect the treatment of the ACL with
- * respect to inheritance of an ACL. This flag is ignored if the
- * SaclPresent flag is not set.
- *
- * SE_SELF_RELATIVE - This boolean flag, when set, indicates that the security
- * descriptor is in self-relative form. In this form, all fields of the
- * security descriptor are contiguous in memory and all pointer fields are
- * expressed as offsets from the beginning of the security descriptor.
- */
-enum {
- SE_OWNER_DEFAULTED = cpu_to_le16(0x0001),
- SE_GROUP_DEFAULTED = cpu_to_le16(0x0002),
- SE_DACL_PRESENT = cpu_to_le16(0x0004),
- SE_DACL_DEFAULTED = cpu_to_le16(0x0008),
-
- SE_SACL_PRESENT = cpu_to_le16(0x0010),
- SE_SACL_DEFAULTED = cpu_to_le16(0x0020),
-
- SE_DACL_AUTO_INHERIT_REQ = cpu_to_le16(0x0100),
- SE_SACL_AUTO_INHERIT_REQ = cpu_to_le16(0x0200),
- SE_DACL_AUTO_INHERITED = cpu_to_le16(0x0400),
- SE_SACL_AUTO_INHERITED = cpu_to_le16(0x0800),
-
- SE_DACL_PROTECTED = cpu_to_le16(0x1000),
- SE_SACL_PROTECTED = cpu_to_le16(0x2000),
- SE_RM_CONTROL_VALID = cpu_to_le16(0x4000),
- SE_SELF_RELATIVE = cpu_to_le16(0x8000)
-} __attribute__ ((__packed__));
-
-typedef le16 SECURITY_DESCRIPTOR_CONTROL;
-
-/*
- * Self-relative security descriptor. Contains the owner and group SIDs as well
- * as the sacl and dacl ACLs inside the security descriptor itself.
- */
-typedef struct {
- u8 revision; /* Revision level of the security descriptor. */
- u8 alignment;
- SECURITY_DESCRIPTOR_CONTROL control; /* Flags qualifying the type of
- the descriptor as well as the following fields. */
- le32 owner; /* Byte offset to a SID representing an object's
- owner. If this is NULL, no owner SID is present in
- the descriptor. */
- le32 group; /* Byte offset to a SID representing an object's
- primary group. If this is NULL, no primary group
- SID is present in the descriptor. */
- le32 sacl; /* Byte offset to a system ACL. Only valid, if
- SE_SACL_PRESENT is set in the control field. If
- SE_SACL_PRESENT is set but sacl is NULL, a NULL ACL
- is specified. */
- le32 dacl; /* Byte offset to a discretionary ACL. Only valid, if
- SE_DACL_PRESENT is set in the control field. If
- SE_DACL_PRESENT is set but dacl is NULL, a NULL ACL
- (unconditionally granting access) is specified. */
-/* sizeof() = 0x14 bytes */
-} __attribute__ ((__packed__)) SECURITY_DESCRIPTOR_RELATIVE;
-
-/*
- * Absolute security descriptor. Does not contain the owner and group SIDs, nor
- * the sacl and dacl ACLs inside the security descriptor. Instead, it contains
- * pointers to these structures in memory. Obviously, absolute security
- * descriptors are only useful for in memory representations of security
- * descriptors. On disk, a self-relative security descriptor is used.
- */
-typedef struct {
- u8 revision; /* Revision level of the security descriptor. */
- u8 alignment;
- SECURITY_DESCRIPTOR_CONTROL control; /* Flags qualifying the type of
- the descriptor as well as the following fields. */
- SID *owner; /* Points to a SID representing an object's owner. If
- this is NULL, no owner SID is present in the
- descriptor. */
- SID *group; /* Points to a SID representing an object's primary
- group. If this is NULL, no primary group SID is
- present in the descriptor. */
- ACL *sacl; /* Points to a system ACL. Only valid, if
- SE_SACL_PRESENT is set in the control field. If
- SE_SACL_PRESENT is set but sacl is NULL, a NULL ACL
- is specified. */
- ACL *dacl; /* Points to a discretionary ACL. Only valid, if
- SE_DACL_PRESENT is set in the control field. If
- SE_DACL_PRESENT is set but dacl is NULL, a NULL ACL
- (unconditionally granting access) is specified. */
-} __attribute__ ((__packed__)) SECURITY_DESCRIPTOR;
-
-/*
- * Current constants for security descriptors.
- */
-typedef enum {
- /* Current revision. */
- SECURITY_DESCRIPTOR_REVISION = 1,
- SECURITY_DESCRIPTOR_REVISION1 = 1,
-
- /* The sizes of both the absolute and relative security descriptors is
- the same as pointers, at least on ia32 architecture are 32-bit. */
- SECURITY_DESCRIPTOR_MIN_LENGTH = sizeof(SECURITY_DESCRIPTOR),
-} SECURITY_DESCRIPTOR_CONSTANTS;
-
-/*
- * Attribute: Security descriptor (0x50). A standard self-relative security
- * descriptor.
- *
- * NOTE: Can be resident or non-resident.
- * NOTE: Not used in NTFS 3.0+, as security descriptors are stored centrally
- * in FILE_Secure and the correct descriptor is found using the security_id
- * from the standard information attribute.
- */
-typedef SECURITY_DESCRIPTOR_RELATIVE SECURITY_DESCRIPTOR_ATTR;
-
-/*
- * On NTFS 3.0+, all security descriptors are stored in FILE_Secure. Only one
- * referenced instance of each unique security descriptor is stored.
- *
- * FILE_Secure contains no unnamed data attribute, i.e. it has zero length. It
- * does, however, contain two indexes ($SDH and $SII) as well as a named data
- * stream ($SDS).
- *
- * Every unique security descriptor is assigned a unique security identifier
- * (security_id, not to be confused with a SID). The security_id is unique for
- * the NTFS volume and is used as an index into the $SII index, which maps
- * security_ids to the security descriptor's storage location within the $SDS
- * data attribute. The $SII index is sorted by ascending security_id.
- *
- * A simple hash is computed from each security descriptor. This hash is used
- * as an index into the $SDH index, which maps security descriptor hashes to
- * the security descriptor's storage location within the $SDS data attribute.
- * The $SDH index is sorted by security descriptor hash and is stored in a B+
- * tree. When searching $SDH (with the intent of determining whether or not a
- * new security descriptor is already present in the $SDS data stream), if a
- * matching hash is found, but the security descriptors do not match, the
- * search in the $SDH index is continued, searching for a next matching hash.
- *
- * When a precise match is found, the security_id coresponding to the security
- * descriptor in the $SDS attribute is read from the found $SDH index entry and
- * is stored in the $STANDARD_INFORMATION attribute of the file/directory to
- * which the security descriptor is being applied. The $STANDARD_INFORMATION
- * attribute is present in all base mft records (i.e. in all files and
- * directories).
- *
- * If a match is not found, the security descriptor is assigned a new unique
- * security_id and is added to the $SDS data attribute. Then, entries
- * referencing the this security descriptor in the $SDS data attribute are
- * added to the $SDH and $SII indexes.
- *
- * Note: Entries are never deleted from FILE_Secure, even if nothing
- * references an entry any more.
- */
-
-/*
- * This header precedes each security descriptor in the $SDS data stream.
- * This is also the index entry data part of both the $SII and $SDH indexes.
- */
-typedef struct {
- le32 hash; /* Hash of the security descriptor. */
- le32 security_id; /* The security_id assigned to the descriptor. */
- le64 offset; /* Byte offset of this entry in the $SDS stream. */
- le32 length; /* Size in bytes of this entry in $SDS stream. */
-} __attribute__ ((__packed__)) SECURITY_DESCRIPTOR_HEADER;
-
-/*
- * The $SDS data stream contains the security descriptors, aligned on 16-byte
- * boundaries, sorted by security_id in a B+ tree. Security descriptors cannot
- * cross 256kib boundaries (this restriction is imposed by the Windows cache
- * manager). Each security descriptor is contained in a SDS_ENTRY structure.
- * Also, each security descriptor is stored twice in the $SDS stream with a
- * fixed offset of 0x40000 bytes (256kib, the Windows cache manager's max size)
- * between them; i.e. if a SDS_ENTRY specifies an offset of 0x51d0, then the
- * first copy of the security descriptor will be at offset 0x51d0 in the
- * $SDS data stream and the second copy will be at offset 0x451d0.
- */
-typedef struct {
-/*Ofs*/
-/* 0 SECURITY_DESCRIPTOR_HEADER; -- Unfolded here as gcc doesn't like
- unnamed structs. */
- le32 hash; /* Hash of the security descriptor. */
- le32 security_id; /* The security_id assigned to the descriptor. */
- le64 offset; /* Byte offset of this entry in the $SDS stream. */
- le32 length; /* Size in bytes of this entry in $SDS stream. */
-/* 20*/ SECURITY_DESCRIPTOR_RELATIVE sid; /* The self-relative security
- descriptor. */
-} __attribute__ ((__packed__)) SDS_ENTRY;
-
-/*
- * The index entry key used in the $SII index. The collation type is
- * COLLATION_NTOFS_ULONG.
- */
-typedef struct {
- le32 security_id; /* The security_id assigned to the descriptor. */
-} __attribute__ ((__packed__)) SII_INDEX_KEY;
-
-/*
- * The index entry key used in the $SDH index. The keys are sorted first by
- * hash and then by security_id. The collation rule is
- * COLLATION_NTOFS_SECURITY_HASH.
- */
-typedef struct {
- le32 hash; /* Hash of the security descriptor. */
- le32 security_id; /* The security_id assigned to the descriptor. */
-} __attribute__ ((__packed__)) SDH_INDEX_KEY;
-
-/*
- * Attribute: Volume name (0x60).
- *
- * NOTE: Always resident.
- * NOTE: Present only in FILE_Volume.
- */
-typedef struct {
- ntfschar name[0]; /* The name of the volume in Unicode. */
-} __attribute__ ((__packed__)) VOLUME_NAME;
-
-/*
- * Possible flags for the volume (16-bit).
- */
-enum {
- VOLUME_IS_DIRTY = cpu_to_le16(0x0001),
- VOLUME_RESIZE_LOG_FILE = cpu_to_le16(0x0002),
- VOLUME_UPGRADE_ON_MOUNT = cpu_to_le16(0x0004),
- VOLUME_MOUNTED_ON_NT4 = cpu_to_le16(0x0008),
-
- VOLUME_DELETE_USN_UNDERWAY = cpu_to_le16(0x0010),
- VOLUME_REPAIR_OBJECT_ID = cpu_to_le16(0x0020),
-
- VOLUME_CHKDSK_UNDERWAY = cpu_to_le16(0x4000),
- VOLUME_MODIFIED_BY_CHKDSK = cpu_to_le16(0x8000),
-
- VOLUME_FLAGS_MASK = cpu_to_le16(0xc03f),
-
- /* To make our life easier when checking if we must mount read-only. */
- VOLUME_MUST_MOUNT_RO_MASK = cpu_to_le16(0xc027),
-} __attribute__ ((__packed__));
-
-typedef le16 VOLUME_FLAGS;
-
-/*
- * Attribute: Volume information (0x70).
- *
- * NOTE: Always resident.
- * NOTE: Present only in FILE_Volume.
- * NOTE: Windows 2000 uses NTFS 3.0 while Windows NT4 service pack 6a uses
- * NTFS 1.2. I haven't personally seen other values yet.
- */
-typedef struct {
- le64 reserved; /* Not used (yet?). */
- u8 major_ver; /* Major version of the ntfs format. */
- u8 minor_ver; /* Minor version of the ntfs format. */
- VOLUME_FLAGS flags; /* Bit array of VOLUME_* flags. */
-} __attribute__ ((__packed__)) VOLUME_INFORMATION;
-
-/*
- * Attribute: Data attribute (0x80).
- *
- * NOTE: Can be resident or non-resident.
- *
- * Data contents of a file (i.e. the unnamed stream) or of a named stream.
- */
-typedef struct {
- u8 data[0]; /* The file's data contents. */
-} __attribute__ ((__packed__)) DATA_ATTR;
-
-/*
- * Index header flags (8-bit).
- */
-enum {
- /*
- * When index header is in an index root attribute:
- */
- SMALL_INDEX = 0, /* The index is small enough to fit inside the index
- root attribute and there is no index allocation
- attribute present. */
- LARGE_INDEX = 1, /* The index is too large to fit in the index root
- attribute and/or an index allocation attribute is
- present. */
- /*
- * When index header is in an index block, i.e. is part of index
- * allocation attribute:
- */
- LEAF_NODE = 0, /* This is a leaf node, i.e. there are no more nodes
- branching off it. */
- INDEX_NODE = 1, /* This node indexes other nodes, i.e. it is not a leaf
- node. */
- NODE_MASK = 1, /* Mask for accessing the *_NODE bits. */
-} __attribute__ ((__packed__));
-
-typedef u8 INDEX_HEADER_FLAGS;
-
-/*
- * This is the header for indexes, describing the INDEX_ENTRY records, which
- * follow the INDEX_HEADER. Together the index header and the index entries
- * make up a complete index.
- *
- * IMPORTANT NOTE: The offset, length and size structure members are counted
- * relative to the start of the index header structure and not relative to the
- * start of the index root or index allocation structures themselves.
- */
-typedef struct {
- le32 entries_offset; /* Byte offset to first INDEX_ENTRY
- aligned to 8-byte boundary. */
- le32 index_length; /* Data size of the index in bytes,
- i.e. bytes used from allocated
- size, aligned to 8-byte boundary. */
- le32 allocated_size; /* Byte size of this index (block),
- multiple of 8 bytes. */
- /* NOTE: For the index root attribute, the above two numbers are always
- equal, as the attribute is resident and it is resized as needed. In
- the case of the index allocation attribute the attribute is not
- resident and hence the allocated_size is a fixed value and must
- equal the index_block_size specified by the INDEX_ROOT attribute
- corresponding to the INDEX_ALLOCATION attribute this INDEX_BLOCK
- belongs to. */
- INDEX_HEADER_FLAGS flags; /* Bit field of INDEX_HEADER_FLAGS. */
- u8 reserved[3]; /* Reserved/align to 8-byte boundary. */
-} __attribute__ ((__packed__)) INDEX_HEADER;
-
-/*
- * Attribute: Index root (0x90).
- *
- * NOTE: Always resident.
- *
- * This is followed by a sequence of index entries (INDEX_ENTRY structures)
- * as described by the index header.
- *
- * When a directory is small enough to fit inside the index root then this
- * is the only attribute describing the directory. When the directory is too
- * large to fit in the index root, on the other hand, two additional attributes
- * are present: an index allocation attribute, containing sub-nodes of the B+
- * directory tree (see below), and a bitmap attribute, describing which virtual
- * cluster numbers (vcns) in the index allocation attribute are in use by an
- * index block.
- *
- * NOTE: The root directory (FILE_root) contains an entry for itself. Other
- * directories do not contain entries for themselves, though.
- */
-typedef struct {
- ATTR_TYPE type; /* Type of the indexed attribute. Is
- $FILE_NAME for directories, zero
- for view indexes. No other values
- allowed. */
- COLLATION_RULE collation_rule; /* Collation rule used to sort the
- index entries. If type is $FILE_NAME,
- this must be COLLATION_FILE_NAME. */
- le32 index_block_size; /* Size of each index block in bytes (in
- the index allocation attribute). */
- u8 clusters_per_index_block; /* Cluster size of each index block (in
- the index allocation attribute), when
- an index block is >= than a cluster,
- otherwise this will be the log of
- the size (like how the encoding of
- the mft record size and the index
- record size found in the boot sector
- work). Has to be a power of 2. */
- u8 reserved[3]; /* Reserved/align to 8-byte boundary. */
- INDEX_HEADER index; /* Index header describing the
- following index entries. */
-} __attribute__ ((__packed__)) INDEX_ROOT;
-
-/*
- * Attribute: Index allocation (0xa0).
- *
- * NOTE: Always non-resident (doesn't make sense to be resident anyway!).
- *
- * This is an array of index blocks. Each index block starts with an
- * INDEX_BLOCK structure containing an index header, followed by a sequence of
- * index entries (INDEX_ENTRY structures), as described by the INDEX_HEADER.
- */
-typedef struct {
-/* 0 NTFS_RECORD; -- Unfolded here as gcc doesn't like unnamed structs. */
- NTFS_RECORD_TYPE magic; /* Magic is "INDX". */
- le16 usa_ofs; /* See NTFS_RECORD definition. */
- le16 usa_count; /* See NTFS_RECORD definition. */
-
-/* 8*/ sle64 lsn; /* $LogFile sequence number of the last
- modification of this index block. */
-/* 16*/ leVCN index_block_vcn; /* Virtual cluster number of the index block.
- If the cluster_size on the volume is <= the
- index_block_size of the directory,
- index_block_vcn counts in units of clusters,
- and in units of sectors otherwise. */
-/* 24*/ INDEX_HEADER index; /* Describes the following index entries. */
-/* sizeof()= 40 (0x28) bytes */
-/*
- * When creating the index block, we place the update sequence array at this
- * offset, i.e. before we start with the index entries. This also makes sense,
- * otherwise we could run into problems with the update sequence array
- * containing in itself the last two bytes of a sector which would mean that
- * multi sector transfer protection wouldn't work. As you can't protect data
- * by overwriting it since you then can't get it back...
- * When reading use the data from the ntfs record header.
- */
-} __attribute__ ((__packed__)) INDEX_BLOCK;
-
-typedef INDEX_BLOCK INDEX_ALLOCATION;
-
-/*
- * The system file FILE_Extend/$Reparse contains an index named $R listing
- * all reparse points on the volume. The index entry keys are as defined
- * below. Note, that there is no index data associated with the index entries.
- *
- * The index entries are sorted by the index key file_id. The collation rule is
- * COLLATION_NTOFS_ULONGS. FIXME: Verify whether the reparse_tag is not the
- * primary key / is not a key at all. (AIA)
- */
-typedef struct {
- le32 reparse_tag; /* Reparse point type (inc. flags). */
- leMFT_REF file_id; /* Mft record of the file containing the
- reparse point attribute. */
-} __attribute__ ((__packed__)) REPARSE_INDEX_KEY;
-
-/*
- * Quota flags (32-bit).
- *
- * The user quota flags. Names explain meaning.
- */
-enum {
- QUOTA_FLAG_DEFAULT_LIMITS = cpu_to_le32(0x00000001),
- QUOTA_FLAG_LIMIT_REACHED = cpu_to_le32(0x00000002),
- QUOTA_FLAG_ID_DELETED = cpu_to_le32(0x00000004),
-
- QUOTA_FLAG_USER_MASK = cpu_to_le32(0x00000007),
- /* This is a bit mask for the user quota flags. */
-
- /*
- * These flags are only present in the quota defaults index entry, i.e.
- * in the entry where owner_id = QUOTA_DEFAULTS_ID.
- */
- QUOTA_FLAG_TRACKING_ENABLED = cpu_to_le32(0x00000010),
- QUOTA_FLAG_ENFORCEMENT_ENABLED = cpu_to_le32(0x00000020),
- QUOTA_FLAG_TRACKING_REQUESTED = cpu_to_le32(0x00000040),
- QUOTA_FLAG_LOG_THRESHOLD = cpu_to_le32(0x00000080),
-
- QUOTA_FLAG_LOG_LIMIT = cpu_to_le32(0x00000100),
- QUOTA_FLAG_OUT_OF_DATE = cpu_to_le32(0x00000200),
- QUOTA_FLAG_CORRUPT = cpu_to_le32(0x00000400),
- QUOTA_FLAG_PENDING_DELETES = cpu_to_le32(0x00000800),
-};
-
-typedef le32 QUOTA_FLAGS;
-
-/*
- * The system file FILE_Extend/$Quota contains two indexes $O and $Q. Quotas
- * are on a per volume and per user basis.
- *
- * The $Q index contains one entry for each existing user_id on the volume. The
- * index key is the user_id of the user/group owning this quota control entry,
- * i.e. the key is the owner_id. The user_id of the owner of a file, i.e. the
- * owner_id, is found in the standard information attribute. The collation rule
- * for $Q is COLLATION_NTOFS_ULONG.
- *
- * The $O index contains one entry for each user/group who has been assigned
- * a quota on that volume. The index key holds the SID of the user_id the
- * entry belongs to, i.e. the owner_id. The collation rule for $O is
- * COLLATION_NTOFS_SID.
- *
- * The $O index entry data is the user_id of the user corresponding to the SID.
- * This user_id is used as an index into $Q to find the quota control entry
- * associated with the SID.
- *
- * The $Q index entry data is the quota control entry and is defined below.
- */
-typedef struct {
- le32 version; /* Currently equals 2. */
- QUOTA_FLAGS flags; /* Flags describing this quota entry. */
- le64 bytes_used; /* How many bytes of the quota are in use. */
- sle64 change_time; /* Last time this quota entry was changed. */
- sle64 threshold; /* Soft quota (-1 if not limited). */
- sle64 limit; /* Hard quota (-1 if not limited). */
- sle64 exceeded_time; /* How long the soft quota has been exceeded. */
- SID sid; /* The SID of the user/object associated with
- this quota entry. Equals zero for the quota
- defaults entry (and in fact on a WinXP
- volume, it is not present at all). */
-} __attribute__ ((__packed__)) QUOTA_CONTROL_ENTRY;
-
-/*
- * Predefined owner_id values (32-bit).
- */
-enum {
- QUOTA_INVALID_ID = cpu_to_le32(0x00000000),
- QUOTA_DEFAULTS_ID = cpu_to_le32(0x00000001),
- QUOTA_FIRST_USER_ID = cpu_to_le32(0x00000100),
-};
-
-/*
- * Current constants for quota control entries.
- */
-typedef enum {
- /* Current version. */
- QUOTA_VERSION = 2,
-} QUOTA_CONTROL_ENTRY_CONSTANTS;
-
-/*
- * Index entry flags (16-bit).
- */
-enum {
- INDEX_ENTRY_NODE = cpu_to_le16(1), /* This entry contains a
- sub-node, i.e. a reference to an index block in form of
- a virtual cluster number (see below). */
- INDEX_ENTRY_END = cpu_to_le16(2), /* This signifies the last
- entry in an index block. The index entry does not
- represent a file but it can point to a sub-node. */
-
- INDEX_ENTRY_SPACE_FILLER = cpu_to_le16(0xffff), /* gcc: Force
- enum bit width to 16-bit. */
-} __attribute__ ((__packed__));
-
-typedef le16 INDEX_ENTRY_FLAGS;
-
-/*
- * This the index entry header (see below).
- */
-typedef struct {
-/* 0*/ union {
- struct { /* Only valid when INDEX_ENTRY_END is not set. */
- leMFT_REF indexed_file; /* The mft reference of the file
- described by this index
- entry. Used for directory
- indexes. */
- } __attribute__ ((__packed__)) dir;
- struct { /* Used for views/indexes to find the entry's data. */
- le16 data_offset; /* Data byte offset from this
- INDEX_ENTRY. Follows the
- index key. */
- le16 data_length; /* Data length in bytes. */
- le32 reservedV; /* Reserved (zero). */
- } __attribute__ ((__packed__)) vi;
- } __attribute__ ((__packed__)) data;
-/* 8*/ le16 length; /* Byte size of this index entry, multiple of
- 8-bytes. */
-/* 10*/ le16 key_length; /* Byte size of the key value, which is in the
- index entry. It follows field reserved. Not
- multiple of 8-bytes. */
-/* 12*/ INDEX_ENTRY_FLAGS flags; /* Bit field of INDEX_ENTRY_* flags. */
-/* 14*/ le16 reserved; /* Reserved/align to 8-byte boundary. */
-/* sizeof() = 16 bytes */
-} __attribute__ ((__packed__)) INDEX_ENTRY_HEADER;
-
-/*
- * This is an index entry. A sequence of such entries follows each INDEX_HEADER
- * structure. Together they make up a complete index. The index follows either
- * an index root attribute or an index allocation attribute.
- *
- * NOTE: Before NTFS 3.0 only filename attributes were indexed.
- */
-typedef struct {
-/*Ofs*/
-/* 0 INDEX_ENTRY_HEADER; -- Unfolded here as gcc dislikes unnamed structs. */
- union {
- struct { /* Only valid when INDEX_ENTRY_END is not set. */
- leMFT_REF indexed_file; /* The mft reference of the file
- described by this index
- entry. Used for directory
- indexes. */
- } __attribute__ ((__packed__)) dir;
- struct { /* Used for views/indexes to find the entry's data. */
- le16 data_offset; /* Data byte offset from this
- INDEX_ENTRY. Follows the
- index key. */
- le16 data_length; /* Data length in bytes. */
- le32 reservedV; /* Reserved (zero). */
- } __attribute__ ((__packed__)) vi;
- } __attribute__ ((__packed__)) data;
- le16 length; /* Byte size of this index entry, multiple of
- 8-bytes. */
- le16 key_length; /* Byte size of the key value, which is in the
- index entry. It follows field reserved. Not
- multiple of 8-bytes. */
- INDEX_ENTRY_FLAGS flags; /* Bit field of INDEX_ENTRY_* flags. */
- le16 reserved; /* Reserved/align to 8-byte boundary. */
-
-/* 16*/ union { /* The key of the indexed attribute. NOTE: Only present
- if INDEX_ENTRY_END bit in flags is not set. NOTE: On
- NTFS versions before 3.0 the only valid key is the
- FILE_NAME_ATTR. On NTFS 3.0+ the following
- additional index keys are defined: */
- FILE_NAME_ATTR file_name;/* $I30 index in directories. */
- SII_INDEX_KEY sii; /* $SII index in $Secure. */
- SDH_INDEX_KEY sdh; /* $SDH index in $Secure. */
- GUID object_id; /* $O index in FILE_Extend/$ObjId: The
- object_id of the mft record found in
- the data part of the index. */
- REPARSE_INDEX_KEY reparse; /* $R index in
- FILE_Extend/$Reparse. */
- SID sid; /* $O index in FILE_Extend/$Quota:
- SID of the owner of the user_id. */
- le32 owner_id; /* $Q index in FILE_Extend/$Quota:
- user_id of the owner of the quota
- control entry in the data part of
- the index. */
- } __attribute__ ((__packed__)) key;
- /* The (optional) index data is inserted here when creating. */
- // leVCN vcn; /* If INDEX_ENTRY_NODE bit in flags is set, the last
- // eight bytes of this index entry contain the virtual
- // cluster number of the index block that holds the
- // entries immediately preceding the current entry (the
- // vcn references the corresponding cluster in the data
- // of the non-resident index allocation attribute). If
- // the key_length is zero, then the vcn immediately
- // follows the INDEX_ENTRY_HEADER. Regardless of
- // key_length, the address of the 8-byte boundary
- // aligned vcn of INDEX_ENTRY{_HEADER} *ie is given by
- // (char*)ie + le16_to_cpu(ie*)->length) - sizeof(VCN),
- // where sizeof(VCN) can be hardcoded as 8 if wanted. */
-} __attribute__ ((__packed__)) INDEX_ENTRY;
-
-/*
- * Attribute: Bitmap (0xb0).
- *
- * Contains an array of bits (aka a bitfield).
- *
- * When used in conjunction with the index allocation attribute, each bit
- * corresponds to one index block within the index allocation attribute. Thus
- * the number of bits in the bitmap * index block size / cluster size is the
- * number of clusters in the index allocation attribute.
- */
-typedef struct {
- u8 bitmap[0]; /* Array of bits. */
-} __attribute__ ((__packed__)) BITMAP_ATTR;
-
-/*
- * The reparse point tag defines the type of the reparse point. It also
- * includes several flags, which further describe the reparse point.
- *
- * The reparse point tag is an unsigned 32-bit value divided in three parts:
- *
- * 1. The least significant 16 bits (i.e. bits 0 to 15) specifiy the type of
- * the reparse point.
- * 2. The 13 bits after this (i.e. bits 16 to 28) are reserved for future use.
- * 3. The most significant three bits are flags describing the reparse point.
- * They are defined as follows:
- * bit 29: Name surrogate bit. If set, the filename is an alias for
- * another object in the system.
- * bit 30: High-latency bit. If set, accessing the first byte of data will
- * be slow. (E.g. the data is stored on a tape drive.)
- * bit 31: Microsoft bit. If set, the tag is owned by Microsoft. User
- * defined tags have to use zero here.
- *
- * These are the predefined reparse point tags:
- */
-enum {
- IO_REPARSE_TAG_IS_ALIAS = cpu_to_le32(0x20000000),
- IO_REPARSE_TAG_IS_HIGH_LATENCY = cpu_to_le32(0x40000000),
- IO_REPARSE_TAG_IS_MICROSOFT = cpu_to_le32(0x80000000),
-
- IO_REPARSE_TAG_RESERVED_ZERO = cpu_to_le32(0x00000000),
- IO_REPARSE_TAG_RESERVED_ONE = cpu_to_le32(0x00000001),
- IO_REPARSE_TAG_RESERVED_RANGE = cpu_to_le32(0x00000001),
-
- IO_REPARSE_TAG_NSS = cpu_to_le32(0x68000005),
- IO_REPARSE_TAG_NSS_RECOVER = cpu_to_le32(0x68000006),
- IO_REPARSE_TAG_SIS = cpu_to_le32(0x68000007),
- IO_REPARSE_TAG_DFS = cpu_to_le32(0x68000008),
-
- IO_REPARSE_TAG_MOUNT_POINT = cpu_to_le32(0x88000003),
-
- IO_REPARSE_TAG_HSM = cpu_to_le32(0xa8000004),
-
- IO_REPARSE_TAG_SYMBOLIC_LINK = cpu_to_le32(0xe8000000),
-
- IO_REPARSE_TAG_VALID_VALUES = cpu_to_le32(0xe000ffff),
-};
-
-/*
- * Attribute: Reparse point (0xc0).
- *
- * NOTE: Can be resident or non-resident.
- */
-typedef struct {
- le32 reparse_tag; /* Reparse point type (inc. flags). */
- le16 reparse_data_length; /* Byte size of reparse data. */
- le16 reserved; /* Align to 8-byte boundary. */
- u8 reparse_data[0]; /* Meaning depends on reparse_tag. */
-} __attribute__ ((__packed__)) REPARSE_POINT;
-
-/*
- * Attribute: Extended attribute (EA) information (0xd0).
- *
- * NOTE: Always resident. (Is this true???)
- */
-typedef struct {
- le16 ea_length; /* Byte size of the packed extended
- attributes. */
- le16 need_ea_count; /* The number of extended attributes which have
- the NEED_EA bit set. */
- le32 ea_query_length; /* Byte size of the buffer required to query
- the extended attributes when calling
- ZwQueryEaFile() in Windows NT/2k. I.e. the
- byte size of the unpacked extended
- attributes. */
-} __attribute__ ((__packed__)) EA_INFORMATION;
-
-/*
- * Extended attribute flags (8-bit).
- */
-enum {
- NEED_EA = 0x80 /* If set the file to which the EA belongs
- cannot be interpreted without understanding
- the associates extended attributes. */
-} __attribute__ ((__packed__));
-
-typedef u8 EA_FLAGS;
-
-/*
- * Attribute: Extended attribute (EA) (0xe0).
- *
- * NOTE: Can be resident or non-resident.
- *
- * Like the attribute list and the index buffer list, the EA attribute value is
- * a sequence of EA_ATTR variable length records.
- */
-typedef struct {
- le32 next_entry_offset; /* Offset to the next EA_ATTR. */
- EA_FLAGS flags; /* Flags describing the EA. */
- u8 ea_name_length; /* Length of the name of the EA in bytes
- excluding the '\0' byte terminator. */
- le16 ea_value_length; /* Byte size of the EA's value. */
- u8 ea_name[0]; /* Name of the EA. Note this is ASCII, not
- Unicode and it is zero terminated. */
- u8 ea_value[0]; /* The value of the EA. Immediately follows
- the name. */
-} __attribute__ ((__packed__)) EA_ATTR;
-
-/*
- * Attribute: Property set (0xf0).
- *
- * Intended to support Native Structure Storage (NSS) - a feature removed from
- * NTFS 3.0 during beta testing.
- */
-typedef struct {
- /* Irrelevant as feature unused. */
-} __attribute__ ((__packed__)) PROPERTY_SET;
-
-/*
- * Attribute: Logged utility stream (0x100).
- *
- * NOTE: Can be resident or non-resident.
- *
- * Operations on this attribute are logged to the journal ($LogFile) like
- * normal metadata changes.
- *
- * Used by the Encrypting File System (EFS). All encrypted files have this
- * attribute with the name $EFS.
- */
-typedef struct {
- /* Can be anything the creator chooses. */
- /* EFS uses it as follows: */
- // FIXME: Type this info, verifying it along the way. (AIA)
-} __attribute__ ((__packed__)) LOGGED_UTILITY_STREAM, EFS_ATTR;
-
-#endif /* _LINUX_NTFS_LAYOUT_H */
diff --git a/fs/ntfs/lcnalloc.c b/fs/ntfs/lcnalloc.c
deleted file mode 100644
index eda9972e6159..000000000000
--- a/fs/ntfs/lcnalloc.c
+++ /dev/null
@@ -1,1000 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * lcnalloc.c - Cluster (de)allocation code. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2004-2005 Anton Altaparmakov
- */
-
-#ifdef NTFS_RW
-
-#include <linux/pagemap.h>
-
-#include "lcnalloc.h"
-#include "debug.h"
-#include "bitmap.h"
-#include "inode.h"
-#include "volume.h"
-#include "attrib.h"
-#include "malloc.h"
-#include "aops.h"
-#include "ntfs.h"
-
-/**
- * ntfs_cluster_free_from_rl_nolock - free clusters from runlist
- * @vol: mounted ntfs volume on which to free the clusters
- * @rl: runlist describing the clusters to free
- *
- * Free all the clusters described by the runlist @rl on the volume @vol. In
- * the case of an error being returned, at least some of the clusters were not
- * freed.
- *
- * Return 0 on success and -errno on error.
- *
- * Locking: - The volume lcn bitmap must be locked for writing on entry and is
- * left locked on return.
- */
-int ntfs_cluster_free_from_rl_nolock(ntfs_volume *vol,
- const runlist_element *rl)
-{
- struct inode *lcnbmp_vi = vol->lcnbmp_ino;
- int ret = 0;
-
- ntfs_debug("Entering.");
- if (!rl)
- return 0;
- for (; rl->length; rl++) {
- int err;
-
- if (rl->lcn < 0)
- continue;
- err = ntfs_bitmap_clear_run(lcnbmp_vi, rl->lcn, rl->length);
- if (unlikely(err && (!ret || ret == -ENOMEM) && ret != err))
- ret = err;
- }
- ntfs_debug("Done.");
- return ret;
-}
-
-/**
- * ntfs_cluster_alloc - allocate clusters on an ntfs volume
- * @vol: mounted ntfs volume on which to allocate the clusters
- * @start_vcn: vcn to use for the first allocated cluster
- * @count: number of clusters to allocate
- * @start_lcn: starting lcn at which to allocate the clusters (or -1 if none)
- * @zone: zone from which to allocate the clusters
- * @is_extension: if 'true', this is an attribute extension
- *
- * Allocate @count clusters preferably starting at cluster @start_lcn or at the
- * current allocator position if @start_lcn is -1, on the mounted ntfs volume
- * @vol. @zone is either DATA_ZONE for allocation of normal clusters or
- * MFT_ZONE for allocation of clusters for the master file table, i.e. the
- * $MFT/$DATA attribute.
- *
- * @start_vcn specifies the vcn of the first allocated cluster. This makes
- * merging the resulting runlist with the old runlist easier.
- *
- * If @is_extension is 'true', the caller is allocating clusters to extend an
- * attribute and if it is 'false', the caller is allocating clusters to fill a
- * hole in an attribute. Practically the difference is that if @is_extension
- * is 'true' the returned runlist will be terminated with LCN_ENOENT and if
- * @is_extension is 'false' the runlist will be terminated with
- * LCN_RL_NOT_MAPPED.
- *
- * You need to check the return value with IS_ERR(). If this is false, the
- * function was successful and the return value is a runlist describing the
- * allocated cluster(s). If IS_ERR() is true, the function failed and
- * PTR_ERR() gives you the error code.
- *
- * Notes on the allocation algorithm
- * =================================
- *
- * There are two data zones. First is the area between the end of the mft zone
- * and the end of the volume, and second is the area between the start of the
- * volume and the start of the mft zone. On unmodified/standard NTFS 1.x
- * volumes, the second data zone does not exist due to the mft zone being
- * expanded to cover the start of the volume in order to reserve space for the
- * mft bitmap attribute.
- *
- * This is not the prettiest function but the complexity stems from the need of
- * implementing the mft vs data zoned approach and from the fact that we have
- * access to the lcn bitmap in portions of up to 8192 bytes at a time, so we
- * need to cope with crossing over boundaries of two buffers. Further, the
- * fact that the allocator allows for caller supplied hints as to the location
- * of where allocation should begin and the fact that the allocator keeps track
- * of where in the data zones the next natural allocation should occur,
- * contribute to the complexity of the function. But it should all be
- * worthwhile, because this allocator should: 1) be a full implementation of
- * the MFT zone approach used by Windows NT, 2) cause reduction in
- * fragmentation, and 3) be speedy in allocations (the code is not optimized
- * for speed, but the algorithm is, so further speed improvements are probably
- * possible).
- *
- * FIXME: We should be monitoring cluster allocation and increment the MFT zone
- * size dynamically but this is something for the future. We will just cause
- * heavier fragmentation by not doing it and I am not even sure Windows would
- * grow the MFT zone dynamically, so it might even be correct not to do this.
- * The overhead in doing dynamic MFT zone expansion would be very large and
- * unlikely worth the effort. (AIA)
- *
- * TODO: I have added in double the required zone position pointer wrap around
- * logic which can be optimized to having only one of the two logic sets.
- * However, having the double logic will work fine, but if we have only one of
- * the sets and we get it wrong somewhere, then we get into trouble, so
- * removing the duplicate logic requires _very_ careful consideration of _all_
- * possible code paths. So at least for now, I am leaving the double logic -
- * better safe than sorry... (AIA)
- *
- * Locking: - The volume lcn bitmap must be unlocked on entry and is unlocked
- * on return.
- * - This function takes the volume lcn bitmap lock for writing and
- * modifies the bitmap contents.
- */
-runlist_element *ntfs_cluster_alloc(ntfs_volume *vol, const VCN start_vcn,
- const s64 count, const LCN start_lcn,
- const NTFS_CLUSTER_ALLOCATION_ZONES zone,
- const bool is_extension)
-{
- LCN zone_start, zone_end, bmp_pos, bmp_initial_pos, last_read_pos, lcn;
- LCN prev_lcn = 0, prev_run_len = 0, mft_zone_size;
- s64 clusters;
- loff_t i_size;
- struct inode *lcnbmp_vi;
- runlist_element *rl = NULL;
- struct address_space *mapping;
- struct page *page = NULL;
- u8 *buf, *byte;
- int err = 0, rlpos, rlsize, buf_size;
- u8 pass, done_zones, search_zone, need_writeback = 0, bit;
-
- ntfs_debug("Entering for start_vcn 0x%llx, count 0x%llx, start_lcn "
- "0x%llx, zone %s_ZONE.", (unsigned long long)start_vcn,
- (unsigned long long)count,
- (unsigned long long)start_lcn,
- zone == MFT_ZONE ? "MFT" : "DATA");
- BUG_ON(!vol);
- lcnbmp_vi = vol->lcnbmp_ino;
- BUG_ON(!lcnbmp_vi);
- BUG_ON(start_vcn < 0);
- BUG_ON(count < 0);
- BUG_ON(start_lcn < -1);
- BUG_ON(zone < FIRST_ZONE);
- BUG_ON(zone > LAST_ZONE);
-
- /* Return NULL if @count is zero. */
- if (!count)
- return NULL;
- /* Take the lcnbmp lock for writing. */
- down_write(&vol->lcnbmp_lock);
- /*
- * If no specific @start_lcn was requested, use the current data zone
- * position, otherwise use the requested @start_lcn but make sure it
- * lies outside the mft zone. Also set done_zones to 0 (no zones done)
- * and pass depending on whether we are starting inside a zone (1) or
- * at the beginning of a zone (2). If requesting from the MFT_ZONE,
- * we either start at the current position within the mft zone or at
- * the specified position. If the latter is out of bounds then we start
- * at the beginning of the MFT_ZONE.
- */
- done_zones = 0;
- pass = 1;
- /*
- * zone_start and zone_end are the current search range. search_zone
- * is 1 for mft zone, 2 for data zone 1 (end of mft zone till end of
- * volume) and 4 for data zone 2 (start of volume till start of mft
- * zone).
- */
- zone_start = start_lcn;
- if (zone_start < 0) {
- if (zone == DATA_ZONE)
- zone_start = vol->data1_zone_pos;
- else
- zone_start = vol->mft_zone_pos;
- if (!zone_start) {
- /*
- * Zone starts at beginning of volume which means a
- * single pass is sufficient.
- */
- pass = 2;
- }
- } else if (zone == DATA_ZONE && zone_start >= vol->mft_zone_start &&
- zone_start < vol->mft_zone_end) {
- zone_start = vol->mft_zone_end;
- /*
- * Starting at beginning of data1_zone which means a single
- * pass in this zone is sufficient.
- */
- pass = 2;
- } else if (zone == MFT_ZONE && (zone_start < vol->mft_zone_start ||
- zone_start >= vol->mft_zone_end)) {
- zone_start = vol->mft_lcn;
- if (!vol->mft_zone_end)
- zone_start = 0;
- /*
- * Starting at beginning of volume which means a single pass
- * is sufficient.
- */
- pass = 2;
- }
- if (zone == MFT_ZONE) {
- zone_end = vol->mft_zone_end;
- search_zone = 1;
- } else /* if (zone == DATA_ZONE) */ {
- /* Skip searching the mft zone. */
- done_zones |= 1;
- if (zone_start >= vol->mft_zone_end) {
- zone_end = vol->nr_clusters;
- search_zone = 2;
- } else {
- zone_end = vol->mft_zone_start;
- search_zone = 4;
- }
- }
- /*
- * bmp_pos is the current bit position inside the bitmap. We use
- * bmp_initial_pos to determine whether or not to do a zone switch.
- */
- bmp_pos = bmp_initial_pos = zone_start;
-
- /* Loop until all clusters are allocated, i.e. clusters == 0. */
- clusters = count;
- rlpos = rlsize = 0;
- mapping = lcnbmp_vi->i_mapping;
- i_size = i_size_read(lcnbmp_vi);
- while (1) {
- ntfs_debug("Start of outer while loop: done_zones 0x%x, "
- "search_zone %i, pass %i, zone_start 0x%llx, "
- "zone_end 0x%llx, bmp_initial_pos 0x%llx, "
- "bmp_pos 0x%llx, rlpos %i, rlsize %i.",
- done_zones, search_zone, pass,
- (unsigned long long)zone_start,
- (unsigned long long)zone_end,
- (unsigned long long)bmp_initial_pos,
- (unsigned long long)bmp_pos, rlpos, rlsize);
- /* Loop until we run out of free clusters. */
- last_read_pos = bmp_pos >> 3;
- ntfs_debug("last_read_pos 0x%llx.",
- (unsigned long long)last_read_pos);
- if (last_read_pos > i_size) {
- ntfs_debug("End of attribute reached. "
- "Skipping to zone_pass_done.");
- goto zone_pass_done;
- }
- if (likely(page)) {
- if (need_writeback) {
- ntfs_debug("Marking page dirty.");
- flush_dcache_page(page);
- set_page_dirty(page);
- need_writeback = 0;
- }
- ntfs_unmap_page(page);
- }
- page = ntfs_map_page(mapping, last_read_pos >>
- PAGE_SHIFT);
- if (IS_ERR(page)) {
- err = PTR_ERR(page);
- ntfs_error(vol->sb, "Failed to map page.");
- goto out;
- }
- buf_size = last_read_pos & ~PAGE_MASK;
- buf = page_address(page) + buf_size;
- buf_size = PAGE_SIZE - buf_size;
- if (unlikely(last_read_pos + buf_size > i_size))
- buf_size = i_size - last_read_pos;
- buf_size <<= 3;
- lcn = bmp_pos & 7;
- bmp_pos &= ~(LCN)7;
- ntfs_debug("Before inner while loop: buf_size %i, lcn 0x%llx, "
- "bmp_pos 0x%llx, need_writeback %i.", buf_size,
- (unsigned long long)lcn,
- (unsigned long long)bmp_pos, need_writeback);
- while (lcn < buf_size && lcn + bmp_pos < zone_end) {
- byte = buf + (lcn >> 3);
- ntfs_debug("In inner while loop: buf_size %i, "
- "lcn 0x%llx, bmp_pos 0x%llx, "
- "need_writeback %i, byte ofs 0x%x, "
- "*byte 0x%x.", buf_size,
- (unsigned long long)lcn,
- (unsigned long long)bmp_pos,
- need_writeback,
- (unsigned int)(lcn >> 3),
- (unsigned int)*byte);
- /* Skip full bytes. */
- if (*byte == 0xff) {
- lcn = (lcn + 8) & ~(LCN)7;
- ntfs_debug("Continuing while loop 1.");
- continue;
- }
- bit = 1 << (lcn & 7);
- ntfs_debug("bit 0x%x.", bit);
- /* If the bit is already set, go onto the next one. */
- if (*byte & bit) {
- lcn++;
- ntfs_debug("Continuing while loop 2.");
- continue;
- }
- /*
- * Allocate more memory if needed, including space for
- * the terminator element.
- * ntfs_malloc_nofs() operates on whole pages only.
- */
- if ((rlpos + 2) * sizeof(*rl) > rlsize) {
- runlist_element *rl2;
-
- ntfs_debug("Reallocating memory.");
- if (!rl)
- ntfs_debug("First free bit is at LCN "
- "0x%llx.",
- (unsigned long long)
- (lcn + bmp_pos));
- rl2 = ntfs_malloc_nofs(rlsize + (int)PAGE_SIZE);
- if (unlikely(!rl2)) {
- err = -ENOMEM;
- ntfs_error(vol->sb, "Failed to "
- "allocate memory.");
- goto out;
- }
- memcpy(rl2, rl, rlsize);
- ntfs_free(rl);
- rl = rl2;
- rlsize += PAGE_SIZE;
- ntfs_debug("Reallocated memory, rlsize 0x%x.",
- rlsize);
- }
- /* Allocate the bitmap bit. */
- *byte |= bit;
- /* We need to write this bitmap page to disk. */
- need_writeback = 1;
- ntfs_debug("*byte 0x%x, need_writeback is set.",
- (unsigned int)*byte);
- /*
- * Coalesce with previous run if adjacent LCNs.
- * Otherwise, append a new run.
- */
- ntfs_debug("Adding run (lcn 0x%llx, len 0x%llx), "
- "prev_lcn 0x%llx, lcn 0x%llx, "
- "bmp_pos 0x%llx, prev_run_len 0x%llx, "
- "rlpos %i.",
- (unsigned long long)(lcn + bmp_pos),
- 1ULL, (unsigned long long)prev_lcn,
- (unsigned long long)lcn,
- (unsigned long long)bmp_pos,
- (unsigned long long)prev_run_len,
- rlpos);
- if (prev_lcn == lcn + bmp_pos - prev_run_len && rlpos) {
- ntfs_debug("Coalescing to run (lcn 0x%llx, "
- "len 0x%llx).",
- (unsigned long long)
- rl[rlpos - 1].lcn,
- (unsigned long long)
- rl[rlpos - 1].length);
- rl[rlpos - 1].length = ++prev_run_len;
- ntfs_debug("Run now (lcn 0x%llx, len 0x%llx), "
- "prev_run_len 0x%llx.",
- (unsigned long long)
- rl[rlpos - 1].lcn,
- (unsigned long long)
- rl[rlpos - 1].length,
- (unsigned long long)
- prev_run_len);
- } else {
- if (likely(rlpos)) {
- ntfs_debug("Adding new run, (previous "
- "run lcn 0x%llx, "
- "len 0x%llx).",
- (unsigned long long)
- rl[rlpos - 1].lcn,
- (unsigned long long)
- rl[rlpos - 1].length);
- rl[rlpos].vcn = rl[rlpos - 1].vcn +
- prev_run_len;
- } else {
- ntfs_debug("Adding new run, is first "
- "run.");
- rl[rlpos].vcn = start_vcn;
- }
- rl[rlpos].lcn = prev_lcn = lcn + bmp_pos;
- rl[rlpos].length = prev_run_len = 1;
- rlpos++;
- }
- /* Done? */
- if (!--clusters) {
- LCN tc;
- /*
- * Update the current zone position. Positions
- * of already scanned zones have been updated
- * during the respective zone switches.
- */
- tc = lcn + bmp_pos + 1;
- ntfs_debug("Done. Updating current zone "
- "position, tc 0x%llx, "
- "search_zone %i.",
- (unsigned long long)tc,
- search_zone);
- switch (search_zone) {
- case 1:
- ntfs_debug("Before checks, "
- "vol->mft_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->mft_zone_pos);
- if (tc >= vol->mft_zone_end) {
- vol->mft_zone_pos =
- vol->mft_lcn;
- if (!vol->mft_zone_end)
- vol->mft_zone_pos = 0;
- } else if ((bmp_initial_pos >=
- vol->mft_zone_pos ||
- tc > vol->mft_zone_pos)
- && tc >= vol->mft_lcn)
- vol->mft_zone_pos = tc;
- ntfs_debug("After checks, "
- "vol->mft_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->mft_zone_pos);
- break;
- case 2:
- ntfs_debug("Before checks, "
- "vol->data1_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->data1_zone_pos);
- if (tc >= vol->nr_clusters)
- vol->data1_zone_pos =
- vol->mft_zone_end;
- else if ((bmp_initial_pos >=
- vol->data1_zone_pos ||
- tc > vol->data1_zone_pos)
- && tc >= vol->mft_zone_end)
- vol->data1_zone_pos = tc;
- ntfs_debug("After checks, "
- "vol->data1_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->data1_zone_pos);
- break;
- case 4:
- ntfs_debug("Before checks, "
- "vol->data2_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->data2_zone_pos);
- if (tc >= vol->mft_zone_start)
- vol->data2_zone_pos = 0;
- else if (bmp_initial_pos >=
- vol->data2_zone_pos ||
- tc > vol->data2_zone_pos)
- vol->data2_zone_pos = tc;
- ntfs_debug("After checks, "
- "vol->data2_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->data2_zone_pos);
- break;
- default:
- BUG();
- }
- ntfs_debug("Finished. Going to out.");
- goto out;
- }
- lcn++;
- }
- bmp_pos += buf_size;
- ntfs_debug("After inner while loop: buf_size 0x%x, lcn "
- "0x%llx, bmp_pos 0x%llx, need_writeback %i.",
- buf_size, (unsigned long long)lcn,
- (unsigned long long)bmp_pos, need_writeback);
- if (bmp_pos < zone_end) {
- ntfs_debug("Continuing outer while loop, "
- "bmp_pos 0x%llx, zone_end 0x%llx.",
- (unsigned long long)bmp_pos,
- (unsigned long long)zone_end);
- continue;
- }
-zone_pass_done: /* Finished with the current zone pass. */
- ntfs_debug("At zone_pass_done, pass %i.", pass);
- if (pass == 1) {
- /*
- * Now do pass 2, scanning the first part of the zone
- * we omitted in pass 1.
- */
- pass = 2;
- zone_end = zone_start;
- switch (search_zone) {
- case 1: /* mft_zone */
- zone_start = vol->mft_zone_start;
- break;
- case 2: /* data1_zone */
- zone_start = vol->mft_zone_end;
- break;
- case 4: /* data2_zone */
- zone_start = 0;
- break;
- default:
- BUG();
- }
- /* Sanity check. */
- if (zone_end < zone_start)
- zone_end = zone_start;
- bmp_pos = zone_start;
- ntfs_debug("Continuing outer while loop, pass 2, "
- "zone_start 0x%llx, zone_end 0x%llx, "
- "bmp_pos 0x%llx.",
- (unsigned long long)zone_start,
- (unsigned long long)zone_end,
- (unsigned long long)bmp_pos);
- continue;
- } /* pass == 2 */
-done_zones_check:
- ntfs_debug("At done_zones_check, search_zone %i, done_zones "
- "before 0x%x, done_zones after 0x%x.",
- search_zone, done_zones,
- done_zones | search_zone);
- done_zones |= search_zone;
- if (done_zones < 7) {
- ntfs_debug("Switching zone.");
- /* Now switch to the next zone we haven't done yet. */
- pass = 1;
- switch (search_zone) {
- case 1:
- ntfs_debug("Switching from mft zone to data1 "
- "zone.");
- /* Update mft zone position. */
- if (rlpos) {
- LCN tc;
-
- ntfs_debug("Before checks, "
- "vol->mft_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->mft_zone_pos);
- tc = rl[rlpos - 1].lcn +
- rl[rlpos - 1].length;
- if (tc >= vol->mft_zone_end) {
- vol->mft_zone_pos =
- vol->mft_lcn;
- if (!vol->mft_zone_end)
- vol->mft_zone_pos = 0;
- } else if ((bmp_initial_pos >=
- vol->mft_zone_pos ||
- tc > vol->mft_zone_pos)
- && tc >= vol->mft_lcn)
- vol->mft_zone_pos = tc;
- ntfs_debug("After checks, "
- "vol->mft_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->mft_zone_pos);
- }
- /* Switch from mft zone to data1 zone. */
-switch_to_data1_zone: search_zone = 2;
- zone_start = bmp_initial_pos =
- vol->data1_zone_pos;
- zone_end = vol->nr_clusters;
- if (zone_start == vol->mft_zone_end)
- pass = 2;
- if (zone_start >= zone_end) {
- vol->data1_zone_pos = zone_start =
- vol->mft_zone_end;
- pass = 2;
- }
- break;
- case 2:
- ntfs_debug("Switching from data1 zone to "
- "data2 zone.");
- /* Update data1 zone position. */
- if (rlpos) {
- LCN tc;
-
- ntfs_debug("Before checks, "
- "vol->data1_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->data1_zone_pos);
- tc = rl[rlpos - 1].lcn +
- rl[rlpos - 1].length;
- if (tc >= vol->nr_clusters)
- vol->data1_zone_pos =
- vol->mft_zone_end;
- else if ((bmp_initial_pos >=
- vol->data1_zone_pos ||
- tc > vol->data1_zone_pos)
- && tc >= vol->mft_zone_end)
- vol->data1_zone_pos = tc;
- ntfs_debug("After checks, "
- "vol->data1_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->data1_zone_pos);
- }
- /* Switch from data1 zone to data2 zone. */
- search_zone = 4;
- zone_start = bmp_initial_pos =
- vol->data2_zone_pos;
- zone_end = vol->mft_zone_start;
- if (!zone_start)
- pass = 2;
- if (zone_start >= zone_end) {
- vol->data2_zone_pos = zone_start =
- bmp_initial_pos = 0;
- pass = 2;
- }
- break;
- case 4:
- ntfs_debug("Switching from data2 zone to "
- "data1 zone.");
- /* Update data2 zone position. */
- if (rlpos) {
- LCN tc;
-
- ntfs_debug("Before checks, "
- "vol->data2_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->data2_zone_pos);
- tc = rl[rlpos - 1].lcn +
- rl[rlpos - 1].length;
- if (tc >= vol->mft_zone_start)
- vol->data2_zone_pos = 0;
- else if (bmp_initial_pos >=
- vol->data2_zone_pos ||
- tc > vol->data2_zone_pos)
- vol->data2_zone_pos = tc;
- ntfs_debug("After checks, "
- "vol->data2_zone_pos "
- "0x%llx.",
- (unsigned long long)
- vol->data2_zone_pos);
- }
- /* Switch from data2 zone to data1 zone. */
- goto switch_to_data1_zone;
- default:
- BUG();
- }
- ntfs_debug("After zone switch, search_zone %i, "
- "pass %i, bmp_initial_pos 0x%llx, "
- "zone_start 0x%llx, zone_end 0x%llx.",
- search_zone, pass,
- (unsigned long long)bmp_initial_pos,
- (unsigned long long)zone_start,
- (unsigned long long)zone_end);
- bmp_pos = zone_start;
- if (zone_start == zone_end) {
- ntfs_debug("Empty zone, going to "
- "done_zones_check.");
- /* Empty zone. Don't bother searching it. */
- goto done_zones_check;
- }
- ntfs_debug("Continuing outer while loop.");
- continue;
- } /* done_zones == 7 */
- ntfs_debug("All zones are finished.");
- /*
- * All zones are finished! If DATA_ZONE, shrink mft zone. If
- * MFT_ZONE, we have really run out of space.
- */
- mft_zone_size = vol->mft_zone_end - vol->mft_zone_start;
- ntfs_debug("vol->mft_zone_start 0x%llx, vol->mft_zone_end "
- "0x%llx, mft_zone_size 0x%llx.",
- (unsigned long long)vol->mft_zone_start,
- (unsigned long long)vol->mft_zone_end,
- (unsigned long long)mft_zone_size);
- if (zone == MFT_ZONE || mft_zone_size <= 0) {
- ntfs_debug("No free clusters left, going to out.");
- /* Really no more space left on device. */
- err = -ENOSPC;
- goto out;
- } /* zone == DATA_ZONE && mft_zone_size > 0 */
- ntfs_debug("Shrinking mft zone.");
- zone_end = vol->mft_zone_end;
- mft_zone_size >>= 1;
- if (mft_zone_size > 0)
- vol->mft_zone_end = vol->mft_zone_start + mft_zone_size;
- else /* mft zone and data2 zone no longer exist. */
- vol->data2_zone_pos = vol->mft_zone_start =
- vol->mft_zone_end = 0;
- if (vol->mft_zone_pos >= vol->mft_zone_end) {
- vol->mft_zone_pos = vol->mft_lcn;
- if (!vol->mft_zone_end)
- vol->mft_zone_pos = 0;
- }
- bmp_pos = zone_start = bmp_initial_pos =
- vol->data1_zone_pos = vol->mft_zone_end;
- search_zone = 2;
- pass = 2;
- done_zones &= ~2;
- ntfs_debug("After shrinking mft zone, mft_zone_size 0x%llx, "
- "vol->mft_zone_start 0x%llx, "
- "vol->mft_zone_end 0x%llx, "
- "vol->mft_zone_pos 0x%llx, search_zone 2, "
- "pass 2, dones_zones 0x%x, zone_start 0x%llx, "
- "zone_end 0x%llx, vol->data1_zone_pos 0x%llx, "
- "continuing outer while loop.",
- (unsigned long long)mft_zone_size,
- (unsigned long long)vol->mft_zone_start,
- (unsigned long long)vol->mft_zone_end,
- (unsigned long long)vol->mft_zone_pos,
- done_zones, (unsigned long long)zone_start,
- (unsigned long long)zone_end,
- (unsigned long long)vol->data1_zone_pos);
- }
- ntfs_debug("After outer while loop.");
-out:
- ntfs_debug("At out.");
- /* Add runlist terminator element. */
- if (likely(rl)) {
- rl[rlpos].vcn = rl[rlpos - 1].vcn + rl[rlpos - 1].length;
- rl[rlpos].lcn = is_extension ? LCN_ENOENT : LCN_RL_NOT_MAPPED;
- rl[rlpos].length = 0;
- }
- if (likely(page && !IS_ERR(page))) {
- if (need_writeback) {
- ntfs_debug("Marking page dirty.");
- flush_dcache_page(page);
- set_page_dirty(page);
- need_writeback = 0;
- }
- ntfs_unmap_page(page);
- }
- if (likely(!err)) {
- up_write(&vol->lcnbmp_lock);
- ntfs_debug("Done.");
- return rl;
- }
- ntfs_error(vol->sb, "Failed to allocate clusters, aborting "
- "(error %i).", err);
- if (rl) {
- int err2;
-
- if (err == -ENOSPC)
- ntfs_debug("Not enough space to complete allocation, "
- "err -ENOSPC, first free lcn 0x%llx, "
- "could allocate up to 0x%llx "
- "clusters.",
- (unsigned long long)rl[0].lcn,
- (unsigned long long)(count - clusters));
- /* Deallocate all allocated clusters. */
- ntfs_debug("Attempting rollback...");
- err2 = ntfs_cluster_free_from_rl_nolock(vol, rl);
- if (err2) {
- ntfs_error(vol->sb, "Failed to rollback (error %i). "
- "Leaving inconsistent metadata! "
- "Unmount and run chkdsk.", err2);
- NVolSetErrors(vol);
- }
- /* Free the runlist. */
- ntfs_free(rl);
- } else if (err == -ENOSPC)
- ntfs_debug("No space left at all, err = -ENOSPC, first free "
- "lcn = 0x%llx.",
- (long long)vol->data1_zone_pos);
- up_write(&vol->lcnbmp_lock);
- return ERR_PTR(err);
-}
-
-/**
- * __ntfs_cluster_free - free clusters on an ntfs volume
- * @ni: ntfs inode whose runlist describes the clusters to free
- * @start_vcn: vcn in the runlist of @ni at which to start freeing clusters
- * @count: number of clusters to free or -1 for all clusters
- * @ctx: active attribute search context if present or NULL if not
- * @is_rollback: true if this is a rollback operation
- *
- * Free @count clusters starting at the cluster @start_vcn in the runlist
- * described by the vfs inode @ni.
- *
- * If @count is -1, all clusters from @start_vcn to the end of the runlist are
- * deallocated. Thus, to completely free all clusters in a runlist, use
- * @start_vcn = 0 and @count = -1.
- *
- * If @ctx is specified, it is an active search context of @ni and its base mft
- * record. This is needed when __ntfs_cluster_free() encounters unmapped
- * runlist fragments and allows their mapping. If you do not have the mft
- * record mapped, you can specify @ctx as NULL and __ntfs_cluster_free() will
- * perform the necessary mapping and unmapping.
- *
- * Note, __ntfs_cluster_free() saves the state of @ctx on entry and restores it
- * before returning. Thus, @ctx will be left pointing to the same attribute on
- * return as on entry. However, the actual pointers in @ctx may point to
- * different memory locations on return, so you must remember to reset any
- * cached pointers from the @ctx, i.e. after the call to __ntfs_cluster_free(),
- * you will probably want to do:
- * m = ctx->mrec;
- * a = ctx->attr;
- * Assuming you cache ctx->attr in a variable @a of type ATTR_RECORD * and that
- * you cache ctx->mrec in a variable @m of type MFT_RECORD *.
- *
- * @is_rollback should always be 'false', it is for internal use to rollback
- * errors. You probably want to use ntfs_cluster_free() instead.
- *
- * Note, __ntfs_cluster_free() does not modify the runlist, so you have to
- * remove from the runlist or mark sparse the freed runs later.
- *
- * Return the number of deallocated clusters (not counting sparse ones) on
- * success and -errno on error.
- *
- * WARNING: If @ctx is supplied, regardless of whether success or failure is
- * returned, you need to check IS_ERR(@ctx->mrec) and if 'true' the @ctx
- * is no longer valid, i.e. you need to either call
- * ntfs_attr_reinit_search_ctx() or ntfs_attr_put_search_ctx() on it.
- * In that case PTR_ERR(@ctx->mrec) will give you the error code for
- * why the mapping of the old inode failed.
- *
- * Locking: - The runlist described by @ni must be locked for writing on entry
- * and is locked on return. Note the runlist may be modified when
- * needed runlist fragments need to be mapped.
- * - The volume lcn bitmap must be unlocked on entry and is unlocked
- * on return.
- * - This function takes the volume lcn bitmap lock for writing and
- * modifies the bitmap contents.
- * - If @ctx is NULL, the base mft record of @ni must not be mapped on
- * entry and it will be left unmapped on return.
- * - If @ctx is not NULL, the base mft record must be mapped on entry
- * and it will be left mapped on return.
- */
-s64 __ntfs_cluster_free(ntfs_inode *ni, const VCN start_vcn, s64 count,
- ntfs_attr_search_ctx *ctx, const bool is_rollback)
-{
- s64 delta, to_free, total_freed, real_freed;
- ntfs_volume *vol;
- struct inode *lcnbmp_vi;
- runlist_element *rl;
- int err;
-
- BUG_ON(!ni);
- ntfs_debug("Entering for i_ino 0x%lx, start_vcn 0x%llx, count "
- "0x%llx.%s", ni->mft_no, (unsigned long long)start_vcn,
- (unsigned long long)count,
- is_rollback ? " (rollback)" : "");
- vol = ni->vol;
- lcnbmp_vi = vol->lcnbmp_ino;
- BUG_ON(!lcnbmp_vi);
- BUG_ON(start_vcn < 0);
- BUG_ON(count < -1);
- /*
- * Lock the lcn bitmap for writing but only if not rolling back. We
- * must hold the lock all the way including through rollback otherwise
- * rollback is not possible because once we have cleared a bit and
- * dropped the lock, anyone could have set the bit again, thus
- * allocating the cluster for another use.
- */
- if (likely(!is_rollback))
- down_write(&vol->lcnbmp_lock);
-
- total_freed = real_freed = 0;
-
- rl = ntfs_attr_find_vcn_nolock(ni, start_vcn, ctx);
- if (IS_ERR(rl)) {
- if (!is_rollback)
- ntfs_error(vol->sb, "Failed to find first runlist "
- "element (error %li), aborting.",
- PTR_ERR(rl));
- err = PTR_ERR(rl);
- goto err_out;
- }
- if (unlikely(rl->lcn < LCN_HOLE)) {
- if (!is_rollback)
- ntfs_error(vol->sb, "First runlist element has "
- "invalid lcn, aborting.");
- err = -EIO;
- goto err_out;
- }
- /* Find the starting cluster inside the run that needs freeing. */
- delta = start_vcn - rl->vcn;
-
- /* The number of clusters in this run that need freeing. */
- to_free = rl->length - delta;
- if (count >= 0 && to_free > count)
- to_free = count;
-
- if (likely(rl->lcn >= 0)) {
- /* Do the actual freeing of the clusters in this run. */
- err = ntfs_bitmap_set_bits_in_run(lcnbmp_vi, rl->lcn + delta,
- to_free, likely(!is_rollback) ? 0 : 1);
- if (unlikely(err)) {
- if (!is_rollback)
- ntfs_error(vol->sb, "Failed to clear first run "
- "(error %i), aborting.", err);
- goto err_out;
- }
- /* We have freed @to_free real clusters. */
- real_freed = to_free;
- };
- /* Go to the next run and adjust the number of clusters left to free. */
- ++rl;
- if (count >= 0)
- count -= to_free;
-
- /* Keep track of the total "freed" clusters, including sparse ones. */
- total_freed = to_free;
- /*
- * Loop over the remaining runs, using @count as a capping value, and
- * free them.
- */
- for (; rl->length && count != 0; ++rl) {
- if (unlikely(rl->lcn < LCN_HOLE)) {
- VCN vcn;
-
- /* Attempt to map runlist. */
- vcn = rl->vcn;
- rl = ntfs_attr_find_vcn_nolock(ni, vcn, ctx);
- if (IS_ERR(rl)) {
- err = PTR_ERR(rl);
- if (!is_rollback)
- ntfs_error(vol->sb, "Failed to map "
- "runlist fragment or "
- "failed to find "
- "subsequent runlist "
- "element.");
- goto err_out;
- }
- if (unlikely(rl->lcn < LCN_HOLE)) {
- if (!is_rollback)
- ntfs_error(vol->sb, "Runlist element "
- "has invalid lcn "
- "(0x%llx).",
- (unsigned long long)
- rl->lcn);
- err = -EIO;
- goto err_out;
- }
- }
- /* The number of clusters in this run that need freeing. */
- to_free = rl->length;
- if (count >= 0 && to_free > count)
- to_free = count;
-
- if (likely(rl->lcn >= 0)) {
- /* Do the actual freeing of the clusters in the run. */
- err = ntfs_bitmap_set_bits_in_run(lcnbmp_vi, rl->lcn,
- to_free, likely(!is_rollback) ? 0 : 1);
- if (unlikely(err)) {
- if (!is_rollback)
- ntfs_error(vol->sb, "Failed to clear "
- "subsequent run.");
- goto err_out;
- }
- /* We have freed @to_free real clusters. */
- real_freed += to_free;
- }
- /* Adjust the number of clusters left to free. */
- if (count >= 0)
- count -= to_free;
-
- /* Update the total done clusters. */
- total_freed += to_free;
- }
- if (likely(!is_rollback))
- up_write(&vol->lcnbmp_lock);
-
- BUG_ON(count > 0);
-
- /* We are done. Return the number of actually freed clusters. */
- ntfs_debug("Done.");
- return real_freed;
-err_out:
- if (is_rollback)
- return err;
- /* If no real clusters were freed, no need to rollback. */
- if (!real_freed) {
- up_write(&vol->lcnbmp_lock);
- return err;
- }
- /*
- * Attempt to rollback and if that succeeds just return the error code.
- * If rollback fails, set the volume errors flag, emit an error
- * message, and return the error code.
- */
- delta = __ntfs_cluster_free(ni, start_vcn, total_freed, ctx, true);
- if (delta < 0) {
- ntfs_error(vol->sb, "Failed to rollback (error %i). Leaving "
- "inconsistent metadata! Unmount and run "
- "chkdsk.", (int)delta);
- NVolSetErrors(vol);
- }
- up_write(&vol->lcnbmp_lock);
- ntfs_error(vol->sb, "Aborting (error %i).", err);
- return err;
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/lcnalloc.h b/fs/ntfs/lcnalloc.h
deleted file mode 100644
index 1589a6d8434b..000000000000
--- a/fs/ntfs/lcnalloc.h
+++ /dev/null
@@ -1,131 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * lcnalloc.h - Exports for NTFS kernel cluster (de)allocation. Part of the
- * Linux-NTFS project.
- *
- * Copyright (c) 2004-2005 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_LCNALLOC_H
-#define _LINUX_NTFS_LCNALLOC_H
-
-#ifdef NTFS_RW
-
-#include <linux/fs.h>
-
-#include "attrib.h"
-#include "types.h"
-#include "inode.h"
-#include "runlist.h"
-#include "volume.h"
-
-typedef enum {
- FIRST_ZONE = 0, /* For sanity checking. */
- MFT_ZONE = 0, /* Allocate from $MFT zone. */
- DATA_ZONE = 1, /* Allocate from $DATA zone. */
- LAST_ZONE = 1, /* For sanity checking. */
-} NTFS_CLUSTER_ALLOCATION_ZONES;
-
-extern runlist_element *ntfs_cluster_alloc(ntfs_volume *vol,
- const VCN start_vcn, const s64 count, const LCN start_lcn,
- const NTFS_CLUSTER_ALLOCATION_ZONES zone,
- const bool is_extension);
-
-extern s64 __ntfs_cluster_free(ntfs_inode *ni, const VCN start_vcn,
- s64 count, ntfs_attr_search_ctx *ctx, const bool is_rollback);
-
-/**
- * ntfs_cluster_free - free clusters on an ntfs volume
- * @ni: ntfs inode whose runlist describes the clusters to free
- * @start_vcn: vcn in the runlist of @ni at which to start freeing clusters
- * @count: number of clusters to free or -1 for all clusters
- * @ctx: active attribute search context if present or NULL if not
- *
- * Free @count clusters starting at the cluster @start_vcn in the runlist
- * described by the ntfs inode @ni.
- *
- * If @count is -1, all clusters from @start_vcn to the end of the runlist are
- * deallocated. Thus, to completely free all clusters in a runlist, use
- * @start_vcn = 0 and @count = -1.
- *
- * If @ctx is specified, it is an active search context of @ni and its base mft
- * record. This is needed when ntfs_cluster_free() encounters unmapped runlist
- * fragments and allows their mapping. If you do not have the mft record
- * mapped, you can specify @ctx as NULL and ntfs_cluster_free() will perform
- * the necessary mapping and unmapping.
- *
- * Note, ntfs_cluster_free() saves the state of @ctx on entry and restores it
- * before returning. Thus, @ctx will be left pointing to the same attribute on
- * return as on entry. However, the actual pointers in @ctx may point to
- * different memory locations on return, so you must remember to reset any
- * cached pointers from the @ctx, i.e. after the call to ntfs_cluster_free(),
- * you will probably want to do:
- * m = ctx->mrec;
- * a = ctx->attr;
- * Assuming you cache ctx->attr in a variable @a of type ATTR_RECORD * and that
- * you cache ctx->mrec in a variable @m of type MFT_RECORD *.
- *
- * Note, ntfs_cluster_free() does not modify the runlist, so you have to remove
- * from the runlist or mark sparse the freed runs later.
- *
- * Return the number of deallocated clusters (not counting sparse ones) on
- * success and -errno on error.
- *
- * WARNING: If @ctx is supplied, regardless of whether success or failure is
- * returned, you need to check IS_ERR(@ctx->mrec) and if 'true' the @ctx
- * is no longer valid, i.e. you need to either call
- * ntfs_attr_reinit_search_ctx() or ntfs_attr_put_search_ctx() on it.
- * In that case PTR_ERR(@ctx->mrec) will give you the error code for
- * why the mapping of the old inode failed.
- *
- * Locking: - The runlist described by @ni must be locked for writing on entry
- * and is locked on return. Note the runlist may be modified when
- * needed runlist fragments need to be mapped.
- * - The volume lcn bitmap must be unlocked on entry and is unlocked
- * on return.
- * - This function takes the volume lcn bitmap lock for writing and
- * modifies the bitmap contents.
- * - If @ctx is NULL, the base mft record of @ni must not be mapped on
- * entry and it will be left unmapped on return.
- * - If @ctx is not NULL, the base mft record must be mapped on entry
- * and it will be left mapped on return.
- */
-static inline s64 ntfs_cluster_free(ntfs_inode *ni, const VCN start_vcn,
- s64 count, ntfs_attr_search_ctx *ctx)
-{
- return __ntfs_cluster_free(ni, start_vcn, count, ctx, false);
-}
-
-extern int ntfs_cluster_free_from_rl_nolock(ntfs_volume *vol,
- const runlist_element *rl);
-
-/**
- * ntfs_cluster_free_from_rl - free clusters from runlist
- * @vol: mounted ntfs volume on which to free the clusters
- * @rl: runlist describing the clusters to free
- *
- * Free all the clusters described by the runlist @rl on the volume @vol. In
- * the case of an error being returned, at least some of the clusters were not
- * freed.
- *
- * Return 0 on success and -errno on error.
- *
- * Locking: - This function takes the volume lcn bitmap lock for writing and
- * modifies the bitmap contents.
- * - The caller must have locked the runlist @rl for reading or
- * writing.
- */
-static inline int ntfs_cluster_free_from_rl(ntfs_volume *vol,
- const runlist_element *rl)
-{
- int ret;
-
- down_write(&vol->lcnbmp_lock);
- ret = ntfs_cluster_free_from_rl_nolock(vol, rl);
- up_write(&vol->lcnbmp_lock);
- return ret;
-}
-
-#endif /* NTFS_RW */
-
-#endif /* defined _LINUX_NTFS_LCNALLOC_H */
diff --git a/fs/ntfs/logfile.c b/fs/ntfs/logfile.c
deleted file mode 100644
index 6ce60ffc6ac0..000000000000
--- a/fs/ntfs/logfile.c
+++ /dev/null
@@ -1,849 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * logfile.c - NTFS kernel journal handling. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2002-2007 Anton Altaparmakov
- */
-
-#ifdef NTFS_RW
-
-#include <linux/types.h>
-#include <linux/fs.h>
-#include <linux/highmem.h>
-#include <linux/buffer_head.h>
-#include <linux/bitops.h>
-#include <linux/log2.h>
-#include <linux/bio.h>
-
-#include "attrib.h"
-#include "aops.h"
-#include "debug.h"
-#include "logfile.h"
-#include "malloc.h"
-#include "volume.h"
-#include "ntfs.h"
-
-/**
- * ntfs_check_restart_page_header - check the page header for consistency
- * @vi: $LogFile inode to which the restart page header belongs
- * @rp: restart page header to check
- * @pos: position in @vi at which the restart page header resides
- *
- * Check the restart page header @rp for consistency and return 'true' if it is
- * consistent and 'false' otherwise.
- *
- * This function only needs NTFS_BLOCK_SIZE bytes in @rp, i.e. it does not
- * require the full restart page.
- */
-static bool ntfs_check_restart_page_header(struct inode *vi,
- RESTART_PAGE_HEADER *rp, s64 pos)
-{
- u32 logfile_system_page_size, logfile_log_page_size;
- u16 ra_ofs, usa_count, usa_ofs, usa_end = 0;
- bool have_usa = true;
-
- ntfs_debug("Entering.");
- /*
- * If the system or log page sizes are smaller than the ntfs block size
- * or either is not a power of 2 we cannot handle this log file.
- */
- logfile_system_page_size = le32_to_cpu(rp->system_page_size);
- logfile_log_page_size = le32_to_cpu(rp->log_page_size);
- if (logfile_system_page_size < NTFS_BLOCK_SIZE ||
- logfile_log_page_size < NTFS_BLOCK_SIZE ||
- logfile_system_page_size &
- (logfile_system_page_size - 1) ||
- !is_power_of_2(logfile_log_page_size)) {
- ntfs_error(vi->i_sb, "$LogFile uses unsupported page size.");
- return false;
- }
- /*
- * We must be either at !pos (1st restart page) or at pos = system page
- * size (2nd restart page).
- */
- if (pos && pos != logfile_system_page_size) {
- ntfs_error(vi->i_sb, "Found restart area in incorrect "
- "position in $LogFile.");
- return false;
- }
- /* We only know how to handle version 1.1. */
- if (sle16_to_cpu(rp->major_ver) != 1 ||
- sle16_to_cpu(rp->minor_ver) != 1) {
- ntfs_error(vi->i_sb, "$LogFile version %i.%i is not "
- "supported. (This driver supports version "
- "1.1 only.)", (int)sle16_to_cpu(rp->major_ver),
- (int)sle16_to_cpu(rp->minor_ver));
- return false;
- }
- /*
- * If chkdsk has been run the restart page may not be protected by an
- * update sequence array.
- */
- if (ntfs_is_chkd_record(rp->magic) && !le16_to_cpu(rp->usa_count)) {
- have_usa = false;
- goto skip_usa_checks;
- }
- /* Verify the size of the update sequence array. */
- usa_count = 1 + (logfile_system_page_size >> NTFS_BLOCK_SIZE_BITS);
- if (usa_count != le16_to_cpu(rp->usa_count)) {
- ntfs_error(vi->i_sb, "$LogFile restart page specifies "
- "inconsistent update sequence array count.");
- return false;
- }
- /* Verify the position of the update sequence array. */
- usa_ofs = le16_to_cpu(rp->usa_ofs);
- usa_end = usa_ofs + usa_count * sizeof(u16);
- if (usa_ofs < sizeof(RESTART_PAGE_HEADER) ||
- usa_end > NTFS_BLOCK_SIZE - sizeof(u16)) {
- ntfs_error(vi->i_sb, "$LogFile restart page specifies "
- "inconsistent update sequence array offset.");
- return false;
- }
-skip_usa_checks:
- /*
- * Verify the position of the restart area. It must be:
- * - aligned to 8-byte boundary,
- * - after the update sequence array, and
- * - within the system page size.
- */
- ra_ofs = le16_to_cpu(rp->restart_area_offset);
- if (ra_ofs & 7 || (have_usa ? ra_ofs < usa_end :
- ra_ofs < sizeof(RESTART_PAGE_HEADER)) ||
- ra_ofs > logfile_system_page_size) {
- ntfs_error(vi->i_sb, "$LogFile restart page specifies "
- "inconsistent restart area offset.");
- return false;
- }
- /*
- * Only restart pages modified by chkdsk are allowed to have chkdsk_lsn
- * set.
- */
- if (!ntfs_is_chkd_record(rp->magic) && sle64_to_cpu(rp->chkdsk_lsn)) {
- ntfs_error(vi->i_sb, "$LogFile restart page is not modified "
- "by chkdsk but a chkdsk LSN is specified.");
- return false;
- }
- ntfs_debug("Done.");
- return true;
-}
-
-/**
- * ntfs_check_restart_area - check the restart area for consistency
- * @vi: $LogFile inode to which the restart page belongs
- * @rp: restart page whose restart area to check
- *
- * Check the restart area of the restart page @rp for consistency and return
- * 'true' if it is consistent and 'false' otherwise.
- *
- * This function assumes that the restart page header has already been
- * consistency checked.
- *
- * This function only needs NTFS_BLOCK_SIZE bytes in @rp, i.e. it does not
- * require the full restart page.
- */
-static bool ntfs_check_restart_area(struct inode *vi, RESTART_PAGE_HEADER *rp)
-{
- u64 file_size;
- RESTART_AREA *ra;
- u16 ra_ofs, ra_len, ca_ofs;
- u8 fs_bits;
-
- ntfs_debug("Entering.");
- ra_ofs = le16_to_cpu(rp->restart_area_offset);
- ra = (RESTART_AREA*)((u8*)rp + ra_ofs);
- /*
- * Everything before ra->file_size must be before the first word
- * protected by an update sequence number. This ensures that it is
- * safe to access ra->client_array_offset.
- */
- if (ra_ofs + offsetof(RESTART_AREA, file_size) >
- NTFS_BLOCK_SIZE - sizeof(u16)) {
- ntfs_error(vi->i_sb, "$LogFile restart area specifies "
- "inconsistent file offset.");
- return false;
- }
- /*
- * Now that we can access ra->client_array_offset, make sure everything
- * up to the log client array is before the first word protected by an
- * update sequence number. This ensures we can access all of the
- * restart area elements safely. Also, the client array offset must be
- * aligned to an 8-byte boundary.
- */
- ca_ofs = le16_to_cpu(ra->client_array_offset);
- if (((ca_ofs + 7) & ~7) != ca_ofs ||
- ra_ofs + ca_ofs > NTFS_BLOCK_SIZE - sizeof(u16)) {
- ntfs_error(vi->i_sb, "$LogFile restart area specifies "
- "inconsistent client array offset.");
- return false;
- }
- /*
- * The restart area must end within the system page size both when
- * calculated manually and as specified by ra->restart_area_length.
- * Also, the calculated length must not exceed the specified length.
- */
- ra_len = ca_ofs + le16_to_cpu(ra->log_clients) *
- sizeof(LOG_CLIENT_RECORD);
- if (ra_ofs + ra_len > le32_to_cpu(rp->system_page_size) ||
- ra_ofs + le16_to_cpu(ra->restart_area_length) >
- le32_to_cpu(rp->system_page_size) ||
- ra_len > le16_to_cpu(ra->restart_area_length)) {
- ntfs_error(vi->i_sb, "$LogFile restart area is out of bounds "
- "of the system page size specified by the "
- "restart page header and/or the specified "
- "restart area length is inconsistent.");
- return false;
- }
- /*
- * The ra->client_free_list and ra->client_in_use_list must be either
- * LOGFILE_NO_CLIENT or less than ra->log_clients or they are
- * overflowing the client array.
- */
- if ((ra->client_free_list != LOGFILE_NO_CLIENT &&
- le16_to_cpu(ra->client_free_list) >=
- le16_to_cpu(ra->log_clients)) ||
- (ra->client_in_use_list != LOGFILE_NO_CLIENT &&
- le16_to_cpu(ra->client_in_use_list) >=
- le16_to_cpu(ra->log_clients))) {
- ntfs_error(vi->i_sb, "$LogFile restart area specifies "
- "overflowing client free and/or in use lists.");
- return false;
- }
- /*
- * Check ra->seq_number_bits against ra->file_size for consistency.
- * We cannot just use ffs() because the file size is not a power of 2.
- */
- file_size = (u64)sle64_to_cpu(ra->file_size);
- fs_bits = 0;
- while (file_size) {
- file_size >>= 1;
- fs_bits++;
- }
- if (le32_to_cpu(ra->seq_number_bits) != 67 - fs_bits) {
- ntfs_error(vi->i_sb, "$LogFile restart area specifies "
- "inconsistent sequence number bits.");
- return false;
- }
- /* The log record header length must be a multiple of 8. */
- if (((le16_to_cpu(ra->log_record_header_length) + 7) & ~7) !=
- le16_to_cpu(ra->log_record_header_length)) {
- ntfs_error(vi->i_sb, "$LogFile restart area specifies "
- "inconsistent log record header length.");
- return false;
- }
- /* Dito for the log page data offset. */
- if (((le16_to_cpu(ra->log_page_data_offset) + 7) & ~7) !=
- le16_to_cpu(ra->log_page_data_offset)) {
- ntfs_error(vi->i_sb, "$LogFile restart area specifies "
- "inconsistent log page data offset.");
- return false;
- }
- ntfs_debug("Done.");
- return true;
-}
-
-/**
- * ntfs_check_log_client_array - check the log client array for consistency
- * @vi: $LogFile inode to which the restart page belongs
- * @rp: restart page whose log client array to check
- *
- * Check the log client array of the restart page @rp for consistency and
- * return 'true' if it is consistent and 'false' otherwise.
- *
- * This function assumes that the restart page header and the restart area have
- * already been consistency checked.
- *
- * Unlike ntfs_check_restart_page_header() and ntfs_check_restart_area(), this
- * function needs @rp->system_page_size bytes in @rp, i.e. it requires the full
- * restart page and the page must be multi sector transfer deprotected.
- */
-static bool ntfs_check_log_client_array(struct inode *vi,
- RESTART_PAGE_HEADER *rp)
-{
- RESTART_AREA *ra;
- LOG_CLIENT_RECORD *ca, *cr;
- u16 nr_clients, idx;
- bool in_free_list, idx_is_first;
-
- ntfs_debug("Entering.");
- ra = (RESTART_AREA*)((u8*)rp + le16_to_cpu(rp->restart_area_offset));
- ca = (LOG_CLIENT_RECORD*)((u8*)ra +
- le16_to_cpu(ra->client_array_offset));
- /*
- * Check the ra->client_free_list first and then check the
- * ra->client_in_use_list. Check each of the log client records in
- * each of the lists and check that the array does not overflow the
- * ra->log_clients value. Also keep track of the number of records
- * visited as there cannot be more than ra->log_clients records and
- * that way we detect eventual loops in within a list.
- */
- nr_clients = le16_to_cpu(ra->log_clients);
- idx = le16_to_cpu(ra->client_free_list);
- in_free_list = true;
-check_list:
- for (idx_is_first = true; idx != LOGFILE_NO_CLIENT_CPU; nr_clients--,
- idx = le16_to_cpu(cr->next_client)) {
- if (!nr_clients || idx >= le16_to_cpu(ra->log_clients))
- goto err_out;
- /* Set @cr to the current log client record. */
- cr = ca + idx;
- /* The first log client record must not have a prev_client. */
- if (idx_is_first) {
- if (cr->prev_client != LOGFILE_NO_CLIENT)
- goto err_out;
- idx_is_first = false;
- }
- }
- /* Switch to and check the in use list if we just did the free list. */
- if (in_free_list) {
- in_free_list = false;
- idx = le16_to_cpu(ra->client_in_use_list);
- goto check_list;
- }
- ntfs_debug("Done.");
- return true;
-err_out:
- ntfs_error(vi->i_sb, "$LogFile log client array is corrupt.");
- return false;
-}
-
-/**
- * ntfs_check_and_load_restart_page - check the restart page for consistency
- * @vi: $LogFile inode to which the restart page belongs
- * @rp: restart page to check
- * @pos: position in @vi at which the restart page resides
- * @wrp: [OUT] copy of the multi sector transfer deprotected restart page
- * @lsn: [OUT] set to the current logfile lsn on success
- *
- * Check the restart page @rp for consistency and return 0 if it is consistent
- * and -errno otherwise. The restart page may have been modified by chkdsk in
- * which case its magic is CHKD instead of RSTR.
- *
- * This function only needs NTFS_BLOCK_SIZE bytes in @rp, i.e. it does not
- * require the full restart page.
- *
- * If @wrp is not NULL, on success, *@wrp will point to a buffer containing a
- * copy of the complete multi sector transfer deprotected page. On failure,
- * *@wrp is undefined.
- *
- * Simillarly, if @lsn is not NULL, on success *@lsn will be set to the current
- * logfile lsn according to this restart page. On failure, *@lsn is undefined.
- *
- * The following error codes are defined:
- * -EINVAL - The restart page is inconsistent.
- * -ENOMEM - Not enough memory to load the restart page.
- * -EIO - Failed to reading from $LogFile.
- */
-static int ntfs_check_and_load_restart_page(struct inode *vi,
- RESTART_PAGE_HEADER *rp, s64 pos, RESTART_PAGE_HEADER **wrp,
- LSN *lsn)
-{
- RESTART_AREA *ra;
- RESTART_PAGE_HEADER *trp;
- int size, err;
-
- ntfs_debug("Entering.");
- /* Check the restart page header for consistency. */
- if (!ntfs_check_restart_page_header(vi, rp, pos)) {
- /* Error output already done inside the function. */
- return -EINVAL;
- }
- /* Check the restart area for consistency. */
- if (!ntfs_check_restart_area(vi, rp)) {
- /* Error output already done inside the function. */
- return -EINVAL;
- }
- ra = (RESTART_AREA*)((u8*)rp + le16_to_cpu(rp->restart_area_offset));
- /*
- * Allocate a buffer to store the whole restart page so we can multi
- * sector transfer deprotect it.
- */
- trp = ntfs_malloc_nofs(le32_to_cpu(rp->system_page_size));
- if (!trp) {
- ntfs_error(vi->i_sb, "Failed to allocate memory for $LogFile "
- "restart page buffer.");
- return -ENOMEM;
- }
- /*
- * Read the whole of the restart page into the buffer. If it fits
- * completely inside @rp, just copy it from there. Otherwise map all
- * the required pages and copy the data from them.
- */
- size = PAGE_SIZE - (pos & ~PAGE_MASK);
- if (size >= le32_to_cpu(rp->system_page_size)) {
- memcpy(trp, rp, le32_to_cpu(rp->system_page_size));
- } else {
- pgoff_t idx;
- struct page *page;
- int have_read, to_read;
-
- /* First copy what we already have in @rp. */
- memcpy(trp, rp, size);
- /* Copy the remaining data one page at a time. */
- have_read = size;
- to_read = le32_to_cpu(rp->system_page_size) - size;
- idx = (pos + size) >> PAGE_SHIFT;
- BUG_ON((pos + size) & ~PAGE_MASK);
- do {
- page = ntfs_map_page(vi->i_mapping, idx);
- if (IS_ERR(page)) {
- ntfs_error(vi->i_sb, "Error mapping $LogFile "
- "page (index %lu).", idx);
- err = PTR_ERR(page);
- if (err != -EIO && err != -ENOMEM)
- err = -EIO;
- goto err_out;
- }
- size = min_t(int, to_read, PAGE_SIZE);
- memcpy((u8*)trp + have_read, page_address(page), size);
- ntfs_unmap_page(page);
- have_read += size;
- to_read -= size;
- idx++;
- } while (to_read > 0);
- }
- /*
- * Perform the multi sector transfer deprotection on the buffer if the
- * restart page is protected.
- */
- if ((!ntfs_is_chkd_record(trp->magic) || le16_to_cpu(trp->usa_count))
- && post_read_mst_fixup((NTFS_RECORD*)trp,
- le32_to_cpu(rp->system_page_size))) {
- /*
- * A multi sector tranfer error was detected. We only need to
- * abort if the restart page contents exceed the multi sector
- * transfer fixup of the first sector.
- */
- if (le16_to_cpu(rp->restart_area_offset) +
- le16_to_cpu(ra->restart_area_length) >
- NTFS_BLOCK_SIZE - sizeof(u16)) {
- ntfs_error(vi->i_sb, "Multi sector transfer error "
- "detected in $LogFile restart page.");
- err = -EINVAL;
- goto err_out;
- }
- }
- /*
- * If the restart page is modified by chkdsk or there are no active
- * logfile clients, the logfile is consistent. Otherwise, need to
- * check the log client records for consistency, too.
- */
- err = 0;
- if (ntfs_is_rstr_record(rp->magic) &&
- ra->client_in_use_list != LOGFILE_NO_CLIENT) {
- if (!ntfs_check_log_client_array(vi, trp)) {
- err = -EINVAL;
- goto err_out;
- }
- }
- if (lsn) {
- if (ntfs_is_rstr_record(rp->magic))
- *lsn = sle64_to_cpu(ra->current_lsn);
- else /* if (ntfs_is_chkd_record(rp->magic)) */
- *lsn = sle64_to_cpu(rp->chkdsk_lsn);
- }
- ntfs_debug("Done.");
- if (wrp)
- *wrp = trp;
- else {
-err_out:
- ntfs_free(trp);
- }
- return err;
-}
-
-/**
- * ntfs_check_logfile - check the journal for consistency
- * @log_vi: struct inode of loaded journal $LogFile to check
- * @rp: [OUT] on success this is a copy of the current restart page
- *
- * Check the $LogFile journal for consistency and return 'true' if it is
- * consistent and 'false' if not. On success, the current restart page is
- * returned in *@rp. Caller must call ntfs_free(*@rp) when finished with it.
- *
- * At present we only check the two restart pages and ignore the log record
- * pages.
- *
- * Note that the MstProtected flag is not set on the $LogFile inode and hence
- * when reading pages they are not deprotected. This is because we do not know
- * if the $LogFile was created on a system with a different page size to ours
- * yet and mst deprotection would fail if our page size is smaller.
- */
-bool ntfs_check_logfile(struct inode *log_vi, RESTART_PAGE_HEADER **rp)
-{
- s64 size, pos;
- LSN rstr1_lsn, rstr2_lsn;
- ntfs_volume *vol = NTFS_SB(log_vi->i_sb);
- struct address_space *mapping = log_vi->i_mapping;
- struct page *page = NULL;
- u8 *kaddr = NULL;
- RESTART_PAGE_HEADER *rstr1_ph = NULL;
- RESTART_PAGE_HEADER *rstr2_ph = NULL;
- int log_page_size, err;
- bool logfile_is_empty = true;
- u8 log_page_bits;
-
- ntfs_debug("Entering.");
- /* An empty $LogFile must have been clean before it got emptied. */
- if (NVolLogFileEmpty(vol))
- goto is_empty;
- size = i_size_read(log_vi);
- /* Make sure the file doesn't exceed the maximum allowed size. */
- if (size > MaxLogFileSize)
- size = MaxLogFileSize;
- /*
- * Truncate size to a multiple of the page cache size or the default
- * log page size if the page cache size is between the default log page
- * log page size if the page cache size is between the default log page
- * size and twice that.
- */
- if (PAGE_SIZE >= DefaultLogPageSize && PAGE_SIZE <=
- DefaultLogPageSize * 2)
- log_page_size = DefaultLogPageSize;
- else
- log_page_size = PAGE_SIZE;
- /*
- * Use ntfs_ffs() instead of ffs() to enable the compiler to
- * optimize log_page_size and log_page_bits into constants.
- */
- log_page_bits = ntfs_ffs(log_page_size) - 1;
- size &= ~(s64)(log_page_size - 1);
- /*
- * Ensure the log file is big enough to store at least the two restart
- * pages and the minimum number of log record pages.
- */
- if (size < log_page_size * 2 || (size - log_page_size * 2) >>
- log_page_bits < MinLogRecordPages) {
- ntfs_error(vol->sb, "$LogFile is too small.");
- return false;
- }
- /*
- * Read through the file looking for a restart page. Since the restart
- * page header is at the beginning of a page we only need to search at
- * what could be the beginning of a page (for each page size) rather
- * than scanning the whole file byte by byte. If all potential places
- * contain empty and uninitialzed records, the log file can be assumed
- * to be empty.
- */
- for (pos = 0; pos < size; pos <<= 1) {
- pgoff_t idx = pos >> PAGE_SHIFT;
- if (!page || page->index != idx) {
- if (page)
- ntfs_unmap_page(page);
- page = ntfs_map_page(mapping, idx);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Error mapping $LogFile "
- "page (index %lu).", idx);
- goto err_out;
- }
- }
- kaddr = (u8*)page_address(page) + (pos & ~PAGE_MASK);
- /*
- * A non-empty block means the logfile is not empty while an
- * empty block after a non-empty block has been encountered
- * means we are done.
- */
- if (!ntfs_is_empty_recordp((le32*)kaddr))
- logfile_is_empty = false;
- else if (!logfile_is_empty)
- break;
- /*
- * A log record page means there cannot be a restart page after
- * this so no need to continue searching.
- */
- if (ntfs_is_rcrd_recordp((le32*)kaddr))
- break;
- /* If not a (modified by chkdsk) restart page, continue. */
- if (!ntfs_is_rstr_recordp((le32*)kaddr) &&
- !ntfs_is_chkd_recordp((le32*)kaddr)) {
- if (!pos)
- pos = NTFS_BLOCK_SIZE >> 1;
- continue;
- }
- /*
- * Check the (modified by chkdsk) restart page for consistency
- * and get a copy of the complete multi sector transfer
- * deprotected restart page.
- */
- err = ntfs_check_and_load_restart_page(log_vi,
- (RESTART_PAGE_HEADER*)kaddr, pos,
- !rstr1_ph ? &rstr1_ph : &rstr2_ph,
- !rstr1_ph ? &rstr1_lsn : &rstr2_lsn);
- if (!err) {
- /*
- * If we have now found the first (modified by chkdsk)
- * restart page, continue looking for the second one.
- */
- if (!pos) {
- pos = NTFS_BLOCK_SIZE >> 1;
- continue;
- }
- /*
- * We have now found the second (modified by chkdsk)
- * restart page, so we can stop looking.
- */
- break;
- }
- /*
- * Error output already done inside the function. Note, we do
- * not abort if the restart page was invalid as we might still
- * find a valid one further in the file.
- */
- if (err != -EINVAL) {
- ntfs_unmap_page(page);
- goto err_out;
- }
- /* Continue looking. */
- if (!pos)
- pos = NTFS_BLOCK_SIZE >> 1;
- }
- if (page)
- ntfs_unmap_page(page);
- if (logfile_is_empty) {
- NVolSetLogFileEmpty(vol);
-is_empty:
- ntfs_debug("Done. ($LogFile is empty.)");
- return true;
- }
- if (!rstr1_ph) {
- BUG_ON(rstr2_ph);
- ntfs_error(vol->sb, "Did not find any restart pages in "
- "$LogFile and it was not empty.");
- return false;
- }
- /* If both restart pages were found, use the more recent one. */
- if (rstr2_ph) {
- /*
- * If the second restart area is more recent, switch to it.
- * Otherwise just throw it away.
- */
- if (rstr2_lsn > rstr1_lsn) {
- ntfs_debug("Using second restart page as it is more "
- "recent.");
- ntfs_free(rstr1_ph);
- rstr1_ph = rstr2_ph;
- /* rstr1_lsn = rstr2_lsn; */
- } else {
- ntfs_debug("Using first restart page as it is more "
- "recent.");
- ntfs_free(rstr2_ph);
- }
- rstr2_ph = NULL;
- }
- /* All consistency checks passed. */
- if (rp)
- *rp = rstr1_ph;
- else
- ntfs_free(rstr1_ph);
- ntfs_debug("Done.");
- return true;
-err_out:
- if (rstr1_ph)
- ntfs_free(rstr1_ph);
- return false;
-}
-
-/**
- * ntfs_is_logfile_clean - check in the journal if the volume is clean
- * @log_vi: struct inode of loaded journal $LogFile to check
- * @rp: copy of the current restart page
- *
- * Analyze the $LogFile journal and return 'true' if it indicates the volume was
- * shutdown cleanly and 'false' if not.
- *
- * At present we only look at the two restart pages and ignore the log record
- * pages. This is a little bit crude in that there will be a very small number
- * of cases where we think that a volume is dirty when in fact it is clean.
- * This should only affect volumes that have not been shutdown cleanly but did
- * not have any pending, non-check-pointed i/o, i.e. they were completely idle
- * at least for the five seconds preceding the unclean shutdown.
- *
- * This function assumes that the $LogFile journal has already been consistency
- * checked by a call to ntfs_check_logfile() and in particular if the $LogFile
- * is empty this function requires that NVolLogFileEmpty() is true otherwise an
- * empty volume will be reported as dirty.
- */
-bool ntfs_is_logfile_clean(struct inode *log_vi, const RESTART_PAGE_HEADER *rp)
-{
- ntfs_volume *vol = NTFS_SB(log_vi->i_sb);
- RESTART_AREA *ra;
-
- ntfs_debug("Entering.");
- /* An empty $LogFile must have been clean before it got emptied. */
- if (NVolLogFileEmpty(vol)) {
- ntfs_debug("Done. ($LogFile is empty.)");
- return true;
- }
- BUG_ON(!rp);
- if (!ntfs_is_rstr_record(rp->magic) &&
- !ntfs_is_chkd_record(rp->magic)) {
- ntfs_error(vol->sb, "Restart page buffer is invalid. This is "
- "probably a bug in that the $LogFile should "
- "have been consistency checked before calling "
- "this function.");
- return false;
- }
- ra = (RESTART_AREA*)((u8*)rp + le16_to_cpu(rp->restart_area_offset));
- /*
- * If the $LogFile has active clients, i.e. it is open, and we do not
- * have the RESTART_VOLUME_IS_CLEAN bit set in the restart area flags,
- * we assume there was an unclean shutdown.
- */
- if (ra->client_in_use_list != LOGFILE_NO_CLIENT &&
- !(ra->flags & RESTART_VOLUME_IS_CLEAN)) {
- ntfs_debug("Done. $LogFile indicates a dirty shutdown.");
- return false;
- }
- /* $LogFile indicates a clean shutdown. */
- ntfs_debug("Done. $LogFile indicates a clean shutdown.");
- return true;
-}
-
-/**
- * ntfs_empty_logfile - empty the contents of the $LogFile journal
- * @log_vi: struct inode of loaded journal $LogFile to empty
- *
- * Empty the contents of the $LogFile journal @log_vi and return 'true' on
- * success and 'false' on error.
- *
- * This function assumes that the $LogFile journal has already been consistency
- * checked by a call to ntfs_check_logfile() and that ntfs_is_logfile_clean()
- * has been used to ensure that the $LogFile is clean.
- */
-bool ntfs_empty_logfile(struct inode *log_vi)
-{
- VCN vcn, end_vcn;
- ntfs_inode *log_ni = NTFS_I(log_vi);
- ntfs_volume *vol = log_ni->vol;
- struct super_block *sb = vol->sb;
- runlist_element *rl;
- unsigned long flags;
- unsigned block_size, block_size_bits;
- int err;
- bool should_wait = true;
-
- ntfs_debug("Entering.");
- if (NVolLogFileEmpty(vol)) {
- ntfs_debug("Done.");
- return true;
- }
- /*
- * We cannot use ntfs_attr_set() because we may be still in the middle
- * of a mount operation. Thus we do the emptying by hand by first
- * zapping the page cache pages for the $LogFile/$DATA attribute and
- * then emptying each of the buffers in each of the clusters specified
- * by the runlist by hand.
- */
- block_size = sb->s_blocksize;
- block_size_bits = sb->s_blocksize_bits;
- vcn = 0;
- read_lock_irqsave(&log_ni->size_lock, flags);
- end_vcn = (log_ni->initialized_size + vol->cluster_size_mask) >>
- vol->cluster_size_bits;
- read_unlock_irqrestore(&log_ni->size_lock, flags);
- truncate_inode_pages(log_vi->i_mapping, 0);
- down_write(&log_ni->runlist.lock);
- rl = log_ni->runlist.rl;
- if (unlikely(!rl || vcn < rl->vcn || !rl->length)) {
-map_vcn:
- err = ntfs_map_runlist_nolock(log_ni, vcn, NULL);
- if (err) {
- ntfs_error(sb, "Failed to map runlist fragment (error "
- "%d).", -err);
- goto err;
- }
- rl = log_ni->runlist.rl;
- BUG_ON(!rl || vcn < rl->vcn || !rl->length);
- }
- /* Seek to the runlist element containing @vcn. */
- while (rl->length && vcn >= rl[1].vcn)
- rl++;
- do {
- LCN lcn;
- sector_t block, end_block;
- s64 len;
-
- /*
- * If this run is not mapped map it now and start again as the
- * runlist will have been updated.
- */
- lcn = rl->lcn;
- if (unlikely(lcn == LCN_RL_NOT_MAPPED)) {
- vcn = rl->vcn;
- goto map_vcn;
- }
- /* If this run is not valid abort with an error. */
- if (unlikely(!rl->length || lcn < LCN_HOLE))
- goto rl_err;
- /* Skip holes. */
- if (lcn == LCN_HOLE)
- continue;
- block = lcn << vol->cluster_size_bits >> block_size_bits;
- len = rl->length;
- if (rl[1].vcn > end_vcn)
- len = end_vcn - rl->vcn;
- end_block = (lcn + len) << vol->cluster_size_bits >>
- block_size_bits;
- /* Iterate over the blocks in the run and empty them. */
- do {
- struct buffer_head *bh;
-
- /* Obtain the buffer, possibly not uptodate. */
- bh = sb_getblk(sb, block);
- BUG_ON(!bh);
- /* Setup buffer i/o submission. */
- lock_buffer(bh);
- bh->b_end_io = end_buffer_write_sync;
- get_bh(bh);
- /* Set the entire contents of the buffer to 0xff. */
- memset(bh->b_data, -1, block_size);
- if (!buffer_uptodate(bh))
- set_buffer_uptodate(bh);
- if (buffer_dirty(bh))
- clear_buffer_dirty(bh);
- /*
- * Submit the buffer and wait for i/o to complete but
- * only for the first buffer so we do not miss really
- * serious i/o errors. Once the first buffer has
- * completed ignore errors afterwards as we can assume
- * that if one buffer worked all of them will work.
- */
- submit_bh(REQ_OP_WRITE, bh);
- if (should_wait) {
- should_wait = false;
- wait_on_buffer(bh);
- if (unlikely(!buffer_uptodate(bh)))
- goto io_err;
- }
- brelse(bh);
- } while (++block < end_block);
- } while ((++rl)->vcn < end_vcn);
- up_write(&log_ni->runlist.lock);
- /*
- * Zap the pages again just in case any got instantiated whilst we were
- * emptying the blocks by hand. FIXME: We may not have completed
- * writing to all the buffer heads yet so this may happen too early.
- * We really should use a kernel thread to do the emptying
- * asynchronously and then we can also set the volume dirty and output
- * an error message if emptying should fail.
- */
- truncate_inode_pages(log_vi->i_mapping, 0);
- /* Set the flag so we do not have to do it again on remount. */
- NVolSetLogFileEmpty(vol);
- ntfs_debug("Done.");
- return true;
-io_err:
- ntfs_error(sb, "Failed to write buffer. Unmount and run chkdsk.");
- goto dirty_err;
-rl_err:
- ntfs_error(sb, "Runlist is corrupt. Unmount and run chkdsk.");
-dirty_err:
- NVolSetErrors(vol);
- err = -EIO;
-err:
- up_write(&log_ni->runlist.lock);
- ntfs_error(sb, "Failed to fill $LogFile with 0xff bytes (error %d).",
- -err);
- return false;
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/logfile.h b/fs/ntfs/logfile.h
deleted file mode 100644
index 429d4909cc72..000000000000
--- a/fs/ntfs/logfile.h
+++ /dev/null
@@ -1,295 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * logfile.h - Defines for NTFS kernel journal ($LogFile) handling. Part of
- * the Linux-NTFS project.
- *
- * Copyright (c) 2000-2005 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_LOGFILE_H
-#define _LINUX_NTFS_LOGFILE_H
-
-#ifdef NTFS_RW
-
-#include <linux/fs.h>
-
-#include "types.h"
-#include "endian.h"
-#include "layout.h"
-
-/*
- * Journal ($LogFile) organization:
- *
- * Two restart areas present in the first two pages (restart pages, one restart
- * area in each page). When the volume is dismounted they should be identical,
- * except for the update sequence array which usually has a different update
- * sequence number.
- *
- * These are followed by log records organized in pages headed by a log record
- * header going up to log file size. Not all pages contain log records when a
- * volume is first formatted, but as the volume ages, all records will be used.
- * When the log file fills up, the records at the beginning are purged (by
- * modifying the oldest_lsn to a higher value presumably) and writing begins
- * at the beginning of the file. Effectively, the log file is viewed as a
- * circular entity.
- *
- * NOTE: Windows NT, 2000, and XP all use log file version 1.1 but they accept
- * versions <= 1.x, including 0.-1. (Yes, that is a minus one in there!) We
- * probably only want to support 1.1 as this seems to be the current version
- * and we don't know how that differs from the older versions. The only
- * exception is if the journal is clean as marked by the two restart pages
- * then it doesn't matter whether we are on an earlier version. We can just
- * reinitialize the logfile and start again with version 1.1.
- */
-
-/* Some $LogFile related constants. */
-#define MaxLogFileSize 0x100000000ULL
-#define DefaultLogPageSize 4096
-#define MinLogRecordPages 48
-
-/*
- * Log file restart page header (begins the restart area).
- */
-typedef struct {
-/*Ofs*/
-/* 0 NTFS_RECORD; -- Unfolded here as gcc doesn't like unnamed structs. */
-/* 0*/ NTFS_RECORD_TYPE magic; /* The magic is "RSTR". */
-/* 4*/ le16 usa_ofs; /* See NTFS_RECORD definition in layout.h.
- When creating, set this to be immediately
- after this header structure (without any
- alignment). */
-/* 6*/ le16 usa_count; /* See NTFS_RECORD definition in layout.h. */
-
-/* 8*/ leLSN chkdsk_lsn; /* The last log file sequence number found by
- chkdsk. Only used when the magic is changed
- to "CHKD". Otherwise this is zero. */
-/* 16*/ le32 system_page_size; /* Byte size of system pages when the log file
- was created, has to be >= 512 and a power of
- 2. Use this to calculate the required size
- of the usa (usa_count) and add it to usa_ofs.
- Then verify that the result is less than the
- value of the restart_area_offset. */
-/* 20*/ le32 log_page_size; /* Byte size of log file pages, has to be >=
- 512 and a power of 2. The default is 4096
- and is used when the system page size is
- between 4096 and 8192. Otherwise this is
- set to the system page size instead. */
-/* 24*/ le16 restart_area_offset;/* Byte offset from the start of this header to
- the RESTART_AREA. Value has to be aligned
- to 8-byte boundary. When creating, set this
- to be after the usa. */
-/* 26*/ sle16 minor_ver; /* Log file minor version. Only check if major
- version is 1. */
-/* 28*/ sle16 major_ver; /* Log file major version. We only support
- version 1.1. */
-/* sizeof() = 30 (0x1e) bytes */
-} __attribute__ ((__packed__)) RESTART_PAGE_HEADER;
-
-/*
- * Constant for the log client indices meaning that there are no client records
- * in this particular client array. Also inside the client records themselves,
- * this means that there are no client records preceding or following this one.
- */
-#define LOGFILE_NO_CLIENT cpu_to_le16(0xffff)
-#define LOGFILE_NO_CLIENT_CPU 0xffff
-
-/*
- * These are the so far known RESTART_AREA_* flags (16-bit) which contain
- * information about the log file in which they are present.
- */
-enum {
- RESTART_VOLUME_IS_CLEAN = cpu_to_le16(0x0002),
- RESTART_SPACE_FILLER = cpu_to_le16(0xffff), /* gcc: Force enum bit width to 16. */
-} __attribute__ ((__packed__));
-
-typedef le16 RESTART_AREA_FLAGS;
-
-/*
- * Log file restart area record. The offset of this record is found by adding
- * the offset of the RESTART_PAGE_HEADER to the restart_area_offset value found
- * in it. See notes at restart_area_offset above.
- */
-typedef struct {
-/*Ofs*/
-/* 0*/ leLSN current_lsn; /* The current, i.e. last LSN inside the log
- when the restart area was last written.
- This happens often but what is the interval?
- Is it just fixed time or is it every time a
- check point is written or somethine else?
- On create set to 0. */
-/* 8*/ le16 log_clients; /* Number of log client records in the array of
- log client records which follows this
- restart area. Must be 1. */
-/* 10*/ le16 client_free_list; /* The index of the first free log client record
- in the array of log client records.
- LOGFILE_NO_CLIENT means that there are no
- free log client records in the array.
- If != LOGFILE_NO_CLIENT, check that
- log_clients > client_free_list. On Win2k
- and presumably earlier, on a clean volume
- this is != LOGFILE_NO_CLIENT, and it should
- be 0, i.e. the first (and only) client
- record is free and thus the logfile is
- closed and hence clean. A dirty volume
- would have left the logfile open and hence
- this would be LOGFILE_NO_CLIENT. On WinXP
- and presumably later, the logfile is always
- open, even on clean shutdown so this should
- always be LOGFILE_NO_CLIENT. */
-/* 12*/ le16 client_in_use_list;/* The index of the first in-use log client
- record in the array of log client records.
- LOGFILE_NO_CLIENT means that there are no
- in-use log client records in the array. If
- != LOGFILE_NO_CLIENT check that log_clients
- > client_in_use_list. On Win2k and
- presumably earlier, on a clean volume this
- is LOGFILE_NO_CLIENT, i.e. there are no
- client records in use and thus the logfile
- is closed and hence clean. A dirty volume
- would have left the logfile open and hence
- this would be != LOGFILE_NO_CLIENT, and it
- should be 0, i.e. the first (and only)
- client record is in use. On WinXP and
- presumably later, the logfile is always
- open, even on clean shutdown so this should
- always be 0. */
-/* 14*/ RESTART_AREA_FLAGS flags;/* Flags modifying LFS behaviour. On Win2k
- and presumably earlier this is always 0. On
- WinXP and presumably later, if the logfile
- was shutdown cleanly, the second bit,
- RESTART_VOLUME_IS_CLEAN, is set. This bit
- is cleared when the volume is mounted by
- WinXP and set when the volume is dismounted,
- thus if the logfile is dirty, this bit is
- clear. Thus we don't need to check the
- Windows version to determine if the logfile
- is clean. Instead if the logfile is closed,
- we know it must be clean. If it is open and
- this bit is set, we also know it must be
- clean. If on the other hand the logfile is
- open and this bit is clear, we can be almost
- certain that the logfile is dirty. */
-/* 16*/ le32 seq_number_bits; /* How many bits to use for the sequence
- number. This is calculated as 67 - the
- number of bits required to store the logfile
- size in bytes and this can be used in with
- the specified file_size as a consistency
- check. */
-/* 20*/ le16 restart_area_length;/* Length of the restart area including the
- client array. Following checks required if
- version matches. Otherwise, skip them.
- restart_area_offset + restart_area_length
- has to be <= system_page_size. Also,
- restart_area_length has to be >=
- client_array_offset + (log_clients *
- sizeof(log client record)). */
-/* 22*/ le16 client_array_offset;/* Offset from the start of this record to
- the first log client record if versions are
- matched. When creating, set this to be
- after this restart area structure, aligned
- to 8-bytes boundary. If the versions do not
- match, this is ignored and the offset is
- assumed to be (sizeof(RESTART_AREA) + 7) &
- ~7, i.e. rounded up to first 8-byte
- boundary. Either way, client_array_offset
- has to be aligned to an 8-byte boundary.
- Also, restart_area_offset +
- client_array_offset has to be <= 510.
- Finally, client_array_offset + (log_clients
- * sizeof(log client record)) has to be <=
- system_page_size. On Win2k and presumably
- earlier, this is 0x30, i.e. immediately
- following this record. On WinXP and
- presumably later, this is 0x40, i.e. there
- are 16 extra bytes between this record and
- the client array. This probably means that
- the RESTART_AREA record is actually bigger
- in WinXP and later. */
-/* 24*/ sle64 file_size; /* Usable byte size of the log file. If the
- restart_area_offset + the offset of the
- file_size are > 510 then corruption has
- occurred. This is the very first check when
- starting with the restart_area as if it
- fails it means that some of the above values
- will be corrupted by the multi sector
- transfer protection. The file_size has to
- be rounded down to be a multiple of the
- log_page_size in the RESTART_PAGE_HEADER and
- then it has to be at least big enough to
- store the two restart pages and 48 (0x30)
- log record pages. */
-/* 32*/ le32 last_lsn_data_length;/* Length of data of last LSN, not including
- the log record header. On create set to
- 0. */
-/* 36*/ le16 log_record_header_length;/* Byte size of the log record header.
- If the version matches then check that the
- value of log_record_header_length is a
- multiple of 8, i.e.
- (log_record_header_length + 7) & ~7 ==
- log_record_header_length. When creating set
- it to sizeof(LOG_RECORD_HEADER), aligned to
- 8 bytes. */
-/* 38*/ le16 log_page_data_offset;/* Offset to the start of data in a log record
- page. Must be a multiple of 8. On create
- set it to immediately after the update
- sequence array of the log record page. */
-/* 40*/ le32 restart_log_open_count;/* A counter that gets incremented every
- time the logfile is restarted which happens
- at mount time when the logfile is opened.
- When creating set to a random value. Win2k
- sets it to the low 32 bits of the current
- system time in NTFS format (see time.h). */
-/* 44*/ le32 reserved; /* Reserved/alignment to 8-byte boundary. */
-/* sizeof() = 48 (0x30) bytes */
-} __attribute__ ((__packed__)) RESTART_AREA;
-
-/*
- * Log client record. The offset of this record is found by adding the offset
- * of the RESTART_AREA to the client_array_offset value found in it.
- */
-typedef struct {
-/*Ofs*/
-/* 0*/ leLSN oldest_lsn; /* Oldest LSN needed by this client. On create
- set to 0. */
-/* 8*/ leLSN client_restart_lsn;/* LSN at which this client needs to restart
- the volume, i.e. the current position within
- the log file. At present, if clean this
- should = current_lsn in restart area but it
- probably also = current_lsn when dirty most
- of the time. At create set to 0. */
-/* 16*/ le16 prev_client; /* The offset to the previous log client record
- in the array of log client records.
- LOGFILE_NO_CLIENT means there is no previous
- client record, i.e. this is the first one.
- This is always LOGFILE_NO_CLIENT. */
-/* 18*/ le16 next_client; /* The offset to the next log client record in
- the array of log client records.
- LOGFILE_NO_CLIENT means there are no next
- client records, i.e. this is the last one.
- This is always LOGFILE_NO_CLIENT. */
-/* 20*/ le16 seq_number; /* On Win2k and presumably earlier, this is set
- to zero every time the logfile is restarted
- and it is incremented when the logfile is
- closed at dismount time. Thus it is 0 when
- dirty and 1 when clean. On WinXP and
- presumably later, this is always 0. */
-/* 22*/ u8 reserved[6]; /* Reserved/alignment. */
-/* 28*/ le32 client_name_length;/* Length of client name in bytes. Should
- always be 8. */
-/* 32*/ ntfschar client_name[64];/* Name of the client in Unicode. Should
- always be "NTFS" with the remaining bytes
- set to 0. */
-/* sizeof() = 160 (0xa0) bytes */
-} __attribute__ ((__packed__)) LOG_CLIENT_RECORD;
-
-extern bool ntfs_check_logfile(struct inode *log_vi,
- RESTART_PAGE_HEADER **rp);
-
-extern bool ntfs_is_logfile_clean(struct inode *log_vi,
- const RESTART_PAGE_HEADER *rp);
-
-extern bool ntfs_empty_logfile(struct inode *log_vi);
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_LOGFILE_H */
diff --git a/fs/ntfs/malloc.h b/fs/ntfs/malloc.h
deleted file mode 100644
index 7068425735f1..000000000000
--- a/fs/ntfs/malloc.h
+++ /dev/null
@@ -1,77 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * malloc.h - NTFS kernel memory handling. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2005 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_MALLOC_H
-#define _LINUX_NTFS_MALLOC_H
-
-#include <linux/vmalloc.h>
-#include <linux/slab.h>
-#include <linux/highmem.h>
-
-/**
- * __ntfs_malloc - allocate memory in multiples of pages
- * @size: number of bytes to allocate
- * @gfp_mask: extra flags for the allocator
- *
- * Internal function. You probably want ntfs_malloc_nofs()...
- *
- * Allocates @size bytes of memory, rounded up to multiples of PAGE_SIZE and
- * returns a pointer to the allocated memory.
- *
- * If there was insufficient memory to complete the request, return NULL.
- * Depending on @gfp_mask the allocation may be guaranteed to succeed.
- */
-static inline void *__ntfs_malloc(unsigned long size, gfp_t gfp_mask)
-{
- if (likely(size <= PAGE_SIZE)) {
- BUG_ON(!size);
- /* kmalloc() has per-CPU caches so is faster for now. */
- return kmalloc(PAGE_SIZE, gfp_mask & ~__GFP_HIGHMEM);
- /* return (void *)__get_free_page(gfp_mask); */
- }
- if (likely((size >> PAGE_SHIFT) < totalram_pages()))
- return __vmalloc(size, gfp_mask);
- return NULL;
-}
-
-/**
- * ntfs_malloc_nofs - allocate memory in multiples of pages
- * @size: number of bytes to allocate
- *
- * Allocates @size bytes of memory, rounded up to multiples of PAGE_SIZE and
- * returns a pointer to the allocated memory.
- *
- * If there was insufficient memory to complete the request, return NULL.
- */
-static inline void *ntfs_malloc_nofs(unsigned long size)
-{
- return __ntfs_malloc(size, GFP_NOFS | __GFP_HIGHMEM);
-}
-
-/**
- * ntfs_malloc_nofs_nofail - allocate memory in multiples of pages
- * @size: number of bytes to allocate
- *
- * Allocates @size bytes of memory, rounded up to multiples of PAGE_SIZE and
- * returns a pointer to the allocated memory.
- *
- * This function guarantees that the allocation will succeed. It will sleep
- * for as long as it takes to complete the allocation.
- *
- * If there was insufficient memory to complete the request, return NULL.
- */
-static inline void *ntfs_malloc_nofs_nofail(unsigned long size)
-{
- return __ntfs_malloc(size, GFP_NOFS | __GFP_HIGHMEM | __GFP_NOFAIL);
-}
-
-static inline void ntfs_free(void *addr)
-{
- kvfree(addr);
-}
-
-#endif /* _LINUX_NTFS_MALLOC_H */
diff --git a/fs/ntfs/mft.c b/fs/ntfs/mft.c
deleted file mode 100644
index f7bf5ce960cc..000000000000
--- a/fs/ntfs/mft.c
+++ /dev/null
@@ -1,2906 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/**
- * mft.c - NTFS kernel mft record operations. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2012 Anton Altaparmakov and Tuxera Inc.
- * Copyright (c) 2002 Richard Russon
- */
-
-#include <linux/buffer_head.h>
-#include <linux/slab.h>
-#include <linux/swap.h>
-#include <linux/bio.h>
-
-#include "attrib.h"
-#include "aops.h"
-#include "bitmap.h"
-#include "debug.h"
-#include "dir.h"
-#include "lcnalloc.h"
-#include "malloc.h"
-#include "mft.h"
-#include "ntfs.h"
-
-#define MAX_BHS (PAGE_SIZE / NTFS_BLOCK_SIZE)
-
-/**
- * map_mft_record_page - map the page in which a specific mft record resides
- * @ni: ntfs inode whose mft record page to map
- *
- * This maps the page in which the mft record of the ntfs inode @ni is situated
- * and returns a pointer to the mft record within the mapped page.
- *
- * Return value needs to be checked with IS_ERR() and if that is true PTR_ERR()
- * contains the negative error code returned.
- */
-static inline MFT_RECORD *map_mft_record_page(ntfs_inode *ni)
-{
- loff_t i_size;
- ntfs_volume *vol = ni->vol;
- struct inode *mft_vi = vol->mft_ino;
- struct page *page;
- unsigned long index, end_index;
- unsigned ofs;
-
- BUG_ON(ni->page);
- /*
- * The index into the page cache and the offset within the page cache
- * page of the wanted mft record. FIXME: We need to check for
- * overflowing the unsigned long, but I don't think we would ever get
- * here if the volume was that big...
- */
- index = (u64)ni->mft_no << vol->mft_record_size_bits >>
- PAGE_SHIFT;
- ofs = (ni->mft_no << vol->mft_record_size_bits) & ~PAGE_MASK;
-
- i_size = i_size_read(mft_vi);
- /* The maximum valid index into the page cache for $MFT's data. */
- end_index = i_size >> PAGE_SHIFT;
-
- /* If the wanted index is out of bounds the mft record doesn't exist. */
- if (unlikely(index >= end_index)) {
- if (index > end_index || (i_size & ~PAGE_MASK) < ofs +
- vol->mft_record_size) {
- page = ERR_PTR(-ENOENT);
- ntfs_error(vol->sb, "Attempt to read mft record 0x%lx, "
- "which is beyond the end of the mft. "
- "This is probably a bug in the ntfs "
- "driver.", ni->mft_no);
- goto err_out;
- }
- }
- /* Read, map, and pin the page. */
- page = ntfs_map_page(mft_vi->i_mapping, index);
- if (!IS_ERR(page)) {
- /* Catch multi sector transfer fixup errors. */
- if (likely(ntfs_is_mft_recordp((le32*)(page_address(page) +
- ofs)))) {
- ni->page = page;
- ni->page_ofs = ofs;
- return page_address(page) + ofs;
- }
- ntfs_error(vol->sb, "Mft record 0x%lx is corrupt. "
- "Run chkdsk.", ni->mft_no);
- ntfs_unmap_page(page);
- page = ERR_PTR(-EIO);
- NVolSetErrors(vol);
- }
-err_out:
- ni->page = NULL;
- ni->page_ofs = 0;
- return (void*)page;
-}
-
-/**
- * map_mft_record - map, pin and lock an mft record
- * @ni: ntfs inode whose MFT record to map
- *
- * First, take the mrec_lock mutex. We might now be sleeping, while waiting
- * for the mutex if it was already locked by someone else.
- *
- * The page of the record is mapped using map_mft_record_page() before being
- * returned to the caller.
- *
- * This in turn uses ntfs_map_page() to get the page containing the wanted mft
- * record (it in turn calls read_cache_page() which reads it in from disk if
- * necessary, increments the use count on the page so that it cannot disappear
- * under us and returns a reference to the page cache page).
- *
- * If read_cache_page() invokes ntfs_readpage() to load the page from disk, it
- * sets PG_locked and clears PG_uptodate on the page. Once I/O has completed
- * and the post-read mst fixups on each mft record in the page have been
- * performed, the page gets PG_uptodate set and PG_locked cleared (this is done
- * in our asynchronous I/O completion handler end_buffer_read_mft_async()).
- * ntfs_map_page() waits for PG_locked to become clear and checks if
- * PG_uptodate is set and returns an error code if not. This provides
- * sufficient protection against races when reading/using the page.
- *
- * However there is the write mapping to think about. Doing the above described
- * checking here will be fine, because when initiating the write we will set
- * PG_locked and clear PG_uptodate making sure nobody is touching the page
- * contents. Doing the locking this way means that the commit to disk code in
- * the page cache code paths is automatically sufficiently locked with us as
- * we will not touch a page that has been locked or is not uptodate. The only
- * locking problem then is them locking the page while we are accessing it.
- *
- * So that code will end up having to own the mrec_lock of all mft
- * records/inodes present in the page before I/O can proceed. In that case we
- * wouldn't need to bother with PG_locked and PG_uptodate as nobody will be
- * accessing anything without owning the mrec_lock mutex. But we do need to
- * use them because of the read_cache_page() invocation and the code becomes so
- * much simpler this way that it is well worth it.
- *
- * The mft record is now ours and we return a pointer to it. You need to check
- * the returned pointer with IS_ERR() and if that is true, PTR_ERR() will return
- * the error code.
- *
- * NOTE: Caller is responsible for setting the mft record dirty before calling
- * unmap_mft_record(). This is obviously only necessary if the caller really
- * modified the mft record...
- * Q: Do we want to recycle one of the VFS inode state bits instead?
- * A: No, the inode ones mean we want to change the mft record, not we want to
- * write it out.
- */
-MFT_RECORD *map_mft_record(ntfs_inode *ni)
-{
- MFT_RECORD *m;
-
- ntfs_debug("Entering for mft_no 0x%lx.", ni->mft_no);
-
- /* Make sure the ntfs inode doesn't go away. */
- atomic_inc(&ni->count);
-
- /* Serialize access to this mft record. */
- mutex_lock(&ni->mrec_lock);
-
- m = map_mft_record_page(ni);
- if (!IS_ERR(m))
- return m;
-
- mutex_unlock(&ni->mrec_lock);
- atomic_dec(&ni->count);
- ntfs_error(ni->vol->sb, "Failed with error code %lu.", -PTR_ERR(m));
- return m;
-}
-
-/**
- * unmap_mft_record_page - unmap the page in which a specific mft record resides
- * @ni: ntfs inode whose mft record page to unmap
- *
- * This unmaps the page in which the mft record of the ntfs inode @ni is
- * situated and returns. This is a NOOP if highmem is not configured.
- *
- * The unmap happens via ntfs_unmap_page() which in turn decrements the use
- * count on the page thus releasing it from the pinned state.
- *
- * We do not actually unmap the page from memory of course, as that will be
- * done by the page cache code itself when memory pressure increases or
- * whatever.
- */
-static inline void unmap_mft_record_page(ntfs_inode *ni)
-{
- BUG_ON(!ni->page);
-
- // TODO: If dirty, blah...
- ntfs_unmap_page(ni->page);
- ni->page = NULL;
- ni->page_ofs = 0;
- return;
-}
-
-/**
- * unmap_mft_record - release a mapped mft record
- * @ni: ntfs inode whose MFT record to unmap
- *
- * We release the page mapping and the mrec_lock mutex which unmaps the mft
- * record and releases it for others to get hold of. We also release the ntfs
- * inode by decrementing the ntfs inode reference count.
- *
- * NOTE: If caller has modified the mft record, it is imperative to set the mft
- * record dirty BEFORE calling unmap_mft_record().
- */
-void unmap_mft_record(ntfs_inode *ni)
-{
- struct page *page = ni->page;
-
- BUG_ON(!page);
-
- ntfs_debug("Entering for mft_no 0x%lx.", ni->mft_no);
-
- unmap_mft_record_page(ni);
- mutex_unlock(&ni->mrec_lock);
- atomic_dec(&ni->count);
- /*
- * If pure ntfs_inode, i.e. no vfs inode attached, we leave it to
- * ntfs_clear_extent_inode() in the extent inode case, and to the
- * caller in the non-extent, yet pure ntfs inode case, to do the actual
- * tear down of all structures and freeing of all allocated memory.
- */
- return;
-}
-
-/**
- * map_extent_mft_record - load an extent inode and attach it to its base
- * @base_ni: base ntfs inode
- * @mref: mft reference of the extent inode to load
- * @ntfs_ino: on successful return, pointer to the ntfs_inode structure
- *
- * Load the extent mft record @mref and attach it to its base inode @base_ni.
- * Return the mapped extent mft record if IS_ERR(result) is false. Otherwise
- * PTR_ERR(result) gives the negative error code.
- *
- * On successful return, @ntfs_ino contains a pointer to the ntfs_inode
- * structure of the mapped extent inode.
- */
-MFT_RECORD *map_extent_mft_record(ntfs_inode *base_ni, MFT_REF mref,
- ntfs_inode **ntfs_ino)
-{
- MFT_RECORD *m;
- ntfs_inode *ni = NULL;
- ntfs_inode **extent_nis = NULL;
- int i;
- unsigned long mft_no = MREF(mref);
- u16 seq_no = MSEQNO(mref);
- bool destroy_ni = false;
-
- ntfs_debug("Mapping extent mft record 0x%lx (base mft record 0x%lx).",
- mft_no, base_ni->mft_no);
- /* Make sure the base ntfs inode doesn't go away. */
- atomic_inc(&base_ni->count);
- /*
- * Check if this extent inode has already been added to the base inode,
- * in which case just return it. If not found, add it to the base
- * inode before returning it.
- */
- mutex_lock(&base_ni->extent_lock);
- if (base_ni->nr_extents > 0) {
- extent_nis = base_ni->ext.extent_ntfs_inos;
- for (i = 0; i < base_ni->nr_extents; i++) {
- if (mft_no != extent_nis[i]->mft_no)
- continue;
- ni = extent_nis[i];
- /* Make sure the ntfs inode doesn't go away. */
- atomic_inc(&ni->count);
- break;
- }
- }
- if (likely(ni != NULL)) {
- mutex_unlock(&base_ni->extent_lock);
- atomic_dec(&base_ni->count);
- /* We found the record; just have to map and return it. */
- m = map_mft_record(ni);
- /* map_mft_record() has incremented this on success. */
- atomic_dec(&ni->count);
- if (!IS_ERR(m)) {
- /* Verify the sequence number. */
- if (likely(le16_to_cpu(m->sequence_number) == seq_no)) {
- ntfs_debug("Done 1.");
- *ntfs_ino = ni;
- return m;
- }
- unmap_mft_record(ni);
- ntfs_error(base_ni->vol->sb, "Found stale extent mft "
- "reference! Corrupt filesystem. "
- "Run chkdsk.");
- return ERR_PTR(-EIO);
- }
-map_err_out:
- ntfs_error(base_ni->vol->sb, "Failed to map extent "
- "mft record, error code %ld.", -PTR_ERR(m));
- return m;
- }
- /* Record wasn't there. Get a new ntfs inode and initialize it. */
- ni = ntfs_new_extent_inode(base_ni->vol->sb, mft_no);
- if (unlikely(!ni)) {
- mutex_unlock(&base_ni->extent_lock);
- atomic_dec(&base_ni->count);
- return ERR_PTR(-ENOMEM);
- }
- ni->vol = base_ni->vol;
- ni->seq_no = seq_no;
- ni->nr_extents = -1;
- ni->ext.base_ntfs_ino = base_ni;
- /* Now map the record. */
- m = map_mft_record(ni);
- if (IS_ERR(m)) {
- mutex_unlock(&base_ni->extent_lock);
- atomic_dec(&base_ni->count);
- ntfs_clear_extent_inode(ni);
- goto map_err_out;
- }
- /* Verify the sequence number if it is present. */
- if (seq_no && (le16_to_cpu(m->sequence_number) != seq_no)) {
- ntfs_error(base_ni->vol->sb, "Found stale extent mft "
- "reference! Corrupt filesystem. Run chkdsk.");
- destroy_ni = true;
- m = ERR_PTR(-EIO);
- goto unm_err_out;
- }
- /* Attach extent inode to base inode, reallocating memory if needed. */
- if (!(base_ni->nr_extents & 3)) {
- ntfs_inode **tmp;
- int new_size = (base_ni->nr_extents + 4) * sizeof(ntfs_inode *);
-
- tmp = kmalloc(new_size, GFP_NOFS);
- if (unlikely(!tmp)) {
- ntfs_error(base_ni->vol->sb, "Failed to allocate "
- "internal buffer.");
- destroy_ni = true;
- m = ERR_PTR(-ENOMEM);
- goto unm_err_out;
- }
- if (base_ni->nr_extents) {
- BUG_ON(!base_ni->ext.extent_ntfs_inos);
- memcpy(tmp, base_ni->ext.extent_ntfs_inos, new_size -
- 4 * sizeof(ntfs_inode *));
- kfree(base_ni->ext.extent_ntfs_inos);
- }
- base_ni->ext.extent_ntfs_inos = tmp;
- }
- base_ni->ext.extent_ntfs_inos[base_ni->nr_extents++] = ni;
- mutex_unlock(&base_ni->extent_lock);
- atomic_dec(&base_ni->count);
- ntfs_debug("Done 2.");
- *ntfs_ino = ni;
- return m;
-unm_err_out:
- unmap_mft_record(ni);
- mutex_unlock(&base_ni->extent_lock);
- atomic_dec(&base_ni->count);
- /*
- * If the extent inode was not attached to the base inode we need to
- * release it or we will leak memory.
- */
- if (destroy_ni)
- ntfs_clear_extent_inode(ni);
- return m;
-}
-
-#ifdef NTFS_RW
-
-/**
- * __mark_mft_record_dirty - set the mft record and the page containing it dirty
- * @ni: ntfs inode describing the mapped mft record
- *
- * Internal function. Users should call mark_mft_record_dirty() instead.
- *
- * Set the mapped (extent) mft record of the (base or extent) ntfs inode @ni,
- * as well as the page containing the mft record, dirty. Also, mark the base
- * vfs inode dirty. This ensures that any changes to the mft record are
- * written out to disk.
- *
- * NOTE: We only set I_DIRTY_DATASYNC (and not I_DIRTY_PAGES)
- * on the base vfs inode, because even though file data may have been modified,
- * it is dirty in the inode meta data rather than the data page cache of the
- * inode, and thus there are no data pages that need writing out. Therefore, a
- * full mark_inode_dirty() is overkill. A mark_inode_dirty_sync(), on the
- * other hand, is not sufficient, because ->write_inode needs to be called even
- * in case of fdatasync. This needs to happen or the file data would not
- * necessarily hit the device synchronously, even though the vfs inode has the
- * O_SYNC flag set. Also, I_DIRTY_DATASYNC simply "feels" better than just
- * I_DIRTY_SYNC, since the file data has not actually hit the block device yet,
- * which is not what I_DIRTY_SYNC on its own would suggest.
- */
-void __mark_mft_record_dirty(ntfs_inode *ni)
-{
- ntfs_inode *base_ni;
-
- ntfs_debug("Entering for inode 0x%lx.", ni->mft_no);
- BUG_ON(NInoAttr(ni));
- mark_ntfs_record_dirty(ni->page, ni->page_ofs);
- /* Determine the base vfs inode and mark it dirty, too. */
- mutex_lock(&ni->extent_lock);
- if (likely(ni->nr_extents >= 0))
- base_ni = ni;
- else
- base_ni = ni->ext.base_ntfs_ino;
- mutex_unlock(&ni->extent_lock);
- __mark_inode_dirty(VFS_I(base_ni), I_DIRTY_DATASYNC);
-}
-
-static const char *ntfs_please_email = "Please email "
- "linux-ntfs-dev@lists.sourceforge.net and say that you saw "
- "this message. Thank you.";
-
-/**
- * ntfs_sync_mft_mirror_umount - synchronise an mft record to the mft mirror
- * @vol: ntfs volume on which the mft record to synchronize resides
- * @mft_no: mft record number of mft record to synchronize
- * @m: mapped, mst protected (extent) mft record to synchronize
- *
- * Write the mapped, mst protected (extent) mft record @m with mft record
- * number @mft_no to the mft mirror ($MFTMirr) of the ntfs volume @vol,
- * bypassing the page cache and the $MFTMirr inode itself.
- *
- * This function is only for use at umount time when the mft mirror inode has
- * already been disposed off. We BUG() if we are called while the mft mirror
- * inode is still attached to the volume.
- *
- * On success return 0. On error return -errno.
- *
- * NOTE: This function is not implemented yet as I am not convinced it can
- * actually be triggered considering the sequence of commits we do in super.c::
- * ntfs_put_super(). But just in case we provide this place holder as the
- * alternative would be either to BUG() or to get a NULL pointer dereference
- * and Oops.
- */
-static int ntfs_sync_mft_mirror_umount(ntfs_volume *vol,
- const unsigned long mft_no, MFT_RECORD *m)
-{
- BUG_ON(vol->mftmirr_ino);
- ntfs_error(vol->sb, "Umount time mft mirror syncing is not "
- "implemented yet. %s", ntfs_please_email);
- return -EOPNOTSUPP;
-}
-
-/**
- * ntfs_sync_mft_mirror - synchronize an mft record to the mft mirror
- * @vol: ntfs volume on which the mft record to synchronize resides
- * @mft_no: mft record number of mft record to synchronize
- * @m: mapped, mst protected (extent) mft record to synchronize
- * @sync: if true, wait for i/o completion
- *
- * Write the mapped, mst protected (extent) mft record @m with mft record
- * number @mft_no to the mft mirror ($MFTMirr) of the ntfs volume @vol.
- *
- * On success return 0. On error return -errno and set the volume errors flag
- * in the ntfs volume @vol.
- *
- * NOTE: We always perform synchronous i/o and ignore the @sync parameter.
- *
- * TODO: If @sync is false, want to do truly asynchronous i/o, i.e. just
- * schedule i/o via ->writepage or do it via kntfsd or whatever.
- */
-int ntfs_sync_mft_mirror(ntfs_volume *vol, const unsigned long mft_no,
- MFT_RECORD *m, int sync)
-{
- struct page *page;
- unsigned int blocksize = vol->sb->s_blocksize;
- int max_bhs = vol->mft_record_size / blocksize;
- struct buffer_head *bhs[MAX_BHS];
- struct buffer_head *bh, *head;
- u8 *kmirr;
- runlist_element *rl;
- unsigned int block_start, block_end, m_start, m_end, page_ofs;
- int i_bhs, nr_bhs, err = 0;
- unsigned char blocksize_bits = vol->sb->s_blocksize_bits;
-
- ntfs_debug("Entering for inode 0x%lx.", mft_no);
- BUG_ON(!max_bhs);
- if (WARN_ON(max_bhs > MAX_BHS))
- return -EINVAL;
- if (unlikely(!vol->mftmirr_ino)) {
- /* This could happen during umount... */
- err = ntfs_sync_mft_mirror_umount(vol, mft_no, m);
- if (likely(!err))
- return err;
- goto err_out;
- }
- /* Get the page containing the mirror copy of the mft record @m. */
- page = ntfs_map_page(vol->mftmirr_ino->i_mapping, mft_no >>
- (PAGE_SHIFT - vol->mft_record_size_bits));
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to map mft mirror page.");
- err = PTR_ERR(page);
- goto err_out;
- }
- lock_page(page);
- BUG_ON(!PageUptodate(page));
- ClearPageUptodate(page);
- /* Offset of the mft mirror record inside the page. */
- page_ofs = (mft_no << vol->mft_record_size_bits) & ~PAGE_MASK;
- /* The address in the page of the mirror copy of the mft record @m. */
- kmirr = page_address(page) + page_ofs;
- /* Copy the mst protected mft record to the mirror. */
- memcpy(kmirr, m, vol->mft_record_size);
- /* Create uptodate buffers if not present. */
- if (unlikely(!page_has_buffers(page))) {
- struct buffer_head *tail;
-
- bh = head = alloc_page_buffers(page, blocksize, true);
- do {
- set_buffer_uptodate(bh);
- tail = bh;
- bh = bh->b_this_page;
- } while (bh);
- tail->b_this_page = head;
- attach_page_private(page, head);
- }
- bh = head = page_buffers(page);
- BUG_ON(!bh);
- rl = NULL;
- nr_bhs = 0;
- block_start = 0;
- m_start = kmirr - (u8*)page_address(page);
- m_end = m_start + vol->mft_record_size;
- do {
- block_end = block_start + blocksize;
- /* If the buffer is outside the mft record, skip it. */
- if (block_end <= m_start)
- continue;
- if (unlikely(block_start >= m_end))
- break;
- /* Need to map the buffer if it is not mapped already. */
- if (unlikely(!buffer_mapped(bh))) {
- VCN vcn;
- LCN lcn;
- unsigned int vcn_ofs;
-
- bh->b_bdev = vol->sb->s_bdev;
- /* Obtain the vcn and offset of the current block. */
- vcn = ((VCN)mft_no << vol->mft_record_size_bits) +
- (block_start - m_start);
- vcn_ofs = vcn & vol->cluster_size_mask;
- vcn >>= vol->cluster_size_bits;
- if (!rl) {
- down_read(&NTFS_I(vol->mftmirr_ino)->
- runlist.lock);
- rl = NTFS_I(vol->mftmirr_ino)->runlist.rl;
- /*
- * $MFTMirr always has the whole of its runlist
- * in memory.
- */
- BUG_ON(!rl);
- }
- /* Seek to element containing target vcn. */
- while (rl->length && rl[1].vcn <= vcn)
- rl++;
- lcn = ntfs_rl_vcn_to_lcn(rl, vcn);
- /* For $MFTMirr, only lcn >= 0 is a successful remap. */
- if (likely(lcn >= 0)) {
- /* Setup buffer head to correct block. */
- bh->b_blocknr = ((lcn <<
- vol->cluster_size_bits) +
- vcn_ofs) >> blocksize_bits;
- set_buffer_mapped(bh);
- } else {
- bh->b_blocknr = -1;
- ntfs_error(vol->sb, "Cannot write mft mirror "
- "record 0x%lx because its "
- "location on disk could not "
- "be determined (error code "
- "%lli).", mft_no,
- (long long)lcn);
- err = -EIO;
- }
- }
- BUG_ON(!buffer_uptodate(bh));
- BUG_ON(!nr_bhs && (m_start != block_start));
- BUG_ON(nr_bhs >= max_bhs);
- bhs[nr_bhs++] = bh;
- BUG_ON((nr_bhs >= max_bhs) && (m_end != block_end));
- } while (block_start = block_end, (bh = bh->b_this_page) != head);
- if (unlikely(rl))
- up_read(&NTFS_I(vol->mftmirr_ino)->runlist.lock);
- if (likely(!err)) {
- /* Lock buffers and start synchronous write i/o on them. */
- for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) {
- struct buffer_head *tbh = bhs[i_bhs];
-
- if (!trylock_buffer(tbh))
- BUG();
- BUG_ON(!buffer_uptodate(tbh));
- clear_buffer_dirty(tbh);
- get_bh(tbh);
- tbh->b_end_io = end_buffer_write_sync;
- submit_bh(REQ_OP_WRITE, tbh);
- }
- /* Wait on i/o completion of buffers. */
- for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) {
- struct buffer_head *tbh = bhs[i_bhs];
-
- wait_on_buffer(tbh);
- if (unlikely(!buffer_uptodate(tbh))) {
- err = -EIO;
- /*
- * Set the buffer uptodate so the page and
- * buffer states do not become out of sync.
- */
- set_buffer_uptodate(tbh);
- }
- }
- } else /* if (unlikely(err)) */ {
- /* Clean the buffers. */
- for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++)
- clear_buffer_dirty(bhs[i_bhs]);
- }
- /* Current state: all buffers are clean, unlocked, and uptodate. */
- /* Remove the mst protection fixups again. */
- post_write_mst_fixup((NTFS_RECORD*)kmirr);
- flush_dcache_page(page);
- SetPageUptodate(page);
- unlock_page(page);
- ntfs_unmap_page(page);
- if (likely(!err)) {
- ntfs_debug("Done.");
- } else {
- ntfs_error(vol->sb, "I/O error while writing mft mirror "
- "record 0x%lx!", mft_no);
-err_out:
- ntfs_error(vol->sb, "Failed to synchronize $MFTMirr (error "
- "code %i). Volume will be left marked dirty "
- "on umount. Run ntfsfix on the partition "
- "after umounting to correct this.", -err);
- NVolSetErrors(vol);
- }
- return err;
-}
-
-/**
- * write_mft_record_nolock - write out a mapped (extent) mft record
- * @ni: ntfs inode describing the mapped (extent) mft record
- * @m: mapped (extent) mft record to write
- * @sync: if true, wait for i/o completion
- *
- * Write the mapped (extent) mft record @m described by the (regular or extent)
- * ntfs inode @ni to backing store. If the mft record @m has a counterpart in
- * the mft mirror, that is also updated.
- *
- * We only write the mft record if the ntfs inode @ni is dirty and the first
- * buffer belonging to its mft record is dirty, too. We ignore the dirty state
- * of subsequent buffers because we could have raced with
- * fs/ntfs/aops.c::mark_ntfs_record_dirty().
- *
- * On success, clean the mft record and return 0. On error, leave the mft
- * record dirty and return -errno.
- *
- * NOTE: We always perform synchronous i/o and ignore the @sync parameter.
- * However, if the mft record has a counterpart in the mft mirror and @sync is
- * true, we write the mft record, wait for i/o completion, and only then write
- * the mft mirror copy. This ensures that if the system crashes either the mft
- * or the mft mirror will contain a self-consistent mft record @m. If @sync is
- * false on the other hand, we start i/o on both and then wait for completion
- * on them. This provides a speedup but no longer guarantees that you will end
- * up with a self-consistent mft record in the case of a crash but if you asked
- * for asynchronous writing you probably do not care about that anyway.
- *
- * TODO: If @sync is false, want to do truly asynchronous i/o, i.e. just
- * schedule i/o via ->writepage or do it via kntfsd or whatever.
- */
-int write_mft_record_nolock(ntfs_inode *ni, MFT_RECORD *m, int sync)
-{
- ntfs_volume *vol = ni->vol;
- struct page *page = ni->page;
- unsigned int blocksize = vol->sb->s_blocksize;
- unsigned char blocksize_bits = vol->sb->s_blocksize_bits;
- int max_bhs = vol->mft_record_size / blocksize;
- struct buffer_head *bhs[MAX_BHS];
- struct buffer_head *bh, *head;
- runlist_element *rl;
- unsigned int block_start, block_end, m_start, m_end;
- int i_bhs, nr_bhs, err = 0;
-
- ntfs_debug("Entering for inode 0x%lx.", ni->mft_no);
- BUG_ON(NInoAttr(ni));
- BUG_ON(!max_bhs);
- BUG_ON(!PageLocked(page));
- if (WARN_ON(max_bhs > MAX_BHS)) {
- err = -EINVAL;
- goto err_out;
- }
- /*
- * If the ntfs_inode is clean no need to do anything. If it is dirty,
- * mark it as clean now so that it can be redirtied later on if needed.
- * There is no danger of races since the caller is holding the locks
- * for the mft record @m and the page it is in.
- */
- if (!NInoTestClearDirty(ni))
- goto done;
- bh = head = page_buffers(page);
- BUG_ON(!bh);
- rl = NULL;
- nr_bhs = 0;
- block_start = 0;
- m_start = ni->page_ofs;
- m_end = m_start + vol->mft_record_size;
- do {
- block_end = block_start + blocksize;
- /* If the buffer is outside the mft record, skip it. */
- if (block_end <= m_start)
- continue;
- if (unlikely(block_start >= m_end))
- break;
- /*
- * If this block is not the first one in the record, we ignore
- * the buffer's dirty state because we could have raced with a
- * parallel mark_ntfs_record_dirty().
- */
- if (block_start == m_start) {
- /* This block is the first one in the record. */
- if (!buffer_dirty(bh)) {
- BUG_ON(nr_bhs);
- /* Clean records are not written out. */
- break;
- }
- }
- /* Need to map the buffer if it is not mapped already. */
- if (unlikely(!buffer_mapped(bh))) {
- VCN vcn;
- LCN lcn;
- unsigned int vcn_ofs;
-
- bh->b_bdev = vol->sb->s_bdev;
- /* Obtain the vcn and offset of the current block. */
- vcn = ((VCN)ni->mft_no << vol->mft_record_size_bits) +
- (block_start - m_start);
- vcn_ofs = vcn & vol->cluster_size_mask;
- vcn >>= vol->cluster_size_bits;
- if (!rl) {
- down_read(&NTFS_I(vol->mft_ino)->runlist.lock);
- rl = NTFS_I(vol->mft_ino)->runlist.rl;
- BUG_ON(!rl);
- }
- /* Seek to element containing target vcn. */
- while (rl->length && rl[1].vcn <= vcn)
- rl++;
- lcn = ntfs_rl_vcn_to_lcn(rl, vcn);
- /* For $MFT, only lcn >= 0 is a successful remap. */
- if (likely(lcn >= 0)) {
- /* Setup buffer head to correct block. */
- bh->b_blocknr = ((lcn <<
- vol->cluster_size_bits) +
- vcn_ofs) >> blocksize_bits;
- set_buffer_mapped(bh);
- } else {
- bh->b_blocknr = -1;
- ntfs_error(vol->sb, "Cannot write mft record "
- "0x%lx because its location "
- "on disk could not be "
- "determined (error code %lli).",
- ni->mft_no, (long long)lcn);
- err = -EIO;
- }
- }
- BUG_ON(!buffer_uptodate(bh));
- BUG_ON(!nr_bhs && (m_start != block_start));
- BUG_ON(nr_bhs >= max_bhs);
- bhs[nr_bhs++] = bh;
- BUG_ON((nr_bhs >= max_bhs) && (m_end != block_end));
- } while (block_start = block_end, (bh = bh->b_this_page) != head);
- if (unlikely(rl))
- up_read(&NTFS_I(vol->mft_ino)->runlist.lock);
- if (!nr_bhs)
- goto done;
- if (unlikely(err))
- goto cleanup_out;
- /* Apply the mst protection fixups. */
- err = pre_write_mst_fixup((NTFS_RECORD*)m, vol->mft_record_size);
- if (err) {
- ntfs_error(vol->sb, "Failed to apply mst fixups!");
- goto cleanup_out;
- }
- flush_dcache_mft_record_page(ni);
- /* Lock buffers and start synchronous write i/o on them. */
- for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) {
- struct buffer_head *tbh = bhs[i_bhs];
-
- if (!trylock_buffer(tbh))
- BUG();
- BUG_ON(!buffer_uptodate(tbh));
- clear_buffer_dirty(tbh);
- get_bh(tbh);
- tbh->b_end_io = end_buffer_write_sync;
- submit_bh(REQ_OP_WRITE, tbh);
- }
- /* Synchronize the mft mirror now if not @sync. */
- if (!sync && ni->mft_no < vol->mftmirr_size)
- ntfs_sync_mft_mirror(vol, ni->mft_no, m, sync);
- /* Wait on i/o completion of buffers. */
- for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) {
- struct buffer_head *tbh = bhs[i_bhs];
-
- wait_on_buffer(tbh);
- if (unlikely(!buffer_uptodate(tbh))) {
- err = -EIO;
- /*
- * Set the buffer uptodate so the page and buffer
- * states do not become out of sync.
- */
- if (PageUptodate(page))
- set_buffer_uptodate(tbh);
- }
- }
- /* If @sync, now synchronize the mft mirror. */
- if (sync && ni->mft_no < vol->mftmirr_size)
- ntfs_sync_mft_mirror(vol, ni->mft_no, m, sync);
- /* Remove the mst protection fixups again. */
- post_write_mst_fixup((NTFS_RECORD*)m);
- flush_dcache_mft_record_page(ni);
- if (unlikely(err)) {
- /* I/O error during writing. This is really bad! */
- ntfs_error(vol->sb, "I/O error while writing mft record "
- "0x%lx! Marking base inode as bad. You "
- "should unmount the volume and run chkdsk.",
- ni->mft_no);
- goto err_out;
- }
-done:
- ntfs_debug("Done.");
- return 0;
-cleanup_out:
- /* Clean the buffers. */
- for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++)
- clear_buffer_dirty(bhs[i_bhs]);
-err_out:
- /*
- * Current state: all buffers are clean, unlocked, and uptodate.
- * The caller should mark the base inode as bad so that no more i/o
- * happens. ->clear_inode() will still be invoked so all extent inodes
- * and other allocated memory will be freed.
- */
- if (err == -ENOMEM) {
- ntfs_error(vol->sb, "Not enough memory to write mft record. "
- "Redirtying so the write is retried later.");
- mark_mft_record_dirty(ni);
- err = 0;
- } else
- NVolSetErrors(vol);
- return err;
-}
-
-/**
- * ntfs_may_write_mft_record - check if an mft record may be written out
- * @vol: [IN] ntfs volume on which the mft record to check resides
- * @mft_no: [IN] mft record number of the mft record to check
- * @m: [IN] mapped mft record to check
- * @locked_ni: [OUT] caller has to unlock this ntfs inode if one is returned
- *
- * Check if the mapped (base or extent) mft record @m with mft record number
- * @mft_no belonging to the ntfs volume @vol may be written out. If necessary
- * and possible the ntfs inode of the mft record is locked and the base vfs
- * inode is pinned. The locked ntfs inode is then returned in @locked_ni. The
- * caller is responsible for unlocking the ntfs inode and unpinning the base
- * vfs inode.
- *
- * Return 'true' if the mft record may be written out and 'false' if not.
- *
- * The caller has locked the page and cleared the uptodate flag on it which
- * means that we can safely write out any dirty mft records that do not have
- * their inodes in icache as determined by ilookup5() as anyone
- * opening/creating such an inode would block when attempting to map the mft
- * record in read_cache_page() until we are finished with the write out.
- *
- * Here is a description of the tests we perform:
- *
- * If the inode is found in icache we know the mft record must be a base mft
- * record. If it is dirty, we do not write it and return 'false' as the vfs
- * inode write paths will result in the access times being updated which would
- * cause the base mft record to be redirtied and written out again. (We know
- * the access time update will modify the base mft record because Windows
- * chkdsk complains if the standard information attribute is not in the base
- * mft record.)
- *
- * If the inode is in icache and not dirty, we attempt to lock the mft record
- * and if we find the lock was already taken, it is not safe to write the mft
- * record and we return 'false'.
- *
- * If we manage to obtain the lock we have exclusive access to the mft record,
- * which also allows us safe writeout of the mft record. We then set
- * @locked_ni to the locked ntfs inode and return 'true'.
- *
- * Note we cannot just lock the mft record and sleep while waiting for the lock
- * because this would deadlock due to lock reversal (normally the mft record is
- * locked before the page is locked but we already have the page locked here
- * when we try to lock the mft record).
- *
- * If the inode is not in icache we need to perform further checks.
- *
- * If the mft record is not a FILE record or it is a base mft record, we can
- * safely write it and return 'true'.
- *
- * We now know the mft record is an extent mft record. We check if the inode
- * corresponding to its base mft record is in icache and obtain a reference to
- * it if it is. If it is not, we can safely write it and return 'true'.
- *
- * We now have the base inode for the extent mft record. We check if it has an
- * ntfs inode for the extent mft record attached and if not it is safe to write
- * the extent mft record and we return 'true'.
- *
- * The ntfs inode for the extent mft record is attached to the base inode so we
- * attempt to lock the extent mft record and if we find the lock was already
- * taken, it is not safe to write the extent mft record and we return 'false'.
- *
- * If we manage to obtain the lock we have exclusive access to the extent mft
- * record, which also allows us safe writeout of the extent mft record. We
- * set the ntfs inode of the extent mft record clean and then set @locked_ni to
- * the now locked ntfs inode and return 'true'.
- *
- * Note, the reason for actually writing dirty mft records here and not just
- * relying on the vfs inode dirty code paths is that we can have mft records
- * modified without them ever having actual inodes in memory. Also we can have
- * dirty mft records with clean ntfs inodes in memory. None of the described
- * cases would result in the dirty mft records being written out if we only
- * relied on the vfs inode dirty code paths. And these cases can really occur
- * during allocation of new mft records and in particular when the
- * initialized_size of the $MFT/$DATA attribute is extended and the new space
- * is initialized using ntfs_mft_record_format(). The clean inode can then
- * appear if the mft record is reused for a new inode before it got written
- * out.
- */
-bool ntfs_may_write_mft_record(ntfs_volume *vol, const unsigned long mft_no,
- const MFT_RECORD *m, ntfs_inode **locked_ni)
-{
- struct super_block *sb = vol->sb;
- struct inode *mft_vi = vol->mft_ino;
- struct inode *vi;
- ntfs_inode *ni, *eni, **extent_nis;
- int i;
- ntfs_attr na;
-
- ntfs_debug("Entering for inode 0x%lx.", mft_no);
- /*
- * Normally we do not return a locked inode so set @locked_ni to NULL.
- */
- BUG_ON(!locked_ni);
- *locked_ni = NULL;
- /*
- * Check if the inode corresponding to this mft record is in the VFS
- * inode cache and obtain a reference to it if it is.
- */
- ntfs_debug("Looking for inode 0x%lx in icache.", mft_no);
- na.mft_no = mft_no;
- na.name = NULL;
- na.name_len = 0;
- na.type = AT_UNUSED;
- /*
- * Optimize inode 0, i.e. $MFT itself, since we have it in memory and
- * we get here for it rather often.
- */
- if (!mft_no) {
- /* Balance the below iput(). */
- vi = igrab(mft_vi);
- BUG_ON(vi != mft_vi);
- } else {
- /*
- * Have to use ilookup5_nowait() since ilookup5() waits for the
- * inode lock which causes ntfs to deadlock when a concurrent
- * inode write via the inode dirty code paths and the page
- * dirty code path of the inode dirty code path when writing
- * $MFT occurs.
- */
- vi = ilookup5_nowait(sb, mft_no, ntfs_test_inode, &na);
- }
- if (vi) {
- ntfs_debug("Base inode 0x%lx is in icache.", mft_no);
- /* The inode is in icache. */
- ni = NTFS_I(vi);
- /* Take a reference to the ntfs inode. */
- atomic_inc(&ni->count);
- /* If the inode is dirty, do not write this record. */
- if (NInoDirty(ni)) {
- ntfs_debug("Inode 0x%lx is dirty, do not write it.",
- mft_no);
- atomic_dec(&ni->count);
- iput(vi);
- return false;
- }
- ntfs_debug("Inode 0x%lx is not dirty.", mft_no);
- /* The inode is not dirty, try to take the mft record lock. */
- if (unlikely(!mutex_trylock(&ni->mrec_lock))) {
- ntfs_debug("Mft record 0x%lx is already locked, do "
- "not write it.", mft_no);
- atomic_dec(&ni->count);
- iput(vi);
- return false;
- }
- ntfs_debug("Managed to lock mft record 0x%lx, write it.",
- mft_no);
- /*
- * The write has to occur while we hold the mft record lock so
- * return the locked ntfs inode.
- */
- *locked_ni = ni;
- return true;
- }
- ntfs_debug("Inode 0x%lx is not in icache.", mft_no);
- /* The inode is not in icache. */
- /* Write the record if it is not a mft record (type "FILE"). */
- if (!ntfs_is_mft_record(m->magic)) {
- ntfs_debug("Mft record 0x%lx is not a FILE record, write it.",
- mft_no);
- return true;
- }
- /* Write the mft record if it is a base inode. */
- if (!m->base_mft_record) {
- ntfs_debug("Mft record 0x%lx is a base record, write it.",
- mft_no);
- return true;
- }
- /*
- * This is an extent mft record. Check if the inode corresponding to
- * its base mft record is in icache and obtain a reference to it if it
- * is.
- */
- na.mft_no = MREF_LE(m->base_mft_record);
- ntfs_debug("Mft record 0x%lx is an extent record. Looking for base "
- "inode 0x%lx in icache.", mft_no, na.mft_no);
- if (!na.mft_no) {
- /* Balance the below iput(). */
- vi = igrab(mft_vi);
- BUG_ON(vi != mft_vi);
- } else
- vi = ilookup5_nowait(sb, na.mft_no, ntfs_test_inode,
- &na);
- if (!vi) {
- /*
- * The base inode is not in icache, write this extent mft
- * record.
- */
- ntfs_debug("Base inode 0x%lx is not in icache, write the "
- "extent record.", na.mft_no);
- return true;
- }
- ntfs_debug("Base inode 0x%lx is in icache.", na.mft_no);
- /*
- * The base inode is in icache. Check if it has the extent inode
- * corresponding to this extent mft record attached.
- */
- ni = NTFS_I(vi);
- mutex_lock(&ni->extent_lock);
- if (ni->nr_extents <= 0) {
- /*
- * The base inode has no attached extent inodes, write this
- * extent mft record.
- */
- mutex_unlock(&ni->extent_lock);
- iput(vi);
- ntfs_debug("Base inode 0x%lx has no attached extent inodes, "
- "write the extent record.", na.mft_no);
- return true;
- }
- /* Iterate over the attached extent inodes. */
- extent_nis = ni->ext.extent_ntfs_inos;
- for (eni = NULL, i = 0; i < ni->nr_extents; ++i) {
- if (mft_no == extent_nis[i]->mft_no) {
- /*
- * Found the extent inode corresponding to this extent
- * mft record.
- */
- eni = extent_nis[i];
- break;
- }
- }
- /*
- * If the extent inode was not attached to the base inode, write this
- * extent mft record.
- */
- if (!eni) {
- mutex_unlock(&ni->extent_lock);
- iput(vi);
- ntfs_debug("Extent inode 0x%lx is not attached to its base "
- "inode 0x%lx, write the extent record.",
- mft_no, na.mft_no);
- return true;
- }
- ntfs_debug("Extent inode 0x%lx is attached to its base inode 0x%lx.",
- mft_no, na.mft_no);
- /* Take a reference to the extent ntfs inode. */
- atomic_inc(&eni->count);
- mutex_unlock(&ni->extent_lock);
- /*
- * Found the extent inode coresponding to this extent mft record.
- * Try to take the mft record lock.
- */
- if (unlikely(!mutex_trylock(&eni->mrec_lock))) {
- atomic_dec(&eni->count);
- iput(vi);
- ntfs_debug("Extent mft record 0x%lx is already locked, do "
- "not write it.", mft_no);
- return false;
- }
- ntfs_debug("Managed to lock extent mft record 0x%lx, write it.",
- mft_no);
- if (NInoTestClearDirty(eni))
- ntfs_debug("Extent inode 0x%lx is dirty, marking it clean.",
- mft_no);
- /*
- * The write has to occur while we hold the mft record lock so return
- * the locked extent ntfs inode.
- */
- *locked_ni = eni;
- return true;
-}
-
-static const char *es = " Leaving inconsistent metadata. Unmount and run "
- "chkdsk.";
-
-/**
- * ntfs_mft_bitmap_find_and_alloc_free_rec_nolock - see name
- * @vol: volume on which to search for a free mft record
- * @base_ni: open base inode if allocating an extent mft record or NULL
- *
- * Search for a free mft record in the mft bitmap attribute on the ntfs volume
- * @vol.
- *
- * If @base_ni is NULL start the search at the default allocator position.
- *
- * If @base_ni is not NULL start the search at the mft record after the base
- * mft record @base_ni.
- *
- * Return the free mft record on success and -errno on error. An error code of
- * -ENOSPC means that there are no free mft records in the currently
- * initialized mft bitmap.
- *
- * Locking: Caller must hold vol->mftbmp_lock for writing.
- */
-static int ntfs_mft_bitmap_find_and_alloc_free_rec_nolock(ntfs_volume *vol,
- ntfs_inode *base_ni)
-{
- s64 pass_end, ll, data_pos, pass_start, ofs, bit;
- unsigned long flags;
- struct address_space *mftbmp_mapping;
- u8 *buf, *byte;
- struct page *page;
- unsigned int page_ofs, size;
- u8 pass, b;
-
- ntfs_debug("Searching for free mft record in the currently "
- "initialized mft bitmap.");
- mftbmp_mapping = vol->mftbmp_ino->i_mapping;
- /*
- * Set the end of the pass making sure we do not overflow the mft
- * bitmap.
- */
- read_lock_irqsave(&NTFS_I(vol->mft_ino)->size_lock, flags);
- pass_end = NTFS_I(vol->mft_ino)->allocated_size >>
- vol->mft_record_size_bits;
- read_unlock_irqrestore(&NTFS_I(vol->mft_ino)->size_lock, flags);
- read_lock_irqsave(&NTFS_I(vol->mftbmp_ino)->size_lock, flags);
- ll = NTFS_I(vol->mftbmp_ino)->initialized_size << 3;
- read_unlock_irqrestore(&NTFS_I(vol->mftbmp_ino)->size_lock, flags);
- if (pass_end > ll)
- pass_end = ll;
- pass = 1;
- if (!base_ni)
- data_pos = vol->mft_data_pos;
- else
- data_pos = base_ni->mft_no + 1;
- if (data_pos < 24)
- data_pos = 24;
- if (data_pos >= pass_end) {
- data_pos = 24;
- pass = 2;
- /* This happens on a freshly formatted volume. */
- if (data_pos >= pass_end)
- return -ENOSPC;
- }
- pass_start = data_pos;
- ntfs_debug("Starting bitmap search: pass %u, pass_start 0x%llx, "
- "pass_end 0x%llx, data_pos 0x%llx.", pass,
- (long long)pass_start, (long long)pass_end,
- (long long)data_pos);
- /* Loop until a free mft record is found. */
- for (; pass <= 2;) {
- /* Cap size to pass_end. */
- ofs = data_pos >> 3;
- page_ofs = ofs & ~PAGE_MASK;
- size = PAGE_SIZE - page_ofs;
- ll = ((pass_end + 7) >> 3) - ofs;
- if (size > ll)
- size = ll;
- size <<= 3;
- /*
- * If we are still within the active pass, search the next page
- * for a zero bit.
- */
- if (size) {
- page = ntfs_map_page(mftbmp_mapping,
- ofs >> PAGE_SHIFT);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to read mft "
- "bitmap, aborting.");
- return PTR_ERR(page);
- }
- buf = (u8*)page_address(page) + page_ofs;
- bit = data_pos & 7;
- data_pos &= ~7ull;
- ntfs_debug("Before inner for loop: size 0x%x, "
- "data_pos 0x%llx, bit 0x%llx", size,
- (long long)data_pos, (long long)bit);
- for (; bit < size && data_pos + bit < pass_end;
- bit &= ~7ull, bit += 8) {
- byte = buf + (bit >> 3);
- if (*byte == 0xff)
- continue;
- b = ffz((unsigned long)*byte);
- if (b < 8 && b >= (bit & 7)) {
- ll = data_pos + (bit & ~7ull) + b;
- if (unlikely(ll > (1ll << 32))) {
- ntfs_unmap_page(page);
- return -ENOSPC;
- }
- *byte |= 1 << b;
- flush_dcache_page(page);
- set_page_dirty(page);
- ntfs_unmap_page(page);
- ntfs_debug("Done. (Found and "
- "allocated mft record "
- "0x%llx.)",
- (long long)ll);
- return ll;
- }
- }
- ntfs_debug("After inner for loop: size 0x%x, "
- "data_pos 0x%llx, bit 0x%llx", size,
- (long long)data_pos, (long long)bit);
- data_pos += size;
- ntfs_unmap_page(page);
- /*
- * If the end of the pass has not been reached yet,
- * continue searching the mft bitmap for a zero bit.
- */
- if (data_pos < pass_end)
- continue;
- }
- /* Do the next pass. */
- if (++pass == 2) {
- /*
- * Starting the second pass, in which we scan the first
- * part of the zone which we omitted earlier.
- */
- pass_end = pass_start;
- data_pos = pass_start = 24;
- ntfs_debug("pass %i, pass_start 0x%llx, pass_end "
- "0x%llx.", pass, (long long)pass_start,
- (long long)pass_end);
- if (data_pos >= pass_end)
- break;
- }
- }
- /* No free mft records in currently initialized mft bitmap. */
- ntfs_debug("Done. (No free mft records left in currently initialized "
- "mft bitmap.)");
- return -ENOSPC;
-}
-
-/**
- * ntfs_mft_bitmap_extend_allocation_nolock - extend mft bitmap by a cluster
- * @vol: volume on which to extend the mft bitmap attribute
- *
- * Extend the mft bitmap attribute on the ntfs volume @vol by one cluster.
- *
- * Note: Only changes allocated_size, i.e. does not touch initialized_size or
- * data_size.
- *
- * Return 0 on success and -errno on error.
- *
- * Locking: - Caller must hold vol->mftbmp_lock for writing.
- * - This function takes NTFS_I(vol->mftbmp_ino)->runlist.lock for
- * writing and releases it before returning.
- * - This function takes vol->lcnbmp_lock for writing and releases it
- * before returning.
- */
-static int ntfs_mft_bitmap_extend_allocation_nolock(ntfs_volume *vol)
-{
- LCN lcn;
- s64 ll;
- unsigned long flags;
- struct page *page;
- ntfs_inode *mft_ni, *mftbmp_ni;
- runlist_element *rl, *rl2 = NULL;
- ntfs_attr_search_ctx *ctx = NULL;
- MFT_RECORD *mrec;
- ATTR_RECORD *a = NULL;
- int ret, mp_size;
- u32 old_alen = 0;
- u8 *b, tb;
- struct {
- u8 added_cluster:1;
- u8 added_run:1;
- u8 mp_rebuilt:1;
- } status = { 0, 0, 0 };
-
- ntfs_debug("Extending mft bitmap allocation.");
- mft_ni = NTFS_I(vol->mft_ino);
- mftbmp_ni = NTFS_I(vol->mftbmp_ino);
- /*
- * Determine the last lcn of the mft bitmap. The allocated size of the
- * mft bitmap cannot be zero so we are ok to do this.
- */
- down_write(&mftbmp_ni->runlist.lock);
- read_lock_irqsave(&mftbmp_ni->size_lock, flags);
- ll = mftbmp_ni->allocated_size;
- read_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
- rl = ntfs_attr_find_vcn_nolock(mftbmp_ni,
- (ll - 1) >> vol->cluster_size_bits, NULL);
- if (IS_ERR(rl) || unlikely(!rl->length || rl->lcn < 0)) {
- up_write(&mftbmp_ni->runlist.lock);
- ntfs_error(vol->sb, "Failed to determine last allocated "
- "cluster of mft bitmap attribute.");
- if (!IS_ERR(rl))
- ret = -EIO;
- else
- ret = PTR_ERR(rl);
- return ret;
- }
- lcn = rl->lcn + rl->length;
- ntfs_debug("Last lcn of mft bitmap attribute is 0x%llx.",
- (long long)lcn);
- /*
- * Attempt to get the cluster following the last allocated cluster by
- * hand as it may be in the MFT zone so the allocator would not give it
- * to us.
- */
- ll = lcn >> 3;
- page = ntfs_map_page(vol->lcnbmp_ino->i_mapping,
- ll >> PAGE_SHIFT);
- if (IS_ERR(page)) {
- up_write(&mftbmp_ni->runlist.lock);
- ntfs_error(vol->sb, "Failed to read from lcn bitmap.");
- return PTR_ERR(page);
- }
- b = (u8*)page_address(page) + (ll & ~PAGE_MASK);
- tb = 1 << (lcn & 7ull);
- down_write(&vol->lcnbmp_lock);
- if (*b != 0xff && !(*b & tb)) {
- /* Next cluster is free, allocate it. */
- *b |= tb;
- flush_dcache_page(page);
- set_page_dirty(page);
- up_write(&vol->lcnbmp_lock);
- ntfs_unmap_page(page);
- /* Update the mft bitmap runlist. */
- rl->length++;
- rl[1].vcn++;
- status.added_cluster = 1;
- ntfs_debug("Appending one cluster to mft bitmap.");
- } else {
- up_write(&vol->lcnbmp_lock);
- ntfs_unmap_page(page);
- /* Allocate a cluster from the DATA_ZONE. */
- rl2 = ntfs_cluster_alloc(vol, rl[1].vcn, 1, lcn, DATA_ZONE,
- true);
- if (IS_ERR(rl2)) {
- up_write(&mftbmp_ni->runlist.lock);
- ntfs_error(vol->sb, "Failed to allocate a cluster for "
- "the mft bitmap.");
- return PTR_ERR(rl2);
- }
- rl = ntfs_runlists_merge(mftbmp_ni->runlist.rl, rl2);
- if (IS_ERR(rl)) {
- up_write(&mftbmp_ni->runlist.lock);
- ntfs_error(vol->sb, "Failed to merge runlists for mft "
- "bitmap.");
- if (ntfs_cluster_free_from_rl(vol, rl2)) {
- ntfs_error(vol->sb, "Failed to deallocate "
- "allocated cluster.%s", es);
- NVolSetErrors(vol);
- }
- ntfs_free(rl2);
- return PTR_ERR(rl);
- }
- mftbmp_ni->runlist.rl = rl;
- status.added_run = 1;
- ntfs_debug("Adding one run to mft bitmap.");
- /* Find the last run in the new runlist. */
- for (; rl[1].length; rl++)
- ;
- }
- /*
- * Update the attribute record as well. Note: @rl is the last
- * (non-terminator) runlist element of mft bitmap.
- */
- mrec = map_mft_record(mft_ni);
- if (IS_ERR(mrec)) {
- ntfs_error(vol->sb, "Failed to map mft record.");
- ret = PTR_ERR(mrec);
- goto undo_alloc;
- }
- ctx = ntfs_attr_get_search_ctx(mft_ni, mrec);
- if (unlikely(!ctx)) {
- ntfs_error(vol->sb, "Failed to get search context.");
- ret = -ENOMEM;
- goto undo_alloc;
- }
- ret = ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name,
- mftbmp_ni->name_len, CASE_SENSITIVE, rl[1].vcn, NULL,
- 0, ctx);
- if (unlikely(ret)) {
- ntfs_error(vol->sb, "Failed to find last attribute extent of "
- "mft bitmap attribute.");
- if (ret == -ENOENT)
- ret = -EIO;
- goto undo_alloc;
- }
- a = ctx->attr;
- ll = sle64_to_cpu(a->data.non_resident.lowest_vcn);
- /* Search back for the previous last allocated cluster of mft bitmap. */
- for (rl2 = rl; rl2 > mftbmp_ni->runlist.rl; rl2--) {
- if (ll >= rl2->vcn)
- break;
- }
- BUG_ON(ll < rl2->vcn);
- BUG_ON(ll >= rl2->vcn + rl2->length);
- /* Get the size for the new mapping pairs array for this extent. */
- mp_size = ntfs_get_size_for_mapping_pairs(vol, rl2, ll, -1);
- if (unlikely(mp_size <= 0)) {
- ntfs_error(vol->sb, "Get size for mapping pairs failed for "
- "mft bitmap attribute extent.");
- ret = mp_size;
- if (!ret)
- ret = -EIO;
- goto undo_alloc;
- }
- /* Expand the attribute record if necessary. */
- old_alen = le32_to_cpu(a->length);
- ret = ntfs_attr_record_resize(ctx->mrec, a, mp_size +
- le16_to_cpu(a->data.non_resident.mapping_pairs_offset));
- if (unlikely(ret)) {
- if (ret != -ENOSPC) {
- ntfs_error(vol->sb, "Failed to resize attribute "
- "record for mft bitmap attribute.");
- goto undo_alloc;
- }
- // TODO: Deal with this by moving this extent to a new mft
- // record or by starting a new extent in a new mft record or by
- // moving other attributes out of this mft record.
- // Note: It will need to be a special mft record and if none of
- // those are available it gets rather complicated...
- ntfs_error(vol->sb, "Not enough space in this mft record to "
- "accommodate extended mft bitmap attribute "
- "extent. Cannot handle this yet.");
- ret = -EOPNOTSUPP;
- goto undo_alloc;
- }
- status.mp_rebuilt = 1;
- /* Generate the mapping pairs array directly into the attr record. */
- ret = ntfs_mapping_pairs_build(vol, (u8*)a +
- le16_to_cpu(a->data.non_resident.mapping_pairs_offset),
- mp_size, rl2, ll, -1, NULL);
- if (unlikely(ret)) {
- ntfs_error(vol->sb, "Failed to build mapping pairs array for "
- "mft bitmap attribute.");
- goto undo_alloc;
- }
- /* Update the highest_vcn. */
- a->data.non_resident.highest_vcn = cpu_to_sle64(rl[1].vcn - 1);
- /*
- * We now have extended the mft bitmap allocated_size by one cluster.
- * Reflect this in the ntfs_inode structure and the attribute record.
- */
- if (a->data.non_resident.lowest_vcn) {
- /*
- * We are not in the first attribute extent, switch to it, but
- * first ensure the changes will make it to disk later.
- */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_reinit_search_ctx(ctx);
- ret = ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name,
- mftbmp_ni->name_len, CASE_SENSITIVE, 0, NULL,
- 0, ctx);
- if (unlikely(ret)) {
- ntfs_error(vol->sb, "Failed to find first attribute "
- "extent of mft bitmap attribute.");
- goto restore_undo_alloc;
- }
- a = ctx->attr;
- }
- write_lock_irqsave(&mftbmp_ni->size_lock, flags);
- mftbmp_ni->allocated_size += vol->cluster_size;
- a->data.non_resident.allocated_size =
- cpu_to_sle64(mftbmp_ni->allocated_size);
- write_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
- /* Ensure the changes make it to disk. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(mft_ni);
- up_write(&mftbmp_ni->runlist.lock);
- ntfs_debug("Done.");
- return 0;
-restore_undo_alloc:
- ntfs_attr_reinit_search_ctx(ctx);
- if (ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name,
- mftbmp_ni->name_len, CASE_SENSITIVE, rl[1].vcn, NULL,
- 0, ctx)) {
- ntfs_error(vol->sb, "Failed to find last attribute extent of "
- "mft bitmap attribute.%s", es);
- write_lock_irqsave(&mftbmp_ni->size_lock, flags);
- mftbmp_ni->allocated_size += vol->cluster_size;
- write_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(mft_ni);
- up_write(&mftbmp_ni->runlist.lock);
- /*
- * The only thing that is now wrong is ->allocated_size of the
- * base attribute extent which chkdsk should be able to fix.
- */
- NVolSetErrors(vol);
- return ret;
- }
- a = ctx->attr;
- a->data.non_resident.highest_vcn = cpu_to_sle64(rl[1].vcn - 2);
-undo_alloc:
- if (status.added_cluster) {
- /* Truncate the last run in the runlist by one cluster. */
- rl->length--;
- rl[1].vcn--;
- } else if (status.added_run) {
- lcn = rl->lcn;
- /* Remove the last run from the runlist. */
- rl->lcn = rl[1].lcn;
- rl->length = 0;
- }
- /* Deallocate the cluster. */
- down_write(&vol->lcnbmp_lock);
- if (ntfs_bitmap_clear_bit(vol->lcnbmp_ino, lcn)) {
- ntfs_error(vol->sb, "Failed to free allocated cluster.%s", es);
- NVolSetErrors(vol);
- }
- up_write(&vol->lcnbmp_lock);
- if (status.mp_rebuilt) {
- if (ntfs_mapping_pairs_build(vol, (u8*)a + le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset),
- old_alen - le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset),
- rl2, ll, -1, NULL)) {
- ntfs_error(vol->sb, "Failed to restore mapping pairs "
- "array.%s", es);
- NVolSetErrors(vol);
- }
- if (ntfs_attr_record_resize(ctx->mrec, a, old_alen)) {
- ntfs_error(vol->sb, "Failed to restore attribute "
- "record.%s", es);
- NVolSetErrors(vol);
- }
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- }
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (!IS_ERR(mrec))
- unmap_mft_record(mft_ni);
- up_write(&mftbmp_ni->runlist.lock);
- return ret;
-}
-
-/**
- * ntfs_mft_bitmap_extend_initialized_nolock - extend mftbmp initialized data
- * @vol: volume on which to extend the mft bitmap attribute
- *
- * Extend the initialized portion of the mft bitmap attribute on the ntfs
- * volume @vol by 8 bytes.
- *
- * Note: Only changes initialized_size and data_size, i.e. requires that
- * allocated_size is big enough to fit the new initialized_size.
- *
- * Return 0 on success and -error on error.
- *
- * Locking: Caller must hold vol->mftbmp_lock for writing.
- */
-static int ntfs_mft_bitmap_extend_initialized_nolock(ntfs_volume *vol)
-{
- s64 old_data_size, old_initialized_size;
- unsigned long flags;
- struct inode *mftbmp_vi;
- ntfs_inode *mft_ni, *mftbmp_ni;
- ntfs_attr_search_ctx *ctx;
- MFT_RECORD *mrec;
- ATTR_RECORD *a;
- int ret;
-
- ntfs_debug("Extending mft bitmap initiailized (and data) size.");
- mft_ni = NTFS_I(vol->mft_ino);
- mftbmp_vi = vol->mftbmp_ino;
- mftbmp_ni = NTFS_I(mftbmp_vi);
- /* Get the attribute record. */
- mrec = map_mft_record(mft_ni);
- if (IS_ERR(mrec)) {
- ntfs_error(vol->sb, "Failed to map mft record.");
- return PTR_ERR(mrec);
- }
- ctx = ntfs_attr_get_search_ctx(mft_ni, mrec);
- if (unlikely(!ctx)) {
- ntfs_error(vol->sb, "Failed to get search context.");
- ret = -ENOMEM;
- goto unm_err_out;
- }
- ret = ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name,
- mftbmp_ni->name_len, CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(ret)) {
- ntfs_error(vol->sb, "Failed to find first attribute extent of "
- "mft bitmap attribute.");
- if (ret == -ENOENT)
- ret = -EIO;
- goto put_err_out;
- }
- a = ctx->attr;
- write_lock_irqsave(&mftbmp_ni->size_lock, flags);
- old_data_size = i_size_read(mftbmp_vi);
- old_initialized_size = mftbmp_ni->initialized_size;
- /*
- * We can simply update the initialized_size before filling the space
- * with zeroes because the caller is holding the mft bitmap lock for
- * writing which ensures that no one else is trying to access the data.
- */
- mftbmp_ni->initialized_size += 8;
- a->data.non_resident.initialized_size =
- cpu_to_sle64(mftbmp_ni->initialized_size);
- if (mftbmp_ni->initialized_size > old_data_size) {
- i_size_write(mftbmp_vi, mftbmp_ni->initialized_size);
- a->data.non_resident.data_size =
- cpu_to_sle64(mftbmp_ni->initialized_size);
- }
- write_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
- /* Ensure the changes make it to disk. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(mft_ni);
- /* Initialize the mft bitmap attribute value with zeroes. */
- ret = ntfs_attr_set(mftbmp_ni, old_initialized_size, 8, 0);
- if (likely(!ret)) {
- ntfs_debug("Done. (Wrote eight initialized bytes to mft "
- "bitmap.");
- return 0;
- }
- ntfs_error(vol->sb, "Failed to write to mft bitmap.");
- /* Try to recover from the error. */
- mrec = map_mft_record(mft_ni);
- if (IS_ERR(mrec)) {
- ntfs_error(vol->sb, "Failed to map mft record.%s", es);
- NVolSetErrors(vol);
- return ret;
- }
- ctx = ntfs_attr_get_search_ctx(mft_ni, mrec);
- if (unlikely(!ctx)) {
- ntfs_error(vol->sb, "Failed to get search context.%s", es);
- NVolSetErrors(vol);
- goto unm_err_out;
- }
- if (ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name,
- mftbmp_ni->name_len, CASE_SENSITIVE, 0, NULL, 0, ctx)) {
- ntfs_error(vol->sb, "Failed to find first attribute extent of "
- "mft bitmap attribute.%s", es);
- NVolSetErrors(vol);
-put_err_out:
- ntfs_attr_put_search_ctx(ctx);
-unm_err_out:
- unmap_mft_record(mft_ni);
- goto err_out;
- }
- a = ctx->attr;
- write_lock_irqsave(&mftbmp_ni->size_lock, flags);
- mftbmp_ni->initialized_size = old_initialized_size;
- a->data.non_resident.initialized_size =
- cpu_to_sle64(old_initialized_size);
- if (i_size_read(mftbmp_vi) != old_data_size) {
- i_size_write(mftbmp_vi, old_data_size);
- a->data.non_resident.data_size = cpu_to_sle64(old_data_size);
- }
- write_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(mft_ni);
-#ifdef DEBUG
- read_lock_irqsave(&mftbmp_ni->size_lock, flags);
- ntfs_debug("Restored status of mftbmp: allocated_size 0x%llx, "
- "data_size 0x%llx, initialized_size 0x%llx.",
- (long long)mftbmp_ni->allocated_size,
- (long long)i_size_read(mftbmp_vi),
- (long long)mftbmp_ni->initialized_size);
- read_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
-#endif /* DEBUG */
-err_out:
- return ret;
-}
-
-/**
- * ntfs_mft_data_extend_allocation_nolock - extend mft data attribute
- * @vol: volume on which to extend the mft data attribute
- *
- * Extend the mft data attribute on the ntfs volume @vol by 16 mft records
- * worth of clusters or if not enough space for this by one mft record worth
- * of clusters.
- *
- * Note: Only changes allocated_size, i.e. does not touch initialized_size or
- * data_size.
- *
- * Return 0 on success and -errno on error.
- *
- * Locking: - Caller must hold vol->mftbmp_lock for writing.
- * - This function takes NTFS_I(vol->mft_ino)->runlist.lock for
- * writing and releases it before returning.
- * - This function calls functions which take vol->lcnbmp_lock for
- * writing and release it before returning.
- */
-static int ntfs_mft_data_extend_allocation_nolock(ntfs_volume *vol)
-{
- LCN lcn;
- VCN old_last_vcn;
- s64 min_nr, nr, ll;
- unsigned long flags;
- ntfs_inode *mft_ni;
- runlist_element *rl, *rl2;
- ntfs_attr_search_ctx *ctx = NULL;
- MFT_RECORD *mrec;
- ATTR_RECORD *a = NULL;
- int ret, mp_size;
- u32 old_alen = 0;
- bool mp_rebuilt = false;
-
- ntfs_debug("Extending mft data allocation.");
- mft_ni = NTFS_I(vol->mft_ino);
- /*
- * Determine the preferred allocation location, i.e. the last lcn of
- * the mft data attribute. The allocated size of the mft data
- * attribute cannot be zero so we are ok to do this.
- */
- down_write(&mft_ni->runlist.lock);
- read_lock_irqsave(&mft_ni->size_lock, flags);
- ll = mft_ni->allocated_size;
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
- rl = ntfs_attr_find_vcn_nolock(mft_ni,
- (ll - 1) >> vol->cluster_size_bits, NULL);
- if (IS_ERR(rl) || unlikely(!rl->length || rl->lcn < 0)) {
- up_write(&mft_ni->runlist.lock);
- ntfs_error(vol->sb, "Failed to determine last allocated "
- "cluster of mft data attribute.");
- if (!IS_ERR(rl))
- ret = -EIO;
- else
- ret = PTR_ERR(rl);
- return ret;
- }
- lcn = rl->lcn + rl->length;
- ntfs_debug("Last lcn of mft data attribute is 0x%llx.", (long long)lcn);
- /* Minimum allocation is one mft record worth of clusters. */
- min_nr = vol->mft_record_size >> vol->cluster_size_bits;
- if (!min_nr)
- min_nr = 1;
- /* Want to allocate 16 mft records worth of clusters. */
- nr = vol->mft_record_size << 4 >> vol->cluster_size_bits;
- if (!nr)
- nr = min_nr;
- /* Ensure we do not go above 2^32-1 mft records. */
- read_lock_irqsave(&mft_ni->size_lock, flags);
- ll = mft_ni->allocated_size;
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
- if (unlikely((ll + (nr << vol->cluster_size_bits)) >>
- vol->mft_record_size_bits >= (1ll << 32))) {
- nr = min_nr;
- if (unlikely((ll + (nr << vol->cluster_size_bits)) >>
- vol->mft_record_size_bits >= (1ll << 32))) {
- ntfs_warning(vol->sb, "Cannot allocate mft record "
- "because the maximum number of inodes "
- "(2^32) has already been reached.");
- up_write(&mft_ni->runlist.lock);
- return -ENOSPC;
- }
- }
- ntfs_debug("Trying mft data allocation with %s cluster count %lli.",
- nr > min_nr ? "default" : "minimal", (long long)nr);
- old_last_vcn = rl[1].vcn;
- do {
- rl2 = ntfs_cluster_alloc(vol, old_last_vcn, nr, lcn, MFT_ZONE,
- true);
- if (!IS_ERR(rl2))
- break;
- if (PTR_ERR(rl2) != -ENOSPC || nr == min_nr) {
- ntfs_error(vol->sb, "Failed to allocate the minimal "
- "number of clusters (%lli) for the "
- "mft data attribute.", (long long)nr);
- up_write(&mft_ni->runlist.lock);
- return PTR_ERR(rl2);
- }
- /*
- * There is not enough space to do the allocation, but there
- * might be enough space to do a minimal allocation so try that
- * before failing.
- */
- nr = min_nr;
- ntfs_debug("Retrying mft data allocation with minimal cluster "
- "count %lli.", (long long)nr);
- } while (1);
- rl = ntfs_runlists_merge(mft_ni->runlist.rl, rl2);
- if (IS_ERR(rl)) {
- up_write(&mft_ni->runlist.lock);
- ntfs_error(vol->sb, "Failed to merge runlists for mft data "
- "attribute.");
- if (ntfs_cluster_free_from_rl(vol, rl2)) {
- ntfs_error(vol->sb, "Failed to deallocate clusters "
- "from the mft data attribute.%s", es);
- NVolSetErrors(vol);
- }
- ntfs_free(rl2);
- return PTR_ERR(rl);
- }
- mft_ni->runlist.rl = rl;
- ntfs_debug("Allocated %lli clusters.", (long long)nr);
- /* Find the last run in the new runlist. */
- for (; rl[1].length; rl++)
- ;
- /* Update the attribute record as well. */
- mrec = map_mft_record(mft_ni);
- if (IS_ERR(mrec)) {
- ntfs_error(vol->sb, "Failed to map mft record.");
- ret = PTR_ERR(mrec);
- goto undo_alloc;
- }
- ctx = ntfs_attr_get_search_ctx(mft_ni, mrec);
- if (unlikely(!ctx)) {
- ntfs_error(vol->sb, "Failed to get search context.");
- ret = -ENOMEM;
- goto undo_alloc;
- }
- ret = ntfs_attr_lookup(mft_ni->type, mft_ni->name, mft_ni->name_len,
- CASE_SENSITIVE, rl[1].vcn, NULL, 0, ctx);
- if (unlikely(ret)) {
- ntfs_error(vol->sb, "Failed to find last attribute extent of "
- "mft data attribute.");
- if (ret == -ENOENT)
- ret = -EIO;
- goto undo_alloc;
- }
- a = ctx->attr;
- ll = sle64_to_cpu(a->data.non_resident.lowest_vcn);
- /* Search back for the previous last allocated cluster of mft bitmap. */
- for (rl2 = rl; rl2 > mft_ni->runlist.rl; rl2--) {
- if (ll >= rl2->vcn)
- break;
- }
- BUG_ON(ll < rl2->vcn);
- BUG_ON(ll >= rl2->vcn + rl2->length);
- /* Get the size for the new mapping pairs array for this extent. */
- mp_size = ntfs_get_size_for_mapping_pairs(vol, rl2, ll, -1);
- if (unlikely(mp_size <= 0)) {
- ntfs_error(vol->sb, "Get size for mapping pairs failed for "
- "mft data attribute extent.");
- ret = mp_size;
- if (!ret)
- ret = -EIO;
- goto undo_alloc;
- }
- /* Expand the attribute record if necessary. */
- old_alen = le32_to_cpu(a->length);
- ret = ntfs_attr_record_resize(ctx->mrec, a, mp_size +
- le16_to_cpu(a->data.non_resident.mapping_pairs_offset));
- if (unlikely(ret)) {
- if (ret != -ENOSPC) {
- ntfs_error(vol->sb, "Failed to resize attribute "
- "record for mft data attribute.");
- goto undo_alloc;
- }
- // TODO: Deal with this by moving this extent to a new mft
- // record or by starting a new extent in a new mft record or by
- // moving other attributes out of this mft record.
- // Note: Use the special reserved mft records and ensure that
- // this extent is not required to find the mft record in
- // question. If no free special records left we would need to
- // move an existing record away, insert ours in its place, and
- // then place the moved record into the newly allocated space
- // and we would then need to update all references to this mft
- // record appropriately. This is rather complicated...
- ntfs_error(vol->sb, "Not enough space in this mft record to "
- "accommodate extended mft data attribute "
- "extent. Cannot handle this yet.");
- ret = -EOPNOTSUPP;
- goto undo_alloc;
- }
- mp_rebuilt = true;
- /* Generate the mapping pairs array directly into the attr record. */
- ret = ntfs_mapping_pairs_build(vol, (u8*)a +
- le16_to_cpu(a->data.non_resident.mapping_pairs_offset),
- mp_size, rl2, ll, -1, NULL);
- if (unlikely(ret)) {
- ntfs_error(vol->sb, "Failed to build mapping pairs array of "
- "mft data attribute.");
- goto undo_alloc;
- }
- /* Update the highest_vcn. */
- a->data.non_resident.highest_vcn = cpu_to_sle64(rl[1].vcn - 1);
- /*
- * We now have extended the mft data allocated_size by nr clusters.
- * Reflect this in the ntfs_inode structure and the attribute record.
- * @rl is the last (non-terminator) runlist element of mft data
- * attribute.
- */
- if (a->data.non_resident.lowest_vcn) {
- /*
- * We are not in the first attribute extent, switch to it, but
- * first ensure the changes will make it to disk later.
- */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_reinit_search_ctx(ctx);
- ret = ntfs_attr_lookup(mft_ni->type, mft_ni->name,
- mft_ni->name_len, CASE_SENSITIVE, 0, NULL, 0,
- ctx);
- if (unlikely(ret)) {
- ntfs_error(vol->sb, "Failed to find first attribute "
- "extent of mft data attribute.");
- goto restore_undo_alloc;
- }
- a = ctx->attr;
- }
- write_lock_irqsave(&mft_ni->size_lock, flags);
- mft_ni->allocated_size += nr << vol->cluster_size_bits;
- a->data.non_resident.allocated_size =
- cpu_to_sle64(mft_ni->allocated_size);
- write_unlock_irqrestore(&mft_ni->size_lock, flags);
- /* Ensure the changes make it to disk. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(mft_ni);
- up_write(&mft_ni->runlist.lock);
- ntfs_debug("Done.");
- return 0;
-restore_undo_alloc:
- ntfs_attr_reinit_search_ctx(ctx);
- if (ntfs_attr_lookup(mft_ni->type, mft_ni->name, mft_ni->name_len,
- CASE_SENSITIVE, rl[1].vcn, NULL, 0, ctx)) {
- ntfs_error(vol->sb, "Failed to find last attribute extent of "
- "mft data attribute.%s", es);
- write_lock_irqsave(&mft_ni->size_lock, flags);
- mft_ni->allocated_size += nr << vol->cluster_size_bits;
- write_unlock_irqrestore(&mft_ni->size_lock, flags);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(mft_ni);
- up_write(&mft_ni->runlist.lock);
- /*
- * The only thing that is now wrong is ->allocated_size of the
- * base attribute extent which chkdsk should be able to fix.
- */
- NVolSetErrors(vol);
- return ret;
- }
- ctx->attr->data.non_resident.highest_vcn =
- cpu_to_sle64(old_last_vcn - 1);
-undo_alloc:
- if (ntfs_cluster_free(mft_ni, old_last_vcn, -1, ctx) < 0) {
- ntfs_error(vol->sb, "Failed to free clusters from mft data "
- "attribute.%s", es);
- NVolSetErrors(vol);
- }
- a = ctx->attr;
- if (ntfs_rl_truncate_nolock(vol, &mft_ni->runlist, old_last_vcn)) {
- ntfs_error(vol->sb, "Failed to truncate mft data attribute "
- "runlist.%s", es);
- NVolSetErrors(vol);
- }
- if (mp_rebuilt && !IS_ERR(ctx->mrec)) {
- if (ntfs_mapping_pairs_build(vol, (u8*)a + le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset),
- old_alen - le16_to_cpu(
- a->data.non_resident.mapping_pairs_offset),
- rl2, ll, -1, NULL)) {
- ntfs_error(vol->sb, "Failed to restore mapping pairs "
- "array.%s", es);
- NVolSetErrors(vol);
- }
- if (ntfs_attr_record_resize(ctx->mrec, a, old_alen)) {
- ntfs_error(vol->sb, "Failed to restore attribute "
- "record.%s", es);
- NVolSetErrors(vol);
- }
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- } else if (IS_ERR(ctx->mrec)) {
- ntfs_error(vol->sb, "Failed to restore attribute search "
- "context.%s", es);
- NVolSetErrors(vol);
- }
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (!IS_ERR(mrec))
- unmap_mft_record(mft_ni);
- up_write(&mft_ni->runlist.lock);
- return ret;
-}
-
-/**
- * ntfs_mft_record_layout - layout an mft record into a memory buffer
- * @vol: volume to which the mft record will belong
- * @mft_no: mft reference specifying the mft record number
- * @m: destination buffer of size >= @vol->mft_record_size bytes
- *
- * Layout an empty, unused mft record with the mft record number @mft_no into
- * the buffer @m. The volume @vol is needed because the mft record structure
- * was modified in NTFS 3.1 so we need to know which volume version this mft
- * record will be used on.
- *
- * Return 0 on success and -errno on error.
- */
-static int ntfs_mft_record_layout(const ntfs_volume *vol, const s64 mft_no,
- MFT_RECORD *m)
-{
- ATTR_RECORD *a;
-
- ntfs_debug("Entering for mft record 0x%llx.", (long long)mft_no);
- if (mft_no >= (1ll << 32)) {
- ntfs_error(vol->sb, "Mft record number 0x%llx exceeds "
- "maximum of 2^32.", (long long)mft_no);
- return -ERANGE;
- }
- /* Start by clearing the whole mft record to gives us a clean slate. */
- memset(m, 0, vol->mft_record_size);
- /* Aligned to 2-byte boundary. */
- if (vol->major_ver < 3 || (vol->major_ver == 3 && !vol->minor_ver))
- m->usa_ofs = cpu_to_le16((sizeof(MFT_RECORD_OLD) + 1) & ~1);
- else {
- m->usa_ofs = cpu_to_le16((sizeof(MFT_RECORD) + 1) & ~1);
- /*
- * Set the NTFS 3.1+ specific fields while we know that the
- * volume version is 3.1+.
- */
- m->reserved = 0;
- m->mft_record_number = cpu_to_le32((u32)mft_no);
- }
- m->magic = magic_FILE;
- if (vol->mft_record_size >= NTFS_BLOCK_SIZE)
- m->usa_count = cpu_to_le16(vol->mft_record_size /
- NTFS_BLOCK_SIZE + 1);
- else {
- m->usa_count = cpu_to_le16(1);
- ntfs_warning(vol->sb, "Sector size is bigger than mft record "
- "size. Setting usa_count to 1. If chkdsk "
- "reports this as corruption, please email "
- "linux-ntfs-dev@lists.sourceforge.net stating "
- "that you saw this message and that the "
- "modified filesystem created was corrupt. "
- "Thank you.");
- }
- /* Set the update sequence number to 1. */
- *(le16*)((u8*)m + le16_to_cpu(m->usa_ofs)) = cpu_to_le16(1);
- m->lsn = 0;
- m->sequence_number = cpu_to_le16(1);
- m->link_count = 0;
- /*
- * Place the attributes straight after the update sequence array,
- * aligned to 8-byte boundary.
- */
- m->attrs_offset = cpu_to_le16((le16_to_cpu(m->usa_ofs) +
- (le16_to_cpu(m->usa_count) << 1) + 7) & ~7);
- m->flags = 0;
- /*
- * Using attrs_offset plus eight bytes (for the termination attribute).
- * attrs_offset is already aligned to 8-byte boundary, so no need to
- * align again.
- */
- m->bytes_in_use = cpu_to_le32(le16_to_cpu(m->attrs_offset) + 8);
- m->bytes_allocated = cpu_to_le32(vol->mft_record_size);
- m->base_mft_record = 0;
- m->next_attr_instance = 0;
- /* Add the termination attribute. */
- a = (ATTR_RECORD*)((u8*)m + le16_to_cpu(m->attrs_offset));
- a->type = AT_END;
- a->length = 0;
- ntfs_debug("Done.");
- return 0;
-}
-
-/**
- * ntfs_mft_record_format - format an mft record on an ntfs volume
- * @vol: volume on which to format the mft record
- * @mft_no: mft record number to format
- *
- * Format the mft record @mft_no in $MFT/$DATA, i.e. lay out an empty, unused
- * mft record into the appropriate place of the mft data attribute. This is
- * used when extending the mft data attribute.
- *
- * Return 0 on success and -errno on error.
- */
-static int ntfs_mft_record_format(const ntfs_volume *vol, const s64 mft_no)
-{
- loff_t i_size;
- struct inode *mft_vi = vol->mft_ino;
- struct page *page;
- MFT_RECORD *m;
- pgoff_t index, end_index;
- unsigned int ofs;
- int err;
-
- ntfs_debug("Entering for mft record 0x%llx.", (long long)mft_no);
- /*
- * The index into the page cache and the offset within the page cache
- * page of the wanted mft record.
- */
- index = mft_no << vol->mft_record_size_bits >> PAGE_SHIFT;
- ofs = (mft_no << vol->mft_record_size_bits) & ~PAGE_MASK;
- /* The maximum valid index into the page cache for $MFT's data. */
- i_size = i_size_read(mft_vi);
- end_index = i_size >> PAGE_SHIFT;
- if (unlikely(index >= end_index)) {
- if (unlikely(index > end_index || ofs + vol->mft_record_size >=
- (i_size & ~PAGE_MASK))) {
- ntfs_error(vol->sb, "Tried to format non-existing mft "
- "record 0x%llx.", (long long)mft_no);
- return -ENOENT;
- }
- }
- /* Read, map, and pin the page containing the mft record. */
- page = ntfs_map_page(mft_vi->i_mapping, index);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to map page containing mft record "
- "to format 0x%llx.", (long long)mft_no);
- return PTR_ERR(page);
- }
- lock_page(page);
- BUG_ON(!PageUptodate(page));
- ClearPageUptodate(page);
- m = (MFT_RECORD*)((u8*)page_address(page) + ofs);
- err = ntfs_mft_record_layout(vol, mft_no, m);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Failed to layout mft record 0x%llx.",
- (long long)mft_no);
- SetPageUptodate(page);
- unlock_page(page);
- ntfs_unmap_page(page);
- return err;
- }
- flush_dcache_page(page);
- SetPageUptodate(page);
- unlock_page(page);
- /*
- * Make sure the mft record is written out to disk. We could use
- * ilookup5() to check if an inode is in icache and so on but this is
- * unnecessary as ntfs_writepage() will write the dirty record anyway.
- */
- mark_ntfs_record_dirty(page, ofs);
- ntfs_unmap_page(page);
- ntfs_debug("Done.");
- return 0;
-}
-
-/**
- * ntfs_mft_record_alloc - allocate an mft record on an ntfs volume
- * @vol: [IN] volume on which to allocate the mft record
- * @mode: [IN] mode if want a file or directory, i.e. base inode or 0
- * @base_ni: [IN] open base inode if allocating an extent mft record or NULL
- * @mrec: [OUT] on successful return this is the mapped mft record
- *
- * Allocate an mft record in $MFT/$DATA of an open ntfs volume @vol.
- *
- * If @base_ni is NULL make the mft record a base mft record, i.e. a file or
- * direvctory inode, and allocate it at the default allocator position. In
- * this case @mode is the file mode as given to us by the caller. We in
- * particular use @mode to distinguish whether a file or a directory is being
- * created (S_IFDIR(mode) and S_IFREG(mode), respectively).
- *
- * If @base_ni is not NULL make the allocated mft record an extent record,
- * allocate it starting at the mft record after the base mft record and attach
- * the allocated and opened ntfs inode to the base inode @base_ni. In this
- * case @mode must be 0 as it is meaningless for extent inodes.
- *
- * You need to check the return value with IS_ERR(). If false, the function
- * was successful and the return value is the now opened ntfs inode of the
- * allocated mft record. *@mrec is then set to the allocated, mapped, pinned,
- * and locked mft record. If IS_ERR() is true, the function failed and the
- * error code is obtained from PTR_ERR(return value). *@mrec is undefined in
- * this case.
- *
- * Allocation strategy:
- *
- * To find a free mft record, we scan the mft bitmap for a zero bit. To
- * optimize this we start scanning at the place specified by @base_ni or if
- * @base_ni is NULL we start where we last stopped and we perform wrap around
- * when we reach the end. Note, we do not try to allocate mft records below
- * number 24 because numbers 0 to 15 are the defined system files anyway and 16
- * to 24 are special in that they are used for storing extension mft records
- * for the $DATA attribute of $MFT. This is required to avoid the possibility
- * of creating a runlist with a circular dependency which once written to disk
- * can never be read in again. Windows will only use records 16 to 24 for
- * normal files if the volume is completely out of space. We never use them
- * which means that when the volume is really out of space we cannot create any
- * more files while Windows can still create up to 8 small files. We can start
- * doing this at some later time, it does not matter much for now.
- *
- * When scanning the mft bitmap, we only search up to the last allocated mft
- * record. If there are no free records left in the range 24 to number of
- * allocated mft records, then we extend the $MFT/$DATA attribute in order to
- * create free mft records. We extend the allocated size of $MFT/$DATA by 16
- * records at a time or one cluster, if cluster size is above 16kiB. If there
- * is not sufficient space to do this, we try to extend by a single mft record
- * or one cluster, if cluster size is above the mft record size.
- *
- * No matter how many mft records we allocate, we initialize only the first
- * allocated mft record, incrementing mft data size and initialized size
- * accordingly, open an ntfs_inode for it and return it to the caller, unless
- * there are less than 24 mft records, in which case we allocate and initialize
- * mft records until we reach record 24 which we consider as the first free mft
- * record for use by normal files.
- *
- * If during any stage we overflow the initialized data in the mft bitmap, we
- * extend the initialized size (and data size) by 8 bytes, allocating another
- * cluster if required. The bitmap data size has to be at least equal to the
- * number of mft records in the mft, but it can be bigger, in which case the
- * superflous bits are padded with zeroes.
- *
- * Thus, when we return successfully (IS_ERR() is false), we will have:
- * - initialized / extended the mft bitmap if necessary,
- * - initialized / extended the mft data if necessary,
- * - set the bit corresponding to the mft record being allocated in the
- * mft bitmap,
- * - opened an ntfs_inode for the allocated mft record, and we will have
- * - returned the ntfs_inode as well as the allocated mapped, pinned, and
- * locked mft record.
- *
- * On error, the volume will be left in a consistent state and no record will
- * be allocated. If rolling back a partial operation fails, we may leave some
- * inconsistent metadata in which case we set NVolErrors() so the volume is
- * left dirty when unmounted.
- *
- * Note, this function cannot make use of most of the normal functions, like
- * for example for attribute resizing, etc, because when the run list overflows
- * the base mft record and an attribute list is used, it is very important that
- * the extension mft records used to store the $DATA attribute of $MFT can be
- * reached without having to read the information contained inside them, as
- * this would make it impossible to find them in the first place after the
- * volume is unmounted. $MFT/$BITMAP probably does not need to follow this
- * rule because the bitmap is not essential for finding the mft records, but on
- * the other hand, handling the bitmap in this special way would make life
- * easier because otherwise there might be circular invocations of functions
- * when reading the bitmap.
- */
-ntfs_inode *ntfs_mft_record_alloc(ntfs_volume *vol, const int mode,
- ntfs_inode *base_ni, MFT_RECORD **mrec)
-{
- s64 ll, bit, old_data_initialized, old_data_size;
- unsigned long flags;
- struct inode *vi;
- struct page *page;
- ntfs_inode *mft_ni, *mftbmp_ni, *ni;
- ntfs_attr_search_ctx *ctx;
- MFT_RECORD *m;
- ATTR_RECORD *a;
- pgoff_t index;
- unsigned int ofs;
- int err;
- le16 seq_no, usn;
- bool record_formatted = false;
-
- if (base_ni) {
- ntfs_debug("Entering (allocating an extent mft record for "
- "base mft record 0x%llx).",
- (long long)base_ni->mft_no);
- /* @mode and @base_ni are mutually exclusive. */
- BUG_ON(mode);
- } else
- ntfs_debug("Entering (allocating a base mft record).");
- if (mode) {
- /* @mode and @base_ni are mutually exclusive. */
- BUG_ON(base_ni);
- /* We only support creation of normal files and directories. */
- if (!S_ISREG(mode) && !S_ISDIR(mode))
- return ERR_PTR(-EOPNOTSUPP);
- }
- BUG_ON(!mrec);
- mft_ni = NTFS_I(vol->mft_ino);
- mftbmp_ni = NTFS_I(vol->mftbmp_ino);
- down_write(&vol->mftbmp_lock);
- bit = ntfs_mft_bitmap_find_and_alloc_free_rec_nolock(vol, base_ni);
- if (bit >= 0) {
- ntfs_debug("Found and allocated free record (#1), bit 0x%llx.",
- (long long)bit);
- goto have_alloc_rec;
- }
- if (bit != -ENOSPC) {
- up_write(&vol->mftbmp_lock);
- return ERR_PTR(bit);
- }
- /*
- * No free mft records left. If the mft bitmap already covers more
- * than the currently used mft records, the next records are all free,
- * so we can simply allocate the first unused mft record.
- * Note: We also have to make sure that the mft bitmap at least covers
- * the first 24 mft records as they are special and whilst they may not
- * be in use, we do not allocate from them.
- */
- read_lock_irqsave(&mft_ni->size_lock, flags);
- ll = mft_ni->initialized_size >> vol->mft_record_size_bits;
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
- read_lock_irqsave(&mftbmp_ni->size_lock, flags);
- old_data_initialized = mftbmp_ni->initialized_size;
- read_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
- if (old_data_initialized << 3 > ll && old_data_initialized > 3) {
- bit = ll;
- if (bit < 24)
- bit = 24;
- if (unlikely(bit >= (1ll << 32)))
- goto max_err_out;
- ntfs_debug("Found free record (#2), bit 0x%llx.",
- (long long)bit);
- goto found_free_rec;
- }
- /*
- * The mft bitmap needs to be expanded until it covers the first unused
- * mft record that we can allocate.
- * Note: The smallest mft record we allocate is mft record 24.
- */
- bit = old_data_initialized << 3;
- if (unlikely(bit >= (1ll << 32)))
- goto max_err_out;
- read_lock_irqsave(&mftbmp_ni->size_lock, flags);
- old_data_size = mftbmp_ni->allocated_size;
- ntfs_debug("Status of mftbmp before extension: allocated_size 0x%llx, "
- "data_size 0x%llx, initialized_size 0x%llx.",
- (long long)old_data_size,
- (long long)i_size_read(vol->mftbmp_ino),
- (long long)old_data_initialized);
- read_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
- if (old_data_initialized + 8 > old_data_size) {
- /* Need to extend bitmap by one more cluster. */
- ntfs_debug("mftbmp: initialized_size + 8 > allocated_size.");
- err = ntfs_mft_bitmap_extend_allocation_nolock(vol);
- if (unlikely(err)) {
- up_write(&vol->mftbmp_lock);
- goto err_out;
- }
-#ifdef DEBUG
- read_lock_irqsave(&mftbmp_ni->size_lock, flags);
- ntfs_debug("Status of mftbmp after allocation extension: "
- "allocated_size 0x%llx, data_size 0x%llx, "
- "initialized_size 0x%llx.",
- (long long)mftbmp_ni->allocated_size,
- (long long)i_size_read(vol->mftbmp_ino),
- (long long)mftbmp_ni->initialized_size);
- read_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
-#endif /* DEBUG */
- }
- /*
- * We now have sufficient allocated space, extend the initialized_size
- * as well as the data_size if necessary and fill the new space with
- * zeroes.
- */
- err = ntfs_mft_bitmap_extend_initialized_nolock(vol);
- if (unlikely(err)) {
- up_write(&vol->mftbmp_lock);
- goto err_out;
- }
-#ifdef DEBUG
- read_lock_irqsave(&mftbmp_ni->size_lock, flags);
- ntfs_debug("Status of mftbmp after initialized extension: "
- "allocated_size 0x%llx, data_size 0x%llx, "
- "initialized_size 0x%llx.",
- (long long)mftbmp_ni->allocated_size,
- (long long)i_size_read(vol->mftbmp_ino),
- (long long)mftbmp_ni->initialized_size);
- read_unlock_irqrestore(&mftbmp_ni->size_lock, flags);
-#endif /* DEBUG */
- ntfs_debug("Found free record (#3), bit 0x%llx.", (long long)bit);
-found_free_rec:
- /* @bit is the found free mft record, allocate it in the mft bitmap. */
- ntfs_debug("At found_free_rec.");
- err = ntfs_bitmap_set_bit(vol->mftbmp_ino, bit);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Failed to allocate bit in mft bitmap.");
- up_write(&vol->mftbmp_lock);
- goto err_out;
- }
- ntfs_debug("Set bit 0x%llx in mft bitmap.", (long long)bit);
-have_alloc_rec:
- /*
- * The mft bitmap is now uptodate. Deal with mft data attribute now.
- * Note, we keep hold of the mft bitmap lock for writing until all
- * modifications to the mft data attribute are complete, too, as they
- * will impact decisions for mft bitmap and mft record allocation done
- * by a parallel allocation and if the lock is not maintained a
- * parallel allocation could allocate the same mft record as this one.
- */
- ll = (bit + 1) << vol->mft_record_size_bits;
- read_lock_irqsave(&mft_ni->size_lock, flags);
- old_data_initialized = mft_ni->initialized_size;
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
- if (ll <= old_data_initialized) {
- ntfs_debug("Allocated mft record already initialized.");
- goto mft_rec_already_initialized;
- }
- ntfs_debug("Initializing allocated mft record.");
- /*
- * The mft record is outside the initialized data. Extend the mft data
- * attribute until it covers the allocated record. The loop is only
- * actually traversed more than once when a freshly formatted volume is
- * first written to so it optimizes away nicely in the common case.
- */
- read_lock_irqsave(&mft_ni->size_lock, flags);
- ntfs_debug("Status of mft data before extension: "
- "allocated_size 0x%llx, data_size 0x%llx, "
- "initialized_size 0x%llx.",
- (long long)mft_ni->allocated_size,
- (long long)i_size_read(vol->mft_ino),
- (long long)mft_ni->initialized_size);
- while (ll > mft_ni->allocated_size) {
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
- err = ntfs_mft_data_extend_allocation_nolock(vol);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Failed to extend mft data "
- "allocation.");
- goto undo_mftbmp_alloc_nolock;
- }
- read_lock_irqsave(&mft_ni->size_lock, flags);
- ntfs_debug("Status of mft data after allocation extension: "
- "allocated_size 0x%llx, data_size 0x%llx, "
- "initialized_size 0x%llx.",
- (long long)mft_ni->allocated_size,
- (long long)i_size_read(vol->mft_ino),
- (long long)mft_ni->initialized_size);
- }
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
- /*
- * Extend mft data initialized size (and data size of course) to reach
- * the allocated mft record, formatting the mft records allong the way.
- * Note: We only modify the ntfs_inode structure as that is all that is
- * needed by ntfs_mft_record_format(). We will update the attribute
- * record itself in one fell swoop later on.
- */
- write_lock_irqsave(&mft_ni->size_lock, flags);
- old_data_initialized = mft_ni->initialized_size;
- old_data_size = vol->mft_ino->i_size;
- while (ll > mft_ni->initialized_size) {
- s64 new_initialized_size, mft_no;
-
- new_initialized_size = mft_ni->initialized_size +
- vol->mft_record_size;
- mft_no = mft_ni->initialized_size >> vol->mft_record_size_bits;
- if (new_initialized_size > i_size_read(vol->mft_ino))
- i_size_write(vol->mft_ino, new_initialized_size);
- write_unlock_irqrestore(&mft_ni->size_lock, flags);
- ntfs_debug("Initializing mft record 0x%llx.",
- (long long)mft_no);
- err = ntfs_mft_record_format(vol, mft_no);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Failed to format mft record.");
- goto undo_data_init;
- }
- write_lock_irqsave(&mft_ni->size_lock, flags);
- mft_ni->initialized_size = new_initialized_size;
- }
- write_unlock_irqrestore(&mft_ni->size_lock, flags);
- record_formatted = true;
- /* Update the mft data attribute record to reflect the new sizes. */
- m = map_mft_record(mft_ni);
- if (IS_ERR(m)) {
- ntfs_error(vol->sb, "Failed to map mft record.");
- err = PTR_ERR(m);
- goto undo_data_init;
- }
- ctx = ntfs_attr_get_search_ctx(mft_ni, m);
- if (unlikely(!ctx)) {
- ntfs_error(vol->sb, "Failed to get search context.");
- err = -ENOMEM;
- unmap_mft_record(mft_ni);
- goto undo_data_init;
- }
- err = ntfs_attr_lookup(mft_ni->type, mft_ni->name, mft_ni->name_len,
- CASE_SENSITIVE, 0, NULL, 0, ctx);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Failed to find first attribute extent of "
- "mft data attribute.");
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(mft_ni);
- goto undo_data_init;
- }
- a = ctx->attr;
- read_lock_irqsave(&mft_ni->size_lock, flags);
- a->data.non_resident.initialized_size =
- cpu_to_sle64(mft_ni->initialized_size);
- a->data.non_resident.data_size =
- cpu_to_sle64(i_size_read(vol->mft_ino));
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
- /* Ensure the changes make it to disk. */
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(mft_ni);
- read_lock_irqsave(&mft_ni->size_lock, flags);
- ntfs_debug("Status of mft data after mft record initialization: "
- "allocated_size 0x%llx, data_size 0x%llx, "
- "initialized_size 0x%llx.",
- (long long)mft_ni->allocated_size,
- (long long)i_size_read(vol->mft_ino),
- (long long)mft_ni->initialized_size);
- BUG_ON(i_size_read(vol->mft_ino) > mft_ni->allocated_size);
- BUG_ON(mft_ni->initialized_size > i_size_read(vol->mft_ino));
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
-mft_rec_already_initialized:
- /*
- * We can finally drop the mft bitmap lock as the mft data attribute
- * has been fully updated. The only disparity left is that the
- * allocated mft record still needs to be marked as in use to match the
- * set bit in the mft bitmap but this is actually not a problem since
- * this mft record is not referenced from anywhere yet and the fact
- * that it is allocated in the mft bitmap means that no-one will try to
- * allocate it either.
- */
- up_write(&vol->mftbmp_lock);
- /*
- * We now have allocated and initialized the mft record. Calculate the
- * index of and the offset within the page cache page the record is in.
- */
- index = bit << vol->mft_record_size_bits >> PAGE_SHIFT;
- ofs = (bit << vol->mft_record_size_bits) & ~PAGE_MASK;
- /* Read, map, and pin the page containing the mft record. */
- page = ntfs_map_page(vol->mft_ino->i_mapping, index);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to map page containing allocated "
- "mft record 0x%llx.", (long long)bit);
- err = PTR_ERR(page);
- goto undo_mftbmp_alloc;
- }
- lock_page(page);
- BUG_ON(!PageUptodate(page));
- ClearPageUptodate(page);
- m = (MFT_RECORD*)((u8*)page_address(page) + ofs);
- /* If we just formatted the mft record no need to do it again. */
- if (!record_formatted) {
- /* Sanity check that the mft record is really not in use. */
- if (ntfs_is_file_record(m->magic) &&
- (m->flags & MFT_RECORD_IN_USE)) {
- ntfs_error(vol->sb, "Mft record 0x%llx was marked "
- "free in mft bitmap but is marked "
- "used itself. Corrupt filesystem. "
- "Unmount and run chkdsk.",
- (long long)bit);
- err = -EIO;
- SetPageUptodate(page);
- unlock_page(page);
- ntfs_unmap_page(page);
- NVolSetErrors(vol);
- goto undo_mftbmp_alloc;
- }
- /*
- * We need to (re-)format the mft record, preserving the
- * sequence number if it is not zero as well as the update
- * sequence number if it is not zero or -1 (0xffff). This
- * means we do not need to care whether or not something went
- * wrong with the previous mft record.
- */
- seq_no = m->sequence_number;
- usn = *(le16*)((u8*)m + le16_to_cpu(m->usa_ofs));
- err = ntfs_mft_record_layout(vol, bit, m);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Failed to layout allocated mft "
- "record 0x%llx.", (long long)bit);
- SetPageUptodate(page);
- unlock_page(page);
- ntfs_unmap_page(page);
- goto undo_mftbmp_alloc;
- }
- if (seq_no)
- m->sequence_number = seq_no;
- if (usn && le16_to_cpu(usn) != 0xffff)
- *(le16*)((u8*)m + le16_to_cpu(m->usa_ofs)) = usn;
- }
- /* Set the mft record itself in use. */
- m->flags |= MFT_RECORD_IN_USE;
- if (S_ISDIR(mode))
- m->flags |= MFT_RECORD_IS_DIRECTORY;
- flush_dcache_page(page);
- SetPageUptodate(page);
- if (base_ni) {
- MFT_RECORD *m_tmp;
-
- /*
- * Setup the base mft record in the extent mft record. This
- * completes initialization of the allocated extent mft record
- * and we can simply use it with map_extent_mft_record().
- */
- m->base_mft_record = MK_LE_MREF(base_ni->mft_no,
- base_ni->seq_no);
- /*
- * Allocate an extent inode structure for the new mft record,
- * attach it to the base inode @base_ni and map, pin, and lock
- * its, i.e. the allocated, mft record.
- */
- m_tmp = map_extent_mft_record(base_ni, bit, &ni);
- if (IS_ERR(m_tmp)) {
- ntfs_error(vol->sb, "Failed to map allocated extent "
- "mft record 0x%llx.", (long long)bit);
- err = PTR_ERR(m_tmp);
- /* Set the mft record itself not in use. */
- m->flags &= cpu_to_le16(
- ~le16_to_cpu(MFT_RECORD_IN_USE));
- flush_dcache_page(page);
- /* Make sure the mft record is written out to disk. */
- mark_ntfs_record_dirty(page, ofs);
- unlock_page(page);
- ntfs_unmap_page(page);
- goto undo_mftbmp_alloc;
- }
- BUG_ON(m != m_tmp);
- /*
- * Make sure the allocated mft record is written out to disk.
- * No need to set the inode dirty because the caller is going
- * to do that anyway after finishing with the new extent mft
- * record (e.g. at a minimum a new attribute will be added to
- * the mft record.
- */
- mark_ntfs_record_dirty(page, ofs);
- unlock_page(page);
- /*
- * Need to unmap the page since map_extent_mft_record() mapped
- * it as well so we have it mapped twice at the moment.
- */
- ntfs_unmap_page(page);
- } else {
- /*
- * Allocate a new VFS inode and set it up. NOTE: @vi->i_nlink
- * is set to 1 but the mft record->link_count is 0. The caller
- * needs to bear this in mind.
- */
- vi = new_inode(vol->sb);
- if (unlikely(!vi)) {
- err = -ENOMEM;
- /* Set the mft record itself not in use. */
- m->flags &= cpu_to_le16(
- ~le16_to_cpu(MFT_RECORD_IN_USE));
- flush_dcache_page(page);
- /* Make sure the mft record is written out to disk. */
- mark_ntfs_record_dirty(page, ofs);
- unlock_page(page);
- ntfs_unmap_page(page);
- goto undo_mftbmp_alloc;
- }
- vi->i_ino = bit;
-
- /* The owner and group come from the ntfs volume. */
- vi->i_uid = vol->uid;
- vi->i_gid = vol->gid;
-
- /* Initialize the ntfs specific part of @vi. */
- ntfs_init_big_inode(vi);
- ni = NTFS_I(vi);
- /*
- * Set the appropriate mode, attribute type, and name. For
- * directories, also setup the index values to the defaults.
- */
- if (S_ISDIR(mode)) {
- vi->i_mode = S_IFDIR | S_IRWXUGO;
- vi->i_mode &= ~vol->dmask;
-
- NInoSetMstProtected(ni);
- ni->type = AT_INDEX_ALLOCATION;
- ni->name = I30;
- ni->name_len = 4;
-
- ni->itype.index.block_size = 4096;
- ni->itype.index.block_size_bits = ntfs_ffs(4096) - 1;
- ni->itype.index.collation_rule = COLLATION_FILE_NAME;
- if (vol->cluster_size <= ni->itype.index.block_size) {
- ni->itype.index.vcn_size = vol->cluster_size;
- ni->itype.index.vcn_size_bits =
- vol->cluster_size_bits;
- } else {
- ni->itype.index.vcn_size = vol->sector_size;
- ni->itype.index.vcn_size_bits =
- vol->sector_size_bits;
- }
- } else {
- vi->i_mode = S_IFREG | S_IRWXUGO;
- vi->i_mode &= ~vol->fmask;
-
- ni->type = AT_DATA;
- ni->name = NULL;
- ni->name_len = 0;
- }
- if (IS_RDONLY(vi))
- vi->i_mode &= ~S_IWUGO;
-
- /* Set the inode times to the current time. */
- vi->i_atime = vi->i_mtime = vi->i_ctime =
- current_time(vi);
- /*
- * Set the file size to 0, the ntfs inode sizes are set to 0 by
- * the call to ntfs_init_big_inode() below.
- */
- vi->i_size = 0;
- vi->i_blocks = 0;
-
- /* Set the sequence number. */
- vi->i_generation = ni->seq_no = le16_to_cpu(m->sequence_number);
- /*
- * Manually map, pin, and lock the mft record as we already
- * have its page mapped and it is very easy to do.
- */
- atomic_inc(&ni->count);
- mutex_lock(&ni->mrec_lock);
- ni->page = page;
- ni->page_ofs = ofs;
- /*
- * Make sure the allocated mft record is written out to disk.
- * NOTE: We do not set the ntfs inode dirty because this would
- * fail in ntfs_write_inode() because the inode does not have a
- * standard information attribute yet. Also, there is no need
- * to set the inode dirty because the caller is going to do
- * that anyway after finishing with the new mft record (e.g. at
- * a minimum some new attributes will be added to the mft
- * record.
- */
- mark_ntfs_record_dirty(page, ofs);
- unlock_page(page);
-
- /* Add the inode to the inode hash for the superblock. */
- insert_inode_hash(vi);
-
- /* Update the default mft allocation position. */
- vol->mft_data_pos = bit + 1;
- }
- /*
- * Return the opened, allocated inode of the allocated mft record as
- * well as the mapped, pinned, and locked mft record.
- */
- ntfs_debug("Returning opened, allocated %sinode 0x%llx.",
- base_ni ? "extent " : "", (long long)bit);
- *mrec = m;
- return ni;
-undo_data_init:
- write_lock_irqsave(&mft_ni->size_lock, flags);
- mft_ni->initialized_size = old_data_initialized;
- i_size_write(vol->mft_ino, old_data_size);
- write_unlock_irqrestore(&mft_ni->size_lock, flags);
- goto undo_mftbmp_alloc_nolock;
-undo_mftbmp_alloc:
- down_write(&vol->mftbmp_lock);
-undo_mftbmp_alloc_nolock:
- if (ntfs_bitmap_clear_bit(vol->mftbmp_ino, bit)) {
- ntfs_error(vol->sb, "Failed to clear bit in mft bitmap.%s", es);
- NVolSetErrors(vol);
- }
- up_write(&vol->mftbmp_lock);
-err_out:
- return ERR_PTR(err);
-max_err_out:
- ntfs_warning(vol->sb, "Cannot allocate mft record because the maximum "
- "number of inodes (2^32) has already been reached.");
- up_write(&vol->mftbmp_lock);
- return ERR_PTR(-ENOSPC);
-}
-
-/**
- * ntfs_extent_mft_record_free - free an extent mft record on an ntfs volume
- * @ni: ntfs inode of the mapped extent mft record to free
- * @m: mapped extent mft record of the ntfs inode @ni
- *
- * Free the mapped extent mft record @m of the extent ntfs inode @ni.
- *
- * Note that this function unmaps the mft record and closes and destroys @ni
- * internally and hence you cannot use either @ni nor @m any more after this
- * function returns success.
- *
- * On success return 0 and on error return -errno. @ni and @m are still valid
- * in this case and have not been freed.
- *
- * For some errors an error message is displayed and the success code 0 is
- * returned and the volume is then left dirty on umount. This makes sense in
- * case we could not rollback the changes that were already done since the
- * caller no longer wants to reference this mft record so it does not matter to
- * the caller if something is wrong with it as long as it is properly detached
- * from the base inode.
- */
-int ntfs_extent_mft_record_free(ntfs_inode *ni, MFT_RECORD *m)
-{
- unsigned long mft_no = ni->mft_no;
- ntfs_volume *vol = ni->vol;
- ntfs_inode *base_ni;
- ntfs_inode **extent_nis;
- int i, err;
- le16 old_seq_no;
- u16 seq_no;
-
- BUG_ON(NInoAttr(ni));
- BUG_ON(ni->nr_extents != -1);
-
- mutex_lock(&ni->extent_lock);
- base_ni = ni->ext.base_ntfs_ino;
- mutex_unlock(&ni->extent_lock);
-
- BUG_ON(base_ni->nr_extents <= 0);
-
- ntfs_debug("Entering for extent inode 0x%lx, base inode 0x%lx.\n",
- mft_no, base_ni->mft_no);
-
- mutex_lock(&base_ni->extent_lock);
-
- /* Make sure we are holding the only reference to the extent inode. */
- if (atomic_read(&ni->count) > 2) {
- ntfs_error(vol->sb, "Tried to free busy extent inode 0x%lx, "
- "not freeing.", base_ni->mft_no);
- mutex_unlock(&base_ni->extent_lock);
- return -EBUSY;
- }
-
- /* Dissociate the ntfs inode from the base inode. */
- extent_nis = base_ni->ext.extent_ntfs_inos;
- err = -ENOENT;
- for (i = 0; i < base_ni->nr_extents; i++) {
- if (ni != extent_nis[i])
- continue;
- extent_nis += i;
- base_ni->nr_extents--;
- memmove(extent_nis, extent_nis + 1, (base_ni->nr_extents - i) *
- sizeof(ntfs_inode*));
- err = 0;
- break;
- }
-
- mutex_unlock(&base_ni->extent_lock);
-
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Extent inode 0x%lx is not attached to "
- "its base inode 0x%lx.", mft_no,
- base_ni->mft_no);
- BUG();
- }
-
- /*
- * The extent inode is no longer attached to the base inode so no one
- * can get a reference to it any more.
- */
-
- /* Mark the mft record as not in use. */
- m->flags &= ~MFT_RECORD_IN_USE;
-
- /* Increment the sequence number, skipping zero, if it is not zero. */
- old_seq_no = m->sequence_number;
- seq_no = le16_to_cpu(old_seq_no);
- if (seq_no == 0xffff)
- seq_no = 1;
- else if (seq_no)
- seq_no++;
- m->sequence_number = cpu_to_le16(seq_no);
-
- /*
- * Set the ntfs inode dirty and write it out. We do not need to worry
- * about the base inode here since whatever caused the extent mft
- * record to be freed is guaranteed to do it already.
- */
- NInoSetDirty(ni);
- err = write_mft_record(ni, m, 0);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Failed to write mft record 0x%lx, not "
- "freeing.", mft_no);
- goto rollback;
- }
-rollback_error:
- /* Unmap and throw away the now freed extent inode. */
- unmap_extent_mft_record(ni);
- ntfs_clear_extent_inode(ni);
-
- /* Clear the bit in the $MFT/$BITMAP corresponding to this record. */
- down_write(&vol->mftbmp_lock);
- err = ntfs_bitmap_clear_bit(vol->mftbmp_ino, mft_no);
- up_write(&vol->mftbmp_lock);
- if (unlikely(err)) {
- /*
- * The extent inode is gone but we failed to deallocate it in
- * the mft bitmap. Just emit a warning and leave the volume
- * dirty on umount.
- */
- ntfs_error(vol->sb, "Failed to clear bit in mft bitmap.%s", es);
- NVolSetErrors(vol);
- }
- return 0;
-rollback:
- /* Rollback what we did... */
- mutex_lock(&base_ni->extent_lock);
- extent_nis = base_ni->ext.extent_ntfs_inos;
- if (!(base_ni->nr_extents & 3)) {
- int new_size = (base_ni->nr_extents + 4) * sizeof(ntfs_inode*);
-
- extent_nis = kmalloc(new_size, GFP_NOFS);
- if (unlikely(!extent_nis)) {
- ntfs_error(vol->sb, "Failed to allocate internal "
- "buffer during rollback.%s", es);
- mutex_unlock(&base_ni->extent_lock);
- NVolSetErrors(vol);
- goto rollback_error;
- }
- if (base_ni->nr_extents) {
- BUG_ON(!base_ni->ext.extent_ntfs_inos);
- memcpy(extent_nis, base_ni->ext.extent_ntfs_inos,
- new_size - 4 * sizeof(ntfs_inode*));
- kfree(base_ni->ext.extent_ntfs_inos);
- }
- base_ni->ext.extent_ntfs_inos = extent_nis;
- }
- m->flags |= MFT_RECORD_IN_USE;
- m->sequence_number = old_seq_no;
- extent_nis[base_ni->nr_extents++] = ni;
- mutex_unlock(&base_ni->extent_lock);
- mark_mft_record_dirty(ni);
- return err;
-}
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/mft.h b/fs/ntfs/mft.h
deleted file mode 100644
index 49c001af16ed..000000000000
--- a/fs/ntfs/mft.h
+++ /dev/null
@@ -1,110 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * mft.h - Defines for mft record handling in NTFS Linux kernel driver.
- * Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_MFT_H
-#define _LINUX_NTFS_MFT_H
-
-#include <linux/fs.h>
-#include <linux/highmem.h>
-#include <linux/pagemap.h>
-
-#include "inode.h"
-
-extern MFT_RECORD *map_mft_record(ntfs_inode *ni);
-extern void unmap_mft_record(ntfs_inode *ni);
-
-extern MFT_RECORD *map_extent_mft_record(ntfs_inode *base_ni, MFT_REF mref,
- ntfs_inode **ntfs_ino);
-
-static inline void unmap_extent_mft_record(ntfs_inode *ni)
-{
- unmap_mft_record(ni);
- return;
-}
-
-#ifdef NTFS_RW
-
-/**
- * flush_dcache_mft_record_page - flush_dcache_page() for mft records
- * @ni: ntfs inode structure of mft record
- *
- * Call flush_dcache_page() for the page in which an mft record resides.
- *
- * This must be called every time an mft record is modified, just after the
- * modification.
- */
-static inline void flush_dcache_mft_record_page(ntfs_inode *ni)
-{
- flush_dcache_page(ni->page);
-}
-
-extern void __mark_mft_record_dirty(ntfs_inode *ni);
-
-/**
- * mark_mft_record_dirty - set the mft record and the page containing it dirty
- * @ni: ntfs inode describing the mapped mft record
- *
- * Set the mapped (extent) mft record of the (base or extent) ntfs inode @ni,
- * as well as the page containing the mft record, dirty. Also, mark the base
- * vfs inode dirty. This ensures that any changes to the mft record are
- * written out to disk.
- *
- * NOTE: Do not do anything if the mft record is already marked dirty.
- */
-static inline void mark_mft_record_dirty(ntfs_inode *ni)
-{
- if (!NInoTestSetDirty(ni))
- __mark_mft_record_dirty(ni);
-}
-
-extern int ntfs_sync_mft_mirror(ntfs_volume *vol, const unsigned long mft_no,
- MFT_RECORD *m, int sync);
-
-extern int write_mft_record_nolock(ntfs_inode *ni, MFT_RECORD *m, int sync);
-
-/**
- * write_mft_record - write out a mapped (extent) mft record
- * @ni: ntfs inode describing the mapped (extent) mft record
- * @m: mapped (extent) mft record to write
- * @sync: if true, wait for i/o completion
- *
- * This is just a wrapper for write_mft_record_nolock() (see mft.c), which
- * locks the page for the duration of the write. This ensures that there are
- * no race conditions between writing the mft record via the dirty inode code
- * paths and via the page cache write back code paths or between writing
- * neighbouring mft records residing in the same page.
- *
- * Locking the page also serializes us against ->read_folio() if the page is not
- * uptodate.
- *
- * On success, clean the mft record and return 0. On error, leave the mft
- * record dirty and return -errno.
- */
-static inline int write_mft_record(ntfs_inode *ni, MFT_RECORD *m, int sync)
-{
- struct page *page = ni->page;
- int err;
-
- BUG_ON(!page);
- lock_page(page);
- err = write_mft_record_nolock(ni, m, sync);
- unlock_page(page);
- return err;
-}
-
-extern bool ntfs_may_write_mft_record(ntfs_volume *vol,
- const unsigned long mft_no, const MFT_RECORD *m,
- ntfs_inode **locked_ni);
-
-extern ntfs_inode *ntfs_mft_record_alloc(ntfs_volume *vol, const int mode,
- ntfs_inode *base_ni, MFT_RECORD **mrec);
-extern int ntfs_extent_mft_record_free(ntfs_inode *ni, MFT_RECORD *m);
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_MFT_H */
diff --git a/fs/ntfs/mst.c b/fs/ntfs/mst.c
deleted file mode 100644
index 16b3c884abfc..000000000000
--- a/fs/ntfs/mst.c
+++ /dev/null
@@ -1,189 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * mst.c - NTFS multi sector transfer protection handling code. Part of the
- * Linux-NTFS project.
- *
- * Copyright (c) 2001-2004 Anton Altaparmakov
- */
-
-#include "ntfs.h"
-
-/**
- * post_read_mst_fixup - deprotect multi sector transfer protected data
- * @b: pointer to the data to deprotect
- * @size: size in bytes of @b
- *
- * Perform the necessary post read multi sector transfer fixup and detect the
- * presence of incomplete multi sector transfers. - In that case, overwrite the
- * magic of the ntfs record header being processed with "BAAD" (in memory only!)
- * and abort processing.
- *
- * Return 0 on success and -EINVAL on error ("BAAD" magic will be present).
- *
- * NOTE: We consider the absence / invalidity of an update sequence array to
- * mean that the structure is not protected at all and hence doesn't need to
- * be fixed up. Thus, we return success and not failure in this case. This is
- * in contrast to pre_write_mst_fixup(), see below.
- */
-int post_read_mst_fixup(NTFS_RECORD *b, const u32 size)
-{
- u16 usa_ofs, usa_count, usn;
- u16 *usa_pos, *data_pos;
-
- /* Setup the variables. */
- usa_ofs = le16_to_cpu(b->usa_ofs);
- /* Decrement usa_count to get number of fixups. */
- usa_count = le16_to_cpu(b->usa_count) - 1;
- /* Size and alignment checks. */
- if ( size & (NTFS_BLOCK_SIZE - 1) ||
- usa_ofs & 1 ||
- usa_ofs + (usa_count * 2) > size ||
- (size >> NTFS_BLOCK_SIZE_BITS) != usa_count)
- return 0;
- /* Position of usn in update sequence array. */
- usa_pos = (u16*)b + usa_ofs/sizeof(u16);
- /*
- * The update sequence number which has to be equal to each of the
- * u16 values before they are fixed up. Note no need to care for
- * endianness since we are comparing and moving data for on disk
- * structures which means the data is consistent. - If it is
- * consistenty the wrong endianness it doesn't make any difference.
- */
- usn = *usa_pos;
- /*
- * Position in protected data of first u16 that needs fixing up.
- */
- data_pos = (u16*)b + NTFS_BLOCK_SIZE/sizeof(u16) - 1;
- /*
- * Check for incomplete multi sector transfer(s).
- */
- while (usa_count--) {
- if (*data_pos != usn) {
- /*
- * Incomplete multi sector transfer detected! )-:
- * Set the magic to "BAAD" and return failure.
- * Note that magic_BAAD is already converted to le32.
- */
- b->magic = magic_BAAD;
- return -EINVAL;
- }
- data_pos += NTFS_BLOCK_SIZE/sizeof(u16);
- }
- /* Re-setup the variables. */
- usa_count = le16_to_cpu(b->usa_count) - 1;
- data_pos = (u16*)b + NTFS_BLOCK_SIZE/sizeof(u16) - 1;
- /* Fixup all sectors. */
- while (usa_count--) {
- /*
- * Increment position in usa and restore original data from
- * the usa into the data buffer.
- */
- *data_pos = *(++usa_pos);
- /* Increment position in data as well. */
- data_pos += NTFS_BLOCK_SIZE/sizeof(u16);
- }
- return 0;
-}
-
-/**
- * pre_write_mst_fixup - apply multi sector transfer protection
- * @b: pointer to the data to protect
- * @size: size in bytes of @b
- *
- * Perform the necessary pre write multi sector transfer fixup on the data
- * pointer to by @b of @size.
- *
- * Return 0 if fixup applied (success) or -EINVAL if no fixup was performed
- * (assumed not needed). This is in contrast to post_read_mst_fixup() above.
- *
- * NOTE: We consider the absence / invalidity of an update sequence array to
- * mean that the structure is not subject to protection and hence doesn't need
- * to be fixed up. This means that you have to create a valid update sequence
- * array header in the ntfs record before calling this function, otherwise it
- * will fail (the header needs to contain the position of the update sequence
- * array together with the number of elements in the array). You also need to
- * initialise the update sequence number before calling this function
- * otherwise a random word will be used (whatever was in the record at that
- * position at that time).
- */
-int pre_write_mst_fixup(NTFS_RECORD *b, const u32 size)
-{
- le16 *usa_pos, *data_pos;
- u16 usa_ofs, usa_count, usn;
- le16 le_usn;
-
- /* Sanity check + only fixup if it makes sense. */
- if (!b || ntfs_is_baad_record(b->magic) ||
- ntfs_is_hole_record(b->magic))
- return -EINVAL;
- /* Setup the variables. */
- usa_ofs = le16_to_cpu(b->usa_ofs);
- /* Decrement usa_count to get number of fixups. */
- usa_count = le16_to_cpu(b->usa_count) - 1;
- /* Size and alignment checks. */
- if ( size & (NTFS_BLOCK_SIZE - 1) ||
- usa_ofs & 1 ||
- usa_ofs + (usa_count * 2) > size ||
- (size >> NTFS_BLOCK_SIZE_BITS) != usa_count)
- return -EINVAL;
- /* Position of usn in update sequence array. */
- usa_pos = (le16*)((u8*)b + usa_ofs);
- /*
- * Cyclically increment the update sequence number
- * (skipping 0 and -1, i.e. 0xffff).
- */
- usn = le16_to_cpup(usa_pos) + 1;
- if (usn == 0xffff || !usn)
- usn = 1;
- le_usn = cpu_to_le16(usn);
- *usa_pos = le_usn;
- /* Position in data of first u16 that needs fixing up. */
- data_pos = (le16*)b + NTFS_BLOCK_SIZE/sizeof(le16) - 1;
- /* Fixup all sectors. */
- while (usa_count--) {
- /*
- * Increment the position in the usa and save the
- * original data from the data buffer into the usa.
- */
- *(++usa_pos) = *data_pos;
- /* Apply fixup to data. */
- *data_pos = le_usn;
- /* Increment position in data as well. */
- data_pos += NTFS_BLOCK_SIZE/sizeof(le16);
- }
- return 0;
-}
-
-/**
- * post_write_mst_fixup - fast deprotect multi sector transfer protected data
- * @b: pointer to the data to deprotect
- *
- * Perform the necessary post write multi sector transfer fixup, not checking
- * for any errors, because we assume we have just used pre_write_mst_fixup(),
- * thus the data will be fine or we would never have gotten here.
- */
-void post_write_mst_fixup(NTFS_RECORD *b)
-{
- le16 *usa_pos, *data_pos;
-
- u16 usa_ofs = le16_to_cpu(b->usa_ofs);
- u16 usa_count = le16_to_cpu(b->usa_count) - 1;
-
- /* Position of usn in update sequence array. */
- usa_pos = (le16*)b + usa_ofs/sizeof(le16);
-
- /* Position in protected data of first u16 that needs fixing up. */
- data_pos = (le16*)b + NTFS_BLOCK_SIZE/sizeof(le16) - 1;
-
- /* Fixup all sectors. */
- while (usa_count--) {
- /*
- * Increment position in usa and restore original data from
- * the usa into the data buffer.
- */
- *data_pos = *(++usa_pos);
-
- /* Increment position in data as well. */
- data_pos += NTFS_BLOCK_SIZE/sizeof(le16);
- }
-}
diff --git a/fs/ntfs/namei.c b/fs/ntfs/namei.c
deleted file mode 100644
index 4e6a44bc654c..000000000000
--- a/fs/ntfs/namei.c
+++ /dev/null
@@ -1,391 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * namei.c - NTFS kernel directory inode operations. Part of the Linux-NTFS
- * project.
- *
- * Copyright (c) 2001-2006 Anton Altaparmakov
- */
-
-#include <linux/dcache.h>
-#include <linux/exportfs.h>
-#include <linux/security.h>
-#include <linux/slab.h>
-
-#include "attrib.h"
-#include "debug.h"
-#include "dir.h"
-#include "mft.h"
-#include "ntfs.h"
-
-/**
- * ntfs_lookup - find the inode represented by a dentry in a directory inode
- * @dir_ino: directory inode in which to look for the inode
- * @dent: dentry representing the inode to look for
- * @flags: lookup flags
- *
- * In short, ntfs_lookup() looks for the inode represented by the dentry @dent
- * in the directory inode @dir_ino and if found attaches the inode to the
- * dentry @dent.
- *
- * In more detail, the dentry @dent specifies which inode to look for by
- * supplying the name of the inode in @dent->d_name.name. ntfs_lookup()
- * converts the name to Unicode and walks the contents of the directory inode
- * @dir_ino looking for the converted Unicode name. If the name is found in the
- * directory, the corresponding inode is loaded by calling ntfs_iget() on its
- * inode number and the inode is associated with the dentry @dent via a call to
- * d_splice_alias().
- *
- * If the name is not found in the directory, a NULL inode is inserted into the
- * dentry @dent via a call to d_add(). The dentry is then termed a negative
- * dentry.
- *
- * Only if an actual error occurs, do we return an error via ERR_PTR().
- *
- * In order to handle the case insensitivity issues of NTFS with regards to the
- * dcache and the dcache requiring only one dentry per directory, we deal with
- * dentry aliases that only differ in case in ->ntfs_lookup() while maintaining
- * a case sensitive dcache. This means that we get the full benefit of dcache
- * speed when the file/directory is looked up with the same case as returned by
- * ->ntfs_readdir() but that a lookup for any other case (or for the short file
- * name) will not find anything in dcache and will enter ->ntfs_lookup()
- * instead, where we search the directory for a fully matching file name
- * (including case) and if that is not found, we search for a file name that
- * matches with different case and if that has non-POSIX semantics we return
- * that. We actually do only one search (case sensitive) and keep tabs on
- * whether we have found a case insensitive match in the process.
- *
- * To simplify matters for us, we do not treat the short vs long filenames as
- * two hard links but instead if the lookup matches a short filename, we
- * return the dentry for the corresponding long filename instead.
- *
- * There are three cases we need to distinguish here:
- *
- * 1) @dent perfectly matches (i.e. including case) a directory entry with a
- * file name in the WIN32 or POSIX namespaces. In this case
- * ntfs_lookup_inode_by_name() will return with name set to NULL and we
- * just d_splice_alias() @dent.
- * 2) @dent matches (not including case) a directory entry with a file name in
- * the WIN32 namespace. In this case ntfs_lookup_inode_by_name() will return
- * with name set to point to a kmalloc()ed ntfs_name structure containing
- * the properly cased little endian Unicode name. We convert the name to the
- * current NLS code page, search if a dentry with this name already exists
- * and if so return that instead of @dent. At this point things are
- * complicated by the possibility of 'disconnected' dentries due to NFS
- * which we deal with appropriately (see the code comments). The VFS will
- * then destroy the old @dent and use the one we returned. If a dentry is
- * not found, we allocate a new one, d_splice_alias() it, and return it as
- * above.
- * 3) @dent matches either perfectly or not (i.e. we don't care about case) a
- * directory entry with a file name in the DOS namespace. In this case
- * ntfs_lookup_inode_by_name() will return with name set to point to a
- * kmalloc()ed ntfs_name structure containing the mft reference (cpu endian)
- * of the inode. We use the mft reference to read the inode and to find the
- * file name in the WIN32 namespace corresponding to the matched short file
- * name. We then convert the name to the current NLS code page, and proceed
- * searching for a dentry with this name, etc, as in case 2), above.
- *
- * Locking: Caller must hold i_mutex on the directory.
- */
-static struct dentry *ntfs_lookup(struct inode *dir_ino, struct dentry *dent,
- unsigned int flags)
-{
- ntfs_volume *vol = NTFS_SB(dir_ino->i_sb);
- struct inode *dent_inode;
- ntfschar *uname;
- ntfs_name *name = NULL;
- MFT_REF mref;
- unsigned long dent_ino;
- int uname_len;
-
- ntfs_debug("Looking up %pd in directory inode 0x%lx.",
- dent, dir_ino->i_ino);
- /* Convert the name of the dentry to Unicode. */
- uname_len = ntfs_nlstoucs(vol, dent->d_name.name, dent->d_name.len,
- &uname);
- if (uname_len < 0) {
- if (uname_len != -ENAMETOOLONG)
- ntfs_error(vol->sb, "Failed to convert name to "
- "Unicode.");
- return ERR_PTR(uname_len);
- }
- mref = ntfs_lookup_inode_by_name(NTFS_I(dir_ino), uname, uname_len,
- &name);
- kmem_cache_free(ntfs_name_cache, uname);
- if (!IS_ERR_MREF(mref)) {
- dent_ino = MREF(mref);
- ntfs_debug("Found inode 0x%lx. Calling ntfs_iget.", dent_ino);
- dent_inode = ntfs_iget(vol->sb, dent_ino);
- if (!IS_ERR(dent_inode)) {
- /* Consistency check. */
- if (is_bad_inode(dent_inode) || MSEQNO(mref) ==
- NTFS_I(dent_inode)->seq_no ||
- dent_ino == FILE_MFT) {
- /* Perfect WIN32/POSIX match. -- Case 1. */
- if (!name) {
- ntfs_debug("Done. (Case 1.)");
- return d_splice_alias(dent_inode, dent);
- }
- /*
- * We are too indented. Handle imperfect
- * matches and short file names further below.
- */
- goto handle_name;
- }
- ntfs_error(vol->sb, "Found stale reference to inode "
- "0x%lx (reference sequence number = "
- "0x%x, inode sequence number = 0x%x), "
- "returning -EIO. Run chkdsk.",
- dent_ino, MSEQNO(mref),
- NTFS_I(dent_inode)->seq_no);
- iput(dent_inode);
- dent_inode = ERR_PTR(-EIO);
- } else
- ntfs_error(vol->sb, "ntfs_iget(0x%lx) failed with "
- "error code %li.", dent_ino,
- PTR_ERR(dent_inode));
- kfree(name);
- /* Return the error code. */
- return ERR_CAST(dent_inode);
- }
- /* It is guaranteed that @name is no longer allocated at this point. */
- if (MREF_ERR(mref) == -ENOENT) {
- ntfs_debug("Entry was not found, adding negative dentry.");
- /* The dcache will handle negative entries. */
- d_add(dent, NULL);
- ntfs_debug("Done.");
- return NULL;
- }
- ntfs_error(vol->sb, "ntfs_lookup_ino_by_name() failed with error "
- "code %i.", -MREF_ERR(mref));
- return ERR_PTR(MREF_ERR(mref));
- // TODO: Consider moving this lot to a separate function! (AIA)
-handle_name:
- {
- MFT_RECORD *m;
- ntfs_attr_search_ctx *ctx;
- ntfs_inode *ni = NTFS_I(dent_inode);
- int err;
- struct qstr nls_name;
-
- nls_name.name = NULL;
- if (name->type != FILE_NAME_DOS) { /* Case 2. */
- ntfs_debug("Case 2.");
- nls_name.len = (unsigned)ntfs_ucstonls(vol,
- (ntfschar*)&name->name, name->len,
- (unsigned char**)&nls_name.name, 0);
- kfree(name);
- } else /* if (name->type == FILE_NAME_DOS) */ { /* Case 3. */
- FILE_NAME_ATTR *fn;
-
- ntfs_debug("Case 3.");
- kfree(name);
-
- /* Find the WIN32 name corresponding to the matched DOS name. */
- ni = NTFS_I(dent_inode);
- m = map_mft_record(ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- m = NULL;
- ctx = NULL;
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(ni, m);
- if (unlikely(!ctx)) {
- err = -ENOMEM;
- goto err_out;
- }
- do {
- ATTR_RECORD *a;
- u32 val_len;
-
- err = ntfs_attr_lookup(AT_FILE_NAME, NULL, 0, 0, 0,
- NULL, 0, ctx);
- if (unlikely(err)) {
- ntfs_error(vol->sb, "Inode corrupt: No WIN32 "
- "namespace counterpart to DOS "
- "file name. Run chkdsk.");
- if (err == -ENOENT)
- err = -EIO;
- goto err_out;
- }
- /* Consistency checks. */
- a = ctx->attr;
- if (a->non_resident || a->flags)
- goto eio_err_out;
- val_len = le32_to_cpu(a->data.resident.value_length);
- if (le16_to_cpu(a->data.resident.value_offset) +
- val_len > le32_to_cpu(a->length))
- goto eio_err_out;
- fn = (FILE_NAME_ATTR*)((u8*)ctx->attr + le16_to_cpu(
- ctx->attr->data.resident.value_offset));
- if ((u32)(fn->file_name_length * sizeof(ntfschar) +
- sizeof(FILE_NAME_ATTR)) > val_len)
- goto eio_err_out;
- } while (fn->file_name_type != FILE_NAME_WIN32);
-
- /* Convert the found WIN32 name to current NLS code page. */
- nls_name.len = (unsigned)ntfs_ucstonls(vol,
- (ntfschar*)&fn->file_name, fn->file_name_length,
- (unsigned char**)&nls_name.name, 0);
-
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ni);
- }
- m = NULL;
- ctx = NULL;
-
- /* Check if a conversion error occurred. */
- if ((signed)nls_name.len < 0) {
- err = (signed)nls_name.len;
- goto err_out;
- }
- nls_name.hash = full_name_hash(dent, nls_name.name, nls_name.len);
-
- dent = d_add_ci(dent, dent_inode, &nls_name);
- kfree(nls_name.name);
- return dent;
-
-eio_err_out:
- ntfs_error(vol->sb, "Illegal file name attribute. Run chkdsk.");
- err = -EIO;
-err_out:
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- if (m)
- unmap_mft_record(ni);
- iput(dent_inode);
- ntfs_error(vol->sb, "Failed, returning error code %i.", err);
- return ERR_PTR(err);
- }
-}
-
-/**
- * Inode operations for directories.
- */
-const struct inode_operations ntfs_dir_inode_ops = {
- .lookup = ntfs_lookup, /* VFS: Lookup directory. */
-};
-
-/**
- * ntfs_get_parent - find the dentry of the parent of a given directory dentry
- * @child_dent: dentry of the directory whose parent directory to find
- *
- * Find the dentry for the parent directory of the directory specified by the
- * dentry @child_dent. This function is called from
- * fs/exportfs/expfs.c::find_exported_dentry() which in turn is called from the
- * default ->decode_fh() which is export_decode_fh() in the same file.
- *
- * The code is based on the ext3 ->get_parent() implementation found in
- * fs/ext3/namei.c::ext3_get_parent().
- *
- * Note: ntfs_get_parent() is called with @d_inode(child_dent)->i_mutex down.
- *
- * Return the dentry of the parent directory on success or the error code on
- * error (IS_ERR() is true).
- */
-static struct dentry *ntfs_get_parent(struct dentry *child_dent)
-{
- struct inode *vi = d_inode(child_dent);
- ntfs_inode *ni = NTFS_I(vi);
- MFT_RECORD *mrec;
- ntfs_attr_search_ctx *ctx;
- ATTR_RECORD *attr;
- FILE_NAME_ATTR *fn;
- unsigned long parent_ino;
- int err;
-
- ntfs_debug("Entering for inode 0x%lx.", vi->i_ino);
- /* Get the mft record of the inode belonging to the child dentry. */
- mrec = map_mft_record(ni);
- if (IS_ERR(mrec))
- return ERR_CAST(mrec);
- /* Find the first file name attribute in the mft record. */
- ctx = ntfs_attr_get_search_ctx(ni, mrec);
- if (unlikely(!ctx)) {
- unmap_mft_record(ni);
- return ERR_PTR(-ENOMEM);
- }
-try_next:
- err = ntfs_attr_lookup(AT_FILE_NAME, NULL, 0, CASE_SENSITIVE, 0, NULL,
- 0, ctx);
- if (unlikely(err)) {
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ni);
- if (err == -ENOENT)
- ntfs_error(vi->i_sb, "Inode 0x%lx does not have a "
- "file name attribute. Run chkdsk.",
- vi->i_ino);
- return ERR_PTR(err);
- }
- attr = ctx->attr;
- if (unlikely(attr->non_resident))
- goto try_next;
- fn = (FILE_NAME_ATTR *)((u8 *)attr +
- le16_to_cpu(attr->data.resident.value_offset));
- if (unlikely((u8 *)fn + le32_to_cpu(attr->data.resident.value_length) >
- (u8*)attr + le32_to_cpu(attr->length)))
- goto try_next;
- /* Get the inode number of the parent directory. */
- parent_ino = MREF_LE(fn->parent_directory);
- /* Release the search context and the mft record of the child. */
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ni);
-
- return d_obtain_alias(ntfs_iget(vi->i_sb, parent_ino));
-}
-
-static struct inode *ntfs_nfs_get_inode(struct super_block *sb,
- u64 ino, u32 generation)
-{
- struct inode *inode;
-
- inode = ntfs_iget(sb, ino);
- if (!IS_ERR(inode)) {
- if (is_bad_inode(inode) || inode->i_generation != generation) {
- iput(inode);
- inode = ERR_PTR(-ESTALE);
- }
- }
-
- return inode;
-}
-
-static struct dentry *ntfs_fh_to_dentry(struct super_block *sb, struct fid *fid,
- int fh_len, int fh_type)
-{
- return generic_fh_to_dentry(sb, fid, fh_len, fh_type,
- ntfs_nfs_get_inode);
-}
-
-static struct dentry *ntfs_fh_to_parent(struct super_block *sb, struct fid *fid,
- int fh_len, int fh_type)
-{
- return generic_fh_to_parent(sb, fid, fh_len, fh_type,
- ntfs_nfs_get_inode);
-}
-
-/**
- * Export operations allowing NFS exporting of mounted NTFS partitions.
- *
- * We use the default ->encode_fh() for now. Note that they
- * use 32 bits to store the inode number which is an unsigned long so on 64-bit
- * architectures is usually 64 bits so it would all fail horribly on huge
- * volumes. I guess we need to define our own encode and decode fh functions
- * that store 64-bit inode numbers at some point but for now we will ignore the
- * problem...
- *
- * We also use the default ->get_name() helper (used by ->decode_fh() via
- * fs/exportfs/expfs.c::find_exported_dentry()) as that is completely fs
- * independent.
- *
- * The default ->get_parent() just returns -EACCES so we have to provide our
- * own and the default ->get_dentry() is incompatible with NTFS due to not
- * allowing the inode number 0 which is used in NTFS for the system file $MFT
- * and due to using iget() whereas NTFS needs ntfs_iget().
- */
-const struct export_operations ntfs_export_ops = {
- .get_parent = ntfs_get_parent, /* Find the parent of a given
- directory. */
- .fh_to_dentry = ntfs_fh_to_dentry,
- .fh_to_parent = ntfs_fh_to_parent,
-};
diff --git a/fs/ntfs/ntfs.h b/fs/ntfs/ntfs.h
deleted file mode 100644
index e81376ea9152..000000000000
--- a/fs/ntfs/ntfs.h
+++ /dev/null
@@ -1,150 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * ntfs.h - Defines for NTFS Linux kernel driver.
- *
- * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc.
- * Copyright (C) 2002 Richard Russon
- */
-
-#ifndef _LINUX_NTFS_H
-#define _LINUX_NTFS_H
-
-#include <linux/stddef.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/compiler.h>
-#include <linux/fs.h>
-#include <linux/nls.h>
-#include <linux/smp.h>
-#include <linux/pagemap.h>
-
-#include "types.h"
-#include "volume.h"
-#include "layout.h"
-
-typedef enum {
- NTFS_BLOCK_SIZE = 512,
- NTFS_BLOCK_SIZE_BITS = 9,
- NTFS_SB_MAGIC = 0x5346544e, /* 'NTFS' */
- NTFS_MAX_NAME_LEN = 255,
- NTFS_MAX_ATTR_NAME_LEN = 255,
- NTFS_MAX_CLUSTER_SIZE = 64 * 1024, /* 64kiB */
- NTFS_MAX_PAGES_PER_CLUSTER = NTFS_MAX_CLUSTER_SIZE / PAGE_SIZE,
-} NTFS_CONSTANTS;
-
-/* Global variables. */
-
-/* Slab caches (from super.c). */
-extern struct kmem_cache *ntfs_name_cache;
-extern struct kmem_cache *ntfs_inode_cache;
-extern struct kmem_cache *ntfs_big_inode_cache;
-extern struct kmem_cache *ntfs_attr_ctx_cache;
-extern struct kmem_cache *ntfs_index_ctx_cache;
-
-/* The various operations structs defined throughout the driver files. */
-extern const struct address_space_operations ntfs_normal_aops;
-extern const struct address_space_operations ntfs_compressed_aops;
-extern const struct address_space_operations ntfs_mst_aops;
-
-extern const struct file_operations ntfs_file_ops;
-extern const struct inode_operations ntfs_file_inode_ops;
-
-extern const struct file_operations ntfs_dir_ops;
-extern const struct inode_operations ntfs_dir_inode_ops;
-
-extern const struct file_operations ntfs_empty_file_ops;
-extern const struct inode_operations ntfs_empty_inode_ops;
-
-extern const struct export_operations ntfs_export_ops;
-
-/**
- * NTFS_SB - return the ntfs volume given a vfs super block
- * @sb: VFS super block
- *
- * NTFS_SB() returns the ntfs volume associated with the VFS super block @sb.
- */
-static inline ntfs_volume *NTFS_SB(struct super_block *sb)
-{
- return sb->s_fs_info;
-}
-
-/* Declarations of functions and global variables. */
-
-/* From fs/ntfs/compress.c */
-extern int ntfs_read_compressed_block(struct page *page);
-extern int allocate_compression_buffers(void);
-extern void free_compression_buffers(void);
-
-/* From fs/ntfs/super.c */
-#define default_upcase_len 0x10000
-extern struct mutex ntfs_lock;
-
-typedef struct {
- int val;
- char *str;
-} option_t;
-extern const option_t on_errors_arr[];
-
-/* From fs/ntfs/mst.c */
-extern int post_read_mst_fixup(NTFS_RECORD *b, const u32 size);
-extern int pre_write_mst_fixup(NTFS_RECORD *b, const u32 size);
-extern void post_write_mst_fixup(NTFS_RECORD *b);
-
-/* From fs/ntfs/unistr.c */
-extern bool ntfs_are_names_equal(const ntfschar *s1, size_t s1_len,
- const ntfschar *s2, size_t s2_len,
- const IGNORE_CASE_BOOL ic,
- const ntfschar *upcase, const u32 upcase_size);
-extern int ntfs_collate_names(const ntfschar *name1, const u32 name1_len,
- const ntfschar *name2, const u32 name2_len,
- const int err_val, const IGNORE_CASE_BOOL ic,
- const ntfschar *upcase, const u32 upcase_len);
-extern int ntfs_ucsncmp(const ntfschar *s1, const ntfschar *s2, size_t n);
-extern int ntfs_ucsncasecmp(const ntfschar *s1, const ntfschar *s2, size_t n,
- const ntfschar *upcase, const u32 upcase_size);
-extern void ntfs_upcase_name(ntfschar *name, u32 name_len,
- const ntfschar *upcase, const u32 upcase_len);
-extern void ntfs_file_upcase_value(FILE_NAME_ATTR *file_name_attr,
- const ntfschar *upcase, const u32 upcase_len);
-extern int ntfs_file_compare_values(FILE_NAME_ATTR *file_name_attr1,
- FILE_NAME_ATTR *file_name_attr2,
- const int err_val, const IGNORE_CASE_BOOL ic,
- const ntfschar *upcase, const u32 upcase_len);
-extern int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins,
- const int ins_len, ntfschar **outs);
-extern int ntfs_ucstonls(const ntfs_volume *vol, const ntfschar *ins,
- const int ins_len, unsigned char **outs, int outs_len);
-
-/* From fs/ntfs/upcase.c */
-extern ntfschar *generate_default_upcase(void);
-
-static inline int ntfs_ffs(int x)
-{
- int r = 1;
-
- if (!x)
- return 0;
- if (!(x & 0xffff)) {
- x >>= 16;
- r += 16;
- }
- if (!(x & 0xff)) {
- x >>= 8;
- r += 8;
- }
- if (!(x & 0xf)) {
- x >>= 4;
- r += 4;
- }
- if (!(x & 3)) {
- x >>= 2;
- r += 2;
- }
- if (!(x & 1)) {
- x >>= 1;
- r += 1;
- }
- return r;
-}
-
-#endif /* _LINUX_NTFS_H */
diff --git a/fs/ntfs/quota.c b/fs/ntfs/quota.c
deleted file mode 100644
index 9160480222fd..000000000000
--- a/fs/ntfs/quota.c
+++ /dev/null
@@ -1,103 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * quota.c - NTFS kernel quota ($Quota) handling. Part of the Linux-NTFS
- * project.
- *
- * Copyright (c) 2004 Anton Altaparmakov
- */
-
-#ifdef NTFS_RW
-
-#include "index.h"
-#include "quota.h"
-#include "debug.h"
-#include "ntfs.h"
-
-/**
- * ntfs_mark_quotas_out_of_date - mark the quotas out of date on an ntfs volume
- * @vol: ntfs volume on which to mark the quotas out of date
- *
- * Mark the quotas out of date on the ntfs volume @vol and return 'true' on
- * success and 'false' on error.
- */
-bool ntfs_mark_quotas_out_of_date(ntfs_volume *vol)
-{
- ntfs_index_context *ictx;
- QUOTA_CONTROL_ENTRY *qce;
- const le32 qid = QUOTA_DEFAULTS_ID;
- int err;
-
- ntfs_debug("Entering.");
- if (NVolQuotaOutOfDate(vol))
- goto done;
- if (!vol->quota_ino || !vol->quota_q_ino) {
- ntfs_error(vol->sb, "Quota inodes are not open.");
- return false;
- }
- inode_lock(vol->quota_q_ino);
- ictx = ntfs_index_ctx_get(NTFS_I(vol->quota_q_ino));
- if (!ictx) {
- ntfs_error(vol->sb, "Failed to get index context.");
- goto err_out;
- }
- err = ntfs_index_lookup(&qid, sizeof(qid), ictx);
- if (err) {
- if (err == -ENOENT)
- ntfs_error(vol->sb, "Quota defaults entry is not "
- "present.");
- else
- ntfs_error(vol->sb, "Lookup of quota defaults entry "
- "failed.");
- goto err_out;
- }
- if (ictx->data_len < offsetof(QUOTA_CONTROL_ENTRY, sid)) {
- ntfs_error(vol->sb, "Quota defaults entry size is invalid. "
- "Run chkdsk.");
- goto err_out;
- }
- qce = (QUOTA_CONTROL_ENTRY*)ictx->data;
- if (le32_to_cpu(qce->version) != QUOTA_VERSION) {
- ntfs_error(vol->sb, "Quota defaults entry version 0x%x is not "
- "supported.", le32_to_cpu(qce->version));
- goto err_out;
- }
- ntfs_debug("Quota defaults flags = 0x%x.", le32_to_cpu(qce->flags));
- /* If quotas are already marked out of date, no need to do anything. */
- if (qce->flags & QUOTA_FLAG_OUT_OF_DATE)
- goto set_done;
- /*
- * If quota tracking is neither requested, nor enabled and there are no
- * pending deletes, no need to mark the quotas out of date.
- */
- if (!(qce->flags & (QUOTA_FLAG_TRACKING_ENABLED |
- QUOTA_FLAG_TRACKING_REQUESTED |
- QUOTA_FLAG_PENDING_DELETES)))
- goto set_done;
- /*
- * Set the QUOTA_FLAG_OUT_OF_DATE bit thus marking quotas out of date.
- * This is verified on WinXP to be sufficient to cause windows to
- * rescan the volume on boot and update all quota entries.
- */
- qce->flags |= QUOTA_FLAG_OUT_OF_DATE;
- /* Ensure the modified flags are written to disk. */
- ntfs_index_entry_flush_dcache_page(ictx);
- ntfs_index_entry_mark_dirty(ictx);
-set_done:
- ntfs_index_ctx_put(ictx);
- inode_unlock(vol->quota_q_ino);
- /*
- * We set the flag so we do not try to mark the quotas out of date
- * again on remount.
- */
- NVolSetQuotaOutOfDate(vol);
-done:
- ntfs_debug("Done.");
- return true;
-err_out:
- if (ictx)
- ntfs_index_ctx_put(ictx);
- inode_unlock(vol->quota_q_ino);
- return false;
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/quota.h b/fs/ntfs/quota.h
deleted file mode 100644
index fe3132a3d6d2..000000000000
--- a/fs/ntfs/quota.h
+++ /dev/null
@@ -1,21 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * quota.h - Defines for NTFS kernel quota ($Quota) handling. Part of the
- * Linux-NTFS project.
- *
- * Copyright (c) 2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_QUOTA_H
-#define _LINUX_NTFS_QUOTA_H
-
-#ifdef NTFS_RW
-
-#include "types.h"
-#include "volume.h"
-
-extern bool ntfs_mark_quotas_out_of_date(ntfs_volume *vol);
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_QUOTA_H */
diff --git a/fs/ntfs/runlist.c b/fs/ntfs/runlist.c
deleted file mode 100644
index 97932fb5179c..000000000000
--- a/fs/ntfs/runlist.c
+++ /dev/null
@@ -1,1893 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/**
- * runlist.c - NTFS runlist handling code. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2007 Anton Altaparmakov
- * Copyright (c) 2002-2005 Richard Russon
- */
-
-#include "debug.h"
-#include "dir.h"
-#include "endian.h"
-#include "malloc.h"
-#include "ntfs.h"
-
-/**
- * ntfs_rl_mm - runlist memmove
- *
- * It is up to the caller to serialize access to the runlist @base.
- */
-static inline void ntfs_rl_mm(runlist_element *base, int dst, int src,
- int size)
-{
- if (likely((dst != src) && (size > 0)))
- memmove(base + dst, base + src, size * sizeof(*base));
-}
-
-/**
- * ntfs_rl_mc - runlist memory copy
- *
- * It is up to the caller to serialize access to the runlists @dstbase and
- * @srcbase.
- */
-static inline void ntfs_rl_mc(runlist_element *dstbase, int dst,
- runlist_element *srcbase, int src, int size)
-{
- if (likely(size > 0))
- memcpy(dstbase + dst, srcbase + src, size * sizeof(*dstbase));
-}
-
-/**
- * ntfs_rl_realloc - Reallocate memory for runlists
- * @rl: original runlist
- * @old_size: number of runlist elements in the original runlist @rl
- * @new_size: number of runlist elements we need space for
- *
- * As the runlists grow, more memory will be required. To prevent the
- * kernel having to allocate and reallocate large numbers of small bits of
- * memory, this function returns an entire page of memory.
- *
- * It is up to the caller to serialize access to the runlist @rl.
- *
- * N.B. If the new allocation doesn't require a different number of pages in
- * memory, the function will return the original pointer.
- *
- * On success, return a pointer to the newly allocated, or recycled, memory.
- * On error, return -errno. The following error codes are defined:
- * -ENOMEM - Not enough memory to allocate runlist array.
- * -EINVAL - Invalid parameters were passed in.
- */
-static inline runlist_element *ntfs_rl_realloc(runlist_element *rl,
- int old_size, int new_size)
-{
- runlist_element *new_rl;
-
- old_size = PAGE_ALIGN(old_size * sizeof(*rl));
- new_size = PAGE_ALIGN(new_size * sizeof(*rl));
- if (old_size == new_size)
- return rl;
-
- new_rl = ntfs_malloc_nofs(new_size);
- if (unlikely(!new_rl))
- return ERR_PTR(-ENOMEM);
-
- if (likely(rl != NULL)) {
- if (unlikely(old_size > new_size))
- old_size = new_size;
- memcpy(new_rl, rl, old_size);
- ntfs_free(rl);
- }
- return new_rl;
-}
-
-/**
- * ntfs_rl_realloc_nofail - Reallocate memory for runlists
- * @rl: original runlist
- * @old_size: number of runlist elements in the original runlist @rl
- * @new_size: number of runlist elements we need space for
- *
- * As the runlists grow, more memory will be required. To prevent the
- * kernel having to allocate and reallocate large numbers of small bits of
- * memory, this function returns an entire page of memory.
- *
- * This function guarantees that the allocation will succeed. It will sleep
- * for as long as it takes to complete the allocation.
- *
- * It is up to the caller to serialize access to the runlist @rl.
- *
- * N.B. If the new allocation doesn't require a different number of pages in
- * memory, the function will return the original pointer.
- *
- * On success, return a pointer to the newly allocated, or recycled, memory.
- * On error, return -errno. The following error codes are defined:
- * -ENOMEM - Not enough memory to allocate runlist array.
- * -EINVAL - Invalid parameters were passed in.
- */
-static inline runlist_element *ntfs_rl_realloc_nofail(runlist_element *rl,
- int old_size, int new_size)
-{
- runlist_element *new_rl;
-
- old_size = PAGE_ALIGN(old_size * sizeof(*rl));
- new_size = PAGE_ALIGN(new_size * sizeof(*rl));
- if (old_size == new_size)
- return rl;
-
- new_rl = ntfs_malloc_nofs_nofail(new_size);
- BUG_ON(!new_rl);
-
- if (likely(rl != NULL)) {
- if (unlikely(old_size > new_size))
- old_size = new_size;
- memcpy(new_rl, rl, old_size);
- ntfs_free(rl);
- }
- return new_rl;
-}
-
-/**
- * ntfs_are_rl_mergeable - test if two runlists can be joined together
- * @dst: original runlist
- * @src: new runlist to test for mergeability with @dst
- *
- * Test if two runlists can be joined together. For this, their VCNs and LCNs
- * must be adjacent.
- *
- * It is up to the caller to serialize access to the runlists @dst and @src.
- *
- * Return: true Success, the runlists can be merged.
- * false Failure, the runlists cannot be merged.
- */
-static inline bool ntfs_are_rl_mergeable(runlist_element *dst,
- runlist_element *src)
-{
- BUG_ON(!dst);
- BUG_ON(!src);
-
- /* We can merge unmapped regions even if they are misaligned. */
- if ((dst->lcn == LCN_RL_NOT_MAPPED) && (src->lcn == LCN_RL_NOT_MAPPED))
- return true;
- /* If the runs are misaligned, we cannot merge them. */
- if ((dst->vcn + dst->length) != src->vcn)
- return false;
- /* If both runs are non-sparse and contiguous, we can merge them. */
- if ((dst->lcn >= 0) && (src->lcn >= 0) &&
- ((dst->lcn + dst->length) == src->lcn))
- return true;
- /* If we are merging two holes, we can merge them. */
- if ((dst->lcn == LCN_HOLE) && (src->lcn == LCN_HOLE))
- return true;
- /* Cannot merge. */
- return false;
-}
-
-/**
- * __ntfs_rl_merge - merge two runlists without testing if they can be merged
- * @dst: original, destination runlist
- * @src: new runlist to merge with @dst
- *
- * Merge the two runlists, writing into the destination runlist @dst. The
- * caller must make sure the runlists can be merged or this will corrupt the
- * destination runlist.
- *
- * It is up to the caller to serialize access to the runlists @dst and @src.
- */
-static inline void __ntfs_rl_merge(runlist_element *dst, runlist_element *src)
-{
- dst->length += src->length;
-}
-
-/**
- * ntfs_rl_append - append a runlist after a given element
- * @dst: original runlist to be worked on
- * @dsize: number of elements in @dst (including end marker)
- * @src: runlist to be inserted into @dst
- * @ssize: number of elements in @src (excluding end marker)
- * @loc: append the new runlist @src after this element in @dst
- *
- * Append the runlist @src after element @loc in @dst. Merge the right end of
- * the new runlist, if necessary. Adjust the size of the hole before the
- * appended runlist.
- *
- * It is up to the caller to serialize access to the runlists @dst and @src.
- *
- * On success, return a pointer to the new, combined, runlist. Note, both
- * runlists @dst and @src are deallocated before returning so you cannot use
- * the pointers for anything any more. (Strictly speaking the returned runlist
- * may be the same as @dst but this is irrelevant.)
- *
- * On error, return -errno. Both runlists are left unmodified. The following
- * error codes are defined:
- * -ENOMEM - Not enough memory to allocate runlist array.
- * -EINVAL - Invalid parameters were passed in.
- */
-static inline runlist_element *ntfs_rl_append(runlist_element *dst,
- int dsize, runlist_element *src, int ssize, int loc)
-{
- bool right = false; /* Right end of @src needs merging. */
- int marker; /* End of the inserted runs. */
-
- BUG_ON(!dst);
- BUG_ON(!src);
-
- /* First, check if the right hand end needs merging. */
- if ((loc + 1) < dsize)
- right = ntfs_are_rl_mergeable(src + ssize - 1, dst + loc + 1);
-
- /* Space required: @dst size + @src size, less one if we merged. */
- dst = ntfs_rl_realloc(dst, dsize, dsize + ssize - right);
- if (IS_ERR(dst))
- return dst;
- /*
- * We are guaranteed to succeed from here so can start modifying the
- * original runlists.
- */
-
- /* First, merge the right hand end, if necessary. */
- if (right)
- __ntfs_rl_merge(src + ssize - 1, dst + loc + 1);
-
- /* First run after the @src runs that have been inserted. */
- marker = loc + ssize + 1;
-
- /* Move the tail of @dst out of the way, then copy in @src. */
- ntfs_rl_mm(dst, marker, loc + 1 + right, dsize - (loc + 1 + right));
- ntfs_rl_mc(dst, loc + 1, src, 0, ssize);
-
- /* Adjust the size of the preceding hole. */
- dst[loc].length = dst[loc + 1].vcn - dst[loc].vcn;
-
- /* We may have changed the length of the file, so fix the end marker */
- if (dst[marker].lcn == LCN_ENOENT)
- dst[marker].vcn = dst[marker - 1].vcn + dst[marker - 1].length;
-
- return dst;
-}
-
-/**
- * ntfs_rl_insert - insert a runlist into another
- * @dst: original runlist to be worked on
- * @dsize: number of elements in @dst (including end marker)
- * @src: new runlist to be inserted
- * @ssize: number of elements in @src (excluding end marker)
- * @loc: insert the new runlist @src before this element in @dst
- *
- * Insert the runlist @src before element @loc in the runlist @dst. Merge the
- * left end of the new runlist, if necessary. Adjust the size of the hole
- * after the inserted runlist.
- *
- * It is up to the caller to serialize access to the runlists @dst and @src.
- *
- * On success, return a pointer to the new, combined, runlist. Note, both
- * runlists @dst and @src are deallocated before returning so you cannot use
- * the pointers for anything any more. (Strictly speaking the returned runlist
- * may be the same as @dst but this is irrelevant.)
- *
- * On error, return -errno. Both runlists are left unmodified. The following
- * error codes are defined:
- * -ENOMEM - Not enough memory to allocate runlist array.
- * -EINVAL - Invalid parameters were passed in.
- */
-static inline runlist_element *ntfs_rl_insert(runlist_element *dst,
- int dsize, runlist_element *src, int ssize, int loc)
-{
- bool left = false; /* Left end of @src needs merging. */
- bool disc = false; /* Discontinuity between @dst and @src. */
- int marker; /* End of the inserted runs. */
-
- BUG_ON(!dst);
- BUG_ON(!src);
-
- /*
- * disc => Discontinuity between the end of @dst and the start of @src.
- * This means we might need to insert a "not mapped" run.
- */
- if (loc == 0)
- disc = (src[0].vcn > 0);
- else {
- s64 merged_length;
-
- left = ntfs_are_rl_mergeable(dst + loc - 1, src);
-
- merged_length = dst[loc - 1].length;
- if (left)
- merged_length += src->length;
-
- disc = (src[0].vcn > dst[loc - 1].vcn + merged_length);
- }
- /*
- * Space required: @dst size + @src size, less one if we merged, plus
- * one if there was a discontinuity.
- */
- dst = ntfs_rl_realloc(dst, dsize, dsize + ssize - left + disc);
- if (IS_ERR(dst))
- return dst;
- /*
- * We are guaranteed to succeed from here so can start modifying the
- * original runlist.
- */
- if (left)
- __ntfs_rl_merge(dst + loc - 1, src);
- /*
- * First run after the @src runs that have been inserted.
- * Nominally, @marker equals @loc + @ssize, i.e. location + number of
- * runs in @src. However, if @left, then the first run in @src has
- * been merged with one in @dst. And if @disc, then @dst and @src do
- * not meet and we need an extra run to fill the gap.
- */
- marker = loc + ssize - left + disc;
-
- /* Move the tail of @dst out of the way, then copy in @src. */
- ntfs_rl_mm(dst, marker, loc, dsize - loc);
- ntfs_rl_mc(dst, loc + disc, src, left, ssize - left);
-
- /* Adjust the VCN of the first run after the insertion... */
- dst[marker].vcn = dst[marker - 1].vcn + dst[marker - 1].length;
- /* ... and the length. */
- if (dst[marker].lcn == LCN_HOLE || dst[marker].lcn == LCN_RL_NOT_MAPPED)
- dst[marker].length = dst[marker + 1].vcn - dst[marker].vcn;
-
- /* Writing beyond the end of the file and there is a discontinuity. */
- if (disc) {
- if (loc > 0) {
- dst[loc].vcn = dst[loc - 1].vcn + dst[loc - 1].length;
- dst[loc].length = dst[loc + 1].vcn - dst[loc].vcn;
- } else {
- dst[loc].vcn = 0;
- dst[loc].length = dst[loc + 1].vcn;
- }
- dst[loc].lcn = LCN_RL_NOT_MAPPED;
- }
- return dst;
-}
-
-/**
- * ntfs_rl_replace - overwrite a runlist element with another runlist
- * @dst: original runlist to be worked on
- * @dsize: number of elements in @dst (including end marker)
- * @src: new runlist to be inserted
- * @ssize: number of elements in @src (excluding end marker)
- * @loc: index in runlist @dst to overwrite with @src
- *
- * Replace the runlist element @dst at @loc with @src. Merge the left and
- * right ends of the inserted runlist, if necessary.
- *
- * It is up to the caller to serialize access to the runlists @dst and @src.
- *
- * On success, return a pointer to the new, combined, runlist. Note, both
- * runlists @dst and @src are deallocated before returning so you cannot use
- * the pointers for anything any more. (Strictly speaking the returned runlist
- * may be the same as @dst but this is irrelevant.)
- *
- * On error, return -errno. Both runlists are left unmodified. The following
- * error codes are defined:
- * -ENOMEM - Not enough memory to allocate runlist array.
- * -EINVAL - Invalid parameters were passed in.
- */
-static inline runlist_element *ntfs_rl_replace(runlist_element *dst,
- int dsize, runlist_element *src, int ssize, int loc)
-{
- signed delta;
- bool left = false; /* Left end of @src needs merging. */
- bool right = false; /* Right end of @src needs merging. */
- int tail; /* Start of tail of @dst. */
- int marker; /* End of the inserted runs. */
-
- BUG_ON(!dst);
- BUG_ON(!src);
-
- /* First, see if the left and right ends need merging. */
- if ((loc + 1) < dsize)
- right = ntfs_are_rl_mergeable(src + ssize - 1, dst + loc + 1);
- if (loc > 0)
- left = ntfs_are_rl_mergeable(dst + loc - 1, src);
- /*
- * Allocate some space. We will need less if the left, right, or both
- * ends get merged. The -1 accounts for the run being replaced.
- */
- delta = ssize - 1 - left - right;
- if (delta > 0) {
- dst = ntfs_rl_realloc(dst, dsize, dsize + delta);
- if (IS_ERR(dst))
- return dst;
- }
- /*
- * We are guaranteed to succeed from here so can start modifying the
- * original runlists.
- */
-
- /* First, merge the left and right ends, if necessary. */
- if (right)
- __ntfs_rl_merge(src + ssize - 1, dst + loc + 1);
- if (left)
- __ntfs_rl_merge(dst + loc - 1, src);
- /*
- * Offset of the tail of @dst. This needs to be moved out of the way
- * to make space for the runs to be copied from @src, i.e. the first
- * run of the tail of @dst.
- * Nominally, @tail equals @loc + 1, i.e. location, skipping the
- * replaced run. However, if @right, then one of @dst's runs is
- * already merged into @src.
- */
- tail = loc + right + 1;
- /*
- * First run after the @src runs that have been inserted, i.e. where
- * the tail of @dst needs to be moved to.
- * Nominally, @marker equals @loc + @ssize, i.e. location + number of
- * runs in @src. However, if @left, then the first run in @src has
- * been merged with one in @dst.
- */
- marker = loc + ssize - left;
-
- /* Move the tail of @dst out of the way, then copy in @src. */
- ntfs_rl_mm(dst, marker, tail, dsize - tail);
- ntfs_rl_mc(dst, loc, src, left, ssize - left);
-
- /* We may have changed the length of the file, so fix the end marker. */
- if (dsize - tail > 0 && dst[marker].lcn == LCN_ENOENT)
- dst[marker].vcn = dst[marker - 1].vcn + dst[marker - 1].length;
- return dst;
-}
-
-/**
- * ntfs_rl_split - insert a runlist into the centre of a hole
- * @dst: original runlist to be worked on
- * @dsize: number of elements in @dst (including end marker)
- * @src: new runlist to be inserted
- * @ssize: number of elements in @src (excluding end marker)
- * @loc: index in runlist @dst at which to split and insert @src
- *
- * Split the runlist @dst at @loc into two and insert @new in between the two
- * fragments. No merging of runlists is necessary. Adjust the size of the
- * holes either side.
- *
- * It is up to the caller to serialize access to the runlists @dst and @src.
- *
- * On success, return a pointer to the new, combined, runlist. Note, both
- * runlists @dst and @src are deallocated before returning so you cannot use
- * the pointers for anything any more. (Strictly speaking the returned runlist
- * may be the same as @dst but this is irrelevant.)
- *
- * On error, return -errno. Both runlists are left unmodified. The following
- * error codes are defined:
- * -ENOMEM - Not enough memory to allocate runlist array.
- * -EINVAL - Invalid parameters were passed in.
- */
-static inline runlist_element *ntfs_rl_split(runlist_element *dst, int dsize,
- runlist_element *src, int ssize, int loc)
-{
- BUG_ON(!dst);
- BUG_ON(!src);
-
- /* Space required: @dst size + @src size + one new hole. */
- dst = ntfs_rl_realloc(dst, dsize, dsize + ssize + 1);
- if (IS_ERR(dst))
- return dst;
- /*
- * We are guaranteed to succeed from here so can start modifying the
- * original runlists.
- */
-
- /* Move the tail of @dst out of the way, then copy in @src. */
- ntfs_rl_mm(dst, loc + 1 + ssize, loc, dsize - loc);
- ntfs_rl_mc(dst, loc + 1, src, 0, ssize);
-
- /* Adjust the size of the holes either size of @src. */
- dst[loc].length = dst[loc+1].vcn - dst[loc].vcn;
- dst[loc+ssize+1].vcn = dst[loc+ssize].vcn + dst[loc+ssize].length;
- dst[loc+ssize+1].length = dst[loc+ssize+2].vcn - dst[loc+ssize+1].vcn;
-
- return dst;
-}
-
-/**
- * ntfs_runlists_merge - merge two runlists into one
- * @drl: original runlist to be worked on
- * @srl: new runlist to be merged into @drl
- *
- * First we sanity check the two runlists @srl and @drl to make sure that they
- * are sensible and can be merged. The runlist @srl must be either after the
- * runlist @drl or completely within a hole (or unmapped region) in @drl.
- *
- * It is up to the caller to serialize access to the runlists @drl and @srl.
- *
- * Merging of runlists is necessary in two cases:
- * 1. When attribute lists are used and a further extent is being mapped.
- * 2. When new clusters are allocated to fill a hole or extend a file.
- *
- * There are four possible ways @srl can be merged. It can:
- * - be inserted at the beginning of a hole,
- * - split the hole in two and be inserted between the two fragments,
- * - be appended at the end of a hole, or it can
- * - replace the whole hole.
- * It can also be appended to the end of the runlist, which is just a variant
- * of the insert case.
- *
- * On success, return a pointer to the new, combined, runlist. Note, both
- * runlists @drl and @srl are deallocated before returning so you cannot use
- * the pointers for anything any more. (Strictly speaking the returned runlist
- * may be the same as @dst but this is irrelevant.)
- *
- * On error, return -errno. Both runlists are left unmodified. The following
- * error codes are defined:
- * -ENOMEM - Not enough memory to allocate runlist array.
- * -EINVAL - Invalid parameters were passed in.
- * -ERANGE - The runlists overlap and cannot be merged.
- */
-runlist_element *ntfs_runlists_merge(runlist_element *drl,
- runlist_element *srl)
-{
- int di, si; /* Current index into @[ds]rl. */
- int sstart; /* First index with lcn > LCN_RL_NOT_MAPPED. */
- int dins; /* Index into @drl at which to insert @srl. */
- int dend, send; /* Last index into @[ds]rl. */
- int dfinal, sfinal; /* The last index into @[ds]rl with
- lcn >= LCN_HOLE. */
- int marker = 0;
- VCN marker_vcn = 0;
-
-#ifdef DEBUG
- ntfs_debug("dst:");
- ntfs_debug_dump_runlist(drl);
- ntfs_debug("src:");
- ntfs_debug_dump_runlist(srl);
-#endif
-
- /* Check for silly calling... */
- if (unlikely(!srl))
- return drl;
- if (IS_ERR(srl) || IS_ERR(drl))
- return ERR_PTR(-EINVAL);
-
- /* Check for the case where the first mapping is being done now. */
- if (unlikely(!drl)) {
- drl = srl;
- /* Complete the source runlist if necessary. */
- if (unlikely(drl[0].vcn)) {
- /* Scan to the end of the source runlist. */
- for (dend = 0; likely(drl[dend].length); dend++)
- ;
- dend++;
- drl = ntfs_rl_realloc(drl, dend, dend + 1);
- if (IS_ERR(drl))
- return drl;
- /* Insert start element at the front of the runlist. */
- ntfs_rl_mm(drl, 1, 0, dend);
- drl[0].vcn = 0;
- drl[0].lcn = LCN_RL_NOT_MAPPED;
- drl[0].length = drl[1].vcn;
- }
- goto finished;
- }
-
- si = di = 0;
-
- /* Skip any unmapped start element(s) in the source runlist. */
- while (srl[si].length && srl[si].lcn < LCN_HOLE)
- si++;
-
- /* Can't have an entirely unmapped source runlist. */
- BUG_ON(!srl[si].length);
-
- /* Record the starting points. */
- sstart = si;
-
- /*
- * Skip forward in @drl until we reach the position where @srl needs to
- * be inserted. If we reach the end of @drl, @srl just needs to be
- * appended to @drl.
- */
- for (; drl[di].length; di++) {
- if (drl[di].vcn + drl[di].length > srl[sstart].vcn)
- break;
- }
- dins = di;
-
- /* Sanity check for illegal overlaps. */
- if ((drl[di].vcn == srl[si].vcn) && (drl[di].lcn >= 0) &&
- (srl[si].lcn >= 0)) {
- ntfs_error(NULL, "Run lists overlap. Cannot merge!");
- return ERR_PTR(-ERANGE);
- }
-
- /* Scan to the end of both runlists in order to know their sizes. */
- for (send = si; srl[send].length; send++)
- ;
- for (dend = di; drl[dend].length; dend++)
- ;
-
- if (srl[send].lcn == LCN_ENOENT)
- marker_vcn = srl[marker = send].vcn;
-
- /* Scan to the last element with lcn >= LCN_HOLE. */
- for (sfinal = send; sfinal >= 0 && srl[sfinal].lcn < LCN_HOLE; sfinal--)
- ;
- for (dfinal = dend; dfinal >= 0 && drl[dfinal].lcn < LCN_HOLE; dfinal--)
- ;
-
- {
- bool start;
- bool finish;
- int ds = dend + 1; /* Number of elements in drl & srl */
- int ss = sfinal - sstart + 1;
-
- start = ((drl[dins].lcn < LCN_RL_NOT_MAPPED) || /* End of file */
- (drl[dins].vcn == srl[sstart].vcn)); /* Start of hole */
- finish = ((drl[dins].lcn >= LCN_RL_NOT_MAPPED) && /* End of file */
- ((drl[dins].vcn + drl[dins].length) <= /* End of hole */
- (srl[send - 1].vcn + srl[send - 1].length)));
-
- /* Or we will lose an end marker. */
- if (finish && !drl[dins].length)
- ss++;
- if (marker && (drl[dins].vcn + drl[dins].length > srl[send - 1].vcn))
- finish = false;
-#if 0
- ntfs_debug("dfinal = %i, dend = %i", dfinal, dend);
- ntfs_debug("sstart = %i, sfinal = %i, send = %i", sstart, sfinal, send);
- ntfs_debug("start = %i, finish = %i", start, finish);
- ntfs_debug("ds = %i, ss = %i, dins = %i", ds, ss, dins);
-#endif
- if (start) {
- if (finish)
- drl = ntfs_rl_replace(drl, ds, srl + sstart, ss, dins);
- else
- drl = ntfs_rl_insert(drl, ds, srl + sstart, ss, dins);
- } else {
- if (finish)
- drl = ntfs_rl_append(drl, ds, srl + sstart, ss, dins);
- else
- drl = ntfs_rl_split(drl, ds, srl + sstart, ss, dins);
- }
- if (IS_ERR(drl)) {
- ntfs_error(NULL, "Merge failed.");
- return drl;
- }
- ntfs_free(srl);
- if (marker) {
- ntfs_debug("Triggering marker code.");
- for (ds = dend; drl[ds].length; ds++)
- ;
- /* We only need to care if @srl ended after @drl. */
- if (drl[ds].vcn <= marker_vcn) {
- int slots = 0;
-
- if (drl[ds].vcn == marker_vcn) {
- ntfs_debug("Old marker = 0x%llx, replacing "
- "with LCN_ENOENT.",
- (unsigned long long)
- drl[ds].lcn);
- drl[ds].lcn = LCN_ENOENT;
- goto finished;
- }
- /*
- * We need to create an unmapped runlist element in
- * @drl or extend an existing one before adding the
- * ENOENT terminator.
- */
- if (drl[ds].lcn == LCN_ENOENT) {
- ds--;
- slots = 1;
- }
- if (drl[ds].lcn != LCN_RL_NOT_MAPPED) {
- /* Add an unmapped runlist element. */
- if (!slots) {
- drl = ntfs_rl_realloc_nofail(drl, ds,
- ds + 2);
- slots = 2;
- }
- ds++;
- /* Need to set vcn if it isn't set already. */
- if (slots != 1)
- drl[ds].vcn = drl[ds - 1].vcn +
- drl[ds - 1].length;
- drl[ds].lcn = LCN_RL_NOT_MAPPED;
- /* We now used up a slot. */
- slots--;
- }
- drl[ds].length = marker_vcn - drl[ds].vcn;
- /* Finally add the ENOENT terminator. */
- ds++;
- if (!slots)
- drl = ntfs_rl_realloc_nofail(drl, ds, ds + 1);
- drl[ds].vcn = marker_vcn;
- drl[ds].lcn = LCN_ENOENT;
- drl[ds].length = (s64)0;
- }
- }
- }
-
-finished:
- /* The merge was completed successfully. */
- ntfs_debug("Merged runlist:");
- ntfs_debug_dump_runlist(drl);
- return drl;
-}
-
-/**
- * ntfs_mapping_pairs_decompress - convert mapping pairs array to runlist
- * @vol: ntfs volume on which the attribute resides
- * @attr: attribute record whose mapping pairs array to decompress
- * @old_rl: optional runlist in which to insert @attr's runlist
- *
- * It is up to the caller to serialize access to the runlist @old_rl.
- *
- * Decompress the attribute @attr's mapping pairs array into a runlist. On
- * success, return the decompressed runlist.
- *
- * If @old_rl is not NULL, decompressed runlist is inserted into the
- * appropriate place in @old_rl and the resultant, combined runlist is
- * returned. The original @old_rl is deallocated.
- *
- * On error, return -errno. @old_rl is left unmodified in that case.
- *
- * The following error codes are defined:
- * -ENOMEM - Not enough memory to allocate runlist array.
- * -EIO - Corrupt runlist.
- * -EINVAL - Invalid parameters were passed in.
- * -ERANGE - The two runlists overlap.
- *
- * FIXME: For now we take the conceptionally simplest approach of creating the
- * new runlist disregarding the already existing one and then splicing the
- * two into one, if that is possible (we check for overlap and discard the new
- * runlist if overlap present before returning ERR_PTR(-ERANGE)).
- */
-runlist_element *ntfs_mapping_pairs_decompress(const ntfs_volume *vol,
- const ATTR_RECORD *attr, runlist_element *old_rl)
-{
- VCN vcn; /* Current vcn. */
- LCN lcn; /* Current lcn. */
- s64 deltaxcn; /* Change in [vl]cn. */
- runlist_element *rl; /* The output runlist. */
- u8 *buf; /* Current position in mapping pairs array. */
- u8 *attr_end; /* End of attribute. */
- int rlsize; /* Size of runlist buffer. */
- u16 rlpos; /* Current runlist position in units of
- runlist_elements. */
- u8 b; /* Current byte offset in buf. */
-
-#ifdef DEBUG
- /* Make sure attr exists and is non-resident. */
- if (!attr || !attr->non_resident || sle64_to_cpu(
- attr->data.non_resident.lowest_vcn) < (VCN)0) {
- ntfs_error(vol->sb, "Invalid arguments.");
- return ERR_PTR(-EINVAL);
- }
-#endif
- /* Start at vcn = lowest_vcn and lcn 0. */
- vcn = sle64_to_cpu(attr->data.non_resident.lowest_vcn);
- lcn = 0;
- /* Get start of the mapping pairs array. */
- buf = (u8*)attr + le16_to_cpu(
- attr->data.non_resident.mapping_pairs_offset);
- attr_end = (u8*)attr + le32_to_cpu(attr->length);
- if (unlikely(buf < (u8*)attr || buf > attr_end)) {
- ntfs_error(vol->sb, "Corrupt attribute.");
- return ERR_PTR(-EIO);
- }
- /* If the mapping pairs array is valid but empty, nothing to do. */
- if (!vcn && !*buf)
- return old_rl;
- /* Current position in runlist array. */
- rlpos = 0;
- /* Allocate first page and set current runlist size to one page. */
- rl = ntfs_malloc_nofs(rlsize = PAGE_SIZE);
- if (unlikely(!rl))
- return ERR_PTR(-ENOMEM);
- /* Insert unmapped starting element if necessary. */
- if (vcn) {
- rl->vcn = 0;
- rl->lcn = LCN_RL_NOT_MAPPED;
- rl->length = vcn;
- rlpos++;
- }
- while (buf < attr_end && *buf) {
- /*
- * Allocate more memory if needed, including space for the
- * not-mapped and terminator elements. ntfs_malloc_nofs()
- * operates on whole pages only.
- */
- if (((rlpos + 3) * sizeof(*old_rl)) > rlsize) {
- runlist_element *rl2;
-
- rl2 = ntfs_malloc_nofs(rlsize + (int)PAGE_SIZE);
- if (unlikely(!rl2)) {
- ntfs_free(rl);
- return ERR_PTR(-ENOMEM);
- }
- memcpy(rl2, rl, rlsize);
- ntfs_free(rl);
- rl = rl2;
- rlsize += PAGE_SIZE;
- }
- /* Enter the current vcn into the current runlist element. */
- rl[rlpos].vcn = vcn;
- /*
- * Get the change in vcn, i.e. the run length in clusters.
- * Doing it this way ensures that we signextend negative values.
- * A negative run length doesn't make any sense, but hey, I
- * didn't make up the NTFS specs and Windows NT4 treats the run
- * length as a signed value so that's how it is...
- */
- b = *buf & 0xf;
- if (b) {
- if (unlikely(buf + b > attr_end))
- goto io_error;
- for (deltaxcn = (s8)buf[b--]; b; b--)
- deltaxcn = (deltaxcn << 8) + buf[b];
- } else { /* The length entry is compulsory. */
- ntfs_error(vol->sb, "Missing length entry in mapping "
- "pairs array.");
- deltaxcn = (s64)-1;
- }
- /*
- * Assume a negative length to indicate data corruption and
- * hence clean-up and return NULL.
- */
- if (unlikely(deltaxcn < 0)) {
- ntfs_error(vol->sb, "Invalid length in mapping pairs "
- "array.");
- goto err_out;
- }
- /*
- * Enter the current run length into the current runlist
- * element.
- */
- rl[rlpos].length = deltaxcn;
- /* Increment the current vcn by the current run length. */
- vcn += deltaxcn;
- /*
- * There might be no lcn change at all, as is the case for
- * sparse clusters on NTFS 3.0+, in which case we set the lcn
- * to LCN_HOLE.
- */
- if (!(*buf & 0xf0))
- rl[rlpos].lcn = LCN_HOLE;
- else {
- /* Get the lcn change which really can be negative. */
- u8 b2 = *buf & 0xf;
- b = b2 + ((*buf >> 4) & 0xf);
- if (buf + b > attr_end)
- goto io_error;
- for (deltaxcn = (s8)buf[b--]; b > b2; b--)
- deltaxcn = (deltaxcn << 8) + buf[b];
- /* Change the current lcn to its new value. */
- lcn += deltaxcn;
-#ifdef DEBUG
- /*
- * On NTFS 1.2-, apparently can have lcn == -1 to
- * indicate a hole. But we haven't verified ourselves
- * whether it is really the lcn or the deltaxcn that is
- * -1. So if either is found give us a message so we
- * can investigate it further!
- */
- if (vol->major_ver < 3) {
- if (unlikely(deltaxcn == (LCN)-1))
- ntfs_error(vol->sb, "lcn delta == -1");
- if (unlikely(lcn == (LCN)-1))
- ntfs_error(vol->sb, "lcn == -1");
- }
-#endif
- /* Check lcn is not below -1. */
- if (unlikely(lcn < (LCN)-1)) {
- ntfs_error(vol->sb, "Invalid LCN < -1 in "
- "mapping pairs array.");
- goto err_out;
- }
- /* Enter the current lcn into the runlist element. */
- rl[rlpos].lcn = lcn;
- }
- /* Get to the next runlist element. */
- rlpos++;
- /* Increment the buffer position to the next mapping pair. */
- buf += (*buf & 0xf) + ((*buf >> 4) & 0xf) + 1;
- }
- if (unlikely(buf >= attr_end))
- goto io_error;
- /*
- * If there is a highest_vcn specified, it must be equal to the final
- * vcn in the runlist - 1, or something has gone badly wrong.
- */
- deltaxcn = sle64_to_cpu(attr->data.non_resident.highest_vcn);
- if (unlikely(deltaxcn && vcn - 1 != deltaxcn)) {
-mpa_err:
- ntfs_error(vol->sb, "Corrupt mapping pairs array in "
- "non-resident attribute.");
- goto err_out;
- }
- /* Setup not mapped runlist element if this is the base extent. */
- if (!attr->data.non_resident.lowest_vcn) {
- VCN max_cluster;
-
- max_cluster = ((sle64_to_cpu(
- attr->data.non_resident.allocated_size) +
- vol->cluster_size - 1) >>
- vol->cluster_size_bits) - 1;
- /*
- * A highest_vcn of zero means this is a single extent
- * attribute so simply terminate the runlist with LCN_ENOENT).
- */
- if (deltaxcn) {
- /*
- * If there is a difference between the highest_vcn and
- * the highest cluster, the runlist is either corrupt
- * or, more likely, there are more extents following
- * this one.
- */
- if (deltaxcn < max_cluster) {
- ntfs_debug("More extents to follow; deltaxcn "
- "= 0x%llx, max_cluster = "
- "0x%llx",
- (unsigned long long)deltaxcn,
- (unsigned long long)
- max_cluster);
- rl[rlpos].vcn = vcn;
- vcn += rl[rlpos].length = max_cluster -
- deltaxcn;
- rl[rlpos].lcn = LCN_RL_NOT_MAPPED;
- rlpos++;
- } else if (unlikely(deltaxcn > max_cluster)) {
- ntfs_error(vol->sb, "Corrupt attribute. "
- "deltaxcn = 0x%llx, "
- "max_cluster = 0x%llx",
- (unsigned long long)deltaxcn,
- (unsigned long long)
- max_cluster);
- goto mpa_err;
- }
- }
- rl[rlpos].lcn = LCN_ENOENT;
- } else /* Not the base extent. There may be more extents to follow. */
- rl[rlpos].lcn = LCN_RL_NOT_MAPPED;
-
- /* Setup terminating runlist element. */
- rl[rlpos].vcn = vcn;
- rl[rlpos].length = (s64)0;
- /* If no existing runlist was specified, we are done. */
- if (!old_rl) {
- ntfs_debug("Mapping pairs array successfully decompressed:");
- ntfs_debug_dump_runlist(rl);
- return rl;
- }
- /* Now combine the new and old runlists checking for overlaps. */
- old_rl = ntfs_runlists_merge(old_rl, rl);
- if (!IS_ERR(old_rl))
- return old_rl;
- ntfs_free(rl);
- ntfs_error(vol->sb, "Failed to merge runlists.");
- return old_rl;
-io_error:
- ntfs_error(vol->sb, "Corrupt attribute.");
-err_out:
- ntfs_free(rl);
- return ERR_PTR(-EIO);
-}
-
-/**
- * ntfs_rl_vcn_to_lcn - convert a vcn into a lcn given a runlist
- * @rl: runlist to use for conversion
- * @vcn: vcn to convert
- *
- * Convert the virtual cluster number @vcn of an attribute into a logical
- * cluster number (lcn) of a device using the runlist @rl to map vcns to their
- * corresponding lcns.
- *
- * It is up to the caller to serialize access to the runlist @rl.
- *
- * Since lcns must be >= 0, we use negative return codes with special meaning:
- *
- * Return code Meaning / Description
- * ==================================================
- * LCN_HOLE Hole / not allocated on disk.
- * LCN_RL_NOT_MAPPED This is part of the runlist which has not been
- * inserted into the runlist yet.
- * LCN_ENOENT There is no such vcn in the attribute.
- *
- * Locking: - The caller must have locked the runlist (for reading or writing).
- * - This function does not touch the lock, nor does it modify the
- * runlist.
- */
-LCN ntfs_rl_vcn_to_lcn(const runlist_element *rl, const VCN vcn)
-{
- int i;
-
- BUG_ON(vcn < 0);
- /*
- * If rl is NULL, assume that we have found an unmapped runlist. The
- * caller can then attempt to map it and fail appropriately if
- * necessary.
- */
- if (unlikely(!rl))
- return LCN_RL_NOT_MAPPED;
-
- /* Catch out of lower bounds vcn. */
- if (unlikely(vcn < rl[0].vcn))
- return LCN_ENOENT;
-
- for (i = 0; likely(rl[i].length); i++) {
- if (unlikely(vcn < rl[i+1].vcn)) {
- if (likely(rl[i].lcn >= (LCN)0))
- return rl[i].lcn + (vcn - rl[i].vcn);
- return rl[i].lcn;
- }
- }
- /*
- * The terminator element is setup to the correct value, i.e. one of
- * LCN_HOLE, LCN_RL_NOT_MAPPED, or LCN_ENOENT.
- */
- if (likely(rl[i].lcn < (LCN)0))
- return rl[i].lcn;
- /* Just in case... We could replace this with BUG() some day. */
- return LCN_ENOENT;
-}
-
-#ifdef NTFS_RW
-
-/**
- * ntfs_rl_find_vcn_nolock - find a vcn in a runlist
- * @rl: runlist to search
- * @vcn: vcn to find
- *
- * Find the virtual cluster number @vcn in the runlist @rl and return the
- * address of the runlist element containing the @vcn on success.
- *
- * Return NULL if @rl is NULL or @vcn is in an unmapped part/out of bounds of
- * the runlist.
- *
- * Locking: The runlist must be locked on entry.
- */
-runlist_element *ntfs_rl_find_vcn_nolock(runlist_element *rl, const VCN vcn)
-{
- BUG_ON(vcn < 0);
- if (unlikely(!rl || vcn < rl[0].vcn))
- return NULL;
- while (likely(rl->length)) {
- if (unlikely(vcn < rl[1].vcn)) {
- if (likely(rl->lcn >= LCN_HOLE))
- return rl;
- return NULL;
- }
- rl++;
- }
- if (likely(rl->lcn == LCN_ENOENT))
- return rl;
- return NULL;
-}
-
-/**
- * ntfs_get_nr_significant_bytes - get number of bytes needed to store a number
- * @n: number for which to get the number of bytes for
- *
- * Return the number of bytes required to store @n unambiguously as
- * a signed number.
- *
- * This is used in the context of the mapping pairs array to determine how
- * many bytes will be needed in the array to store a given logical cluster
- * number (lcn) or a specific run length.
- *
- * Return the number of bytes written. This function cannot fail.
- */
-static inline int ntfs_get_nr_significant_bytes(const s64 n)
-{
- s64 l = n;
- int i;
- s8 j;
-
- i = 0;
- do {
- l >>= 8;
- i++;
- } while (l != 0 && l != -1);
- j = (n >> 8 * (i - 1)) & 0xff;
- /* If the sign bit is wrong, we need an extra byte. */
- if ((n < 0 && j >= 0) || (n > 0 && j < 0))
- i++;
- return i;
-}
-
-/**
- * ntfs_get_size_for_mapping_pairs - get bytes needed for mapping pairs array
- * @vol: ntfs volume (needed for the ntfs version)
- * @rl: locked runlist to determine the size of the mapping pairs of
- * @first_vcn: first vcn which to include in the mapping pairs array
- * @last_vcn: last vcn which to include in the mapping pairs array
- *
- * Walk the locked runlist @rl and calculate the size in bytes of the mapping
- * pairs array corresponding to the runlist @rl, starting at vcn @first_vcn and
- * finishing with vcn @last_vcn.
- *
- * A @last_vcn of -1 means end of runlist and in that case the size of the
- * mapping pairs array corresponding to the runlist starting at vcn @first_vcn
- * and finishing at the end of the runlist is determined.
- *
- * This for example allows us to allocate a buffer of the right size when
- * building the mapping pairs array.
- *
- * If @rl is NULL, just return 1 (for the single terminator byte).
- *
- * Return the calculated size in bytes on success. On error, return -errno.
- * The following error codes are defined:
- * -EINVAL - Run list contains unmapped elements. Make sure to only pass
- * fully mapped runlists to this function.
- * -EIO - The runlist is corrupt.
- *
- * Locking: @rl must be locked on entry (either for reading or writing), it
- * remains locked throughout, and is left locked upon return.
- */
-int ntfs_get_size_for_mapping_pairs(const ntfs_volume *vol,
- const runlist_element *rl, const VCN first_vcn,
- const VCN last_vcn)
-{
- LCN prev_lcn;
- int rls;
- bool the_end = false;
-
- BUG_ON(first_vcn < 0);
- BUG_ON(last_vcn < -1);
- BUG_ON(last_vcn >= 0 && first_vcn > last_vcn);
- if (!rl) {
- BUG_ON(first_vcn);
- BUG_ON(last_vcn > 0);
- return 1;
- }
- /* Skip to runlist element containing @first_vcn. */
- while (rl->length && first_vcn >= rl[1].vcn)
- rl++;
- if (unlikely((!rl->length && first_vcn > rl->vcn) ||
- first_vcn < rl->vcn))
- return -EINVAL;
- prev_lcn = 0;
- /* Always need the termining zero byte. */
- rls = 1;
- /* Do the first partial run if present. */
- if (first_vcn > rl->vcn) {
- s64 delta, length = rl->length;
-
- /* We know rl->length != 0 already. */
- if (unlikely(length < 0 || rl->lcn < LCN_HOLE))
- goto err_out;
- /*
- * If @stop_vcn is given and finishes inside this run, cap the
- * run length.
- */
- if (unlikely(last_vcn >= 0 && rl[1].vcn > last_vcn)) {
- s64 s1 = last_vcn + 1;
- if (unlikely(rl[1].vcn > s1))
- length = s1 - rl->vcn;
- the_end = true;
- }
- delta = first_vcn - rl->vcn;
- /* Header byte + length. */
- rls += 1 + ntfs_get_nr_significant_bytes(length - delta);
- /*
- * If the logical cluster number (lcn) denotes a hole and we
- * are on NTFS 3.0+, we don't store it at all, i.e. we need
- * zero space. On earlier NTFS versions we just store the lcn.
- * Note: this assumes that on NTFS 1.2-, holes are stored with
- * an lcn of -1 and not a delta_lcn of -1 (unless both are -1).
- */
- if (likely(rl->lcn >= 0 || vol->major_ver < 3)) {
- prev_lcn = rl->lcn;
- if (likely(rl->lcn >= 0))
- prev_lcn += delta;
- /* Change in lcn. */
- rls += ntfs_get_nr_significant_bytes(prev_lcn);
- }
- /* Go to next runlist element. */
- rl++;
- }
- /* Do the full runs. */
- for (; rl->length && !the_end; rl++) {
- s64 length = rl->length;
-
- if (unlikely(length < 0 || rl->lcn < LCN_HOLE))
- goto err_out;
- /*
- * If @stop_vcn is given and finishes inside this run, cap the
- * run length.
- */
- if (unlikely(last_vcn >= 0 && rl[1].vcn > last_vcn)) {
- s64 s1 = last_vcn + 1;
- if (unlikely(rl[1].vcn > s1))
- length = s1 - rl->vcn;
- the_end = true;
- }
- /* Header byte + length. */
- rls += 1 + ntfs_get_nr_significant_bytes(length);
- /*
- * If the logical cluster number (lcn) denotes a hole and we
- * are on NTFS 3.0+, we don't store it at all, i.e. we need
- * zero space. On earlier NTFS versions we just store the lcn.
- * Note: this assumes that on NTFS 1.2-, holes are stored with
- * an lcn of -1 and not a delta_lcn of -1 (unless both are -1).
- */
- if (likely(rl->lcn >= 0 || vol->major_ver < 3)) {
- /* Change in lcn. */
- rls += ntfs_get_nr_significant_bytes(rl->lcn -
- prev_lcn);
- prev_lcn = rl->lcn;
- }
- }
- return rls;
-err_out:
- if (rl->lcn == LCN_RL_NOT_MAPPED)
- rls = -EINVAL;
- else
- rls = -EIO;
- return rls;
-}
-
-/**
- * ntfs_write_significant_bytes - write the significant bytes of a number
- * @dst: destination buffer to write to
- * @dst_max: pointer to last byte of destination buffer for bounds checking
- * @n: number whose significant bytes to write
- *
- * Store in @dst, the minimum bytes of the number @n which are required to
- * identify @n unambiguously as a signed number, taking care not to exceed
- * @dest_max, the maximum position within @dst to which we are allowed to
- * write.
- *
- * This is used when building the mapping pairs array of a runlist to compress
- * a given logical cluster number (lcn) or a specific run length to the minimum
- * size possible.
- *
- * Return the number of bytes written on success. On error, i.e. the
- * destination buffer @dst is too small, return -ENOSPC.
- */
-static inline int ntfs_write_significant_bytes(s8 *dst, const s8 *dst_max,
- const s64 n)
-{
- s64 l = n;
- int i;
- s8 j;
-
- i = 0;
- do {
- if (unlikely(dst > dst_max))
- goto err_out;
- *dst++ = l & 0xffll;
- l >>= 8;
- i++;
- } while (l != 0 && l != -1);
- j = (n >> 8 * (i - 1)) & 0xff;
- /* If the sign bit is wrong, we need an extra byte. */
- if (n < 0 && j >= 0) {
- if (unlikely(dst > dst_max))
- goto err_out;
- i++;
- *dst = (s8)-1;
- } else if (n > 0 && j < 0) {
- if (unlikely(dst > dst_max))
- goto err_out;
- i++;
- *dst = (s8)0;
- }
- return i;
-err_out:
- return -ENOSPC;
-}
-
-/**
- * ntfs_mapping_pairs_build - build the mapping pairs array from a runlist
- * @vol: ntfs volume (needed for the ntfs version)
- * @dst: destination buffer to which to write the mapping pairs array
- * @dst_len: size of destination buffer @dst in bytes
- * @rl: locked runlist for which to build the mapping pairs array
- * @first_vcn: first vcn which to include in the mapping pairs array
- * @last_vcn: last vcn which to include in the mapping pairs array
- * @stop_vcn: first vcn outside destination buffer on success or -ENOSPC
- *
- * Create the mapping pairs array from the locked runlist @rl, starting at vcn
- * @first_vcn and finishing with vcn @last_vcn and save the array in @dst.
- * @dst_len is the size of @dst in bytes and it should be at least equal to the
- * value obtained by calling ntfs_get_size_for_mapping_pairs().
- *
- * A @last_vcn of -1 means end of runlist and in that case the mapping pairs
- * array corresponding to the runlist starting at vcn @first_vcn and finishing
- * at the end of the runlist is created.
- *
- * If @rl is NULL, just write a single terminator byte to @dst.
- *
- * On success or -ENOSPC error, if @stop_vcn is not NULL, *@stop_vcn is set to
- * the first vcn outside the destination buffer. Note that on error, @dst has
- * been filled with all the mapping pairs that will fit, thus it can be treated
- * as partial success, in that a new attribute extent needs to be created or
- * the next extent has to be used and the mapping pairs build has to be
- * continued with @first_vcn set to *@stop_vcn.
- *
- * Return 0 on success and -errno on error. The following error codes are
- * defined:
- * -EINVAL - Run list contains unmapped elements. Make sure to only pass
- * fully mapped runlists to this function.
- * -EIO - The runlist is corrupt.
- * -ENOSPC - The destination buffer is too small.
- *
- * Locking: @rl must be locked on entry (either for reading or writing), it
- * remains locked throughout, and is left locked upon return.
- */
-int ntfs_mapping_pairs_build(const ntfs_volume *vol, s8 *dst,
- const int dst_len, const runlist_element *rl,
- const VCN first_vcn, const VCN last_vcn, VCN *const stop_vcn)
-{
- LCN prev_lcn;
- s8 *dst_max, *dst_next;
- int err = -ENOSPC;
- bool the_end = false;
- s8 len_len, lcn_len;
-
- BUG_ON(first_vcn < 0);
- BUG_ON(last_vcn < -1);
- BUG_ON(last_vcn >= 0 && first_vcn > last_vcn);
- BUG_ON(dst_len < 1);
- if (!rl) {
- BUG_ON(first_vcn);
- BUG_ON(last_vcn > 0);
- if (stop_vcn)
- *stop_vcn = 0;
- /* Terminator byte. */
- *dst = 0;
- return 0;
- }
- /* Skip to runlist element containing @first_vcn. */
- while (rl->length && first_vcn >= rl[1].vcn)
- rl++;
- if (unlikely((!rl->length && first_vcn > rl->vcn) ||
- first_vcn < rl->vcn))
- return -EINVAL;
- /*
- * @dst_max is used for bounds checking in
- * ntfs_write_significant_bytes().
- */
- dst_max = dst + dst_len - 1;
- prev_lcn = 0;
- /* Do the first partial run if present. */
- if (first_vcn > rl->vcn) {
- s64 delta, length = rl->length;
-
- /* We know rl->length != 0 already. */
- if (unlikely(length < 0 || rl->lcn < LCN_HOLE))
- goto err_out;
- /*
- * If @stop_vcn is given and finishes inside this run, cap the
- * run length.
- */
- if (unlikely(last_vcn >= 0 && rl[1].vcn > last_vcn)) {
- s64 s1 = last_vcn + 1;
- if (unlikely(rl[1].vcn > s1))
- length = s1 - rl->vcn;
- the_end = true;
- }
- delta = first_vcn - rl->vcn;
- /* Write length. */
- len_len = ntfs_write_significant_bytes(dst + 1, dst_max,
- length - delta);
- if (unlikely(len_len < 0))
- goto size_err;
- /*
- * If the logical cluster number (lcn) denotes a hole and we
- * are on NTFS 3.0+, we don't store it at all, i.e. we need
- * zero space. On earlier NTFS versions we just write the lcn
- * change. FIXME: Do we need to write the lcn change or just
- * the lcn in that case? Not sure as I have never seen this
- * case on NT4. - We assume that we just need to write the lcn
- * change until someone tells us otherwise... (AIA)
- */
- if (likely(rl->lcn >= 0 || vol->major_ver < 3)) {
- prev_lcn = rl->lcn;
- if (likely(rl->lcn >= 0))
- prev_lcn += delta;
- /* Write change in lcn. */
- lcn_len = ntfs_write_significant_bytes(dst + 1 +
- len_len, dst_max, prev_lcn);
- if (unlikely(lcn_len < 0))
- goto size_err;
- } else
- lcn_len = 0;
- dst_next = dst + len_len + lcn_len + 1;
- if (unlikely(dst_next > dst_max))
- goto size_err;
- /* Update header byte. */
- *dst = lcn_len << 4 | len_len;
- /* Position at next mapping pairs array element. */
- dst = dst_next;
- /* Go to next runlist element. */
- rl++;
- }
- /* Do the full runs. */
- for (; rl->length && !the_end; rl++) {
- s64 length = rl->length;
-
- if (unlikely(length < 0 || rl->lcn < LCN_HOLE))
- goto err_out;
- /*
- * If @stop_vcn is given and finishes inside this run, cap the
- * run length.
- */
- if (unlikely(last_vcn >= 0 && rl[1].vcn > last_vcn)) {
- s64 s1 = last_vcn + 1;
- if (unlikely(rl[1].vcn > s1))
- length = s1 - rl->vcn;
- the_end = true;
- }
- /* Write length. */
- len_len = ntfs_write_significant_bytes(dst + 1, dst_max,
- length);
- if (unlikely(len_len < 0))
- goto size_err;
- /*
- * If the logical cluster number (lcn) denotes a hole and we
- * are on NTFS 3.0+, we don't store it at all, i.e. we need
- * zero space. On earlier NTFS versions we just write the lcn
- * change. FIXME: Do we need to write the lcn change or just
- * the lcn in that case? Not sure as I have never seen this
- * case on NT4. - We assume that we just need to write the lcn
- * change until someone tells us otherwise... (AIA)
- */
- if (likely(rl->lcn >= 0 || vol->major_ver < 3)) {
- /* Write change in lcn. */
- lcn_len = ntfs_write_significant_bytes(dst + 1 +
- len_len, dst_max, rl->lcn - prev_lcn);
- if (unlikely(lcn_len < 0))
- goto size_err;
- prev_lcn = rl->lcn;
- } else
- lcn_len = 0;
- dst_next = dst + len_len + lcn_len + 1;
- if (unlikely(dst_next > dst_max))
- goto size_err;
- /* Update header byte. */
- *dst = lcn_len << 4 | len_len;
- /* Position at next mapping pairs array element. */
- dst = dst_next;
- }
- /* Success. */
- err = 0;
-size_err:
- /* Set stop vcn. */
- if (stop_vcn)
- *stop_vcn = rl->vcn;
- /* Add terminator byte. */
- *dst = 0;
- return err;
-err_out:
- if (rl->lcn == LCN_RL_NOT_MAPPED)
- err = -EINVAL;
- else
- err = -EIO;
- return err;
-}
-
-/**
- * ntfs_rl_truncate_nolock - truncate a runlist starting at a specified vcn
- * @vol: ntfs volume (needed for error output)
- * @runlist: runlist to truncate
- * @new_length: the new length of the runlist in VCNs
- *
- * Truncate the runlist described by @runlist as well as the memory buffer
- * holding the runlist elements to a length of @new_length VCNs.
- *
- * If @new_length lies within the runlist, the runlist elements with VCNs of
- * @new_length and above are discarded. As a special case if @new_length is
- * zero, the runlist is discarded and set to NULL.
- *
- * If @new_length lies beyond the runlist, a sparse runlist element is added to
- * the end of the runlist @runlist or if the last runlist element is a sparse
- * one already, this is extended.
- *
- * Note, no checking is done for unmapped runlist elements. It is assumed that
- * the caller has mapped any elements that need to be mapped already.
- *
- * Return 0 on success and -errno on error.
- *
- * Locking: The caller must hold @runlist->lock for writing.
- */
-int ntfs_rl_truncate_nolock(const ntfs_volume *vol, runlist *const runlist,
- const s64 new_length)
-{
- runlist_element *rl;
- int old_size;
-
- ntfs_debug("Entering for new_length 0x%llx.", (long long)new_length);
- BUG_ON(!runlist);
- BUG_ON(new_length < 0);
- rl = runlist->rl;
- if (!new_length) {
- ntfs_debug("Freeing runlist.");
- runlist->rl = NULL;
- if (rl)
- ntfs_free(rl);
- return 0;
- }
- if (unlikely(!rl)) {
- /*
- * Create a runlist consisting of a sparse runlist element of
- * length @new_length followed by a terminator runlist element.
- */
- rl = ntfs_malloc_nofs(PAGE_SIZE);
- if (unlikely(!rl)) {
- ntfs_error(vol->sb, "Not enough memory to allocate "
- "runlist element buffer.");
- return -ENOMEM;
- }
- runlist->rl = rl;
- rl[1].length = rl->vcn = 0;
- rl->lcn = LCN_HOLE;
- rl[1].vcn = rl->length = new_length;
- rl[1].lcn = LCN_ENOENT;
- return 0;
- }
- BUG_ON(new_length < rl->vcn);
- /* Find @new_length in the runlist. */
- while (likely(rl->length && new_length >= rl[1].vcn))
- rl++;
- /*
- * If not at the end of the runlist we need to shrink it.
- * If at the end of the runlist we need to expand it.
- */
- if (rl->length) {
- runlist_element *trl;
- bool is_end;
-
- ntfs_debug("Shrinking runlist.");
- /* Determine the runlist size. */
- trl = rl + 1;
- while (likely(trl->length))
- trl++;
- old_size = trl - runlist->rl + 1;
- /* Truncate the run. */
- rl->length = new_length - rl->vcn;
- /*
- * If a run was partially truncated, make the following runlist
- * element a terminator.
- */
- is_end = false;
- if (rl->length) {
- rl++;
- if (!rl->length)
- is_end = true;
- rl->vcn = new_length;
- rl->length = 0;
- }
- rl->lcn = LCN_ENOENT;
- /* Reallocate memory if necessary. */
- if (!is_end) {
- int new_size = rl - runlist->rl + 1;
- rl = ntfs_rl_realloc(runlist->rl, old_size, new_size);
- if (IS_ERR(rl))
- ntfs_warning(vol->sb, "Failed to shrink "
- "runlist buffer. This just "
- "wastes a bit of memory "
- "temporarily so we ignore it "
- "and return success.");
- else
- runlist->rl = rl;
- }
- } else if (likely(/* !rl->length && */ new_length > rl->vcn)) {
- ntfs_debug("Expanding runlist.");
- /*
- * If there is a previous runlist element and it is a sparse
- * one, extend it. Otherwise need to add a new, sparse runlist
- * element.
- */
- if ((rl > runlist->rl) && ((rl - 1)->lcn == LCN_HOLE))
- (rl - 1)->length = new_length - (rl - 1)->vcn;
- else {
- /* Determine the runlist size. */
- old_size = rl - runlist->rl + 1;
- /* Reallocate memory if necessary. */
- rl = ntfs_rl_realloc(runlist->rl, old_size,
- old_size + 1);
- if (IS_ERR(rl)) {
- ntfs_error(vol->sb, "Failed to expand runlist "
- "buffer, aborting.");
- return PTR_ERR(rl);
- }
- runlist->rl = rl;
- /*
- * Set @rl to the same runlist element in the new
- * runlist as before in the old runlist.
- */
- rl += old_size - 1;
- /* Add a new, sparse runlist element. */
- rl->lcn = LCN_HOLE;
- rl->length = new_length - rl->vcn;
- /* Add a new terminator runlist element. */
- rl++;
- rl->length = 0;
- }
- rl->vcn = new_length;
- rl->lcn = LCN_ENOENT;
- } else /* if (unlikely(!rl->length && new_length == rl->vcn)) */ {
- /* Runlist already has same size as requested. */
- rl->lcn = LCN_ENOENT;
- }
- ntfs_debug("Done.");
- return 0;
-}
-
-/**
- * ntfs_rl_punch_nolock - punch a hole into a runlist
- * @vol: ntfs volume (needed for error output)
- * @runlist: runlist to punch a hole into
- * @start: starting VCN of the hole to be created
- * @length: size of the hole to be created in units of clusters
- *
- * Punch a hole into the runlist @runlist starting at VCN @start and of size
- * @length clusters.
- *
- * Return 0 on success and -errno on error, in which case @runlist has not been
- * modified.
- *
- * If @start and/or @start + @length are outside the runlist return error code
- * -ENOENT.
- *
- * If the runlist contains unmapped or error elements between @start and @start
- * + @length return error code -EINVAL.
- *
- * Locking: The caller must hold @runlist->lock for writing.
- */
-int ntfs_rl_punch_nolock(const ntfs_volume *vol, runlist *const runlist,
- const VCN start, const s64 length)
-{
- const VCN end = start + length;
- s64 delta;
- runlist_element *rl, *rl_end, *rl_real_end, *trl;
- int old_size;
- bool lcn_fixup = false;
-
- ntfs_debug("Entering for start 0x%llx, length 0x%llx.",
- (long long)start, (long long)length);
- BUG_ON(!runlist);
- BUG_ON(start < 0);
- BUG_ON(length < 0);
- BUG_ON(end < 0);
- rl = runlist->rl;
- if (unlikely(!rl)) {
- if (likely(!start && !length))
- return 0;
- return -ENOENT;
- }
- /* Find @start in the runlist. */
- while (likely(rl->length && start >= rl[1].vcn))
- rl++;
- rl_end = rl;
- /* Find @end in the runlist. */
- while (likely(rl_end->length && end >= rl_end[1].vcn)) {
- /* Verify there are no unmapped or error elements. */
- if (unlikely(rl_end->lcn < LCN_HOLE))
- return -EINVAL;
- rl_end++;
- }
- /* Check the last element. */
- if (unlikely(rl_end->length && rl_end->lcn < LCN_HOLE))
- return -EINVAL;
- /* This covers @start being out of bounds, too. */
- if (!rl_end->length && end > rl_end->vcn)
- return -ENOENT;
- if (!length)
- return 0;
- if (!rl->length)
- return -ENOENT;
- rl_real_end = rl_end;
- /* Determine the runlist size. */
- while (likely(rl_real_end->length))
- rl_real_end++;
- old_size = rl_real_end - runlist->rl + 1;
- /* If @start is in a hole simply extend the hole. */
- if (rl->lcn == LCN_HOLE) {
- /*
- * If both @start and @end are in the same sparse run, we are
- * done.
- */
- if (end <= rl[1].vcn) {
- ntfs_debug("Done (requested hole is already sparse).");
- return 0;
- }
-extend_hole:
- /* Extend the hole. */
- rl->length = end - rl->vcn;
- /* If @end is in a hole, merge it with the current one. */
- if (rl_end->lcn == LCN_HOLE) {
- rl_end++;
- rl->length = rl_end->vcn - rl->vcn;
- }
- /* We have done the hole. Now deal with the remaining tail. */
- rl++;
- /* Cut out all runlist elements up to @end. */
- if (rl < rl_end)
- memmove(rl, rl_end, (rl_real_end - rl_end + 1) *
- sizeof(*rl));
- /* Adjust the beginning of the tail if necessary. */
- if (end > rl->vcn) {
- delta = end - rl->vcn;
- rl->vcn = end;
- rl->length -= delta;
- /* Only adjust the lcn if it is real. */
- if (rl->lcn >= 0)
- rl->lcn += delta;
- }
-shrink_allocation:
- /* Reallocate memory if the allocation changed. */
- if (rl < rl_end) {
- rl = ntfs_rl_realloc(runlist->rl, old_size,
- old_size - (rl_end - rl));
- if (IS_ERR(rl))
- ntfs_warning(vol->sb, "Failed to shrink "
- "runlist buffer. This just "
- "wastes a bit of memory "
- "temporarily so we ignore it "
- "and return success.");
- else
- runlist->rl = rl;
- }
- ntfs_debug("Done (extend hole).");
- return 0;
- }
- /*
- * If @start is at the beginning of a run things are easier as there is
- * no need to split the first run.
- */
- if (start == rl->vcn) {
- /*
- * @start is at the beginning of a run.
- *
- * If the previous run is sparse, extend its hole.
- *
- * If @end is not in the same run, switch the run to be sparse
- * and extend the newly created hole.
- *
- * Thus both of these cases reduce the problem to the above
- * case of "@start is in a hole".
- */
- if (rl > runlist->rl && (rl - 1)->lcn == LCN_HOLE) {
- rl--;
- goto extend_hole;
- }
- if (end >= rl[1].vcn) {
- rl->lcn = LCN_HOLE;
- goto extend_hole;
- }
- /*
- * The final case is when @end is in the same run as @start.
- * For this need to split the run into two. One run for the
- * sparse region between the beginning of the old run, i.e.
- * @start, and @end and one for the remaining non-sparse
- * region, i.e. between @end and the end of the old run.
- */
- trl = ntfs_rl_realloc(runlist->rl, old_size, old_size + 1);
- if (IS_ERR(trl))
- goto enomem_out;
- old_size++;
- if (runlist->rl != trl) {
- rl = trl + (rl - runlist->rl);
- rl_end = trl + (rl_end - runlist->rl);
- rl_real_end = trl + (rl_real_end - runlist->rl);
- runlist->rl = trl;
- }
-split_end:
- /* Shift all the runs up by one. */
- memmove(rl + 1, rl, (rl_real_end - rl + 1) * sizeof(*rl));
- /* Finally, setup the two split runs. */
- rl->lcn = LCN_HOLE;
- rl->length = length;
- rl++;
- rl->vcn += length;
- /* Only adjust the lcn if it is real. */
- if (rl->lcn >= 0 || lcn_fixup)
- rl->lcn += length;
- rl->length -= length;
- ntfs_debug("Done (split one).");
- return 0;
- }
- /*
- * @start is neither in a hole nor at the beginning of a run.
- *
- * If @end is in a hole, things are easier as simply truncating the run
- * @start is in to end at @start - 1, deleting all runs after that up
- * to @end, and finally extending the beginning of the run @end is in
- * to be @start is all that is needed.
- */
- if (rl_end->lcn == LCN_HOLE) {
- /* Truncate the run containing @start. */
- rl->length = start - rl->vcn;
- rl++;
- /* Cut out all runlist elements up to @end. */
- if (rl < rl_end)
- memmove(rl, rl_end, (rl_real_end - rl_end + 1) *
- sizeof(*rl));
- /* Extend the beginning of the run @end is in to be @start. */
- rl->vcn = start;
- rl->length = rl[1].vcn - start;
- goto shrink_allocation;
- }
- /*
- * If @end is not in a hole there are still two cases to distinguish.
- * Either @end is or is not in the same run as @start.
- *
- * The second case is easier as it can be reduced to an already solved
- * problem by truncating the run @start is in to end at @start - 1.
- * Then, if @end is in the next run need to split the run into a sparse
- * run followed by a non-sparse run (already covered above) and if @end
- * is not in the next run switching it to be sparse, again reduces the
- * problem to the already covered case of "@start is in a hole".
- */
- if (end >= rl[1].vcn) {
- /*
- * If @end is not in the next run, reduce the problem to the
- * case of "@start is in a hole".
- */
- if (rl[1].length && end >= rl[2].vcn) {
- /* Truncate the run containing @start. */
- rl->length = start - rl->vcn;
- rl++;
- rl->vcn = start;
- rl->lcn = LCN_HOLE;
- goto extend_hole;
- }
- trl = ntfs_rl_realloc(runlist->rl, old_size, old_size + 1);
- if (IS_ERR(trl))
- goto enomem_out;
- old_size++;
- if (runlist->rl != trl) {
- rl = trl + (rl - runlist->rl);
- rl_end = trl + (rl_end - runlist->rl);
- rl_real_end = trl + (rl_real_end - runlist->rl);
- runlist->rl = trl;
- }
- /* Truncate the run containing @start. */
- rl->length = start - rl->vcn;
- rl++;
- /*
- * @end is in the next run, reduce the problem to the case
- * where "@start is at the beginning of a run and @end is in
- * the same run as @start".
- */
- delta = rl->vcn - start;
- rl->vcn = start;
- if (rl->lcn >= 0) {
- rl->lcn -= delta;
- /* Need this in case the lcn just became negative. */
- lcn_fixup = true;
- }
- rl->length += delta;
- goto split_end;
- }
- /*
- * The first case from above, i.e. @end is in the same run as @start.
- * We need to split the run into three. One run for the non-sparse
- * region between the beginning of the old run and @start, one for the
- * sparse region between @start and @end, and one for the remaining
- * non-sparse region, i.e. between @end and the end of the old run.
- */
- trl = ntfs_rl_realloc(runlist->rl, old_size, old_size + 2);
- if (IS_ERR(trl))
- goto enomem_out;
- old_size += 2;
- if (runlist->rl != trl) {
- rl = trl + (rl - runlist->rl);
- rl_end = trl + (rl_end - runlist->rl);
- rl_real_end = trl + (rl_real_end - runlist->rl);
- runlist->rl = trl;
- }
- /* Shift all the runs up by two. */
- memmove(rl + 2, rl, (rl_real_end - rl + 1) * sizeof(*rl));
- /* Finally, setup the three split runs. */
- rl->length = start - rl->vcn;
- rl++;
- rl->vcn = start;
- rl->lcn = LCN_HOLE;
- rl->length = length;
- rl++;
- delta = end - rl->vcn;
- rl->vcn = end;
- rl->lcn += delta;
- rl->length -= delta;
- ntfs_debug("Done (split both).");
- return 0;
-enomem_out:
- ntfs_error(vol->sb, "Not enough memory to extend runlist buffer.");
- return -ENOMEM;
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/runlist.h b/fs/ntfs/runlist.h
deleted file mode 100644
index 38de0a375f59..000000000000
--- a/fs/ntfs/runlist.h
+++ /dev/null
@@ -1,88 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * runlist.h - Defines for runlist handling in NTFS Linux kernel driver.
- * Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2005 Anton Altaparmakov
- * Copyright (c) 2002 Richard Russon
- */
-
-#ifndef _LINUX_NTFS_RUNLIST_H
-#define _LINUX_NTFS_RUNLIST_H
-
-#include "types.h"
-#include "layout.h"
-#include "volume.h"
-
-/**
- * runlist_element - in memory vcn to lcn mapping array element
- * @vcn: starting vcn of the current array element
- * @lcn: starting lcn of the current array element
- * @length: length in clusters of the current array element
- *
- * The last vcn (in fact the last vcn + 1) is reached when length == 0.
- *
- * When lcn == -1 this means that the count vcns starting at vcn are not
- * physically allocated (i.e. this is a hole / data is sparse).
- */
-typedef struct { /* In memory vcn to lcn mapping structure element. */
- VCN vcn; /* vcn = Starting virtual cluster number. */
- LCN lcn; /* lcn = Starting logical cluster number. */
- s64 length; /* Run length in clusters. */
-} runlist_element;
-
-/**
- * runlist - in memory vcn to lcn mapping array including a read/write lock
- * @rl: pointer to an array of runlist elements
- * @lock: read/write spinlock for serializing access to @rl
- *
- */
-typedef struct {
- runlist_element *rl;
- struct rw_semaphore lock;
-} runlist;
-
-static inline void ntfs_init_runlist(runlist *rl)
-{
- rl->rl = NULL;
- init_rwsem(&rl->lock);
-}
-
-typedef enum {
- LCN_HOLE = -1, /* Keep this as highest value or die! */
- LCN_RL_NOT_MAPPED = -2,
- LCN_ENOENT = -3,
- LCN_ENOMEM = -4,
- LCN_EIO = -5,
-} LCN_SPECIAL_VALUES;
-
-extern runlist_element *ntfs_runlists_merge(runlist_element *drl,
- runlist_element *srl);
-
-extern runlist_element *ntfs_mapping_pairs_decompress(const ntfs_volume *vol,
- const ATTR_RECORD *attr, runlist_element *old_rl);
-
-extern LCN ntfs_rl_vcn_to_lcn(const runlist_element *rl, const VCN vcn);
-
-#ifdef NTFS_RW
-
-extern runlist_element *ntfs_rl_find_vcn_nolock(runlist_element *rl,
- const VCN vcn);
-
-extern int ntfs_get_size_for_mapping_pairs(const ntfs_volume *vol,
- const runlist_element *rl, const VCN first_vcn,
- const VCN last_vcn);
-
-extern int ntfs_mapping_pairs_build(const ntfs_volume *vol, s8 *dst,
- const int dst_len, const runlist_element *rl,
- const VCN first_vcn, const VCN last_vcn, VCN *const stop_vcn);
-
-extern int ntfs_rl_truncate_nolock(const ntfs_volume *vol,
- runlist *const runlist, const s64 new_length);
-
-int ntfs_rl_punch_nolock(const ntfs_volume *vol, runlist *const runlist,
- const VCN start, const s64 length);
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_RUNLIST_H */
diff --git a/fs/ntfs/super.c b/fs/ntfs/super.c
deleted file mode 100644
index 001f4e053c85..000000000000
--- a/fs/ntfs/super.c
+++ /dev/null
@@ -1,3194 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * super.c - NTFS kernel super block handling. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2012 Anton Altaparmakov and Tuxera Inc.
- * Copyright (c) 2001,2002 Richard Russon
- */
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-
-#include <linux/stddef.h>
-#include <linux/init.h>
-#include <linux/slab.h>
-#include <linux/string.h>
-#include <linux/spinlock.h>
-#include <linux/blkdev.h> /* For bdev_logical_block_size(). */
-#include <linux/backing-dev.h>
-#include <linux/buffer_head.h>
-#include <linux/vfs.h>
-#include <linux/moduleparam.h>
-#include <linux/bitmap.h>
-
-#include "sysctl.h"
-#include "logfile.h"
-#include "quota.h"
-#include "usnjrnl.h"
-#include "dir.h"
-#include "debug.h"
-#include "index.h"
-#include "inode.h"
-#include "aops.h"
-#include "layout.h"
-#include "malloc.h"
-#include "ntfs.h"
-
-/* Number of mounted filesystems which have compression enabled. */
-static unsigned long ntfs_nr_compression_users;
-
-/* A global default upcase table and a corresponding reference count. */
-static ntfschar *default_upcase;
-static unsigned long ntfs_nr_upcase_users;
-
-/* Error constants/strings used in inode.c::ntfs_show_options(). */
-typedef enum {
- /* One of these must be present, default is ON_ERRORS_CONTINUE. */
- ON_ERRORS_PANIC = 0x01,
- ON_ERRORS_REMOUNT_RO = 0x02,
- ON_ERRORS_CONTINUE = 0x04,
- /* Optional, can be combined with any of the above. */
- ON_ERRORS_RECOVER = 0x10,
-} ON_ERRORS_ACTIONS;
-
-const option_t on_errors_arr[] = {
- { ON_ERRORS_PANIC, "panic" },
- { ON_ERRORS_REMOUNT_RO, "remount-ro", },
- { ON_ERRORS_CONTINUE, "continue", },
- { ON_ERRORS_RECOVER, "recover" },
- { 0, NULL }
-};
-
-/**
- * simple_getbool -
- *
- * Copied from old ntfs driver (which copied from vfat driver).
- */
-static int simple_getbool(char *s, bool *setval)
-{
- if (s) {
- if (!strcmp(s, "1") || !strcmp(s, "yes") || !strcmp(s, "true"))
- *setval = true;
- else if (!strcmp(s, "0") || !strcmp(s, "no") ||
- !strcmp(s, "false"))
- *setval = false;
- else
- return 0;
- } else
- *setval = true;
- return 1;
-}
-
-/**
- * parse_options - parse the (re)mount options
- * @vol: ntfs volume
- * @opt: string containing the (re)mount options
- *
- * Parse the recognized options in @opt for the ntfs volume described by @vol.
- */
-static bool parse_options(ntfs_volume *vol, char *opt)
-{
- char *p, *v, *ov;
- static char *utf8 = "utf8";
- int errors = 0, sloppy = 0;
- kuid_t uid = INVALID_UID;
- kgid_t gid = INVALID_GID;
- umode_t fmask = (umode_t)-1, dmask = (umode_t)-1;
- int mft_zone_multiplier = -1, on_errors = -1;
- int show_sys_files = -1, case_sensitive = -1, disable_sparse = -1;
- struct nls_table *nls_map = NULL, *old_nls;
-
- /* I am lazy... (-8 */
-#define NTFS_GETOPT_WITH_DEFAULT(option, variable, default_value) \
- if (!strcmp(p, option)) { \
- if (!v || !*v) \
- variable = default_value; \
- else { \
- variable = simple_strtoul(ov = v, &v, 0); \
- if (*v) \
- goto needs_val; \
- } \
- }
-#define NTFS_GETOPT(option, variable) \
- if (!strcmp(p, option)) { \
- if (!v || !*v) \
- goto needs_arg; \
- variable = simple_strtoul(ov = v, &v, 0); \
- if (*v) \
- goto needs_val; \
- }
-#define NTFS_GETOPT_UID(option, variable) \
- if (!strcmp(p, option)) { \
- uid_t uid_value; \
- if (!v || !*v) \
- goto needs_arg; \
- uid_value = simple_strtoul(ov = v, &v, 0); \
- if (*v) \
- goto needs_val; \
- variable = make_kuid(current_user_ns(), uid_value); \
- if (!uid_valid(variable)) \
- goto needs_val; \
- }
-#define NTFS_GETOPT_GID(option, variable) \
- if (!strcmp(p, option)) { \
- gid_t gid_value; \
- if (!v || !*v) \
- goto needs_arg; \
- gid_value = simple_strtoul(ov = v, &v, 0); \
- if (*v) \
- goto needs_val; \
- variable = make_kgid(current_user_ns(), gid_value); \
- if (!gid_valid(variable)) \
- goto needs_val; \
- }
-#define NTFS_GETOPT_OCTAL(option, variable) \
- if (!strcmp(p, option)) { \
- if (!v || !*v) \
- goto needs_arg; \
- variable = simple_strtoul(ov = v, &v, 8); \
- if (*v) \
- goto needs_val; \
- }
-#define NTFS_GETOPT_BOOL(option, variable) \
- if (!strcmp(p, option)) { \
- bool val; \
- if (!simple_getbool(v, &val)) \
- goto needs_bool; \
- variable = val; \
- }
-#define NTFS_GETOPT_OPTIONS_ARRAY(option, variable, opt_array) \
- if (!strcmp(p, option)) { \
- int _i; \
- if (!v || !*v) \
- goto needs_arg; \
- ov = v; \
- if (variable == -1) \
- variable = 0; \
- for (_i = 0; opt_array[_i].str && *opt_array[_i].str; _i++) \
- if (!strcmp(opt_array[_i].str, v)) { \
- variable |= opt_array[_i].val; \
- break; \
- } \
- if (!opt_array[_i].str || !*opt_array[_i].str) \
- goto needs_val; \
- }
- if (!opt || !*opt)
- goto no_mount_options;
- ntfs_debug("Entering with mount options string: %s", opt);
- while ((p = strsep(&opt, ","))) {
- if ((v = strchr(p, '=')))
- *v++ = 0;
- NTFS_GETOPT_UID("uid", uid)
- else NTFS_GETOPT_GID("gid", gid)
- else NTFS_GETOPT_OCTAL("umask", fmask = dmask)
- else NTFS_GETOPT_OCTAL("fmask", fmask)
- else NTFS_GETOPT_OCTAL("dmask", dmask)
- else NTFS_GETOPT("mft_zone_multiplier", mft_zone_multiplier)
- else NTFS_GETOPT_WITH_DEFAULT("sloppy", sloppy, true)
- else NTFS_GETOPT_BOOL("show_sys_files", show_sys_files)
- else NTFS_GETOPT_BOOL("case_sensitive", case_sensitive)
- else NTFS_GETOPT_BOOL("disable_sparse", disable_sparse)
- else NTFS_GETOPT_OPTIONS_ARRAY("errors", on_errors,
- on_errors_arr)
- else if (!strcmp(p, "posix") || !strcmp(p, "show_inodes"))
- ntfs_warning(vol->sb, "Ignoring obsolete option %s.",
- p);
- else if (!strcmp(p, "nls") || !strcmp(p, "iocharset")) {
- if (!strcmp(p, "iocharset"))
- ntfs_warning(vol->sb, "Option iocharset is "
- "deprecated. Please use "
- "option nls=<charsetname> in "
- "the future.");
- if (!v || !*v)
- goto needs_arg;
-use_utf8:
- old_nls = nls_map;
- nls_map = load_nls(v);
- if (!nls_map) {
- if (!old_nls) {
- ntfs_error(vol->sb, "NLS character set "
- "%s not found.", v);
- return false;
- }
- ntfs_error(vol->sb, "NLS character set %s not "
- "found. Using previous one %s.",
- v, old_nls->charset);
- nls_map = old_nls;
- } else /* nls_map */ {
- unload_nls(old_nls);
- }
- } else if (!strcmp(p, "utf8")) {
- bool val = false;
- ntfs_warning(vol->sb, "Option utf8 is no longer "
- "supported, using option nls=utf8. Please "
- "use option nls=utf8 in the future and "
- "make sure utf8 is compiled either as a "
- "module or into the kernel.");
- if (!v || !*v)
- val = true;
- else if (!simple_getbool(v, &val))
- goto needs_bool;
- if (val) {
- v = utf8;
- goto use_utf8;
- }
- } else {
- ntfs_error(vol->sb, "Unrecognized mount option %s.", p);
- if (errors < INT_MAX)
- errors++;
- }
-#undef NTFS_GETOPT_OPTIONS_ARRAY
-#undef NTFS_GETOPT_BOOL
-#undef NTFS_GETOPT
-#undef NTFS_GETOPT_WITH_DEFAULT
- }
-no_mount_options:
- if (errors && !sloppy)
- return false;
- if (sloppy)
- ntfs_warning(vol->sb, "Sloppy option given. Ignoring "
- "unrecognized mount option(s) and continuing.");
- /* Keep this first! */
- if (on_errors != -1) {
- if (!on_errors) {
- ntfs_error(vol->sb, "Invalid errors option argument "
- "or bug in options parser.");
- return false;
- }
- }
- if (nls_map) {
- if (vol->nls_map && vol->nls_map != nls_map) {
- ntfs_error(vol->sb, "Cannot change NLS character set "
- "on remount.");
- return false;
- } /* else (!vol->nls_map) */
- ntfs_debug("Using NLS character set %s.", nls_map->charset);
- vol->nls_map = nls_map;
- } else /* (!nls_map) */ {
- if (!vol->nls_map) {
- vol->nls_map = load_nls_default();
- if (!vol->nls_map) {
- ntfs_error(vol->sb, "Failed to load default "
- "NLS character set.");
- return false;
- }
- ntfs_debug("Using default NLS character set (%s).",
- vol->nls_map->charset);
- }
- }
- if (mft_zone_multiplier != -1) {
- if (vol->mft_zone_multiplier && vol->mft_zone_multiplier !=
- mft_zone_multiplier) {
- ntfs_error(vol->sb, "Cannot change mft_zone_multiplier "
- "on remount.");
- return false;
- }
- if (mft_zone_multiplier < 1 || mft_zone_multiplier > 4) {
- ntfs_error(vol->sb, "Invalid mft_zone_multiplier. "
- "Using default value, i.e. 1.");
- mft_zone_multiplier = 1;
- }
- vol->mft_zone_multiplier = mft_zone_multiplier;
- }
- if (!vol->mft_zone_multiplier)
- vol->mft_zone_multiplier = 1;
- if (on_errors != -1)
- vol->on_errors = on_errors;
- if (!vol->on_errors || vol->on_errors == ON_ERRORS_RECOVER)
- vol->on_errors |= ON_ERRORS_CONTINUE;
- if (uid_valid(uid))
- vol->uid = uid;
- if (gid_valid(gid))
- vol->gid = gid;
- if (fmask != (umode_t)-1)
- vol->fmask = fmask;
- if (dmask != (umode_t)-1)
- vol->dmask = dmask;
- if (show_sys_files != -1) {
- if (show_sys_files)
- NVolSetShowSystemFiles(vol);
- else
- NVolClearShowSystemFiles(vol);
- }
- if (case_sensitive != -1) {
- if (case_sensitive)
- NVolSetCaseSensitive(vol);
- else
- NVolClearCaseSensitive(vol);
- }
- if (disable_sparse != -1) {
- if (disable_sparse)
- NVolClearSparseEnabled(vol);
- else {
- if (!NVolSparseEnabled(vol) &&
- vol->major_ver && vol->major_ver < 3)
- ntfs_warning(vol->sb, "Not enabling sparse "
- "support due to NTFS volume "
- "version %i.%i (need at least "
- "version 3.0).", vol->major_ver,
- vol->minor_ver);
- else
- NVolSetSparseEnabled(vol);
- }
- }
- return true;
-needs_arg:
- ntfs_error(vol->sb, "The %s option requires an argument.", p);
- return false;
-needs_bool:
- ntfs_error(vol->sb, "The %s option requires a boolean argument.", p);
- return false;
-needs_val:
- ntfs_error(vol->sb, "Invalid %s option argument: %s", p, ov);
- return false;
-}
-
-#ifdef NTFS_RW
-
-/**
- * ntfs_write_volume_flags - write new flags to the volume information flags
- * @vol: ntfs volume on which to modify the flags
- * @flags: new flags value for the volume information flags
- *
- * Internal function. You probably want to use ntfs_{set,clear}_volume_flags()
- * instead (see below).
- *
- * Replace the volume information flags on the volume @vol with the value
- * supplied in @flags. Note, this overwrites the volume information flags, so
- * make sure to combine the flags you want to modify with the old flags and use
- * the result when calling ntfs_write_volume_flags().
- *
- * Return 0 on success and -errno on error.
- */
-static int ntfs_write_volume_flags(ntfs_volume *vol, const VOLUME_FLAGS flags)
-{
- ntfs_inode *ni = NTFS_I(vol->vol_ino);
- MFT_RECORD *m;
- VOLUME_INFORMATION *vi;
- ntfs_attr_search_ctx *ctx;
- int err;
-
- ntfs_debug("Entering, old flags = 0x%x, new flags = 0x%x.",
- le16_to_cpu(vol->vol_flags), le16_to_cpu(flags));
- if (vol->vol_flags == flags)
- goto done;
- BUG_ON(!ni);
- m = map_mft_record(ni);
- if (IS_ERR(m)) {
- err = PTR_ERR(m);
- goto err_out;
- }
- ctx = ntfs_attr_get_search_ctx(ni, m);
- if (!ctx) {
- err = -ENOMEM;
- goto put_unm_err_out;
- }
- err = ntfs_attr_lookup(AT_VOLUME_INFORMATION, NULL, 0, 0, 0, NULL, 0,
- ctx);
- if (err)
- goto put_unm_err_out;
- vi = (VOLUME_INFORMATION*)((u8*)ctx->attr +
- le16_to_cpu(ctx->attr->data.resident.value_offset));
- vol->vol_flags = vi->flags = flags;
- flush_dcache_mft_record_page(ctx->ntfs_ino);
- mark_mft_record_dirty(ctx->ntfs_ino);
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ni);
-done:
- ntfs_debug("Done.");
- return 0;
-put_unm_err_out:
- if (ctx)
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(ni);
-err_out:
- ntfs_error(vol->sb, "Failed with error code %i.", -err);
- return err;
-}
-
-/**
- * ntfs_set_volume_flags - set bits in the volume information flags
- * @vol: ntfs volume on which to modify the flags
- * @flags: flags to set on the volume
- *
- * Set the bits in @flags in the volume information flags on the volume @vol.
- *
- * Return 0 on success and -errno on error.
- */
-static inline int ntfs_set_volume_flags(ntfs_volume *vol, VOLUME_FLAGS flags)
-{
- flags &= VOLUME_FLAGS_MASK;
- return ntfs_write_volume_flags(vol, vol->vol_flags | flags);
-}
-
-/**
- * ntfs_clear_volume_flags - clear bits in the volume information flags
- * @vol: ntfs volume on which to modify the flags
- * @flags: flags to clear on the volume
- *
- * Clear the bits in @flags in the volume information flags on the volume @vol.
- *
- * Return 0 on success and -errno on error.
- */
-static inline int ntfs_clear_volume_flags(ntfs_volume *vol, VOLUME_FLAGS flags)
-{
- flags &= VOLUME_FLAGS_MASK;
- flags = vol->vol_flags & cpu_to_le16(~le16_to_cpu(flags));
- return ntfs_write_volume_flags(vol, flags);
-}
-
-#endif /* NTFS_RW */
-
-/**
- * ntfs_remount - change the mount options of a mounted ntfs filesystem
- * @sb: superblock of mounted ntfs filesystem
- * @flags: remount flags
- * @opt: remount options string
- *
- * Change the mount options of an already mounted ntfs filesystem.
- *
- * NOTE: The VFS sets the @sb->s_flags remount flags to @flags after
- * ntfs_remount() returns successfully (i.e. returns 0). Otherwise,
- * @sb->s_flags are not changed.
- */
-static int ntfs_remount(struct super_block *sb, int *flags, char *opt)
-{
- ntfs_volume *vol = NTFS_SB(sb);
-
- ntfs_debug("Entering with remount options string: %s", opt);
-
- sync_filesystem(sb);
-
-#ifndef NTFS_RW
- /* For read-only compiled driver, enforce read-only flag. */
- *flags |= SB_RDONLY;
-#else /* NTFS_RW */
- /*
- * For the read-write compiled driver, if we are remounting read-write,
- * make sure there are no volume errors and that no unsupported volume
- * flags are set. Also, empty the logfile journal as it would become
- * stale as soon as something is written to the volume and mark the
- * volume dirty so that chkdsk is run if the volume is not umounted
- * cleanly. Finally, mark the quotas out of date so Windows rescans
- * the volume on boot and updates them.
- *
- * When remounting read-only, mark the volume clean if no volume errors
- * have occurred.
- */
- if (sb_rdonly(sb) && !(*flags & SB_RDONLY)) {
- static const char *es = ". Cannot remount read-write.";
-
- /* Remounting read-write. */
- if (NVolErrors(vol)) {
- ntfs_error(sb, "Volume has errors and is read-only%s",
- es);
- return -EROFS;
- }
- if (vol->vol_flags & VOLUME_IS_DIRTY) {
- ntfs_error(sb, "Volume is dirty and read-only%s", es);
- return -EROFS;
- }
- if (vol->vol_flags & VOLUME_MODIFIED_BY_CHKDSK) {
- ntfs_error(sb, "Volume has been modified by chkdsk "
- "and is read-only%s", es);
- return -EROFS;
- }
- if (vol->vol_flags & VOLUME_MUST_MOUNT_RO_MASK) {
- ntfs_error(sb, "Volume has unsupported flags set "
- "(0x%x) and is read-only%s",
- (unsigned)le16_to_cpu(vol->vol_flags),
- es);
- return -EROFS;
- }
- if (ntfs_set_volume_flags(vol, VOLUME_IS_DIRTY)) {
- ntfs_error(sb, "Failed to set dirty bit in volume "
- "information flags%s", es);
- return -EROFS;
- }
-#if 0
- // TODO: Enable this code once we start modifying anything that
- // is different between NTFS 1.2 and 3.x...
- /* Set NT4 compatibility flag on newer NTFS version volumes. */
- if ((vol->major_ver > 1)) {
- if (ntfs_set_volume_flags(vol, VOLUME_MOUNTED_ON_NT4)) {
- ntfs_error(sb, "Failed to set NT4 "
- "compatibility flag%s", es);
- NVolSetErrors(vol);
- return -EROFS;
- }
- }
-#endif
- if (!ntfs_empty_logfile(vol->logfile_ino)) {
- ntfs_error(sb, "Failed to empty journal $LogFile%s",
- es);
- NVolSetErrors(vol);
- return -EROFS;
- }
- if (!ntfs_mark_quotas_out_of_date(vol)) {
- ntfs_error(sb, "Failed to mark quotas out of date%s",
- es);
- NVolSetErrors(vol);
- return -EROFS;
- }
- if (!ntfs_stamp_usnjrnl(vol)) {
- ntfs_error(sb, "Failed to stamp transaction log "
- "($UsnJrnl)%s", es);
- NVolSetErrors(vol);
- return -EROFS;
- }
- } else if (!sb_rdonly(sb) && (*flags & SB_RDONLY)) {
- /* Remounting read-only. */
- if (!NVolErrors(vol)) {
- if (ntfs_clear_volume_flags(vol, VOLUME_IS_DIRTY))
- ntfs_warning(sb, "Failed to clear dirty bit "
- "in volume information "
- "flags. Run chkdsk.");
- }
- }
-#endif /* NTFS_RW */
-
- // TODO: Deal with *flags.
-
- if (!parse_options(vol, opt))
- return -EINVAL;
-
- ntfs_debug("Done.");
- return 0;
-}
-
-/**
- * is_boot_sector_ntfs - check whether a boot sector is a valid NTFS boot sector
- * @sb: Super block of the device to which @b belongs.
- * @b: Boot sector of device @sb to check.
- * @silent: If 'true', all output will be silenced.
- *
- * is_boot_sector_ntfs() checks whether the boot sector @b is a valid NTFS boot
- * sector. Returns 'true' if it is valid and 'false' if not.
- *
- * @sb is only needed for warning/error output, i.e. it can be NULL when silent
- * is 'true'.
- */
-static bool is_boot_sector_ntfs(const struct super_block *sb,
- const NTFS_BOOT_SECTOR *b, const bool silent)
-{
- /*
- * Check that checksum == sum of u32 values from b to the checksum
- * field. If checksum is zero, no checking is done. We will work when
- * the checksum test fails, since some utilities update the boot sector
- * ignoring the checksum which leaves the checksum out-of-date. We
- * report a warning if this is the case.
- */
- if ((void*)b < (void*)&b->checksum && b->checksum && !silent) {
- le32 *u;
- u32 i;
-
- for (i = 0, u = (le32*)b; u < (le32*)(&b->checksum); ++u)
- i += le32_to_cpup(u);
- if (le32_to_cpu(b->checksum) != i)
- ntfs_warning(sb, "Invalid boot sector checksum.");
- }
- /* Check OEMidentifier is "NTFS " */
- if (b->oem_id != magicNTFS)
- goto not_ntfs;
- /* Check bytes per sector value is between 256 and 4096. */
- if (le16_to_cpu(b->bpb.bytes_per_sector) < 0x100 ||
- le16_to_cpu(b->bpb.bytes_per_sector) > 0x1000)
- goto not_ntfs;
- /* Check sectors per cluster value is valid. */
- switch (b->bpb.sectors_per_cluster) {
- case 1: case 2: case 4: case 8: case 16: case 32: case 64: case 128:
- break;
- default:
- goto not_ntfs;
- }
- /* Check the cluster size is not above the maximum (64kiB). */
- if ((u32)le16_to_cpu(b->bpb.bytes_per_sector) *
- b->bpb.sectors_per_cluster > NTFS_MAX_CLUSTER_SIZE)
- goto not_ntfs;
- /* Check reserved/unused fields are really zero. */
- if (le16_to_cpu(b->bpb.reserved_sectors) ||
- le16_to_cpu(b->bpb.root_entries) ||
- le16_to_cpu(b->bpb.sectors) ||
- le16_to_cpu(b->bpb.sectors_per_fat) ||
- le32_to_cpu(b->bpb.large_sectors) || b->bpb.fats)
- goto not_ntfs;
- /* Check clusters per file mft record value is valid. */
- if ((u8)b->clusters_per_mft_record < 0xe1 ||
- (u8)b->clusters_per_mft_record > 0xf7)
- switch (b->clusters_per_mft_record) {
- case 1: case 2: case 4: case 8: case 16: case 32: case 64:
- break;
- default:
- goto not_ntfs;
- }
- /* Check clusters per index block value is valid. */
- if ((u8)b->clusters_per_index_record < 0xe1 ||
- (u8)b->clusters_per_index_record > 0xf7)
- switch (b->clusters_per_index_record) {
- case 1: case 2: case 4: case 8: case 16: case 32: case 64:
- break;
- default:
- goto not_ntfs;
- }
- /*
- * Check for valid end of sector marker. We will work without it, but
- * many BIOSes will refuse to boot from a bootsector if the magic is
- * incorrect, so we emit a warning.
- */
- if (!silent && b->end_of_sector_marker != cpu_to_le16(0xaa55))
- ntfs_warning(sb, "Invalid end of sector marker.");
- return true;
-not_ntfs:
- return false;
-}
-
-/**
- * read_ntfs_boot_sector - read the NTFS boot sector of a device
- * @sb: super block of device to read the boot sector from
- * @silent: if true, suppress all output
- *
- * Reads the boot sector from the device and validates it. If that fails, tries
- * to read the backup boot sector, first from the end of the device a-la NT4 and
- * later and then from the middle of the device a-la NT3.51 and before.
- *
- * If a valid boot sector is found but it is not the primary boot sector, we
- * repair the primary boot sector silently (unless the device is read-only or
- * the primary boot sector is not accessible).
- *
- * NOTE: To call this function, @sb must have the fields s_dev, the ntfs super
- * block (u.ntfs_sb), nr_blocks and the device flags (s_flags) initialized
- * to their respective values.
- *
- * Return the unlocked buffer head containing the boot sector or NULL on error.
- */
-static struct buffer_head *read_ntfs_boot_sector(struct super_block *sb,
- const int silent)
-{
- const char *read_err_str = "Unable to read %s boot sector.";
- struct buffer_head *bh_primary, *bh_backup;
- sector_t nr_blocks = NTFS_SB(sb)->nr_blocks;
-
- /* Try to read primary boot sector. */
- if ((bh_primary = sb_bread(sb, 0))) {
- if (is_boot_sector_ntfs(sb, (NTFS_BOOT_SECTOR*)
- bh_primary->b_data, silent))
- return bh_primary;
- if (!silent)
- ntfs_error(sb, "Primary boot sector is invalid.");
- } else if (!silent)
- ntfs_error(sb, read_err_str, "primary");
- if (!(NTFS_SB(sb)->on_errors & ON_ERRORS_RECOVER)) {
- if (bh_primary)
- brelse(bh_primary);
- if (!silent)
- ntfs_error(sb, "Mount option errors=recover not used. "
- "Aborting without trying to recover.");
- return NULL;
- }
- /* Try to read NT4+ backup boot sector. */
- if ((bh_backup = sb_bread(sb, nr_blocks - 1))) {
- if (is_boot_sector_ntfs(sb, (NTFS_BOOT_SECTOR*)
- bh_backup->b_data, silent))
- goto hotfix_primary_boot_sector;
- brelse(bh_backup);
- } else if (!silent)
- ntfs_error(sb, read_err_str, "backup");
- /* Try to read NT3.51- backup boot sector. */
- if ((bh_backup = sb_bread(sb, nr_blocks >> 1))) {
- if (is_boot_sector_ntfs(sb, (NTFS_BOOT_SECTOR*)
- bh_backup->b_data, silent))
- goto hotfix_primary_boot_sector;
- if (!silent)
- ntfs_error(sb, "Could not find a valid backup boot "
- "sector.");
- brelse(bh_backup);
- } else if (!silent)
- ntfs_error(sb, read_err_str, "backup");
- /* We failed. Cleanup and return. */
- if (bh_primary)
- brelse(bh_primary);
- return NULL;
-hotfix_primary_boot_sector:
- if (bh_primary) {
- /*
- * If we managed to read sector zero and the volume is not
- * read-only, copy the found, valid backup boot sector to the
- * primary boot sector. Note we only copy the actual boot
- * sector structure, not the actual whole device sector as that
- * may be bigger and would potentially damage the $Boot system
- * file (FIXME: Would be nice to know if the backup boot sector
- * on a large sector device contains the whole boot loader or
- * just the first 512 bytes).
- */
- if (!sb_rdonly(sb)) {
- ntfs_warning(sb, "Hot-fix: Recovering invalid primary "
- "boot sector from backup copy.");
- memcpy(bh_primary->b_data, bh_backup->b_data,
- NTFS_BLOCK_SIZE);
- mark_buffer_dirty(bh_primary);
- sync_dirty_buffer(bh_primary);
- if (buffer_uptodate(bh_primary)) {
- brelse(bh_backup);
- return bh_primary;
- }
- ntfs_error(sb, "Hot-fix: Device write error while "
- "recovering primary boot sector.");
- } else {
- ntfs_warning(sb, "Hot-fix: Recovery of primary boot "
- "sector failed: Read-only mount.");
- }
- brelse(bh_primary);
- }
- ntfs_warning(sb, "Using backup boot sector.");
- return bh_backup;
-}
-
-/**
- * parse_ntfs_boot_sector - parse the boot sector and store the data in @vol
- * @vol: volume structure to initialise with data from boot sector
- * @b: boot sector to parse
- *
- * Parse the ntfs boot sector @b and store all imporant information therein in
- * the ntfs super block @vol. Return 'true' on success and 'false' on error.
- */
-static bool parse_ntfs_boot_sector(ntfs_volume *vol, const NTFS_BOOT_SECTOR *b)
-{
- unsigned int sectors_per_cluster_bits, nr_hidden_sects;
- int clusters_per_mft_record, clusters_per_index_record;
- s64 ll;
-
- vol->sector_size = le16_to_cpu(b->bpb.bytes_per_sector);
- vol->sector_size_bits = ffs(vol->sector_size) - 1;
- ntfs_debug("vol->sector_size = %i (0x%x)", vol->sector_size,
- vol->sector_size);
- ntfs_debug("vol->sector_size_bits = %i (0x%x)", vol->sector_size_bits,
- vol->sector_size_bits);
- if (vol->sector_size < vol->sb->s_blocksize) {
- ntfs_error(vol->sb, "Sector size (%i) is smaller than the "
- "device block size (%lu). This is not "
- "supported. Sorry.", vol->sector_size,
- vol->sb->s_blocksize);
- return false;
- }
- ntfs_debug("sectors_per_cluster = 0x%x", b->bpb.sectors_per_cluster);
- sectors_per_cluster_bits = ffs(b->bpb.sectors_per_cluster) - 1;
- ntfs_debug("sectors_per_cluster_bits = 0x%x",
- sectors_per_cluster_bits);
- nr_hidden_sects = le32_to_cpu(b->bpb.hidden_sectors);
- ntfs_debug("number of hidden sectors = 0x%x", nr_hidden_sects);
- vol->cluster_size = vol->sector_size << sectors_per_cluster_bits;
- vol->cluster_size_mask = vol->cluster_size - 1;
- vol->cluster_size_bits = ffs(vol->cluster_size) - 1;
- ntfs_debug("vol->cluster_size = %i (0x%x)", vol->cluster_size,
- vol->cluster_size);
- ntfs_debug("vol->cluster_size_mask = 0x%x", vol->cluster_size_mask);
- ntfs_debug("vol->cluster_size_bits = %i", vol->cluster_size_bits);
- if (vol->cluster_size < vol->sector_size) {
- ntfs_error(vol->sb, "Cluster size (%i) is smaller than the "
- "sector size (%i). This is not supported. "
- "Sorry.", vol->cluster_size, vol->sector_size);
- return false;
- }
- clusters_per_mft_record = b->clusters_per_mft_record;
- ntfs_debug("clusters_per_mft_record = %i (0x%x)",
- clusters_per_mft_record, clusters_per_mft_record);
- if (clusters_per_mft_record > 0)
- vol->mft_record_size = vol->cluster_size <<
- (ffs(clusters_per_mft_record) - 1);
- else
- /*
- * When mft_record_size < cluster_size, clusters_per_mft_record
- * = -log2(mft_record_size) bytes. mft_record_size normaly is
- * 1024 bytes, which is encoded as 0xF6 (-10 in decimal).
- */
- vol->mft_record_size = 1 << -clusters_per_mft_record;
- vol->mft_record_size_mask = vol->mft_record_size - 1;
- vol->mft_record_size_bits = ffs(vol->mft_record_size) - 1;
- ntfs_debug("vol->mft_record_size = %i (0x%x)", vol->mft_record_size,
- vol->mft_record_size);
- ntfs_debug("vol->mft_record_size_mask = 0x%x",
- vol->mft_record_size_mask);
- ntfs_debug("vol->mft_record_size_bits = %i (0x%x)",
- vol->mft_record_size_bits, vol->mft_record_size_bits);
- /*
- * We cannot support mft record sizes above the PAGE_SIZE since
- * we store $MFT/$DATA, the table of mft records in the page cache.
- */
- if (vol->mft_record_size > PAGE_SIZE) {
- ntfs_error(vol->sb, "Mft record size (%i) exceeds the "
- "PAGE_SIZE on your system (%lu). "
- "This is not supported. Sorry.",
- vol->mft_record_size, PAGE_SIZE);
- return false;
- }
- /* We cannot support mft record sizes below the sector size. */
- if (vol->mft_record_size < vol->sector_size) {
- ntfs_error(vol->sb, "Mft record size (%i) is smaller than the "
- "sector size (%i). This is not supported. "
- "Sorry.", vol->mft_record_size,
- vol->sector_size);
- return false;
- }
- clusters_per_index_record = b->clusters_per_index_record;
- ntfs_debug("clusters_per_index_record = %i (0x%x)",
- clusters_per_index_record, clusters_per_index_record);
- if (clusters_per_index_record > 0)
- vol->index_record_size = vol->cluster_size <<
- (ffs(clusters_per_index_record) - 1);
- else
- /*
- * When index_record_size < cluster_size,
- * clusters_per_index_record = -log2(index_record_size) bytes.
- * index_record_size normaly equals 4096 bytes, which is
- * encoded as 0xF4 (-12 in decimal).
- */
- vol->index_record_size = 1 << -clusters_per_index_record;
- vol->index_record_size_mask = vol->index_record_size - 1;
- vol->index_record_size_bits = ffs(vol->index_record_size) - 1;
- ntfs_debug("vol->index_record_size = %i (0x%x)",
- vol->index_record_size, vol->index_record_size);
- ntfs_debug("vol->index_record_size_mask = 0x%x",
- vol->index_record_size_mask);
- ntfs_debug("vol->index_record_size_bits = %i (0x%x)",
- vol->index_record_size_bits,
- vol->index_record_size_bits);
- /* We cannot support index record sizes below the sector size. */
- if (vol->index_record_size < vol->sector_size) {
- ntfs_error(vol->sb, "Index record size (%i) is smaller than "
- "the sector size (%i). This is not "
- "supported. Sorry.", vol->index_record_size,
- vol->sector_size);
- return false;
- }
- /*
- * Get the size of the volume in clusters and check for 64-bit-ness.
- * Windows currently only uses 32 bits to save the clusters so we do
- * the same as it is much faster on 32-bit CPUs.
- */
- ll = sle64_to_cpu(b->number_of_sectors) >> sectors_per_cluster_bits;
- if ((u64)ll >= 1ULL << 32) {
- ntfs_error(vol->sb, "Cannot handle 64-bit clusters. Sorry.");
- return false;
- }
- vol->nr_clusters = ll;
- ntfs_debug("vol->nr_clusters = 0x%llx", (long long)vol->nr_clusters);
- /*
- * On an architecture where unsigned long is 32-bits, we restrict the
- * volume size to 2TiB (2^41). On a 64-bit architecture, the compiler
- * will hopefully optimize the whole check away.
- */
- if (sizeof(unsigned long) < 8) {
- if ((ll << vol->cluster_size_bits) >= (1ULL << 41)) {
- ntfs_error(vol->sb, "Volume size (%lluTiB) is too "
- "large for this architecture. "
- "Maximum supported is 2TiB. Sorry.",
- (unsigned long long)ll >> (40 -
- vol->cluster_size_bits));
- return false;
- }
- }
- ll = sle64_to_cpu(b->mft_lcn);
- if (ll >= vol->nr_clusters) {
- ntfs_error(vol->sb, "MFT LCN (%lli, 0x%llx) is beyond end of "
- "volume. Weird.", (unsigned long long)ll,
- (unsigned long long)ll);
- return false;
- }
- vol->mft_lcn = ll;
- ntfs_debug("vol->mft_lcn = 0x%llx", (long long)vol->mft_lcn);
- ll = sle64_to_cpu(b->mftmirr_lcn);
- if (ll >= vol->nr_clusters) {
- ntfs_error(vol->sb, "MFTMirr LCN (%lli, 0x%llx) is beyond end "
- "of volume. Weird.", (unsigned long long)ll,
- (unsigned long long)ll);
- return false;
- }
- vol->mftmirr_lcn = ll;
- ntfs_debug("vol->mftmirr_lcn = 0x%llx", (long long)vol->mftmirr_lcn);
-#ifdef NTFS_RW
- /*
- * Work out the size of the mft mirror in number of mft records. If the
- * cluster size is less than or equal to the size taken by four mft
- * records, the mft mirror stores the first four mft records. If the
- * cluster size is bigger than the size taken by four mft records, the
- * mft mirror contains as many mft records as will fit into one
- * cluster.
- */
- if (vol->cluster_size <= (4 << vol->mft_record_size_bits))
- vol->mftmirr_size = 4;
- else
- vol->mftmirr_size = vol->cluster_size >>
- vol->mft_record_size_bits;
- ntfs_debug("vol->mftmirr_size = %i", vol->mftmirr_size);
-#endif /* NTFS_RW */
- vol->serial_no = le64_to_cpu(b->volume_serial_number);
- ntfs_debug("vol->serial_no = 0x%llx",
- (unsigned long long)vol->serial_no);
- return true;
-}
-
-/**
- * ntfs_setup_allocators - initialize the cluster and mft allocators
- * @vol: volume structure for which to setup the allocators
- *
- * Setup the cluster (lcn) and mft allocators to the starting values.
- */
-static void ntfs_setup_allocators(ntfs_volume *vol)
-{
-#ifdef NTFS_RW
- LCN mft_zone_size, mft_lcn;
-#endif /* NTFS_RW */
-
- ntfs_debug("vol->mft_zone_multiplier = 0x%x",
- vol->mft_zone_multiplier);
-#ifdef NTFS_RW
- /* Determine the size of the MFT zone. */
- mft_zone_size = vol->nr_clusters;
- switch (vol->mft_zone_multiplier) { /* % of volume size in clusters */
- case 4:
- mft_zone_size >>= 1; /* 50% */
- break;
- case 3:
- mft_zone_size = (mft_zone_size +
- (mft_zone_size >> 1)) >> 2; /* 37.5% */
- break;
- case 2:
- mft_zone_size >>= 2; /* 25% */
- break;
- /* case 1: */
- default:
- mft_zone_size >>= 3; /* 12.5% */
- break;
- }
- /* Setup the mft zone. */
- vol->mft_zone_start = vol->mft_zone_pos = vol->mft_lcn;
- ntfs_debug("vol->mft_zone_pos = 0x%llx",
- (unsigned long long)vol->mft_zone_pos);
- /*
- * Calculate the mft_lcn for an unmodified NTFS volume (see mkntfs
- * source) and if the actual mft_lcn is in the expected place or even
- * further to the front of the volume, extend the mft_zone to cover the
- * beginning of the volume as well. This is in order to protect the
- * area reserved for the mft bitmap as well within the mft_zone itself.
- * On non-standard volumes we do not protect it as the overhead would
- * be higher than the speed increase we would get by doing it.
- */
- mft_lcn = (8192 + 2 * vol->cluster_size - 1) / vol->cluster_size;
- if (mft_lcn * vol->cluster_size < 16 * 1024)
- mft_lcn = (16 * 1024 + vol->cluster_size - 1) /
- vol->cluster_size;
- if (vol->mft_zone_start <= mft_lcn)
- vol->mft_zone_start = 0;
- ntfs_debug("vol->mft_zone_start = 0x%llx",
- (unsigned long long)vol->mft_zone_start);
- /*
- * Need to cap the mft zone on non-standard volumes so that it does
- * not point outside the boundaries of the volume. We do this by
- * halving the zone size until we are inside the volume.
- */
- vol->mft_zone_end = vol->mft_lcn + mft_zone_size;
- while (vol->mft_zone_end >= vol->nr_clusters) {
- mft_zone_size >>= 1;
- vol->mft_zone_end = vol->mft_lcn + mft_zone_size;
- }
- ntfs_debug("vol->mft_zone_end = 0x%llx",
- (unsigned long long)vol->mft_zone_end);
- /*
- * Set the current position within each data zone to the start of the
- * respective zone.
- */
- vol->data1_zone_pos = vol->mft_zone_end;
- ntfs_debug("vol->data1_zone_pos = 0x%llx",
- (unsigned long long)vol->data1_zone_pos);
- vol->data2_zone_pos = 0;
- ntfs_debug("vol->data2_zone_pos = 0x%llx",
- (unsigned long long)vol->data2_zone_pos);
-
- /* Set the mft data allocation position to mft record 24. */
- vol->mft_data_pos = 24;
- ntfs_debug("vol->mft_data_pos = 0x%llx",
- (unsigned long long)vol->mft_data_pos);
-#endif /* NTFS_RW */
-}
-
-#ifdef NTFS_RW
-
-/**
- * load_and_init_mft_mirror - load and setup the mft mirror inode for a volume
- * @vol: ntfs super block describing device whose mft mirror to load
- *
- * Return 'true' on success or 'false' on error.
- */
-static bool load_and_init_mft_mirror(ntfs_volume *vol)
-{
- struct inode *tmp_ino;
- ntfs_inode *tmp_ni;
-
- ntfs_debug("Entering.");
- /* Get mft mirror inode. */
- tmp_ino = ntfs_iget(vol->sb, FILE_MFTMirr);
- if (IS_ERR(tmp_ino) || is_bad_inode(tmp_ino)) {
- if (!IS_ERR(tmp_ino))
- iput(tmp_ino);
- /* Caller will display error message. */
- return false;
- }
- /*
- * Re-initialize some specifics about $MFTMirr's inode as
- * ntfs_read_inode() will have set up the default ones.
- */
- /* Set uid and gid to root. */
- tmp_ino->i_uid = GLOBAL_ROOT_UID;
- tmp_ino->i_gid = GLOBAL_ROOT_GID;
- /* Regular file. No access for anyone. */
- tmp_ino->i_mode = S_IFREG;
- /* No VFS initiated operations allowed for $MFTMirr. */
- tmp_ino->i_op = &ntfs_empty_inode_ops;
- tmp_ino->i_fop = &ntfs_empty_file_ops;
- /* Put in our special address space operations. */
- tmp_ino->i_mapping->a_ops = &ntfs_mst_aops;
- tmp_ni = NTFS_I(tmp_ino);
- /* The $MFTMirr, like the $MFT is multi sector transfer protected. */
- NInoSetMstProtected(tmp_ni);
- NInoSetSparseDisabled(tmp_ni);
- /*
- * Set up our little cheat allowing us to reuse the async read io
- * completion handler for directories.
- */
- tmp_ni->itype.index.block_size = vol->mft_record_size;
- tmp_ni->itype.index.block_size_bits = vol->mft_record_size_bits;
- vol->mftmirr_ino = tmp_ino;
- ntfs_debug("Done.");
- return true;
-}
-
-/**
- * check_mft_mirror - compare contents of the mft mirror with the mft
- * @vol: ntfs super block describing device whose mft mirror to check
- *
- * Return 'true' on success or 'false' on error.
- *
- * Note, this function also results in the mft mirror runlist being completely
- * mapped into memory. The mft mirror write code requires this and will BUG()
- * should it find an unmapped runlist element.
- */
-static bool check_mft_mirror(ntfs_volume *vol)
-{
- struct super_block *sb = vol->sb;
- ntfs_inode *mirr_ni;
- struct page *mft_page, *mirr_page;
- u8 *kmft, *kmirr;
- runlist_element *rl, rl2[2];
- pgoff_t index;
- int mrecs_per_page, i;
-
- ntfs_debug("Entering.");
- /* Compare contents of $MFT and $MFTMirr. */
- mrecs_per_page = PAGE_SIZE / vol->mft_record_size;
- BUG_ON(!mrecs_per_page);
- BUG_ON(!vol->mftmirr_size);
- mft_page = mirr_page = NULL;
- kmft = kmirr = NULL;
- index = i = 0;
- do {
- u32 bytes;
-
- /* Switch pages if necessary. */
- if (!(i % mrecs_per_page)) {
- if (index) {
- ntfs_unmap_page(mft_page);
- ntfs_unmap_page(mirr_page);
- }
- /* Get the $MFT page. */
- mft_page = ntfs_map_page(vol->mft_ino->i_mapping,
- index);
- if (IS_ERR(mft_page)) {
- ntfs_error(sb, "Failed to read $MFT.");
- return false;
- }
- kmft = page_address(mft_page);
- /* Get the $MFTMirr page. */
- mirr_page = ntfs_map_page(vol->mftmirr_ino->i_mapping,
- index);
- if (IS_ERR(mirr_page)) {
- ntfs_error(sb, "Failed to read $MFTMirr.");
- goto mft_unmap_out;
- }
- kmirr = page_address(mirr_page);
- ++index;
- }
- /* Do not check the record if it is not in use. */
- if (((MFT_RECORD*)kmft)->flags & MFT_RECORD_IN_USE) {
- /* Make sure the record is ok. */
- if (ntfs_is_baad_recordp((le32*)kmft)) {
- ntfs_error(sb, "Incomplete multi sector "
- "transfer detected in mft "
- "record %i.", i);
-mm_unmap_out:
- ntfs_unmap_page(mirr_page);
-mft_unmap_out:
- ntfs_unmap_page(mft_page);
- return false;
- }
- }
- /* Do not check the mirror record if it is not in use. */
- if (((MFT_RECORD*)kmirr)->flags & MFT_RECORD_IN_USE) {
- if (ntfs_is_baad_recordp((le32*)kmirr)) {
- ntfs_error(sb, "Incomplete multi sector "
- "transfer detected in mft "
- "mirror record %i.", i);
- goto mm_unmap_out;
- }
- }
- /* Get the amount of data in the current record. */
- bytes = le32_to_cpu(((MFT_RECORD*)kmft)->bytes_in_use);
- if (bytes < sizeof(MFT_RECORD_OLD) ||
- bytes > vol->mft_record_size ||
- ntfs_is_baad_recordp((le32*)kmft)) {
- bytes = le32_to_cpu(((MFT_RECORD*)kmirr)->bytes_in_use);
- if (bytes < sizeof(MFT_RECORD_OLD) ||
- bytes > vol->mft_record_size ||
- ntfs_is_baad_recordp((le32*)kmirr))
- bytes = vol->mft_record_size;
- }
- /* Compare the two records. */
- if (memcmp(kmft, kmirr, bytes)) {
- ntfs_error(sb, "$MFT and $MFTMirr (record %i) do not "
- "match. Run ntfsfix or chkdsk.", i);
- goto mm_unmap_out;
- }
- kmft += vol->mft_record_size;
- kmirr += vol->mft_record_size;
- } while (++i < vol->mftmirr_size);
- /* Release the last pages. */
- ntfs_unmap_page(mft_page);
- ntfs_unmap_page(mirr_page);
-
- /* Construct the mft mirror runlist by hand. */
- rl2[0].vcn = 0;
- rl2[0].lcn = vol->mftmirr_lcn;
- rl2[0].length = (vol->mftmirr_size * vol->mft_record_size +
- vol->cluster_size - 1) / vol->cluster_size;
- rl2[1].vcn = rl2[0].length;
- rl2[1].lcn = LCN_ENOENT;
- rl2[1].length = 0;
- /*
- * Because we have just read all of the mft mirror, we know we have
- * mapped the full runlist for it.
- */
- mirr_ni = NTFS_I(vol->mftmirr_ino);
- down_read(&mirr_ni->runlist.lock);
- rl = mirr_ni->runlist.rl;
- /* Compare the two runlists. They must be identical. */
- i = 0;
- do {
- if (rl2[i].vcn != rl[i].vcn || rl2[i].lcn != rl[i].lcn ||
- rl2[i].length != rl[i].length) {
- ntfs_error(sb, "$MFTMirr location mismatch. "
- "Run chkdsk.");
- up_read(&mirr_ni->runlist.lock);
- return false;
- }
- } while (rl2[i++].length);
- up_read(&mirr_ni->runlist.lock);
- ntfs_debug("Done.");
- return true;
-}
-
-/**
- * load_and_check_logfile - load and check the logfile inode for a volume
- * @vol: ntfs super block describing device whose logfile to load
- *
- * Return 'true' on success or 'false' on error.
- */
-static bool load_and_check_logfile(ntfs_volume *vol,
- RESTART_PAGE_HEADER **rp)
-{
- struct inode *tmp_ino;
-
- ntfs_debug("Entering.");
- tmp_ino = ntfs_iget(vol->sb, FILE_LogFile);
- if (IS_ERR(tmp_ino) || is_bad_inode(tmp_ino)) {
- if (!IS_ERR(tmp_ino))
- iput(tmp_ino);
- /* Caller will display error message. */
- return false;
- }
- if (!ntfs_check_logfile(tmp_ino, rp)) {
- iput(tmp_ino);
- /* ntfs_check_logfile() will have displayed error output. */
- return false;
- }
- NInoSetSparseDisabled(NTFS_I(tmp_ino));
- vol->logfile_ino = tmp_ino;
- ntfs_debug("Done.");
- return true;
-}
-
-#define NTFS_HIBERFIL_HEADER_SIZE 4096
-
-/**
- * check_windows_hibernation_status - check if Windows is suspended on a volume
- * @vol: ntfs super block of device to check
- *
- * Check if Windows is hibernated on the ntfs volume @vol. This is done by
- * looking for the file hiberfil.sys in the root directory of the volume. If
- * the file is not present Windows is definitely not suspended.
- *
- * If hiberfil.sys exists and is less than 4kiB in size it means Windows is
- * definitely suspended (this volume is not the system volume). Caveat: on a
- * system with many volumes it is possible that the < 4kiB check is bogus but
- * for now this should do fine.
- *
- * If hiberfil.sys exists and is larger than 4kiB in size, we need to read the
- * hiberfil header (which is the first 4kiB). If this begins with "hibr",
- * Windows is definitely suspended. If it is completely full of zeroes,
- * Windows is definitely not hibernated. Any other case is treated as if
- * Windows is suspended. This caters for the above mentioned caveat of a
- * system with many volumes where no "hibr" magic would be present and there is
- * no zero header.
- *
- * Return 0 if Windows is not hibernated on the volume, >0 if Windows is
- * hibernated on the volume, and -errno on error.
- */
-static int check_windows_hibernation_status(ntfs_volume *vol)
-{
- MFT_REF mref;
- struct inode *vi;
- struct page *page;
- u32 *kaddr, *kend;
- ntfs_name *name = NULL;
- int ret = 1;
- static const ntfschar hiberfil[13] = { cpu_to_le16('h'),
- cpu_to_le16('i'), cpu_to_le16('b'),
- cpu_to_le16('e'), cpu_to_le16('r'),
- cpu_to_le16('f'), cpu_to_le16('i'),
- cpu_to_le16('l'), cpu_to_le16('.'),
- cpu_to_le16('s'), cpu_to_le16('y'),
- cpu_to_le16('s'), 0 };
-
- ntfs_debug("Entering.");
- /*
- * Find the inode number for the hibernation file by looking up the
- * filename hiberfil.sys in the root directory.
- */
- inode_lock(vol->root_ino);
- mref = ntfs_lookup_inode_by_name(NTFS_I(vol->root_ino), hiberfil, 12,
- &name);
- inode_unlock(vol->root_ino);
- if (IS_ERR_MREF(mref)) {
- ret = MREF_ERR(mref);
- /* If the file does not exist, Windows is not hibernated. */
- if (ret == -ENOENT) {
- ntfs_debug("hiberfil.sys not present. Windows is not "
- "hibernated on the volume.");
- return 0;
- }
- /* A real error occurred. */
- ntfs_error(vol->sb, "Failed to find inode number for "
- "hiberfil.sys.");
- return ret;
- }
- /* We do not care for the type of match that was found. */
- kfree(name);
- /* Get the inode. */
- vi = ntfs_iget(vol->sb, MREF(mref));
- if (IS_ERR(vi) || is_bad_inode(vi)) {
- if (!IS_ERR(vi))
- iput(vi);
- ntfs_error(vol->sb, "Failed to load hiberfil.sys.");
- return IS_ERR(vi) ? PTR_ERR(vi) : -EIO;
- }
- if (unlikely(i_size_read(vi) < NTFS_HIBERFIL_HEADER_SIZE)) {
- ntfs_debug("hiberfil.sys is smaller than 4kiB (0x%llx). "
- "Windows is hibernated on the volume. This "
- "is not the system volume.", i_size_read(vi));
- goto iput_out;
- }
- page = ntfs_map_page(vi->i_mapping, 0);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to read from hiberfil.sys.");
- ret = PTR_ERR(page);
- goto iput_out;
- }
- kaddr = (u32*)page_address(page);
- if (*(le32*)kaddr == cpu_to_le32(0x72626968)/*'hibr'*/) {
- ntfs_debug("Magic \"hibr\" found in hiberfil.sys. Windows is "
- "hibernated on the volume. This is the "
- "system volume.");
- goto unm_iput_out;
- }
- kend = kaddr + NTFS_HIBERFIL_HEADER_SIZE/sizeof(*kaddr);
- do {
- if (unlikely(*kaddr)) {
- ntfs_debug("hiberfil.sys is larger than 4kiB "
- "(0x%llx), does not contain the "
- "\"hibr\" magic, and does not have a "
- "zero header. Windows is hibernated "
- "on the volume. This is not the "
- "system volume.", i_size_read(vi));
- goto unm_iput_out;
- }
- } while (++kaddr < kend);
- ntfs_debug("hiberfil.sys contains a zero header. Windows is not "
- "hibernated on the volume. This is the system "
- "volume.");
- ret = 0;
-unm_iput_out:
- ntfs_unmap_page(page);
-iput_out:
- iput(vi);
- return ret;
-}
-
-/**
- * load_and_init_quota - load and setup the quota file for a volume if present
- * @vol: ntfs super block describing device whose quota file to load
- *
- * Return 'true' on success or 'false' on error. If $Quota is not present, we
- * leave vol->quota_ino as NULL and return success.
- */
-static bool load_and_init_quota(ntfs_volume *vol)
-{
- MFT_REF mref;
- struct inode *tmp_ino;
- ntfs_name *name = NULL;
- static const ntfschar Quota[7] = { cpu_to_le16('$'),
- cpu_to_le16('Q'), cpu_to_le16('u'),
- cpu_to_le16('o'), cpu_to_le16('t'),
- cpu_to_le16('a'), 0 };
- static ntfschar Q[3] = { cpu_to_le16('$'),
- cpu_to_le16('Q'), 0 };
-
- ntfs_debug("Entering.");
- /*
- * Find the inode number for the quota file by looking up the filename
- * $Quota in the extended system files directory $Extend.
- */
- inode_lock(vol->extend_ino);
- mref = ntfs_lookup_inode_by_name(NTFS_I(vol->extend_ino), Quota, 6,
- &name);
- inode_unlock(vol->extend_ino);
- if (IS_ERR_MREF(mref)) {
- /*
- * If the file does not exist, quotas are disabled and have
- * never been enabled on this volume, just return success.
- */
- if (MREF_ERR(mref) == -ENOENT) {
- ntfs_debug("$Quota not present. Volume does not have "
- "quotas enabled.");
- /*
- * No need to try to set quotas out of date if they are
- * not enabled.
- */
- NVolSetQuotaOutOfDate(vol);
- return true;
- }
- /* A real error occurred. */
- ntfs_error(vol->sb, "Failed to find inode number for $Quota.");
- return false;
- }
- /* We do not care for the type of match that was found. */
- kfree(name);
- /* Get the inode. */
- tmp_ino = ntfs_iget(vol->sb, MREF(mref));
- if (IS_ERR(tmp_ino) || is_bad_inode(tmp_ino)) {
- if (!IS_ERR(tmp_ino))
- iput(tmp_ino);
- ntfs_error(vol->sb, "Failed to load $Quota.");
- return false;
- }
- vol->quota_ino = tmp_ino;
- /* Get the $Q index allocation attribute. */
- tmp_ino = ntfs_index_iget(vol->quota_ino, Q, 2);
- if (IS_ERR(tmp_ino)) {
- ntfs_error(vol->sb, "Failed to load $Quota/$Q index.");
- return false;
- }
- vol->quota_q_ino = tmp_ino;
- ntfs_debug("Done.");
- return true;
-}
-
-/**
- * load_and_init_usnjrnl - load and setup the transaction log if present
- * @vol: ntfs super block describing device whose usnjrnl file to load
- *
- * Return 'true' on success or 'false' on error.
- *
- * If $UsnJrnl is not present or in the process of being disabled, we set
- * NVolUsnJrnlStamped() and return success.
- *
- * If the $UsnJrnl $DATA/$J attribute has a size equal to the lowest valid usn,
- * i.e. transaction logging has only just been enabled or the journal has been
- * stamped and nothing has been logged since, we also set NVolUsnJrnlStamped()
- * and return success.
- */
-static bool load_and_init_usnjrnl(ntfs_volume *vol)
-{
- MFT_REF mref;
- struct inode *tmp_ino;
- ntfs_inode *tmp_ni;
- struct page *page;
- ntfs_name *name = NULL;
- USN_HEADER *uh;
- static const ntfschar UsnJrnl[9] = { cpu_to_le16('$'),
- cpu_to_le16('U'), cpu_to_le16('s'),
- cpu_to_le16('n'), cpu_to_le16('J'),
- cpu_to_le16('r'), cpu_to_le16('n'),
- cpu_to_le16('l'), 0 };
- static ntfschar Max[5] = { cpu_to_le16('$'),
- cpu_to_le16('M'), cpu_to_le16('a'),
- cpu_to_le16('x'), 0 };
- static ntfschar J[3] = { cpu_to_le16('$'),
- cpu_to_le16('J'), 0 };
-
- ntfs_debug("Entering.");
- /*
- * Find the inode number for the transaction log file by looking up the
- * filename $UsnJrnl in the extended system files directory $Extend.
- */
- inode_lock(vol->extend_ino);
- mref = ntfs_lookup_inode_by_name(NTFS_I(vol->extend_ino), UsnJrnl, 8,
- &name);
- inode_unlock(vol->extend_ino);
- if (IS_ERR_MREF(mref)) {
- /*
- * If the file does not exist, transaction logging is disabled,
- * just return success.
- */
- if (MREF_ERR(mref) == -ENOENT) {
- ntfs_debug("$UsnJrnl not present. Volume does not "
- "have transaction logging enabled.");
-not_enabled:
- /*
- * No need to try to stamp the transaction log if
- * transaction logging is not enabled.
- */
- NVolSetUsnJrnlStamped(vol);
- return true;
- }
- /* A real error occurred. */
- ntfs_error(vol->sb, "Failed to find inode number for "
- "$UsnJrnl.");
- return false;
- }
- /* We do not care for the type of match that was found. */
- kfree(name);
- /* Get the inode. */
- tmp_ino = ntfs_iget(vol->sb, MREF(mref));
- if (IS_ERR(tmp_ino) || unlikely(is_bad_inode(tmp_ino))) {
- if (!IS_ERR(tmp_ino))
- iput(tmp_ino);
- ntfs_error(vol->sb, "Failed to load $UsnJrnl.");
- return false;
- }
- vol->usnjrnl_ino = tmp_ino;
- /*
- * If the transaction log is in the process of being deleted, we can
- * ignore it.
- */
- if (unlikely(vol->vol_flags & VOLUME_DELETE_USN_UNDERWAY)) {
- ntfs_debug("$UsnJrnl in the process of being disabled. "
- "Volume does not have transaction logging "
- "enabled.");
- goto not_enabled;
- }
- /* Get the $DATA/$Max attribute. */
- tmp_ino = ntfs_attr_iget(vol->usnjrnl_ino, AT_DATA, Max, 4);
- if (IS_ERR(tmp_ino)) {
- ntfs_error(vol->sb, "Failed to load $UsnJrnl/$DATA/$Max "
- "attribute.");
- return false;
- }
- vol->usnjrnl_max_ino = tmp_ino;
- if (unlikely(i_size_read(tmp_ino) < sizeof(USN_HEADER))) {
- ntfs_error(vol->sb, "Found corrupt $UsnJrnl/$DATA/$Max "
- "attribute (size is 0x%llx but should be at "
- "least 0x%zx bytes).", i_size_read(tmp_ino),
- sizeof(USN_HEADER));
- return false;
- }
- /* Get the $DATA/$J attribute. */
- tmp_ino = ntfs_attr_iget(vol->usnjrnl_ino, AT_DATA, J, 2);
- if (IS_ERR(tmp_ino)) {
- ntfs_error(vol->sb, "Failed to load $UsnJrnl/$DATA/$J "
- "attribute.");
- return false;
- }
- vol->usnjrnl_j_ino = tmp_ino;
- /* Verify $J is non-resident and sparse. */
- tmp_ni = NTFS_I(vol->usnjrnl_j_ino);
- if (unlikely(!NInoNonResident(tmp_ni) || !NInoSparse(tmp_ni))) {
- ntfs_error(vol->sb, "$UsnJrnl/$DATA/$J attribute is resident "
- "and/or not sparse.");
- return false;
- }
- /* Read the USN_HEADER from $DATA/$Max. */
- page = ntfs_map_page(vol->usnjrnl_max_ino->i_mapping, 0);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to read from $UsnJrnl/$DATA/$Max "
- "attribute.");
- return false;
- }
- uh = (USN_HEADER*)page_address(page);
- /* Sanity check the $Max. */
- if (unlikely(sle64_to_cpu(uh->allocation_delta) >
- sle64_to_cpu(uh->maximum_size))) {
- ntfs_error(vol->sb, "Allocation delta (0x%llx) exceeds "
- "maximum size (0x%llx). $UsnJrnl is corrupt.",
- (long long)sle64_to_cpu(uh->allocation_delta),
- (long long)sle64_to_cpu(uh->maximum_size));
- ntfs_unmap_page(page);
- return false;
- }
- /*
- * If the transaction log has been stamped and nothing has been written
- * to it since, we do not need to stamp it.
- */
- if (unlikely(sle64_to_cpu(uh->lowest_valid_usn) >=
- i_size_read(vol->usnjrnl_j_ino))) {
- if (likely(sle64_to_cpu(uh->lowest_valid_usn) ==
- i_size_read(vol->usnjrnl_j_ino))) {
- ntfs_unmap_page(page);
- ntfs_debug("$UsnJrnl is enabled but nothing has been "
- "logged since it was last stamped. "
- "Treating this as if the volume does "
- "not have transaction logging "
- "enabled.");
- goto not_enabled;
- }
- ntfs_error(vol->sb, "$UsnJrnl has lowest valid usn (0x%llx) "
- "which is out of bounds (0x%llx). $UsnJrnl "
- "is corrupt.",
- (long long)sle64_to_cpu(uh->lowest_valid_usn),
- i_size_read(vol->usnjrnl_j_ino));
- ntfs_unmap_page(page);
- return false;
- }
- ntfs_unmap_page(page);
- ntfs_debug("Done.");
- return true;
-}
-
-/**
- * load_and_init_attrdef - load the attribute definitions table for a volume
- * @vol: ntfs super block describing device whose attrdef to load
- *
- * Return 'true' on success or 'false' on error.
- */
-static bool load_and_init_attrdef(ntfs_volume *vol)
-{
- loff_t i_size;
- struct super_block *sb = vol->sb;
- struct inode *ino;
- struct page *page;
- pgoff_t index, max_index;
- unsigned int size;
-
- ntfs_debug("Entering.");
- /* Read attrdef table and setup vol->attrdef and vol->attrdef_size. */
- ino = ntfs_iget(sb, FILE_AttrDef);
- if (IS_ERR(ino) || is_bad_inode(ino)) {
- if (!IS_ERR(ino))
- iput(ino);
- goto failed;
- }
- NInoSetSparseDisabled(NTFS_I(ino));
- /* The size of FILE_AttrDef must be above 0 and fit inside 31 bits. */
- i_size = i_size_read(ino);
- if (i_size <= 0 || i_size > 0x7fffffff)
- goto iput_failed;
- vol->attrdef = (ATTR_DEF*)ntfs_malloc_nofs(i_size);
- if (!vol->attrdef)
- goto iput_failed;
- index = 0;
- max_index = i_size >> PAGE_SHIFT;
- size = PAGE_SIZE;
- while (index < max_index) {
- /* Read the attrdef table and copy it into the linear buffer. */
-read_partial_attrdef_page:
- page = ntfs_map_page(ino->i_mapping, index);
- if (IS_ERR(page))
- goto free_iput_failed;
- memcpy((u8*)vol->attrdef + (index++ << PAGE_SHIFT),
- page_address(page), size);
- ntfs_unmap_page(page);
- };
- if (size == PAGE_SIZE) {
- size = i_size & ~PAGE_MASK;
- if (size)
- goto read_partial_attrdef_page;
- }
- vol->attrdef_size = i_size;
- ntfs_debug("Read %llu bytes from $AttrDef.", i_size);
- iput(ino);
- return true;
-free_iput_failed:
- ntfs_free(vol->attrdef);
- vol->attrdef = NULL;
-iput_failed:
- iput(ino);
-failed:
- ntfs_error(sb, "Failed to initialize attribute definition table.");
- return false;
-}
-
-#endif /* NTFS_RW */
-
-/**
- * load_and_init_upcase - load the upcase table for an ntfs volume
- * @vol: ntfs super block describing device whose upcase to load
- *
- * Return 'true' on success or 'false' on error.
- */
-static bool load_and_init_upcase(ntfs_volume *vol)
-{
- loff_t i_size;
- struct super_block *sb = vol->sb;
- struct inode *ino;
- struct page *page;
- pgoff_t index, max_index;
- unsigned int size;
- int i, max;
-
- ntfs_debug("Entering.");
- /* Read upcase table and setup vol->upcase and vol->upcase_len. */
- ino = ntfs_iget(sb, FILE_UpCase);
- if (IS_ERR(ino) || is_bad_inode(ino)) {
- if (!IS_ERR(ino))
- iput(ino);
- goto upcase_failed;
- }
- /*
- * The upcase size must not be above 64k Unicode characters, must not
- * be zero and must be a multiple of sizeof(ntfschar).
- */
- i_size = i_size_read(ino);
- if (!i_size || i_size & (sizeof(ntfschar) - 1) ||
- i_size > 64ULL * 1024 * sizeof(ntfschar))
- goto iput_upcase_failed;
- vol->upcase = (ntfschar*)ntfs_malloc_nofs(i_size);
- if (!vol->upcase)
- goto iput_upcase_failed;
- index = 0;
- max_index = i_size >> PAGE_SHIFT;
- size = PAGE_SIZE;
- while (index < max_index) {
- /* Read the upcase table and copy it into the linear buffer. */
-read_partial_upcase_page:
- page = ntfs_map_page(ino->i_mapping, index);
- if (IS_ERR(page))
- goto iput_upcase_failed;
- memcpy((char*)vol->upcase + (index++ << PAGE_SHIFT),
- page_address(page), size);
- ntfs_unmap_page(page);
- };
- if (size == PAGE_SIZE) {
- size = i_size & ~PAGE_MASK;
- if (size)
- goto read_partial_upcase_page;
- }
- vol->upcase_len = i_size >> UCHAR_T_SIZE_BITS;
- ntfs_debug("Read %llu bytes from $UpCase (expected %zu bytes).",
- i_size, 64 * 1024 * sizeof(ntfschar));
- iput(ino);
- mutex_lock(&ntfs_lock);
- if (!default_upcase) {
- ntfs_debug("Using volume specified $UpCase since default is "
- "not present.");
- mutex_unlock(&ntfs_lock);
- return true;
- }
- max = default_upcase_len;
- if (max > vol->upcase_len)
- max = vol->upcase_len;
- for (i = 0; i < max; i++)
- if (vol->upcase[i] != default_upcase[i])
- break;
- if (i == max) {
- ntfs_free(vol->upcase);
- vol->upcase = default_upcase;
- vol->upcase_len = max;
- ntfs_nr_upcase_users++;
- mutex_unlock(&ntfs_lock);
- ntfs_debug("Volume specified $UpCase matches default. Using "
- "default.");
- return true;
- }
- mutex_unlock(&ntfs_lock);
- ntfs_debug("Using volume specified $UpCase since it does not match "
- "the default.");
- return true;
-iput_upcase_failed:
- iput(ino);
- ntfs_free(vol->upcase);
- vol->upcase = NULL;
-upcase_failed:
- mutex_lock(&ntfs_lock);
- if (default_upcase) {
- vol->upcase = default_upcase;
- vol->upcase_len = default_upcase_len;
- ntfs_nr_upcase_users++;
- mutex_unlock(&ntfs_lock);
- ntfs_error(sb, "Failed to load $UpCase from the volume. Using "
- "default.");
- return true;
- }
- mutex_unlock(&ntfs_lock);
- ntfs_error(sb, "Failed to initialize upcase table.");
- return false;
-}
-
-/*
- * The lcn and mft bitmap inodes are NTFS-internal inodes with
- * their own special locking rules:
- */
-static struct lock_class_key
- lcnbmp_runlist_lock_key, lcnbmp_mrec_lock_key,
- mftbmp_runlist_lock_key, mftbmp_mrec_lock_key;
-
-/**
- * load_system_files - open the system files using normal functions
- * @vol: ntfs super block describing device whose system files to load
- *
- * Open the system files with normal access functions and complete setting up
- * the ntfs super block @vol.
- *
- * Return 'true' on success or 'false' on error.
- */
-static bool load_system_files(ntfs_volume *vol)
-{
- struct super_block *sb = vol->sb;
- MFT_RECORD *m;
- VOLUME_INFORMATION *vi;
- ntfs_attr_search_ctx *ctx;
-#ifdef NTFS_RW
- RESTART_PAGE_HEADER *rp;
- int err;
-#endif /* NTFS_RW */
-
- ntfs_debug("Entering.");
-#ifdef NTFS_RW
- /* Get mft mirror inode compare the contents of $MFT and $MFTMirr. */
- if (!load_and_init_mft_mirror(vol) || !check_mft_mirror(vol)) {
- static const char *es1 = "Failed to load $MFTMirr";
- static const char *es2 = "$MFTMirr does not match $MFT";
- static const char *es3 = ". Run ntfsfix and/or chkdsk.";
-
- /* If a read-write mount, convert it to a read-only mount. */
- if (!sb_rdonly(sb)) {
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors="
- "continue nor on_errors="
- "remount-ro was specified%s",
- !vol->mftmirr_ino ? es1 : es2,
- es3);
- goto iput_mirr_err_out;
- }
- sb->s_flags |= SB_RDONLY;
- ntfs_error(sb, "%s. Mounting read-only%s",
- !vol->mftmirr_ino ? es1 : es2, es3);
- } else
- ntfs_warning(sb, "%s. Will not be able to remount "
- "read-write%s",
- !vol->mftmirr_ino ? es1 : es2, es3);
- /* This will prevent a read-write remount. */
- NVolSetErrors(vol);
- }
-#endif /* NTFS_RW */
- /* Get mft bitmap attribute inode. */
- vol->mftbmp_ino = ntfs_attr_iget(vol->mft_ino, AT_BITMAP, NULL, 0);
- if (IS_ERR(vol->mftbmp_ino)) {
- ntfs_error(sb, "Failed to load $MFT/$BITMAP attribute.");
- goto iput_mirr_err_out;
- }
- lockdep_set_class(&NTFS_I(vol->mftbmp_ino)->runlist.lock,
- &mftbmp_runlist_lock_key);
- lockdep_set_class(&NTFS_I(vol->mftbmp_ino)->mrec_lock,
- &mftbmp_mrec_lock_key);
- /* Read upcase table and setup @vol->upcase and @vol->upcase_len. */
- if (!load_and_init_upcase(vol))
- goto iput_mftbmp_err_out;
-#ifdef NTFS_RW
- /*
- * Read attribute definitions table and setup @vol->attrdef and
- * @vol->attrdef_size.
- */
- if (!load_and_init_attrdef(vol))
- goto iput_upcase_err_out;
-#endif /* NTFS_RW */
- /*
- * Get the cluster allocation bitmap inode and verify the size, no
- * need for any locking at this stage as we are already running
- * exclusively as we are mount in progress task.
- */
- vol->lcnbmp_ino = ntfs_iget(sb, FILE_Bitmap);
- if (IS_ERR(vol->lcnbmp_ino) || is_bad_inode(vol->lcnbmp_ino)) {
- if (!IS_ERR(vol->lcnbmp_ino))
- iput(vol->lcnbmp_ino);
- goto bitmap_failed;
- }
- lockdep_set_class(&NTFS_I(vol->lcnbmp_ino)->runlist.lock,
- &lcnbmp_runlist_lock_key);
- lockdep_set_class(&NTFS_I(vol->lcnbmp_ino)->mrec_lock,
- &lcnbmp_mrec_lock_key);
-
- NInoSetSparseDisabled(NTFS_I(vol->lcnbmp_ino));
- if ((vol->nr_clusters + 7) >> 3 > i_size_read(vol->lcnbmp_ino)) {
- iput(vol->lcnbmp_ino);
-bitmap_failed:
- ntfs_error(sb, "Failed to load $Bitmap.");
- goto iput_attrdef_err_out;
- }
- /*
- * Get the volume inode and setup our cache of the volume flags and
- * version.
- */
- vol->vol_ino = ntfs_iget(sb, FILE_Volume);
- if (IS_ERR(vol->vol_ino) || is_bad_inode(vol->vol_ino)) {
- if (!IS_ERR(vol->vol_ino))
- iput(vol->vol_ino);
-volume_failed:
- ntfs_error(sb, "Failed to load $Volume.");
- goto iput_lcnbmp_err_out;
- }
- m = map_mft_record(NTFS_I(vol->vol_ino));
- if (IS_ERR(m)) {
-iput_volume_failed:
- iput(vol->vol_ino);
- goto volume_failed;
- }
- if (!(ctx = ntfs_attr_get_search_ctx(NTFS_I(vol->vol_ino), m))) {
- ntfs_error(sb, "Failed to get attribute search context.");
- goto get_ctx_vol_failed;
- }
- if (ntfs_attr_lookup(AT_VOLUME_INFORMATION, NULL, 0, 0, 0, NULL, 0,
- ctx) || ctx->attr->non_resident || ctx->attr->flags) {
-err_put_vol:
- ntfs_attr_put_search_ctx(ctx);
-get_ctx_vol_failed:
- unmap_mft_record(NTFS_I(vol->vol_ino));
- goto iput_volume_failed;
- }
- vi = (VOLUME_INFORMATION*)((char*)ctx->attr +
- le16_to_cpu(ctx->attr->data.resident.value_offset));
- /* Some bounds checks. */
- if ((u8*)vi < (u8*)ctx->attr || (u8*)vi +
- le32_to_cpu(ctx->attr->data.resident.value_length) >
- (u8*)ctx->attr + le32_to_cpu(ctx->attr->length))
- goto err_put_vol;
- /* Copy the volume flags and version to the ntfs_volume structure. */
- vol->vol_flags = vi->flags;
- vol->major_ver = vi->major_ver;
- vol->minor_ver = vi->minor_ver;
- ntfs_attr_put_search_ctx(ctx);
- unmap_mft_record(NTFS_I(vol->vol_ino));
- pr_info("volume version %i.%i.\n", vol->major_ver,
- vol->minor_ver);
- if (vol->major_ver < 3 && NVolSparseEnabled(vol)) {
- ntfs_warning(vol->sb, "Disabling sparse support due to NTFS "
- "volume version %i.%i (need at least version "
- "3.0).", vol->major_ver, vol->minor_ver);
- NVolClearSparseEnabled(vol);
- }
-#ifdef NTFS_RW
- /* Make sure that no unsupported volume flags are set. */
- if (vol->vol_flags & VOLUME_MUST_MOUNT_RO_MASK) {
- static const char *es1a = "Volume is dirty";
- static const char *es1b = "Volume has been modified by chkdsk";
- static const char *es1c = "Volume has unsupported flags set";
- static const char *es2a = ". Run chkdsk and mount in Windows.";
- static const char *es2b = ". Mount in Windows.";
- const char *es1, *es2;
-
- es2 = es2a;
- if (vol->vol_flags & VOLUME_IS_DIRTY)
- es1 = es1a;
- else if (vol->vol_flags & VOLUME_MODIFIED_BY_CHKDSK) {
- es1 = es1b;
- es2 = es2b;
- } else {
- es1 = es1c;
- ntfs_warning(sb, "Unsupported volume flags 0x%x "
- "encountered.",
- (unsigned)le16_to_cpu(vol->vol_flags));
- }
- /* If a read-write mount, convert it to a read-only mount. */
- if (!sb_rdonly(sb)) {
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors="
- "continue nor on_errors="
- "remount-ro was specified%s",
- es1, es2);
- goto iput_vol_err_out;
- }
- sb->s_flags |= SB_RDONLY;
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- } else
- ntfs_warning(sb, "%s. Will not be able to remount "
- "read-write%s", es1, es2);
- /*
- * Do not set NVolErrors() because ntfs_remount() re-checks the
- * flags which we need to do in case any flags have changed.
- */
- }
- /*
- * Get the inode for the logfile, check it and determine if the volume
- * was shutdown cleanly.
- */
- rp = NULL;
- if (!load_and_check_logfile(vol, &rp) ||
- !ntfs_is_logfile_clean(vol->logfile_ino, rp)) {
- static const char *es1a = "Failed to load $LogFile";
- static const char *es1b = "$LogFile is not clean";
- static const char *es2 = ". Mount in Windows.";
- const char *es1;
-
- es1 = !vol->logfile_ino ? es1a : es1b;
- /* If a read-write mount, convert it to a read-only mount. */
- if (!sb_rdonly(sb)) {
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors="
- "continue nor on_errors="
- "remount-ro was specified%s",
- es1, es2);
- if (vol->logfile_ino) {
- BUG_ON(!rp);
- ntfs_free(rp);
- }
- goto iput_logfile_err_out;
- }
- sb->s_flags |= SB_RDONLY;
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- } else
- ntfs_warning(sb, "%s. Will not be able to remount "
- "read-write%s", es1, es2);
- /* This will prevent a read-write remount. */
- NVolSetErrors(vol);
- }
- ntfs_free(rp);
-#endif /* NTFS_RW */
- /* Get the root directory inode so we can do path lookups. */
- vol->root_ino = ntfs_iget(sb, FILE_root);
- if (IS_ERR(vol->root_ino) || is_bad_inode(vol->root_ino)) {
- if (!IS_ERR(vol->root_ino))
- iput(vol->root_ino);
- ntfs_error(sb, "Failed to load root directory.");
- goto iput_logfile_err_out;
- }
-#ifdef NTFS_RW
- /*
- * Check if Windows is suspended to disk on the target volume. If it
- * is hibernated, we must not write *anything* to the disk so set
- * NVolErrors() without setting the dirty volume flag and mount
- * read-only. This will prevent read-write remounting and it will also
- * prevent all writes.
- */
- err = check_windows_hibernation_status(vol);
- if (unlikely(err)) {
- static const char *es1a = "Failed to determine if Windows is "
- "hibernated";
- static const char *es1b = "Windows is hibernated";
- static const char *es2 = ". Run chkdsk.";
- const char *es1;
-
- es1 = err < 0 ? es1a : es1b;
- /* If a read-write mount, convert it to a read-only mount. */
- if (!sb_rdonly(sb)) {
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors="
- "continue nor on_errors="
- "remount-ro was specified%s",
- es1, es2);
- goto iput_root_err_out;
- }
- sb->s_flags |= SB_RDONLY;
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- } else
- ntfs_warning(sb, "%s. Will not be able to remount "
- "read-write%s", es1, es2);
- /* This will prevent a read-write remount. */
- NVolSetErrors(vol);
- }
- /* If (still) a read-write mount, mark the volume dirty. */
- if (!sb_rdonly(sb) && ntfs_set_volume_flags(vol, VOLUME_IS_DIRTY)) {
- static const char *es1 = "Failed to set dirty bit in volume "
- "information flags";
- static const char *es2 = ". Run chkdsk.";
-
- /* Convert to a read-only mount. */
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors=continue nor "
- "on_errors=remount-ro was specified%s",
- es1, es2);
- goto iput_root_err_out;
- }
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- sb->s_flags |= SB_RDONLY;
- /*
- * Do not set NVolErrors() because ntfs_remount() might manage
- * to set the dirty flag in which case all would be well.
- */
- }
-#if 0
- // TODO: Enable this code once we start modifying anything that is
- // different between NTFS 1.2 and 3.x...
- /*
- * If (still) a read-write mount, set the NT4 compatibility flag on
- * newer NTFS version volumes.
- */
- if (!(sb->s_flags & SB_RDONLY) && (vol->major_ver > 1) &&
- ntfs_set_volume_flags(vol, VOLUME_MOUNTED_ON_NT4)) {
- static const char *es1 = "Failed to set NT4 compatibility flag";
- static const char *es2 = ". Run chkdsk.";
-
- /* Convert to a read-only mount. */
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors=continue nor "
- "on_errors=remount-ro was specified%s",
- es1, es2);
- goto iput_root_err_out;
- }
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- sb->s_flags |= SB_RDONLY;
- NVolSetErrors(vol);
- }
-#endif
- /* If (still) a read-write mount, empty the logfile. */
- if (!sb_rdonly(sb) && !ntfs_empty_logfile(vol->logfile_ino)) {
- static const char *es1 = "Failed to empty $LogFile";
- static const char *es2 = ". Mount in Windows.";
-
- /* Convert to a read-only mount. */
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors=continue nor "
- "on_errors=remount-ro was specified%s",
- es1, es2);
- goto iput_root_err_out;
- }
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- sb->s_flags |= SB_RDONLY;
- NVolSetErrors(vol);
- }
-#endif /* NTFS_RW */
- /* If on NTFS versions before 3.0, we are done. */
- if (unlikely(vol->major_ver < 3))
- return true;
- /* NTFS 3.0+ specific initialization. */
- /* Get the security descriptors inode. */
- vol->secure_ino = ntfs_iget(sb, FILE_Secure);
- if (IS_ERR(vol->secure_ino) || is_bad_inode(vol->secure_ino)) {
- if (!IS_ERR(vol->secure_ino))
- iput(vol->secure_ino);
- ntfs_error(sb, "Failed to load $Secure.");
- goto iput_root_err_out;
- }
- // TODO: Initialize security.
- /* Get the extended system files' directory inode. */
- vol->extend_ino = ntfs_iget(sb, FILE_Extend);
- if (IS_ERR(vol->extend_ino) || is_bad_inode(vol->extend_ino) ||
- !S_ISDIR(vol->extend_ino->i_mode)) {
- if (!IS_ERR(vol->extend_ino))
- iput(vol->extend_ino);
- ntfs_error(sb, "Failed to load $Extend.");
- goto iput_sec_err_out;
- }
-#ifdef NTFS_RW
- /* Find the quota file, load it if present, and set it up. */
- if (!load_and_init_quota(vol)) {
- static const char *es1 = "Failed to load $Quota";
- static const char *es2 = ". Run chkdsk.";
-
- /* If a read-write mount, convert it to a read-only mount. */
- if (!sb_rdonly(sb)) {
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors="
- "continue nor on_errors="
- "remount-ro was specified%s",
- es1, es2);
- goto iput_quota_err_out;
- }
- sb->s_flags |= SB_RDONLY;
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- } else
- ntfs_warning(sb, "%s. Will not be able to remount "
- "read-write%s", es1, es2);
- /* This will prevent a read-write remount. */
- NVolSetErrors(vol);
- }
- /* If (still) a read-write mount, mark the quotas out of date. */
- if (!sb_rdonly(sb) && !ntfs_mark_quotas_out_of_date(vol)) {
- static const char *es1 = "Failed to mark quotas out of date";
- static const char *es2 = ". Run chkdsk.";
-
- /* Convert to a read-only mount. */
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors=continue nor "
- "on_errors=remount-ro was specified%s",
- es1, es2);
- goto iput_quota_err_out;
- }
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- sb->s_flags |= SB_RDONLY;
- NVolSetErrors(vol);
- }
- /*
- * Find the transaction log file ($UsnJrnl), load it if present, check
- * it, and set it up.
- */
- if (!load_and_init_usnjrnl(vol)) {
- static const char *es1 = "Failed to load $UsnJrnl";
- static const char *es2 = ". Run chkdsk.";
-
- /* If a read-write mount, convert it to a read-only mount. */
- if (!sb_rdonly(sb)) {
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors="
- "continue nor on_errors="
- "remount-ro was specified%s",
- es1, es2);
- goto iput_usnjrnl_err_out;
- }
- sb->s_flags |= SB_RDONLY;
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- } else
- ntfs_warning(sb, "%s. Will not be able to remount "
- "read-write%s", es1, es2);
- /* This will prevent a read-write remount. */
- NVolSetErrors(vol);
- }
- /* If (still) a read-write mount, stamp the transaction log. */
- if (!sb_rdonly(sb) && !ntfs_stamp_usnjrnl(vol)) {
- static const char *es1 = "Failed to stamp transaction log "
- "($UsnJrnl)";
- static const char *es2 = ". Run chkdsk.";
-
- /* Convert to a read-only mount. */
- if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO |
- ON_ERRORS_CONTINUE))) {
- ntfs_error(sb, "%s and neither on_errors=continue nor "
- "on_errors=remount-ro was specified%s",
- es1, es2);
- goto iput_usnjrnl_err_out;
- }
- ntfs_error(sb, "%s. Mounting read-only%s", es1, es2);
- sb->s_flags |= SB_RDONLY;
- NVolSetErrors(vol);
- }
-#endif /* NTFS_RW */
- return true;
-#ifdef NTFS_RW
-iput_usnjrnl_err_out:
- iput(vol->usnjrnl_j_ino);
- iput(vol->usnjrnl_max_ino);
- iput(vol->usnjrnl_ino);
-iput_quota_err_out:
- iput(vol->quota_q_ino);
- iput(vol->quota_ino);
- iput(vol->extend_ino);
-#endif /* NTFS_RW */
-iput_sec_err_out:
- iput(vol->secure_ino);
-iput_root_err_out:
- iput(vol->root_ino);
-iput_logfile_err_out:
-#ifdef NTFS_RW
- iput(vol->logfile_ino);
-iput_vol_err_out:
-#endif /* NTFS_RW */
- iput(vol->vol_ino);
-iput_lcnbmp_err_out:
- iput(vol->lcnbmp_ino);
-iput_attrdef_err_out:
- vol->attrdef_size = 0;
- if (vol->attrdef) {
- ntfs_free(vol->attrdef);
- vol->attrdef = NULL;
- }
-#ifdef NTFS_RW
-iput_upcase_err_out:
-#endif /* NTFS_RW */
- vol->upcase_len = 0;
- mutex_lock(&ntfs_lock);
- if (vol->upcase == default_upcase) {
- ntfs_nr_upcase_users--;
- vol->upcase = NULL;
- }
- mutex_unlock(&ntfs_lock);
- if (vol->upcase) {
- ntfs_free(vol->upcase);
- vol->upcase = NULL;
- }
-iput_mftbmp_err_out:
- iput(vol->mftbmp_ino);
-iput_mirr_err_out:
-#ifdef NTFS_RW
- iput(vol->mftmirr_ino);
-#endif /* NTFS_RW */
- return false;
-}
-
-/**
- * ntfs_put_super - called by the vfs to unmount a volume
- * @sb: vfs superblock of volume to unmount
- *
- * ntfs_put_super() is called by the VFS (from fs/super.c::do_umount()) when
- * the volume is being unmounted (umount system call has been invoked) and it
- * releases all inodes and memory belonging to the NTFS specific part of the
- * super block.
- */
-static void ntfs_put_super(struct super_block *sb)
-{
- ntfs_volume *vol = NTFS_SB(sb);
-
- ntfs_debug("Entering.");
-
-#ifdef NTFS_RW
- /*
- * Commit all inodes while they are still open in case some of them
- * cause others to be dirtied.
- */
- ntfs_commit_inode(vol->vol_ino);
-
- /* NTFS 3.0+ specific. */
- if (vol->major_ver >= 3) {
- if (vol->usnjrnl_j_ino)
- ntfs_commit_inode(vol->usnjrnl_j_ino);
- if (vol->usnjrnl_max_ino)
- ntfs_commit_inode(vol->usnjrnl_max_ino);
- if (vol->usnjrnl_ino)
- ntfs_commit_inode(vol->usnjrnl_ino);
- if (vol->quota_q_ino)
- ntfs_commit_inode(vol->quota_q_ino);
- if (vol->quota_ino)
- ntfs_commit_inode(vol->quota_ino);
- if (vol->extend_ino)
- ntfs_commit_inode(vol->extend_ino);
- if (vol->secure_ino)
- ntfs_commit_inode(vol->secure_ino);
- }
-
- ntfs_commit_inode(vol->root_ino);
-
- down_write(&vol->lcnbmp_lock);
- ntfs_commit_inode(vol->lcnbmp_ino);
- up_write(&vol->lcnbmp_lock);
-
- down_write(&vol->mftbmp_lock);
- ntfs_commit_inode(vol->mftbmp_ino);
- up_write(&vol->mftbmp_lock);
-
- if (vol->logfile_ino)
- ntfs_commit_inode(vol->logfile_ino);
-
- if (vol->mftmirr_ino)
- ntfs_commit_inode(vol->mftmirr_ino);
- ntfs_commit_inode(vol->mft_ino);
-
- /*
- * If a read-write mount and no volume errors have occurred, mark the
- * volume clean. Also, re-commit all affected inodes.
- */
- if (!sb_rdonly(sb)) {
- if (!NVolErrors(vol)) {
- if (ntfs_clear_volume_flags(vol, VOLUME_IS_DIRTY))
- ntfs_warning(sb, "Failed to clear dirty bit "
- "in volume information "
- "flags. Run chkdsk.");
- ntfs_commit_inode(vol->vol_ino);
- ntfs_commit_inode(vol->root_ino);
- if (vol->mftmirr_ino)
- ntfs_commit_inode(vol->mftmirr_ino);
- ntfs_commit_inode(vol->mft_ino);
- } else {
- ntfs_warning(sb, "Volume has errors. Leaving volume "
- "marked dirty. Run chkdsk.");
- }
- }
-#endif /* NTFS_RW */
-
- iput(vol->vol_ino);
- vol->vol_ino = NULL;
-
- /* NTFS 3.0+ specific clean up. */
- if (vol->major_ver >= 3) {
-#ifdef NTFS_RW
- if (vol->usnjrnl_j_ino) {
- iput(vol->usnjrnl_j_ino);
- vol->usnjrnl_j_ino = NULL;
- }
- if (vol->usnjrnl_max_ino) {
- iput(vol->usnjrnl_max_ino);
- vol->usnjrnl_max_ino = NULL;
- }
- if (vol->usnjrnl_ino) {
- iput(vol->usnjrnl_ino);
- vol->usnjrnl_ino = NULL;
- }
- if (vol->quota_q_ino) {
- iput(vol->quota_q_ino);
- vol->quota_q_ino = NULL;
- }
- if (vol->quota_ino) {
- iput(vol->quota_ino);
- vol->quota_ino = NULL;
- }
-#endif /* NTFS_RW */
- if (vol->extend_ino) {
- iput(vol->extend_ino);
- vol->extend_ino = NULL;
- }
- if (vol->secure_ino) {
- iput(vol->secure_ino);
- vol->secure_ino = NULL;
- }
- }
-
- iput(vol->root_ino);
- vol->root_ino = NULL;
-
- down_write(&vol->lcnbmp_lock);
- iput(vol->lcnbmp_ino);
- vol->lcnbmp_ino = NULL;
- up_write(&vol->lcnbmp_lock);
-
- down_write(&vol->mftbmp_lock);
- iput(vol->mftbmp_ino);
- vol->mftbmp_ino = NULL;
- up_write(&vol->mftbmp_lock);
-
-#ifdef NTFS_RW
- if (vol->logfile_ino) {
- iput(vol->logfile_ino);
- vol->logfile_ino = NULL;
- }
- if (vol->mftmirr_ino) {
- /* Re-commit the mft mirror and mft just in case. */
- ntfs_commit_inode(vol->mftmirr_ino);
- ntfs_commit_inode(vol->mft_ino);
- iput(vol->mftmirr_ino);
- vol->mftmirr_ino = NULL;
- }
- /*
- * We should have no dirty inodes left, due to
- * mft.c::ntfs_mft_writepage() cleaning all the dirty pages as
- * the underlying mft records are written out and cleaned.
- */
- ntfs_commit_inode(vol->mft_ino);
- write_inode_now(vol->mft_ino, 1);
-#endif /* NTFS_RW */
-
- iput(vol->mft_ino);
- vol->mft_ino = NULL;
-
- /* Throw away the table of attribute definitions. */
- vol->attrdef_size = 0;
- if (vol->attrdef) {
- ntfs_free(vol->attrdef);
- vol->attrdef = NULL;
- }
- vol->upcase_len = 0;
- /*
- * Destroy the global default upcase table if necessary. Also decrease
- * the number of upcase users if we are a user.
- */
- mutex_lock(&ntfs_lock);
- if (vol->upcase == default_upcase) {
- ntfs_nr_upcase_users--;
- vol->upcase = NULL;
- }
- if (!ntfs_nr_upcase_users && default_upcase) {
- ntfs_free(default_upcase);
- default_upcase = NULL;
- }
- if (vol->cluster_size <= 4096 && !--ntfs_nr_compression_users)
- free_compression_buffers();
- mutex_unlock(&ntfs_lock);
- if (vol->upcase) {
- ntfs_free(vol->upcase);
- vol->upcase = NULL;
- }
-
- unload_nls(vol->nls_map);
-
- sb->s_fs_info = NULL;
- kfree(vol);
-}
-
-/**
- * get_nr_free_clusters - return the number of free clusters on a volume
- * @vol: ntfs volume for which to obtain free cluster count
- *
- * Calculate the number of free clusters on the mounted NTFS volume @vol. We
- * actually calculate the number of clusters in use instead because this
- * allows us to not care about partial pages as these will be just zero filled
- * and hence not be counted as allocated clusters.
- *
- * The only particularity is that clusters beyond the end of the logical ntfs
- * volume will be marked as allocated to prevent errors which means we have to
- * discount those at the end. This is important as the cluster bitmap always
- * has a size in multiples of 8 bytes, i.e. up to 63 clusters could be outside
- * the logical volume and marked in use when they are not as they do not exist.
- *
- * If any pages cannot be read we assume all clusters in the erroring pages are
- * in use. This means we return an underestimate on errors which is better than
- * an overestimate.
- */
-static s64 get_nr_free_clusters(ntfs_volume *vol)
-{
- s64 nr_free = vol->nr_clusters;
- struct address_space *mapping = vol->lcnbmp_ino->i_mapping;
- struct page *page;
- pgoff_t index, max_index;
-
- ntfs_debug("Entering.");
- /* Serialize accesses to the cluster bitmap. */
- down_read(&vol->lcnbmp_lock);
- /*
- * Convert the number of bits into bytes rounded up, then convert into
- * multiples of PAGE_SIZE, rounding up so that if we have one
- * full and one partial page max_index = 2.
- */
- max_index = (((vol->nr_clusters + 7) >> 3) + PAGE_SIZE - 1) >>
- PAGE_SHIFT;
- /* Use multiples of 4 bytes, thus max_size is PAGE_SIZE / 4. */
- ntfs_debug("Reading $Bitmap, max_index = 0x%lx, max_size = 0x%lx.",
- max_index, PAGE_SIZE / 4);
- for (index = 0; index < max_index; index++) {
- unsigned long *kaddr;
-
- /*
- * Read the page from page cache, getting it from backing store
- * if necessary, and increment the use count.
- */
- page = read_mapping_page(mapping, index, NULL);
- /* Ignore pages which errored synchronously. */
- if (IS_ERR(page)) {
- ntfs_debug("read_mapping_page() error. Skipping "
- "page (index 0x%lx).", index);
- nr_free -= PAGE_SIZE * 8;
- continue;
- }
- kaddr = kmap_atomic(page);
- /*
- * Subtract the number of set bits. If this
- * is the last page and it is partial we don't really care as
- * it just means we do a little extra work but it won't affect
- * the result as all out of range bytes are set to zero by
- * ntfs_readpage().
- */
- nr_free -= bitmap_weight(kaddr,
- PAGE_SIZE * BITS_PER_BYTE);
- kunmap_atomic(kaddr);
- put_page(page);
- }
- ntfs_debug("Finished reading $Bitmap, last index = 0x%lx.", index - 1);
- /*
- * Fixup for eventual bits outside logical ntfs volume (see function
- * description above).
- */
- if (vol->nr_clusters & 63)
- nr_free += 64 - (vol->nr_clusters & 63);
- up_read(&vol->lcnbmp_lock);
- /* If errors occurred we may well have gone below zero, fix this. */
- if (nr_free < 0)
- nr_free = 0;
- ntfs_debug("Exiting.");
- return nr_free;
-}
-
-/**
- * __get_nr_free_mft_records - return the number of free inodes on a volume
- * @vol: ntfs volume for which to obtain free inode count
- * @nr_free: number of mft records in filesystem
- * @max_index: maximum number of pages containing set bits
- *
- * Calculate the number of free mft records (inodes) on the mounted NTFS
- * volume @vol. We actually calculate the number of mft records in use instead
- * because this allows us to not care about partial pages as these will be just
- * zero filled and hence not be counted as allocated mft record.
- *
- * If any pages cannot be read we assume all mft records in the erroring pages
- * are in use. This means we return an underestimate on errors which is better
- * than an overestimate.
- *
- * NOTE: Caller must hold mftbmp_lock rw_semaphore for reading or writing.
- */
-static unsigned long __get_nr_free_mft_records(ntfs_volume *vol,
- s64 nr_free, const pgoff_t max_index)
-{
- struct address_space *mapping = vol->mftbmp_ino->i_mapping;
- struct page *page;
- pgoff_t index;
-
- ntfs_debug("Entering.");
- /* Use multiples of 4 bytes, thus max_size is PAGE_SIZE / 4. */
- ntfs_debug("Reading $MFT/$BITMAP, max_index = 0x%lx, max_size = "
- "0x%lx.", max_index, PAGE_SIZE / 4);
- for (index = 0; index < max_index; index++) {
- unsigned long *kaddr;
-
- /*
- * Read the page from page cache, getting it from backing store
- * if necessary, and increment the use count.
- */
- page = read_mapping_page(mapping, index, NULL);
- /* Ignore pages which errored synchronously. */
- if (IS_ERR(page)) {
- ntfs_debug("read_mapping_page() error. Skipping "
- "page (index 0x%lx).", index);
- nr_free -= PAGE_SIZE * 8;
- continue;
- }
- kaddr = kmap_atomic(page);
- /*
- * Subtract the number of set bits. If this
- * is the last page and it is partial we don't really care as
- * it just means we do a little extra work but it won't affect
- * the result as all out of range bytes are set to zero by
- * ntfs_readpage().
- */
- nr_free -= bitmap_weight(kaddr,
- PAGE_SIZE * BITS_PER_BYTE);
- kunmap_atomic(kaddr);
- put_page(page);
- }
- ntfs_debug("Finished reading $MFT/$BITMAP, last index = 0x%lx.",
- index - 1);
- /* If errors occurred we may well have gone below zero, fix this. */
- if (nr_free < 0)
- nr_free = 0;
- ntfs_debug("Exiting.");
- return nr_free;
-}
-
-/**
- * ntfs_statfs - return information about mounted NTFS volume
- * @dentry: dentry from mounted volume
- * @sfs: statfs structure in which to return the information
- *
- * Return information about the mounted NTFS volume @dentry in the statfs structure
- * pointed to by @sfs (this is initialized with zeros before ntfs_statfs is
- * called). We interpret the values to be correct of the moment in time at
- * which we are called. Most values are variable otherwise and this isn't just
- * the free values but the totals as well. For example we can increase the
- * total number of file nodes if we run out and we can keep doing this until
- * there is no more space on the volume left at all.
- *
- * Called from vfs_statfs which is used to handle the statfs, fstatfs, and
- * ustat system calls.
- *
- * Return 0 on success or -errno on error.
- */
-static int ntfs_statfs(struct dentry *dentry, struct kstatfs *sfs)
-{
- struct super_block *sb = dentry->d_sb;
- s64 size;
- ntfs_volume *vol = NTFS_SB(sb);
- ntfs_inode *mft_ni = NTFS_I(vol->mft_ino);
- pgoff_t max_index;
- unsigned long flags;
-
- ntfs_debug("Entering.");
- /* Type of filesystem. */
- sfs->f_type = NTFS_SB_MAGIC;
- /* Optimal transfer block size. */
- sfs->f_bsize = PAGE_SIZE;
- /*
- * Total data blocks in filesystem in units of f_bsize and since
- * inodes are also stored in data blocs ($MFT is a file) this is just
- * the total clusters.
- */
- sfs->f_blocks = vol->nr_clusters << vol->cluster_size_bits >>
- PAGE_SHIFT;
- /* Free data blocks in filesystem in units of f_bsize. */
- size = get_nr_free_clusters(vol) << vol->cluster_size_bits >>
- PAGE_SHIFT;
- if (size < 0LL)
- size = 0LL;
- /* Free blocks avail to non-superuser, same as above on NTFS. */
- sfs->f_bavail = sfs->f_bfree = size;
- /* Serialize accesses to the inode bitmap. */
- down_read(&vol->mftbmp_lock);
- read_lock_irqsave(&mft_ni->size_lock, flags);
- size = i_size_read(vol->mft_ino) >> vol->mft_record_size_bits;
- /*
- * Convert the maximum number of set bits into bytes rounded up, then
- * convert into multiples of PAGE_SIZE, rounding up so that if we
- * have one full and one partial page max_index = 2.
- */
- max_index = ((((mft_ni->initialized_size >> vol->mft_record_size_bits)
- + 7) >> 3) + PAGE_SIZE - 1) >> PAGE_SHIFT;
- read_unlock_irqrestore(&mft_ni->size_lock, flags);
- /* Number of inodes in filesystem (at this point in time). */
- sfs->f_files = size;
- /* Free inodes in fs (based on current total count). */
- sfs->f_ffree = __get_nr_free_mft_records(vol, size, max_index);
- up_read(&vol->mftbmp_lock);
- /*
- * File system id. This is extremely *nix flavour dependent and even
- * within Linux itself all fs do their own thing. I interpret this to
- * mean a unique id associated with the mounted fs and not the id
- * associated with the filesystem driver, the latter is already given
- * by the filesystem type in sfs->f_type. Thus we use the 64-bit
- * volume serial number splitting it into two 32-bit parts. We enter
- * the least significant 32-bits in f_fsid[0] and the most significant
- * 32-bits in f_fsid[1].
- */
- sfs->f_fsid = u64_to_fsid(vol->serial_no);
- /* Maximum length of filenames. */
- sfs->f_namelen = NTFS_MAX_NAME_LEN;
- return 0;
-}
-
-#ifdef NTFS_RW
-static int ntfs_write_inode(struct inode *vi, struct writeback_control *wbc)
-{
- return __ntfs_write_inode(vi, wbc->sync_mode == WB_SYNC_ALL);
-}
-#endif
-
-/**
- * The complete super operations.
- */
-static const struct super_operations ntfs_sops = {
- .alloc_inode = ntfs_alloc_big_inode, /* VFS: Allocate new inode. */
- .free_inode = ntfs_free_big_inode, /* VFS: Deallocate inode. */
-#ifdef NTFS_RW
- .write_inode = ntfs_write_inode, /* VFS: Write dirty inode to
- disk. */
-#endif /* NTFS_RW */
- .put_super = ntfs_put_super, /* Syscall: umount. */
- .statfs = ntfs_statfs, /* Syscall: statfs */
- .remount_fs = ntfs_remount, /* Syscall: mount -o remount. */
- .evict_inode = ntfs_evict_big_inode, /* VFS: Called when an inode is
- removed from memory. */
- .show_options = ntfs_show_options, /* Show mount options in
- proc. */
-};
-
-/**
- * ntfs_fill_super - mount an ntfs filesystem
- * @sb: super block of ntfs filesystem to mount
- * @opt: string containing the mount options
- * @silent: silence error output
- *
- * ntfs_fill_super() is called by the VFS to mount the device described by @sb
- * with the mount otions in @data with the NTFS filesystem.
- *
- * If @silent is true, remain silent even if errors are detected. This is used
- * during bootup, when the kernel tries to mount the root filesystem with all
- * registered filesystems one after the other until one succeeds. This implies
- * that all filesystems except the correct one will quite correctly and
- * expectedly return an error, but nobody wants to see error messages when in
- * fact this is what is supposed to happen.
- *
- * NOTE: @sb->s_flags contains the mount options flags.
- */
-static int ntfs_fill_super(struct super_block *sb, void *opt, const int silent)
-{
- ntfs_volume *vol;
- struct buffer_head *bh;
- struct inode *tmp_ino;
- int blocksize, result;
-
- /*
- * We do a pretty difficult piece of bootstrap by reading the
- * MFT (and other metadata) from disk into memory. We'll only
- * release this metadata during umount, so the locking patterns
- * observed during bootstrap do not count. So turn off the
- * observation of locking patterns (strictly for this context
- * only) while mounting NTFS. [The validator is still active
- * otherwise, even for this context: it will for example record
- * lock class registrations.]
- */
- lockdep_off();
- ntfs_debug("Entering.");
-#ifndef NTFS_RW
- sb->s_flags |= SB_RDONLY;
-#endif /* ! NTFS_RW */
- /* Allocate a new ntfs_volume and place it in sb->s_fs_info. */
- sb->s_fs_info = kmalloc(sizeof(ntfs_volume), GFP_NOFS);
- vol = NTFS_SB(sb);
- if (!vol) {
- if (!silent)
- ntfs_error(sb, "Allocation of NTFS volume structure "
- "failed. Aborting mount...");
- lockdep_on();
- return -ENOMEM;
- }
- /* Initialize ntfs_volume structure. */
- *vol = (ntfs_volume) {
- .sb = sb,
- /*
- * Default is group and other don't have any access to files or
- * directories while owner has full access. Further, files by
- * default are not executable but directories are of course
- * browseable.
- */
- .fmask = 0177,
- .dmask = 0077,
- };
- init_rwsem(&vol->mftbmp_lock);
- init_rwsem(&vol->lcnbmp_lock);
-
- /* By default, enable sparse support. */
- NVolSetSparseEnabled(vol);
-
- /* Important to get the mount options dealt with now. */
- if (!parse_options(vol, (char*)opt))
- goto err_out_now;
-
- /* We support sector sizes up to the PAGE_SIZE. */
- if (bdev_logical_block_size(sb->s_bdev) > PAGE_SIZE) {
- if (!silent)
- ntfs_error(sb, "Device has unsupported sector size "
- "(%i). The maximum supported sector "
- "size on this architecture is %lu "
- "bytes.",
- bdev_logical_block_size(sb->s_bdev),
- PAGE_SIZE);
- goto err_out_now;
- }
- /*
- * Setup the device access block size to NTFS_BLOCK_SIZE or the hard
- * sector size, whichever is bigger.
- */
- blocksize = sb_min_blocksize(sb, NTFS_BLOCK_SIZE);
- if (blocksize < NTFS_BLOCK_SIZE) {
- if (!silent)
- ntfs_error(sb, "Unable to set device block size.");
- goto err_out_now;
- }
- BUG_ON(blocksize != sb->s_blocksize);
- ntfs_debug("Set device block size to %i bytes (block size bits %i).",
- blocksize, sb->s_blocksize_bits);
- /* Determine the size of the device in units of block_size bytes. */
- vol->nr_blocks = sb_bdev_nr_blocks(sb);
- if (!vol->nr_blocks) {
- if (!silent)
- ntfs_error(sb, "Unable to determine device size.");
- goto err_out_now;
- }
- /* Read the boot sector and return unlocked buffer head to it. */
- if (!(bh = read_ntfs_boot_sector(sb, silent))) {
- if (!silent)
- ntfs_error(sb, "Not an NTFS volume.");
- goto err_out_now;
- }
- /*
- * Extract the data from the boot sector and setup the ntfs volume
- * using it.
- */
- result = parse_ntfs_boot_sector(vol, (NTFS_BOOT_SECTOR*)bh->b_data);
- brelse(bh);
- if (!result) {
- if (!silent)
- ntfs_error(sb, "Unsupported NTFS filesystem.");
- goto err_out_now;
- }
- /*
- * If the boot sector indicates a sector size bigger than the current
- * device block size, switch the device block size to the sector size.
- * TODO: It may be possible to support this case even when the set
- * below fails, we would just be breaking up the i/o for each sector
- * into multiple blocks for i/o purposes but otherwise it should just
- * work. However it is safer to leave disabled until someone hits this
- * error message and then we can get them to try it without the setting
- * so we know for sure that it works.
- */
- if (vol->sector_size > blocksize) {
- blocksize = sb_set_blocksize(sb, vol->sector_size);
- if (blocksize != vol->sector_size) {
- if (!silent)
- ntfs_error(sb, "Unable to set device block "
- "size to sector size (%i).",
- vol->sector_size);
- goto err_out_now;
- }
- BUG_ON(blocksize != sb->s_blocksize);
- vol->nr_blocks = sb_bdev_nr_blocks(sb);
- ntfs_debug("Changed device block size to %i bytes (block size "
- "bits %i) to match volume sector size.",
- blocksize, sb->s_blocksize_bits);
- }
- /* Initialize the cluster and mft allocators. */
- ntfs_setup_allocators(vol);
- /* Setup remaining fields in the super block. */
- sb->s_magic = NTFS_SB_MAGIC;
- /*
- * Ntfs allows 63 bits for the file size, i.e. correct would be:
- * sb->s_maxbytes = ~0ULL >> 1;
- * But the kernel uses a long as the page cache page index which on
- * 32-bit architectures is only 32-bits. MAX_LFS_FILESIZE is kernel
- * defined to the maximum the page cache page index can cope with
- * without overflowing the index or to 2^63 - 1, whichever is smaller.
- */
- sb->s_maxbytes = MAX_LFS_FILESIZE;
- /* Ntfs measures time in 100ns intervals. */
- sb->s_time_gran = 100;
- /*
- * Now load the metadata required for the page cache and our address
- * space operations to function. We do this by setting up a specialised
- * read_inode method and then just calling the normal iget() to obtain
- * the inode for $MFT which is sufficient to allow our normal inode
- * operations and associated address space operations to function.
- */
- sb->s_op = &ntfs_sops;
- tmp_ino = new_inode(sb);
- if (!tmp_ino) {
- if (!silent)
- ntfs_error(sb, "Failed to load essential metadata.");
- goto err_out_now;
- }
- tmp_ino->i_ino = FILE_MFT;
- insert_inode_hash(tmp_ino);
- if (ntfs_read_inode_mount(tmp_ino) < 0) {
- if (!silent)
- ntfs_error(sb, "Failed to load essential metadata.");
- goto iput_tmp_ino_err_out_now;
- }
- mutex_lock(&ntfs_lock);
- /*
- * The current mount is a compression user if the cluster size is
- * less than or equal 4kiB.
- */
- if (vol->cluster_size <= 4096 && !ntfs_nr_compression_users++) {
- result = allocate_compression_buffers();
- if (result) {
- ntfs_error(NULL, "Failed to allocate buffers "
- "for compression engine.");
- ntfs_nr_compression_users--;
- mutex_unlock(&ntfs_lock);
- goto iput_tmp_ino_err_out_now;
- }
- }
- /*
- * Generate the global default upcase table if necessary. Also
- * temporarily increment the number of upcase users to avoid race
- * conditions with concurrent (u)mounts.
- */
- if (!default_upcase)
- default_upcase = generate_default_upcase();
- ntfs_nr_upcase_users++;
- mutex_unlock(&ntfs_lock);
- /*
- * From now on, ignore @silent parameter. If we fail below this line,
- * it will be due to a corrupt fs or a system error, so we report it.
- */
- /*
- * Open the system files with normal access functions and complete
- * setting up the ntfs super block.
- */
- if (!load_system_files(vol)) {
- ntfs_error(sb, "Failed to load system files.");
- goto unl_upcase_iput_tmp_ino_err_out_now;
- }
-
- /* We grab a reference, simulating an ntfs_iget(). */
- ihold(vol->root_ino);
- if ((sb->s_root = d_make_root(vol->root_ino))) {
- ntfs_debug("Exiting, status successful.");
- /* Release the default upcase if it has no users. */
- mutex_lock(&ntfs_lock);
- if (!--ntfs_nr_upcase_users && default_upcase) {
- ntfs_free(default_upcase);
- default_upcase = NULL;
- }
- mutex_unlock(&ntfs_lock);
- sb->s_export_op = &ntfs_export_ops;
- lockdep_on();
- return 0;
- }
- ntfs_error(sb, "Failed to allocate root directory.");
- /* Clean up after the successful load_system_files() call from above. */
- // TODO: Use ntfs_put_super() instead of repeating all this code...
- // FIXME: Should mark the volume clean as the error is most likely
- // -ENOMEM.
- iput(vol->vol_ino);
- vol->vol_ino = NULL;
- /* NTFS 3.0+ specific clean up. */
- if (vol->major_ver >= 3) {
-#ifdef NTFS_RW
- if (vol->usnjrnl_j_ino) {
- iput(vol->usnjrnl_j_ino);
- vol->usnjrnl_j_ino = NULL;
- }
- if (vol->usnjrnl_max_ino) {
- iput(vol->usnjrnl_max_ino);
- vol->usnjrnl_max_ino = NULL;
- }
- if (vol->usnjrnl_ino) {
- iput(vol->usnjrnl_ino);
- vol->usnjrnl_ino = NULL;
- }
- if (vol->quota_q_ino) {
- iput(vol->quota_q_ino);
- vol->quota_q_ino = NULL;
- }
- if (vol->quota_ino) {
- iput(vol->quota_ino);
- vol->quota_ino = NULL;
- }
-#endif /* NTFS_RW */
- if (vol->extend_ino) {
- iput(vol->extend_ino);
- vol->extend_ino = NULL;
- }
- if (vol->secure_ino) {
- iput(vol->secure_ino);
- vol->secure_ino = NULL;
- }
- }
- iput(vol->root_ino);
- vol->root_ino = NULL;
- iput(vol->lcnbmp_ino);
- vol->lcnbmp_ino = NULL;
- iput(vol->mftbmp_ino);
- vol->mftbmp_ino = NULL;
-#ifdef NTFS_RW
- if (vol->logfile_ino) {
- iput(vol->logfile_ino);
- vol->logfile_ino = NULL;
- }
- if (vol->mftmirr_ino) {
- iput(vol->mftmirr_ino);
- vol->mftmirr_ino = NULL;
- }
-#endif /* NTFS_RW */
- /* Throw away the table of attribute definitions. */
- vol->attrdef_size = 0;
- if (vol->attrdef) {
- ntfs_free(vol->attrdef);
- vol->attrdef = NULL;
- }
- vol->upcase_len = 0;
- mutex_lock(&ntfs_lock);
- if (vol->upcase == default_upcase) {
- ntfs_nr_upcase_users--;
- vol->upcase = NULL;
- }
- mutex_unlock(&ntfs_lock);
- if (vol->upcase) {
- ntfs_free(vol->upcase);
- vol->upcase = NULL;
- }
- if (vol->nls_map) {
- unload_nls(vol->nls_map);
- vol->nls_map = NULL;
- }
- /* Error exit code path. */
-unl_upcase_iput_tmp_ino_err_out_now:
- /*
- * Decrease the number of upcase users and destroy the global default
- * upcase table if necessary.
- */
- mutex_lock(&ntfs_lock);
- if (!--ntfs_nr_upcase_users && default_upcase) {
- ntfs_free(default_upcase);
- default_upcase = NULL;
- }
- if (vol->cluster_size <= 4096 && !--ntfs_nr_compression_users)
- free_compression_buffers();
- mutex_unlock(&ntfs_lock);
-iput_tmp_ino_err_out_now:
- iput(tmp_ino);
- if (vol->mft_ino && vol->mft_ino != tmp_ino)
- iput(vol->mft_ino);
- vol->mft_ino = NULL;
- /* Errors at this stage are irrelevant. */
-err_out_now:
- sb->s_fs_info = NULL;
- kfree(vol);
- ntfs_debug("Failed, returning -EINVAL.");
- lockdep_on();
- return -EINVAL;
-}
-
-/*
- * This is a slab cache to optimize allocations and deallocations of Unicode
- * strings of the maximum length allowed by NTFS, which is NTFS_MAX_NAME_LEN
- * (255) Unicode characters + a terminating NULL Unicode character.
- */
-struct kmem_cache *ntfs_name_cache;
-
-/* Slab caches for efficient allocation/deallocation of inodes. */
-struct kmem_cache *ntfs_inode_cache;
-struct kmem_cache *ntfs_big_inode_cache;
-
-/* Init once constructor for the inode slab cache. */
-static void ntfs_big_inode_init_once(void *foo)
-{
- ntfs_inode *ni = (ntfs_inode *)foo;
-
- inode_init_once(VFS_I(ni));
-}
-
-/*
- * Slab caches to optimize allocations and deallocations of attribute search
- * contexts and index contexts, respectively.
- */
-struct kmem_cache *ntfs_attr_ctx_cache;
-struct kmem_cache *ntfs_index_ctx_cache;
-
-/* Driver wide mutex. */
-DEFINE_MUTEX(ntfs_lock);
-
-static struct dentry *ntfs_mount(struct file_system_type *fs_type,
- int flags, const char *dev_name, void *data)
-{
- return mount_bdev(fs_type, flags, dev_name, data, ntfs_fill_super);
-}
-
-static struct file_system_type ntfs_fs_type = {
- .owner = THIS_MODULE,
- .name = "ntfs",
- .mount = ntfs_mount,
- .kill_sb = kill_block_super,
- .fs_flags = FS_REQUIRES_DEV,
-};
-MODULE_ALIAS_FS("ntfs");
-
-/* Stable names for the slab caches. */
-static const char ntfs_index_ctx_cache_name[] = "ntfs_index_ctx_cache";
-static const char ntfs_attr_ctx_cache_name[] = "ntfs_attr_ctx_cache";
-static const char ntfs_name_cache_name[] = "ntfs_name_cache";
-static const char ntfs_inode_cache_name[] = "ntfs_inode_cache";
-static const char ntfs_big_inode_cache_name[] = "ntfs_big_inode_cache";
-
-static int __init init_ntfs_fs(void)
-{
- int err = 0;
-
- /* This may be ugly but it results in pretty output so who cares. (-8 */
- pr_info("driver " NTFS_VERSION " [Flags: R/"
-#ifdef NTFS_RW
- "W"
-#else
- "O"
-#endif
-#ifdef DEBUG
- " DEBUG"
-#endif
-#ifdef MODULE
- " MODULE"
-#endif
- "].\n");
-
- ntfs_debug("Debug messages are enabled.");
-
- ntfs_index_ctx_cache = kmem_cache_create(ntfs_index_ctx_cache_name,
- sizeof(ntfs_index_context), 0 /* offset */,
- SLAB_HWCACHE_ALIGN, NULL /* ctor */);
- if (!ntfs_index_ctx_cache) {
- pr_crit("Failed to create %s!\n", ntfs_index_ctx_cache_name);
- goto ictx_err_out;
- }
- ntfs_attr_ctx_cache = kmem_cache_create(ntfs_attr_ctx_cache_name,
- sizeof(ntfs_attr_search_ctx), 0 /* offset */,
- SLAB_HWCACHE_ALIGN, NULL /* ctor */);
- if (!ntfs_attr_ctx_cache) {
- pr_crit("NTFS: Failed to create %s!\n",
- ntfs_attr_ctx_cache_name);
- goto actx_err_out;
- }
-
- ntfs_name_cache = kmem_cache_create(ntfs_name_cache_name,
- (NTFS_MAX_NAME_LEN+1) * sizeof(ntfschar), 0,
- SLAB_HWCACHE_ALIGN, NULL);
- if (!ntfs_name_cache) {
- pr_crit("Failed to create %s!\n", ntfs_name_cache_name);
- goto name_err_out;
- }
-
- ntfs_inode_cache = kmem_cache_create(ntfs_inode_cache_name,
- sizeof(ntfs_inode), 0,
- SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD, NULL);
- if (!ntfs_inode_cache) {
- pr_crit("Failed to create %s!\n", ntfs_inode_cache_name);
- goto inode_err_out;
- }
-
- ntfs_big_inode_cache = kmem_cache_create(ntfs_big_inode_cache_name,
- sizeof(big_ntfs_inode), 0,
- SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD|
- SLAB_ACCOUNT, ntfs_big_inode_init_once);
- if (!ntfs_big_inode_cache) {
- pr_crit("Failed to create %s!\n", ntfs_big_inode_cache_name);
- goto big_inode_err_out;
- }
-
- /* Register the ntfs sysctls. */
- err = ntfs_sysctl(1);
- if (err) {
- pr_crit("Failed to register NTFS sysctls!\n");
- goto sysctl_err_out;
- }
-
- err = register_filesystem(&ntfs_fs_type);
- if (!err) {
- ntfs_debug("NTFS driver registered successfully.");
- return 0; /* Success! */
- }
- pr_crit("Failed to register NTFS filesystem driver!\n");
-
- /* Unregister the ntfs sysctls. */
- ntfs_sysctl(0);
-sysctl_err_out:
- kmem_cache_destroy(ntfs_big_inode_cache);
-big_inode_err_out:
- kmem_cache_destroy(ntfs_inode_cache);
-inode_err_out:
- kmem_cache_destroy(ntfs_name_cache);
-name_err_out:
- kmem_cache_destroy(ntfs_attr_ctx_cache);
-actx_err_out:
- kmem_cache_destroy(ntfs_index_ctx_cache);
-ictx_err_out:
- if (!err) {
- pr_crit("Aborting NTFS filesystem driver registration...\n");
- err = -ENOMEM;
- }
- return err;
-}
-
-static void __exit exit_ntfs_fs(void)
-{
- ntfs_debug("Unregistering NTFS driver.");
-
- unregister_filesystem(&ntfs_fs_type);
-
- /*
- * Make sure all delayed rcu free inodes are flushed before we
- * destroy cache.
- */
- rcu_barrier();
- kmem_cache_destroy(ntfs_big_inode_cache);
- kmem_cache_destroy(ntfs_inode_cache);
- kmem_cache_destroy(ntfs_name_cache);
- kmem_cache_destroy(ntfs_attr_ctx_cache);
- kmem_cache_destroy(ntfs_index_ctx_cache);
- /* Unregister the ntfs sysctls. */
- ntfs_sysctl(0);
-}
-
-MODULE_AUTHOR("Anton Altaparmakov <anton@tuxera.com>");
-MODULE_DESCRIPTION("NTFS 1.2/3.x driver - Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc.");
-MODULE_VERSION(NTFS_VERSION);
-MODULE_LICENSE("GPL");
-#ifdef DEBUG
-module_param(debug_msgs, bint, 0);
-MODULE_PARM_DESC(debug_msgs, "Enable debug messages.");
-#endif
-
-module_init(init_ntfs_fs)
-module_exit(exit_ntfs_fs)
diff --git a/fs/ntfs/sysctl.c b/fs/ntfs/sysctl.c
deleted file mode 100644
index a030d00af90c..000000000000
--- a/fs/ntfs/sysctl.c
+++ /dev/null
@@ -1,69 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * sysctl.c - Code for sysctl handling in NTFS Linux kernel driver. Part of
- * the Linux-NTFS project. Adapted from the old NTFS driver,
- * Copyright (C) 1997 Martin von Löwis, Régis Duchesne
- *
- * Copyright (c) 2002-2005 Anton Altaparmakov
- */
-
-#ifdef DEBUG
-
-#include <linux/module.h>
-
-#ifdef CONFIG_SYSCTL
-
-#include <linux/proc_fs.h>
-#include <linux/sysctl.h>
-
-#include "sysctl.h"
-#include "debug.h"
-
-/* Definition of the ntfs sysctl. */
-static struct ctl_table ntfs_sysctls[] = {
- {
- .procname = "ntfs-debug",
- .data = &debug_msgs, /* Data pointer and size. */
- .maxlen = sizeof(debug_msgs),
- .mode = 0644, /* Mode, proc handler. */
- .proc_handler = proc_dointvec
- },
- {}
-};
-
-/* Define the parent directory /proc/sys/fs. */
-static struct ctl_table sysctls_root[] = {
- {
- .procname = "fs",
- .mode = 0555,
- .child = ntfs_sysctls
- },
- {}
-};
-
-/* Storage for the sysctls header. */
-static struct ctl_table_header *sysctls_root_table;
-
-/**
- * ntfs_sysctl - add or remove the debug sysctl
- * @add: add (1) or remove (0) the sysctl
- *
- * Add or remove the debug sysctl. Return 0 on success or -errno on error.
- */
-int ntfs_sysctl(int add)
-{
- if (add) {
- BUG_ON(sysctls_root_table);
- sysctls_root_table = register_sysctl_table(sysctls_root);
- if (!sysctls_root_table)
- return -ENOMEM;
- } else {
- BUG_ON(!sysctls_root_table);
- unregister_sysctl_table(sysctls_root_table);
- sysctls_root_table = NULL;
- }
- return 0;
-}
-
-#endif /* CONFIG_SYSCTL */
-#endif /* DEBUG */
diff --git a/fs/ntfs/sysctl.h b/fs/ntfs/sysctl.h
deleted file mode 100644
index 96bb2299d2d5..000000000000
--- a/fs/ntfs/sysctl.h
+++ /dev/null
@@ -1,27 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * sysctl.h - Defines for sysctl handling in NTFS Linux kernel driver. Part of
- * the Linux-NTFS project. Adapted from the old NTFS driver,
- * Copyright (C) 1997 Martin von Löwis, Régis Duchesne
- *
- * Copyright (c) 2002-2004 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_SYSCTL_H
-#define _LINUX_NTFS_SYSCTL_H
-
-
-#if defined(DEBUG) && defined(CONFIG_SYSCTL)
-
-extern int ntfs_sysctl(int add);
-
-#else
-
-/* Just return success. */
-static inline int ntfs_sysctl(int add)
-{
- return 0;
-}
-
-#endif /* DEBUG && CONFIG_SYSCTL */
-#endif /* _LINUX_NTFS_SYSCTL_H */
diff --git a/fs/ntfs/time.h b/fs/ntfs/time.h
deleted file mode 100644
index 6b63261300cc..000000000000
--- a/fs/ntfs/time.h
+++ /dev/null
@@ -1,89 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * time.h - NTFS time conversion functions. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2005 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_TIME_H
-#define _LINUX_NTFS_TIME_H
-
-#include <linux/time.h> /* For current_kernel_time(). */
-#include <asm/div64.h> /* For do_div(). */
-
-#include "endian.h"
-
-#define NTFS_TIME_OFFSET ((s64)(369 * 365 + 89) * 24 * 3600 * 10000000)
-
-/**
- * utc2ntfs - convert Linux UTC time to NTFS time
- * @ts: Linux UTC time to convert to NTFS time
- *
- * Convert the Linux UTC time @ts to its corresponding NTFS time and return
- * that in little endian format.
- *
- * Linux stores time in a struct timespec64 consisting of a time64_t tv_sec
- * and a long tv_nsec where tv_sec is the number of 1-second intervals since
- * 1st January 1970, 00:00:00 UTC and tv_nsec is the number of 1-nano-second
- * intervals since the value of tv_sec.
- *
- * NTFS uses Microsoft's standard time format which is stored in a s64 and is
- * measured as the number of 100-nano-second intervals since 1st January 1601,
- * 00:00:00 UTC.
- */
-static inline sle64 utc2ntfs(const struct timespec64 ts)
-{
- /*
- * Convert the seconds to 100ns intervals, add the nano-seconds
- * converted to 100ns intervals, and then add the NTFS time offset.
- */
- return cpu_to_sle64((s64)ts.tv_sec * 10000000 + ts.tv_nsec / 100 +
- NTFS_TIME_OFFSET);
-}
-
-/**
- * get_current_ntfs_time - get the current time in little endian NTFS format
- *
- * Get the current time from the Linux kernel, convert it to its corresponding
- * NTFS time and return that in little endian format.
- */
-static inline sle64 get_current_ntfs_time(void)
-{
- struct timespec64 ts;
-
- ktime_get_coarse_real_ts64(&ts);
- return utc2ntfs(ts);
-}
-
-/**
- * ntfs2utc - convert NTFS time to Linux time
- * @time: NTFS time (little endian) to convert to Linux UTC
- *
- * Convert the little endian NTFS time @time to its corresponding Linux UTC
- * time and return that in cpu format.
- *
- * Linux stores time in a struct timespec64 consisting of a time64_t tv_sec
- * and a long tv_nsec where tv_sec is the number of 1-second intervals since
- * 1st January 1970, 00:00:00 UTC and tv_nsec is the number of 1-nano-second
- * intervals since the value of tv_sec.
- *
- * NTFS uses Microsoft's standard time format which is stored in a s64 and is
- * measured as the number of 100 nano-second intervals since 1st January 1601,
- * 00:00:00 UTC.
- */
-static inline struct timespec64 ntfs2utc(const sle64 time)
-{
- struct timespec64 ts;
-
- /* Subtract the NTFS time offset. */
- u64 t = (u64)(sle64_to_cpu(time) - NTFS_TIME_OFFSET);
- /*
- * Convert the time to 1-second intervals and the remainder to
- * 1-nano-second intervals.
- */
- ts.tv_nsec = do_div(t, 10000000) * 100;
- ts.tv_sec = t;
- return ts;
-}
-
-#endif /* _LINUX_NTFS_TIME_H */
diff --git a/fs/ntfs/types.h b/fs/ntfs/types.h
deleted file mode 100644
index 9a47859e7a06..000000000000
--- a/fs/ntfs/types.h
+++ /dev/null
@@ -1,55 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * types.h - Defines for NTFS Linux kernel driver specific types.
- * Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2005 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_TYPES_H
-#define _LINUX_NTFS_TYPES_H
-
-#include <linux/types.h>
-
-typedef __le16 le16;
-typedef __le32 le32;
-typedef __le64 le64;
-typedef __u16 __bitwise sle16;
-typedef __u32 __bitwise sle32;
-typedef __u64 __bitwise sle64;
-
-/* 2-byte Unicode character type. */
-typedef le16 ntfschar;
-#define UCHAR_T_SIZE_BITS 1
-
-/*
- * Clusters are signed 64-bit values on NTFS volumes. We define two types, LCN
- * and VCN, to allow for type checking and better code readability.
- */
-typedef s64 VCN;
-typedef sle64 leVCN;
-typedef s64 LCN;
-typedef sle64 leLCN;
-
-/*
- * The NTFS journal $LogFile uses log sequence numbers which are signed 64-bit
- * values. We define our own type LSN, to allow for type checking and better
- * code readability.
- */
-typedef s64 LSN;
-typedef sle64 leLSN;
-
-/*
- * The NTFS transaction log $UsnJrnl uses usn which are signed 64-bit values.
- * We define our own type USN, to allow for type checking and better code
- * readability.
- */
-typedef s64 USN;
-typedef sle64 leUSN;
-
-typedef enum {
- CASE_SENSITIVE = 0,
- IGNORE_CASE = 1,
-} IGNORE_CASE_BOOL;
-
-#endif /* _LINUX_NTFS_TYPES_H */
diff --git a/fs/ntfs/unistr.c b/fs/ntfs/unistr.c
deleted file mode 100644
index a6b6c64f14a9..000000000000
--- a/fs/ntfs/unistr.c
+++ /dev/null
@@ -1,384 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * unistr.c - NTFS Unicode string handling. Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2006 Anton Altaparmakov
- */
-
-#include <linux/slab.h>
-
-#include "types.h"
-#include "debug.h"
-#include "ntfs.h"
-
-/*
- * IMPORTANT
- * =========
- *
- * All these routines assume that the Unicode characters are in little endian
- * encoding inside the strings!!!
- */
-
-/*
- * This is used by the name collation functions to quickly determine what
- * characters are (in)valid.
- */
-static const u8 legal_ansi_char_array[0x40] = {
- 0x00, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10,
- 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10,
-
- 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10,
- 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10,
-
- 0x17, 0x07, 0x18, 0x17, 0x17, 0x17, 0x17, 0x17,
- 0x17, 0x17, 0x18, 0x16, 0x16, 0x17, 0x07, 0x00,
-
- 0x17, 0x17, 0x17, 0x17, 0x17, 0x17, 0x17, 0x17,
- 0x17, 0x17, 0x04, 0x16, 0x18, 0x16, 0x18, 0x18,
-};
-
-/**
- * ntfs_are_names_equal - compare two Unicode names for equality
- * @s1: name to compare to @s2
- * @s1_len: length in Unicode characters of @s1
- * @s2: name to compare to @s1
- * @s2_len: length in Unicode characters of @s2
- * @ic: ignore case bool
- * @upcase: upcase table (only if @ic == IGNORE_CASE)
- * @upcase_size: length in Unicode characters of @upcase (if present)
- *
- * Compare the names @s1 and @s2 and return 'true' (1) if the names are
- * identical, or 'false' (0) if they are not identical. If @ic is IGNORE_CASE,
- * the @upcase table is used to performa a case insensitive comparison.
- */
-bool ntfs_are_names_equal(const ntfschar *s1, size_t s1_len,
- const ntfschar *s2, size_t s2_len, const IGNORE_CASE_BOOL ic,
- const ntfschar *upcase, const u32 upcase_size)
-{
- if (s1_len != s2_len)
- return false;
- if (ic == CASE_SENSITIVE)
- return !ntfs_ucsncmp(s1, s2, s1_len);
- return !ntfs_ucsncasecmp(s1, s2, s1_len, upcase, upcase_size);
-}
-
-/**
- * ntfs_collate_names - collate two Unicode names
- * @name1: first Unicode name to compare
- * @name2: second Unicode name to compare
- * @err_val: if @name1 contains an invalid character return this value
- * @ic: either CASE_SENSITIVE or IGNORE_CASE
- * @upcase: upcase table (ignored if @ic is CASE_SENSITIVE)
- * @upcase_len: upcase table size (ignored if @ic is CASE_SENSITIVE)
- *
- * ntfs_collate_names collates two Unicode names and returns:
- *
- * -1 if the first name collates before the second one,
- * 0 if the names match,
- * 1 if the second name collates before the first one, or
- * @err_val if an invalid character is found in @name1 during the comparison.
- *
- * The following characters are considered invalid: '"', '*', '<', '>' and '?'.
- */
-int ntfs_collate_names(const ntfschar *name1, const u32 name1_len,
- const ntfschar *name2, const u32 name2_len,
- const int err_val, const IGNORE_CASE_BOOL ic,
- const ntfschar *upcase, const u32 upcase_len)
-{
- u32 cnt, min_len;
- u16 c1, c2;
-
- min_len = name1_len;
- if (name1_len > name2_len)
- min_len = name2_len;
- for (cnt = 0; cnt < min_len; ++cnt) {
- c1 = le16_to_cpu(*name1++);
- c2 = le16_to_cpu(*name2++);
- if (ic) {
- if (c1 < upcase_len)
- c1 = le16_to_cpu(upcase[c1]);
- if (c2 < upcase_len)
- c2 = le16_to_cpu(upcase[c2]);
- }
- if (c1 < 64 && legal_ansi_char_array[c1] & 8)
- return err_val;
- if (c1 < c2)
- return -1;
- if (c1 > c2)
- return 1;
- }
- if (name1_len < name2_len)
- return -1;
- if (name1_len == name2_len)
- return 0;
- /* name1_len > name2_len */
- c1 = le16_to_cpu(*name1);
- if (c1 < 64 && legal_ansi_char_array[c1] & 8)
- return err_val;
- return 1;
-}
-
-/**
- * ntfs_ucsncmp - compare two little endian Unicode strings
- * @s1: first string
- * @s2: second string
- * @n: maximum unicode characters to compare
- *
- * Compare the first @n characters of the Unicode strings @s1 and @s2,
- * The strings in little endian format and appropriate le16_to_cpu()
- * conversion is performed on non-little endian machines.
- *
- * The function returns an integer less than, equal to, or greater than zero
- * if @s1 (or the first @n Unicode characters thereof) is found, respectively,
- * to be less than, to match, or be greater than @s2.
- */
-int ntfs_ucsncmp(const ntfschar *s1, const ntfschar *s2, size_t n)
-{
- u16 c1, c2;
- size_t i;
-
- for (i = 0; i < n; ++i) {
- c1 = le16_to_cpu(s1[i]);
- c2 = le16_to_cpu(s2[i]);
- if (c1 < c2)
- return -1;
- if (c1 > c2)
- return 1;
- if (!c1)
- break;
- }
- return 0;
-}
-
-/**
- * ntfs_ucsncasecmp - compare two little endian Unicode strings, ignoring case
- * @s1: first string
- * @s2: second string
- * @n: maximum unicode characters to compare
- * @upcase: upcase table
- * @upcase_size: upcase table size in Unicode characters
- *
- * Compare the first @n characters of the Unicode strings @s1 and @s2,
- * ignoring case. The strings in little endian format and appropriate
- * le16_to_cpu() conversion is performed on non-little endian machines.
- *
- * Each character is uppercased using the @upcase table before the comparison.
- *
- * The function returns an integer less than, equal to, or greater than zero
- * if @s1 (or the first @n Unicode characters thereof) is found, respectively,
- * to be less than, to match, or be greater than @s2.
- */
-int ntfs_ucsncasecmp(const ntfschar *s1, const ntfschar *s2, size_t n,
- const ntfschar *upcase, const u32 upcase_size)
-{
- size_t i;
- u16 c1, c2;
-
- for (i = 0; i < n; ++i) {
- if ((c1 = le16_to_cpu(s1[i])) < upcase_size)
- c1 = le16_to_cpu(upcase[c1]);
- if ((c2 = le16_to_cpu(s2[i])) < upcase_size)
- c2 = le16_to_cpu(upcase[c2]);
- if (c1 < c2)
- return -1;
- if (c1 > c2)
- return 1;
- if (!c1)
- break;
- }
- return 0;
-}
-
-void ntfs_upcase_name(ntfschar *name, u32 name_len, const ntfschar *upcase,
- const u32 upcase_len)
-{
- u32 i;
- u16 u;
-
- for (i = 0; i < name_len; i++)
- if ((u = le16_to_cpu(name[i])) < upcase_len)
- name[i] = upcase[u];
-}
-
-void ntfs_file_upcase_value(FILE_NAME_ATTR *file_name_attr,
- const ntfschar *upcase, const u32 upcase_len)
-{
- ntfs_upcase_name((ntfschar*)&file_name_attr->file_name,
- file_name_attr->file_name_length, upcase, upcase_len);
-}
-
-int ntfs_file_compare_values(FILE_NAME_ATTR *file_name_attr1,
- FILE_NAME_ATTR *file_name_attr2,
- const int err_val, const IGNORE_CASE_BOOL ic,
- const ntfschar *upcase, const u32 upcase_len)
-{
- return ntfs_collate_names((ntfschar*)&file_name_attr1->file_name,
- file_name_attr1->file_name_length,
- (ntfschar*)&file_name_attr2->file_name,
- file_name_attr2->file_name_length,
- err_val, ic, upcase, upcase_len);
-}
-
-/**
- * ntfs_nlstoucs - convert NLS string to little endian Unicode string
- * @vol: ntfs volume which we are working with
- * @ins: input NLS string buffer
- * @ins_len: length of input string in bytes
- * @outs: on return contains the allocated output Unicode string buffer
- *
- * Convert the input string @ins, which is in whatever format the loaded NLS
- * map dictates, into a little endian, 2-byte Unicode string.
- *
- * This function allocates the string and the caller is responsible for
- * calling kmem_cache_free(ntfs_name_cache, *@outs); when finished with it.
- *
- * On success the function returns the number of Unicode characters written to
- * the output string *@outs (>= 0), not counting the terminating Unicode NULL
- * character. *@outs is set to the allocated output string buffer.
- *
- * On error, a negative number corresponding to the error code is returned. In
- * that case the output string is not allocated. Both *@outs and *@outs_len
- * are then undefined.
- *
- * This might look a bit odd due to fast path optimization...
- */
-int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins,
- const int ins_len, ntfschar **outs)
-{
- struct nls_table *nls = vol->nls_map;
- ntfschar *ucs;
- wchar_t wc;
- int i, o, wc_len;
-
- /* We do not trust outside sources. */
- if (likely(ins)) {
- ucs = kmem_cache_alloc(ntfs_name_cache, GFP_NOFS);
- if (likely(ucs)) {
- for (i = o = 0; i < ins_len; i += wc_len) {
- wc_len = nls->char2uni(ins + i, ins_len - i,
- &wc);
- if (likely(wc_len >= 0 &&
- o < NTFS_MAX_NAME_LEN)) {
- if (likely(wc)) {
- ucs[o++] = cpu_to_le16(wc);
- continue;
- } /* else if (!wc) */
- break;
- } /* else if (wc_len < 0 ||
- o >= NTFS_MAX_NAME_LEN) */
- goto name_err;
- }
- ucs[o] = 0;
- *outs = ucs;
- return o;
- } /* else if (!ucs) */
- ntfs_error(vol->sb, "Failed to allocate buffer for converted "
- "name from ntfs_name_cache.");
- return -ENOMEM;
- } /* else if (!ins) */
- ntfs_error(vol->sb, "Received NULL pointer.");
- return -EINVAL;
-name_err:
- kmem_cache_free(ntfs_name_cache, ucs);
- if (wc_len < 0) {
- ntfs_error(vol->sb, "Name using character set %s contains "
- "characters that cannot be converted to "
- "Unicode.", nls->charset);
- i = -EILSEQ;
- } else /* if (o >= NTFS_MAX_NAME_LEN) */ {
- ntfs_error(vol->sb, "Name is too long (maximum length for a "
- "name on NTFS is %d Unicode characters.",
- NTFS_MAX_NAME_LEN);
- i = -ENAMETOOLONG;
- }
- return i;
-}
-
-/**
- * ntfs_ucstonls - convert little endian Unicode string to NLS string
- * @vol: ntfs volume which we are working with
- * @ins: input Unicode string buffer
- * @ins_len: length of input string in Unicode characters
- * @outs: on return contains the (allocated) output NLS string buffer
- * @outs_len: length of output string buffer in bytes
- *
- * Convert the input little endian, 2-byte Unicode string @ins, of length
- * @ins_len into the string format dictated by the loaded NLS.
- *
- * If *@outs is NULL, this function allocates the string and the caller is
- * responsible for calling kfree(*@outs); when finished with it. In this case
- * @outs_len is ignored and can be 0.
- *
- * On success the function returns the number of bytes written to the output
- * string *@outs (>= 0), not counting the terminating NULL byte. If the output
- * string buffer was allocated, *@outs is set to it.
- *
- * On error, a negative number corresponding to the error code is returned. In
- * that case the output string is not allocated. The contents of *@outs are
- * then undefined.
- *
- * This might look a bit odd due to fast path optimization...
- */
-int ntfs_ucstonls(const ntfs_volume *vol, const ntfschar *ins,
- const int ins_len, unsigned char **outs, int outs_len)
-{
- struct nls_table *nls = vol->nls_map;
- unsigned char *ns;
- int i, o, ns_len, wc;
-
- /* We don't trust outside sources. */
- if (ins) {
- ns = *outs;
- ns_len = outs_len;
- if (ns && !ns_len) {
- wc = -ENAMETOOLONG;
- goto conversion_err;
- }
- if (!ns) {
- ns_len = ins_len * NLS_MAX_CHARSET_SIZE;
- ns = kmalloc(ns_len + 1, GFP_NOFS);
- if (!ns)
- goto mem_err_out;
- }
- for (i = o = 0; i < ins_len; i++) {
-retry: wc = nls->uni2char(le16_to_cpu(ins[i]), ns + o,
- ns_len - o);
- if (wc > 0) {
- o += wc;
- continue;
- } else if (!wc)
- break;
- else if (wc == -ENAMETOOLONG && ns != *outs) {
- unsigned char *tc;
- /* Grow in multiples of 64 bytes. */
- tc = kmalloc((ns_len + 64) &
- ~63, GFP_NOFS);
- if (tc) {
- memcpy(tc, ns, ns_len);
- ns_len = ((ns_len + 64) & ~63) - 1;
- kfree(ns);
- ns = tc;
- goto retry;
- } /* No memory so goto conversion_error; */
- } /* wc < 0, real error. */
- goto conversion_err;
- }
- ns[o] = 0;
- *outs = ns;
- return o;
- } /* else (!ins) */
- ntfs_error(vol->sb, "Received NULL pointer.");
- return -EINVAL;
-conversion_err:
- ntfs_error(vol->sb, "Unicode name contains characters that cannot be "
- "converted to character set %s. You might want to "
- "try to use the mount option nls=utf8.", nls->charset);
- if (ns != *outs)
- kfree(ns);
- if (wc != -ENAMETOOLONG)
- wc = -EILSEQ;
- return wc;
-mem_err_out:
- ntfs_error(vol->sb, "Failed to allocate name!");
- return -ENOMEM;
-}
diff --git a/fs/ntfs/upcase.c b/fs/ntfs/upcase.c
deleted file mode 100644
index 4ebe84a78dea..000000000000
--- a/fs/ntfs/upcase.c
+++ /dev/null
@@ -1,73 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * upcase.c - Generate the full NTFS Unicode upcase table in little endian.
- * Part of the Linux-NTFS project.
- *
- * Copyright (c) 2001 Richard Russon <ntfs@flatcap.org>
- * Copyright (c) 2001-2006 Anton Altaparmakov
- */
-
-#include "malloc.h"
-#include "ntfs.h"
-
-ntfschar *generate_default_upcase(void)
-{
- static const int uc_run_table[][3] = { /* Start, End, Add */
- {0x0061, 0x007B, -32}, {0x0451, 0x045D, -80}, {0x1F70, 0x1F72, 74},
- {0x00E0, 0x00F7, -32}, {0x045E, 0x0460, -80}, {0x1F72, 0x1F76, 86},
- {0x00F8, 0x00FF, -32}, {0x0561, 0x0587, -48}, {0x1F76, 0x1F78, 100},
- {0x0256, 0x0258, -205}, {0x1F00, 0x1F08, 8}, {0x1F78, 0x1F7A, 128},
- {0x028A, 0x028C, -217}, {0x1F10, 0x1F16, 8}, {0x1F7A, 0x1F7C, 112},
- {0x03AC, 0x03AD, -38}, {0x1F20, 0x1F28, 8}, {0x1F7C, 0x1F7E, 126},
- {0x03AD, 0x03B0, -37}, {0x1F30, 0x1F38, 8}, {0x1FB0, 0x1FB2, 8},
- {0x03B1, 0x03C2, -32}, {0x1F40, 0x1F46, 8}, {0x1FD0, 0x1FD2, 8},
- {0x03C2, 0x03C3, -31}, {0x1F51, 0x1F52, 8}, {0x1FE0, 0x1FE2, 8},
- {0x03C3, 0x03CC, -32}, {0x1F53, 0x1F54, 8}, {0x1FE5, 0x1FE6, 7},
- {0x03CC, 0x03CD, -64}, {0x1F55, 0x1F56, 8}, {0x2170, 0x2180, -16},
- {0x03CD, 0x03CF, -63}, {0x1F57, 0x1F58, 8}, {0x24D0, 0x24EA, -26},
- {0x0430, 0x0450, -32}, {0x1F60, 0x1F68, 8}, {0xFF41, 0xFF5B, -32},
- {0}
- };
-
- static const int uc_dup_table[][2] = { /* Start, End */
- {0x0100, 0x012F}, {0x01A0, 0x01A6}, {0x03E2, 0x03EF}, {0x04CB, 0x04CC},
- {0x0132, 0x0137}, {0x01B3, 0x01B7}, {0x0460, 0x0481}, {0x04D0, 0x04EB},
- {0x0139, 0x0149}, {0x01CD, 0x01DD}, {0x0490, 0x04BF}, {0x04EE, 0x04F5},
- {0x014A, 0x0178}, {0x01DE, 0x01EF}, {0x04BF, 0x04BF}, {0x04F8, 0x04F9},
- {0x0179, 0x017E}, {0x01F4, 0x01F5}, {0x04C1, 0x04C4}, {0x1E00, 0x1E95},
- {0x018B, 0x018B}, {0x01FA, 0x0218}, {0x04C7, 0x04C8}, {0x1EA0, 0x1EF9},
- {0}
- };
-
- static const int uc_word_table[][2] = { /* Offset, Value */
- {0x00FF, 0x0178}, {0x01AD, 0x01AC}, {0x01F3, 0x01F1}, {0x0269, 0x0196},
- {0x0183, 0x0182}, {0x01B0, 0x01AF}, {0x0253, 0x0181}, {0x026F, 0x019C},
- {0x0185, 0x0184}, {0x01B9, 0x01B8}, {0x0254, 0x0186}, {0x0272, 0x019D},
- {0x0188, 0x0187}, {0x01BD, 0x01BC}, {0x0259, 0x018F}, {0x0275, 0x019F},
- {0x018C, 0x018B}, {0x01C6, 0x01C4}, {0x025B, 0x0190}, {0x0283, 0x01A9},
- {0x0192, 0x0191}, {0x01C9, 0x01C7}, {0x0260, 0x0193}, {0x0288, 0x01AE},
- {0x0199, 0x0198}, {0x01CC, 0x01CA}, {0x0263, 0x0194}, {0x0292, 0x01B7},
- {0x01A8, 0x01A7}, {0x01DD, 0x018E}, {0x0268, 0x0197},
- {0}
- };
-
- int i, r;
- ntfschar *uc;
-
- uc = ntfs_malloc_nofs(default_upcase_len * sizeof(ntfschar));
- if (!uc)
- return uc;
- memset(uc, 0, default_upcase_len * sizeof(ntfschar));
- /* Generate the little endian Unicode upcase table used by ntfs. */
- for (i = 0; i < default_upcase_len; i++)
- uc[i] = cpu_to_le16(i);
- for (r = 0; uc_run_table[r][0]; r++)
- for (i = uc_run_table[r][0]; i < uc_run_table[r][1]; i++)
- le16_add_cpu(&uc[i], uc_run_table[r][2]);
- for (r = 0; uc_dup_table[r][0]; r++)
- for (i = uc_dup_table[r][0]; i < uc_dup_table[r][1]; i += 2)
- le16_add_cpu(&uc[i + 1], -1);
- for (r = 0; uc_word_table[r][0]; r++)
- uc[uc_word_table[r][0]] = cpu_to_le16(uc_word_table[r][1]);
- return uc;
-}
diff --git a/fs/ntfs/usnjrnl.c b/fs/ntfs/usnjrnl.c
deleted file mode 100644
index 9097a0b4ef25..000000000000
--- a/fs/ntfs/usnjrnl.c
+++ /dev/null
@@ -1,70 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * usnjrnl.h - NTFS kernel transaction log ($UsnJrnl) handling. Part of the
- * Linux-NTFS project.
- *
- * Copyright (c) 2005 Anton Altaparmakov
- */
-
-#ifdef NTFS_RW
-
-#include <linux/fs.h>
-#include <linux/highmem.h>
-#include <linux/mm.h>
-
-#include "aops.h"
-#include "debug.h"
-#include "endian.h"
-#include "time.h"
-#include "types.h"
-#include "usnjrnl.h"
-#include "volume.h"
-
-/**
- * ntfs_stamp_usnjrnl - stamp the transaction log ($UsnJrnl) on an ntfs volume
- * @vol: ntfs volume on which to stamp the transaction log
- *
- * Stamp the transaction log ($UsnJrnl) on the ntfs volume @vol and return
- * 'true' on success and 'false' on error.
- *
- * This function assumes that the transaction log has already been loaded and
- * consistency checked by a call to fs/ntfs/super.c::load_and_init_usnjrnl().
- */
-bool ntfs_stamp_usnjrnl(ntfs_volume *vol)
-{
- ntfs_debug("Entering.");
- if (likely(!NVolUsnJrnlStamped(vol))) {
- sle64 stamp;
- struct page *page;
- USN_HEADER *uh;
-
- page = ntfs_map_page(vol->usnjrnl_max_ino->i_mapping, 0);
- if (IS_ERR(page)) {
- ntfs_error(vol->sb, "Failed to read from "
- "$UsnJrnl/$DATA/$Max attribute.");
- return false;
- }
- uh = (USN_HEADER*)page_address(page);
- stamp = get_current_ntfs_time();
- ntfs_debug("Stamping transaction log ($UsnJrnl): old "
- "journal_id 0x%llx, old lowest_valid_usn "
- "0x%llx, new journal_id 0x%llx, new "
- "lowest_valid_usn 0x%llx.",
- (long long)sle64_to_cpu(uh->journal_id),
- (long long)sle64_to_cpu(uh->lowest_valid_usn),
- (long long)sle64_to_cpu(stamp),
- i_size_read(vol->usnjrnl_j_ino));
- uh->lowest_valid_usn =
- cpu_to_sle64(i_size_read(vol->usnjrnl_j_ino));
- uh->journal_id = stamp;
- flush_dcache_page(page);
- set_page_dirty(page);
- ntfs_unmap_page(page);
- /* Set the flag so we do not have to do it again on remount. */
- NVolSetUsnJrnlStamped(vol);
- }
- ntfs_debug("Done.");
- return true;
-}
-
-#endif /* NTFS_RW */
diff --git a/fs/ntfs/usnjrnl.h b/fs/ntfs/usnjrnl.h
deleted file mode 100644
index 85f531b59395..000000000000
--- a/fs/ntfs/usnjrnl.h
+++ /dev/null
@@ -1,191 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * usnjrnl.h - Defines for NTFS kernel transaction log ($UsnJrnl) handling.
- * Part of the Linux-NTFS project.
- *
- * Copyright (c) 2005 Anton Altaparmakov
- */
-
-#ifndef _LINUX_NTFS_USNJRNL_H
-#define _LINUX_NTFS_USNJRNL_H
-
-#ifdef NTFS_RW
-
-#include "types.h"
-#include "endian.h"
-#include "layout.h"
-#include "volume.h"
-
-/*
- * Transaction log ($UsnJrnl) organization:
- *
- * The transaction log records whenever a file is modified in any way. So for
- * example it will record that file "blah" was written to at a particular time
- * but not what was written. If will record that a file was deleted or
- * created, that a file was truncated, etc. See below for all the reason
- * codes used.
- *
- * The transaction log is in the $Extend directory which is in the root
- * directory of each volume. If it is not present it means transaction
- * logging is disabled. If it is present it means transaction logging is
- * either enabled or in the process of being disabled in which case we can
- * ignore it as it will go away as soon as Windows gets its hands on it.
- *
- * To determine whether the transaction logging is enabled or in the process
- * of being disabled, need to check the volume flags in the
- * $VOLUME_INFORMATION attribute in the $Volume system file (which is present
- * in the root directory and has a fixed mft record number, see layout.h).
- * If the flag VOLUME_DELETE_USN_UNDERWAY is set it means the transaction log
- * is in the process of being disabled and if this flag is clear it means the
- * transaction log is enabled.
- *
- * The transaction log consists of two parts; the $DATA/$Max attribute as well
- * as the $DATA/$J attribute. $Max is a header describing the transaction
- * log whilst $J is the transaction log data itself as a sequence of variable
- * sized USN_RECORDs (see below for all the structures).
- *
- * We do not care about transaction logging at this point in time but we still
- * need to let windows know that the transaction log is out of date. To do
- * this we need to stamp the transaction log. This involves setting the
- * lowest_valid_usn field in the $DATA/$Max attribute to the usn to be used
- * for the next added USN_RECORD to the $DATA/$J attribute as well as
- * generating a new journal_id in $DATA/$Max.
- *
- * The journal_id is as of the current version (2.0) of the transaction log
- * simply the 64-bit timestamp of when the journal was either created or last
- * stamped.
- *
- * To determine the next usn there are two ways. The first is to parse
- * $DATA/$J and to find the last USN_RECORD in it and to add its record_length
- * to its usn (which is the byte offset in the $DATA/$J attribute). The
- * second is simply to take the data size of the attribute. Since the usns
- * are simply byte offsets into $DATA/$J, this is exactly the next usn. For
- * obvious reasons we use the second method as it is much simpler and faster.
- *
- * As an aside, note that to actually disable the transaction log, one would
- * need to set the VOLUME_DELETE_USN_UNDERWAY flag (see above), then go
- * through all the mft records on the volume and set the usn field in their
- * $STANDARD_INFORMATION attribute to zero. Once that is done, one would need
- * to delete the transaction log file, i.e. \$Extent\$UsnJrnl, and finally,
- * one would need to clear the VOLUME_DELETE_USN_UNDERWAY flag.
- *
- * Note that if a volume is unmounted whilst the transaction log is being
- * disabled, the process will continue the next time the volume is mounted.
- * This is why we can safely mount read-write when we see a transaction log
- * in the process of being deleted.
- */
-
-/* Some $UsnJrnl related constants. */
-#define UsnJrnlMajorVer 2
-#define UsnJrnlMinorVer 0
-
-/*
- * $DATA/$Max attribute. This is (always?) resident and has a fixed size of
- * 32 bytes. It contains the header describing the transaction log.
- */
-typedef struct {
-/*Ofs*/
-/* 0*/sle64 maximum_size; /* The maximum on-disk size of the $DATA/$J
- attribute. */
-/* 8*/sle64 allocation_delta; /* Number of bytes by which to increase the
- size of the $DATA/$J attribute. */
-/*0x10*/sle64 journal_id; /* Current id of the transaction log. */
-/*0x18*/leUSN lowest_valid_usn; /* Lowest valid usn in $DATA/$J for the
- current journal_id. */
-/* sizeof() = 32 (0x20) bytes */
-} __attribute__ ((__packed__)) USN_HEADER;
-
-/*
- * Reason flags (32-bit). Cumulative flags describing the change(s) to the
- * file since it was last opened. I think the names speak for themselves but
- * if you disagree check out the descriptions in the Linux NTFS project NTFS
- * documentation: http://www.linux-ntfs.org/
- */
-enum {
- USN_REASON_DATA_OVERWRITE = cpu_to_le32(0x00000001),
- USN_REASON_DATA_EXTEND = cpu_to_le32(0x00000002),
- USN_REASON_DATA_TRUNCATION = cpu_to_le32(0x00000004),
- USN_REASON_NAMED_DATA_OVERWRITE = cpu_to_le32(0x00000010),
- USN_REASON_NAMED_DATA_EXTEND = cpu_to_le32(0x00000020),
- USN_REASON_NAMED_DATA_TRUNCATION= cpu_to_le32(0x00000040),
- USN_REASON_FILE_CREATE = cpu_to_le32(0x00000100),
- USN_REASON_FILE_DELETE = cpu_to_le32(0x00000200),
- USN_REASON_EA_CHANGE = cpu_to_le32(0x00000400),
- USN_REASON_SECURITY_CHANGE = cpu_to_le32(0x00000800),
- USN_REASON_RENAME_OLD_NAME = cpu_to_le32(0x00001000),
- USN_REASON_RENAME_NEW_NAME = cpu_to_le32(0x00002000),
- USN_REASON_INDEXABLE_CHANGE = cpu_to_le32(0x00004000),
- USN_REASON_BASIC_INFO_CHANGE = cpu_to_le32(0x00008000),
- USN_REASON_HARD_LINK_CHANGE = cpu_to_le32(0x00010000),
- USN_REASON_COMPRESSION_CHANGE = cpu_to_le32(0x00020000),
- USN_REASON_ENCRYPTION_CHANGE = cpu_to_le32(0x00040000),
- USN_REASON_OBJECT_ID_CHANGE = cpu_to_le32(0x00080000),
- USN_REASON_REPARSE_POINT_CHANGE = cpu_to_le32(0x00100000),
- USN_REASON_STREAM_CHANGE = cpu_to_le32(0x00200000),
- USN_REASON_CLOSE = cpu_to_le32(0x80000000),
-};
-
-typedef le32 USN_REASON_FLAGS;
-
-/*
- * Source info flags (32-bit). Information about the source of the change(s)
- * to the file. For detailed descriptions of what these mean, see the Linux
- * NTFS project NTFS documentation:
- * http://www.linux-ntfs.org/
- */
-enum {
- USN_SOURCE_DATA_MANAGEMENT = cpu_to_le32(0x00000001),
- USN_SOURCE_AUXILIARY_DATA = cpu_to_le32(0x00000002),
- USN_SOURCE_REPLICATION_MANAGEMENT = cpu_to_le32(0x00000004),
-};
-
-typedef le32 USN_SOURCE_INFO_FLAGS;
-
-/*
- * $DATA/$J attribute. This is always non-resident, is marked as sparse, and
- * is of variabled size. It consists of a sequence of variable size
- * USN_RECORDS. The minimum allocated_size is allocation_delta as
- * specified in $DATA/$Max. When the maximum_size specified in $DATA/$Max is
- * exceeded by more than allocation_delta bytes, allocation_delta bytes are
- * allocated and appended to the $DATA/$J attribute and an equal number of
- * bytes at the beginning of the attribute are freed and made sparse. Note the
- * making sparse only happens at volume checkpoints and hence the actual
- * $DATA/$J size can exceed maximum_size + allocation_delta temporarily.
- */
-typedef struct {
-/*Ofs*/
-/* 0*/le32 length; /* Byte size of this record (8-byte
- aligned). */
-/* 4*/le16 major_ver; /* Major version of the transaction log used
- for this record. */
-/* 6*/le16 minor_ver; /* Minor version of the transaction log used
- for this record. */
-/* 8*/leMFT_REF mft_reference;/* The mft reference of the file (or
- directory) described by this record. */
-/*0x10*/leMFT_REF parent_directory;/* The mft reference of the parent
- directory of the file described by this
- record. */
-/*0x18*/leUSN usn; /* The usn of this record. Equals the offset
- within the $DATA/$J attribute. */
-/*0x20*/sle64 time; /* Time when this record was created. */
-/*0x28*/USN_REASON_FLAGS reason;/* Reason flags (see above). */
-/*0x2c*/USN_SOURCE_INFO_FLAGS source_info;/* Source info flags (see above). */
-/*0x30*/le32 security_id; /* File security_id copied from
- $STANDARD_INFORMATION. */
-/*0x34*/FILE_ATTR_FLAGS file_attributes; /* File attributes copied from
- $STANDARD_INFORMATION or $FILE_NAME (not
- sure which). */
-/*0x38*/le16 file_name_size; /* Size of the file name in bytes. */
-/*0x3a*/le16 file_name_offset; /* Offset to the file name in bytes from the
- start of this record. */
-/*0x3c*/ntfschar file_name[0]; /* Use when creating only. When reading use
- file_name_offset to determine the location
- of the name. */
-/* sizeof() = 60 (0x3c) bytes */
-} __attribute__ ((__packed__)) USN_RECORD;
-
-extern bool ntfs_stamp_usnjrnl(ntfs_volume *vol);
-
-#endif /* NTFS_RW */
-
-#endif /* _LINUX_NTFS_USNJRNL_H */
diff --git a/fs/ntfs/volume.h b/fs/ntfs/volume.h
deleted file mode 100644
index 930a9ae8a053..000000000000
--- a/fs/ntfs/volume.h
+++ /dev/null
@@ -1,164 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * volume.h - Defines for volume structures in NTFS Linux kernel driver. Part
- * of the Linux-NTFS project.
- *
- * Copyright (c) 2001-2006 Anton Altaparmakov
- * Copyright (c) 2002 Richard Russon
- */
-
-#ifndef _LINUX_NTFS_VOLUME_H
-#define _LINUX_NTFS_VOLUME_H
-
-#include <linux/rwsem.h>
-#include <linux/uidgid.h>
-
-#include "types.h"
-#include "layout.h"
-
-/*
- * The NTFS in memory super block structure.
- */
-typedef struct {
- /*
- * FIXME: Reorder to have commonly used together element within the
- * same cache line, aiming at a cache line size of 32 bytes. Aim for
- * 64 bytes for less commonly used together elements. Put most commonly
- * used elements to front of structure. Obviously do this only when the
- * structure has stabilized... (AIA)
- */
- /* Device specifics. */
- struct super_block *sb; /* Pointer back to the super_block. */
- LCN nr_blocks; /* Number of sb->s_blocksize bytes
- sized blocks on the device. */
- /* Configuration provided by user at mount time. */
- unsigned long flags; /* Miscellaneous flags, see below. */
- kuid_t uid; /* uid that files will be mounted as. */
- kgid_t gid; /* gid that files will be mounted as. */
- umode_t fmask; /* The mask for file permissions. */
- umode_t dmask; /* The mask for directory
- permissions. */
- u8 mft_zone_multiplier; /* Initial mft zone multiplier. */
- u8 on_errors; /* What to do on filesystem errors. */
- /* NTFS bootsector provided information. */
- u16 sector_size; /* in bytes */
- u8 sector_size_bits; /* log2(sector_size) */
- u32 cluster_size; /* in bytes */
- u32 cluster_size_mask; /* cluster_size - 1 */
- u8 cluster_size_bits; /* log2(cluster_size) */
- u32 mft_record_size; /* in bytes */
- u32 mft_record_size_mask; /* mft_record_size - 1 */
- u8 mft_record_size_bits; /* log2(mft_record_size) */
- u32 index_record_size; /* in bytes */
- u32 index_record_size_mask; /* index_record_size - 1 */
- u8 index_record_size_bits; /* log2(index_record_size) */
- LCN nr_clusters; /* Volume size in clusters == number of
- bits in lcn bitmap. */
- LCN mft_lcn; /* Cluster location of mft data. */
- LCN mftmirr_lcn; /* Cluster location of copy of mft. */
- u64 serial_no; /* The volume serial number. */
- /* Mount specific NTFS information. */
- u32 upcase_len; /* Number of entries in upcase[]. */
- ntfschar *upcase; /* The upcase table. */
-
- s32 attrdef_size; /* Size of the attribute definition
- table in bytes. */
- ATTR_DEF *attrdef; /* Table of attribute definitions.
- Obtained from FILE_AttrDef. */
-
-#ifdef NTFS_RW
- /* Variables used by the cluster and mft allocators. */
- s64 mft_data_pos; /* Mft record number at which to
- allocate the next mft record. */
- LCN mft_zone_start; /* First cluster of the mft zone. */
- LCN mft_zone_end; /* First cluster beyond the mft zone. */
- LCN mft_zone_pos; /* Current position in the mft zone. */
- LCN data1_zone_pos; /* Current position in the first data
- zone. */
- LCN data2_zone_pos; /* Current position in the second data
- zone. */
-#endif /* NTFS_RW */
-
- struct inode *mft_ino; /* The VFS inode of $MFT. */
-
- struct inode *mftbmp_ino; /* Attribute inode for $MFT/$BITMAP. */
- struct rw_semaphore mftbmp_lock; /* Lock for serializing accesses to the
- mft record bitmap ($MFT/$BITMAP). */
-#ifdef NTFS_RW
- struct inode *mftmirr_ino; /* The VFS inode of $MFTMirr. */
- int mftmirr_size; /* Size of mft mirror in mft records. */
-
- struct inode *logfile_ino; /* The VFS inode of $LogFile. */
-#endif /* NTFS_RW */
-
- struct inode *lcnbmp_ino; /* The VFS inode of $Bitmap. */
- struct rw_semaphore lcnbmp_lock; /* Lock for serializing accesses to the
- cluster bitmap ($Bitmap/$DATA). */
-
- struct inode *vol_ino; /* The VFS inode of $Volume. */
- VOLUME_FLAGS vol_flags; /* Volume flags. */
- u8 major_ver; /* Ntfs major version of volume. */
- u8 minor_ver; /* Ntfs minor version of volume. */
-
- struct inode *root_ino; /* The VFS inode of the root
- directory. */
- struct inode *secure_ino; /* The VFS inode of $Secure (NTFS3.0+
- only, otherwise NULL). */
- struct inode *extend_ino; /* The VFS inode of $Extend (NTFS3.0+
- only, otherwise NULL). */
-#ifdef NTFS_RW
- /* $Quota stuff is NTFS3.0+ specific. Unused/NULL otherwise. */
- struct inode *quota_ino; /* The VFS inode of $Quota. */
- struct inode *quota_q_ino; /* Attribute inode for $Quota/$Q. */
- /* $UsnJrnl stuff is NTFS3.0+ specific. Unused/NULL otherwise. */
- struct inode *usnjrnl_ino; /* The VFS inode of $UsnJrnl. */
- struct inode *usnjrnl_max_ino; /* Attribute inode for $UsnJrnl/$Max. */
- struct inode *usnjrnl_j_ino; /* Attribute inode for $UsnJrnl/$J. */
-#endif /* NTFS_RW */
- struct nls_table *nls_map;
-} ntfs_volume;
-
-/*
- * Defined bits for the flags field in the ntfs_volume structure.
- */
-typedef enum {
- NV_Errors, /* 1: Volume has errors, prevent remount rw. */
- NV_ShowSystemFiles, /* 1: Return system files in ntfs_readdir(). */
- NV_CaseSensitive, /* 1: Treat file names as case sensitive and
- create filenames in the POSIX namespace.
- Otherwise be case insensitive but still
- create file names in POSIX namespace. */
- NV_LogFileEmpty, /* 1: $LogFile journal is empty. */
- NV_QuotaOutOfDate, /* 1: $Quota is out of date. */
- NV_UsnJrnlStamped, /* 1: $UsnJrnl has been stamped. */
- NV_SparseEnabled, /* 1: May create sparse files. */
-} ntfs_volume_flags;
-
-/*
- * Macro tricks to expand the NVolFoo(), NVolSetFoo(), and NVolClearFoo()
- * functions.
- */
-#define DEFINE_NVOL_BIT_OPS(flag) \
-static inline int NVol##flag(ntfs_volume *vol) \
-{ \
- return test_bit(NV_##flag, &(vol)->flags); \
-} \
-static inline void NVolSet##flag(ntfs_volume *vol) \
-{ \
- set_bit(NV_##flag, &(vol)->flags); \
-} \
-static inline void NVolClear##flag(ntfs_volume *vol) \
-{ \
- clear_bit(NV_##flag, &(vol)->flags); \
-}
-
-/* Emit the ntfs volume bitops functions. */
-DEFINE_NVOL_BIT_OPS(Errors)
-DEFINE_NVOL_BIT_OPS(ShowSystemFiles)
-DEFINE_NVOL_BIT_OPS(CaseSensitive)
-DEFINE_NVOL_BIT_OPS(LogFileEmpty)
-DEFINE_NVOL_BIT_OPS(QuotaOutOfDate)
-DEFINE_NVOL_BIT_OPS(UsnJrnlStamped)
-DEFINE_NVOL_BIT_OPS(SparseEnabled)
-
-#endif /* _LINUX_NTFS_VOLUME_H */
diff --git a/fs/ntfs3/attrib.c b/fs/ntfs3/attrib.c
index f4573d276a2c..c037f5307657 100644
--- a/fs/ntfs3/attrib.c
+++ b/fs/ntfs3/attrib.c
@@ -815,7 +815,8 @@ int attr_set_size(struct ntfs_inode *ni, enum ATTR_TYPE type,
if (err)
goto out;
- attr = mi_find_attr(mi, NULL, type, name, name_len, &le->id);
+ attr = mi_find_attr(ni, mi, NULL, type, name, name_len,
+ &le->id);
if (!attr) {
err = -EINVAL;
goto bad_inode;
@@ -1047,6 +1048,22 @@ int attr_data_get_block(struct ntfs_inode *ni, CLST vcn, CLST clen, CLST *lcn,
if (err)
goto out;
}
+
+ if (vcn0 < svcn || evcn1 <= vcn0) {
+ struct ATTRIB *attr2;
+ struct ATTR_LIST_ENTRY *le2 = le_b;
+ struct mft_inode *mi2;
+
+ attr2 = ni_find_attr(ni, attr_b, &le2, ATTR_DATA, NULL,
+ 0, &vcn0, &mi2);
+ if (!attr2) {
+ err = -EINVAL;
+ goto out;
+ }
+ err = attr_load_runs(attr2, ni, run, NULL);
+ if (err)
+ goto out;
+ }
}
if (vcn + to_alloc > asize)
@@ -1166,7 +1183,7 @@ int attr_data_get_block(struct ntfs_inode *ni, CLST vcn, CLST clen, CLST *lcn,
goto out;
}
- attr = mi_find_attr(mi, NULL, ATTR_DATA, NULL, 0,
+ attr = mi_find_attr(ni, mi, NULL, ATTR_DATA, NULL, 0,
&le->id);
if (!attr) {
err = -EINVAL;
@@ -1786,7 +1803,7 @@ int attr_allocate_frame(struct ntfs_inode *ni, CLST frame, size_t compr_size,
goto out;
}
- attr = mi_find_attr(mi, NULL, ATTR_DATA, NULL, 0,
+ attr = mi_find_attr(ni, mi, NULL, ATTR_DATA, NULL, 0,
&le->id);
if (!attr) {
err = -EINVAL;
@@ -2031,8 +2048,8 @@ int attr_collapse_range(struct ntfs_inode *ni, u64 vbo, u64 bytes)
}
/* Look for required attribute. */
- attr = mi_find_attr(mi, NULL, ATTR_DATA, NULL,
- 0, &le->id);
+ attr = mi_find_attr(ni, mi, NULL, ATTR_DATA,
+ NULL, 0, &le->id);
if (!attr) {
err = -EINVAL;
goto out;
diff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c
index f2d1df2c988a..81b51d5365f2 100644
--- a/fs/ntfs3/file.c
+++ b/fs/ntfs3/file.c
@@ -390,8 +390,8 @@ static int ntfs_truncate(struct inode *inode, loff_t new_size)
{
struct super_block *sb = inode->i_sb;
struct ntfs_inode *ni = ntfs_i(inode);
- int err, dirty = 0;
u64 new_valid;
+ int err;
if (!S_ISREG(inode->i_mode))
return 0;
@@ -407,7 +407,6 @@ static int ntfs_truncate(struct inode *inode, loff_t new_size)
}
new_valid = ntfs_up_block(sb, min_t(u64, ni->i_valid, new_size));
-
truncate_setsize(inode, new_size);
ni_lock(ni);
@@ -421,20 +420,19 @@ static int ntfs_truncate(struct inode *inode, loff_t new_size)
ni->i_valid = new_valid;
ni_unlock(ni);
+ if (unlikely(err))
+ return err;
ni->std_fa |= FILE_ATTRIBUTE_ARCHIVE;
inode->i_ctime = inode->i_mtime = current_time(inode);
if (!IS_DIRSYNC(inode)) {
- dirty = 1;
+ mark_inode_dirty(inode);
} else {
err = ntfs_sync_inode(inode);
if (err)
return err;
}
- if (dirty)
- mark_inode_dirty(inode);
-
/*ntfs_flush_inodes(inode->i_sb, inode, NULL);*/
return 0;
@@ -1049,25 +1047,28 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from)
ssize_t ret;
struct ntfs_inode *ni = ntfs_i(inode);
+ if (!inode_trylock(inode)) {
+ if (iocb->ki_flags & IOCB_NOWAIT)
+ return -EAGAIN;
+ inode_lock(inode);
+ }
+
if (is_encrypted(ni)) {
ntfs_inode_warn(inode, "encrypted i/o not supported");
- return -EOPNOTSUPP;
+ ret = -EOPNOTSUPP;
+ goto out;
}
if (is_compressed(ni) && (iocb->ki_flags & IOCB_DIRECT)) {
ntfs_inode_warn(inode, "direct i/o + compressed not supported");
- return -EOPNOTSUPP;
+ ret = -EOPNOTSUPP;
+ goto out;
}
if (is_dedup(ni)) {
ntfs_inode_warn(inode, "write into deduplicated not supported");
- return -EOPNOTSUPP;
- }
-
- if (!inode_trylock(inode)) {
- if (iocb->ki_flags & IOCB_NOWAIT)
- return -EAGAIN;
- inode_lock(inode);
+ ret = -EOPNOTSUPP;
+ goto out;
}
ret = generic_write_checks(iocb, from);
diff --git a/fs/ntfs3/frecord.c b/fs/ntfs3/frecord.c
index fcdaea02ba04..a44b7338621e 100644
--- a/fs/ntfs3/frecord.c
+++ b/fs/ntfs3/frecord.c
@@ -75,7 +75,7 @@ struct ATTR_STD_INFO *ni_std(struct ntfs_inode *ni)
{
const struct ATTRIB *attr;
- attr = mi_find_attr(&ni->mi, NULL, ATTR_STD, NULL, 0, NULL);
+ attr = mi_find_attr(ni, &ni->mi, NULL, ATTR_STD, NULL, 0, NULL);
return attr ? resident_data_ex(attr, sizeof(struct ATTR_STD_INFO))
: NULL;
}
@@ -89,7 +89,7 @@ struct ATTR_STD_INFO5 *ni_std5(struct ntfs_inode *ni)
{
const struct ATTRIB *attr;
- attr = mi_find_attr(&ni->mi, NULL, ATTR_STD, NULL, 0, NULL);
+ attr = mi_find_attr(ni, &ni->mi, NULL, ATTR_STD, NULL, 0, NULL);
return attr ? resident_data_ex(attr, sizeof(struct ATTR_STD_INFO5))
: NULL;
@@ -201,7 +201,8 @@ struct ATTRIB *ni_find_attr(struct ntfs_inode *ni, struct ATTRIB *attr,
*mi = &ni->mi;
/* Look for required attribute in primary record. */
- return mi_find_attr(&ni->mi, attr, type, name, name_len, NULL);
+ return mi_find_attr(ni, &ni->mi, attr, type, name, name_len,
+ NULL);
}
/* First look for list entry of required type. */
@@ -217,7 +218,7 @@ struct ATTRIB *ni_find_attr(struct ntfs_inode *ni, struct ATTRIB *attr,
return NULL;
/* Look for required attribute. */
- attr = mi_find_attr(m, NULL, type, name, name_len, &le->id);
+ attr = mi_find_attr(ni, m, NULL, type, name, name_len, &le->id);
if (!attr)
goto out;
@@ -258,7 +259,7 @@ struct ATTRIB *ni_enum_attr_ex(struct ntfs_inode *ni, struct ATTRIB *attr,
if (mi)
*mi = &ni->mi;
/* Enum attributes in primary record. */
- return mi_enum_attr(&ni->mi, attr);
+ return mi_enum_attr(ni, &ni->mi, attr);
}
/* Get next list entry. */
@@ -274,7 +275,7 @@ struct ATTRIB *ni_enum_attr_ex(struct ntfs_inode *ni, struct ATTRIB *attr,
*mi = mi2;
/* Find attribute in loaded record. */
- return rec_find_attr_le(mi2, le2);
+ return rec_find_attr_le(ni, mi2, le2);
}
/*
@@ -292,7 +293,8 @@ struct ATTRIB *ni_load_attr(struct ntfs_inode *ni, enum ATTR_TYPE type,
if (!ni->attr_list.size) {
if (pmi)
*pmi = &ni->mi;
- return mi_find_attr(&ni->mi, NULL, type, name, name_len, NULL);
+ return mi_find_attr(ni, &ni->mi, NULL, type, name, name_len,
+ NULL);
}
le = al_find_ex(ni, NULL, type, name, name_len, NULL);
@@ -318,7 +320,7 @@ struct ATTRIB *ni_load_attr(struct ntfs_inode *ni, enum ATTR_TYPE type,
if (pmi)
*pmi = mi;
- attr = mi_find_attr(mi, NULL, type, name, name_len, &le->id);
+ attr = mi_find_attr(ni, mi, NULL, type, name, name_len, &le->id);
if (!attr)
return NULL;
@@ -399,7 +401,8 @@ int ni_remove_attr(struct ntfs_inode *ni, enum ATTR_TYPE type,
int diff;
if (base_only || type == ATTR_LIST || !ni->attr_list.size) {
- attr = mi_find_attr(&ni->mi, NULL, type, name, name_len, id);
+ attr = mi_find_attr(ni, &ni->mi, NULL, type, name, name_len,
+ id);
if (!attr)
return -ENOENT;
@@ -438,7 +441,7 @@ int ni_remove_attr(struct ntfs_inode *ni, enum ATTR_TYPE type,
al_remove_le(ni, le);
- attr = mi_find_attr(mi, NULL, type, name, name_len, id);
+ attr = mi_find_attr(ni, mi, NULL, type, name, name_len, id);
if (!attr)
return -ENOENT;
@@ -486,7 +489,7 @@ ni_ins_new_attr(struct ntfs_inode *ni, struct mft_inode *mi,
name = le->name;
}
- attr = mi_insert_attr(mi, type, name, name_len, asize, name_off);
+ attr = mi_insert_attr(ni, mi, type, name, name_len, asize, name_off);
if (!attr) {
if (le_added)
al_remove_le(ni, le);
@@ -670,7 +673,7 @@ static int ni_try_remove_attr_list(struct ntfs_inode *ni)
if (err)
return err;
- attr_list = mi_find_attr(&ni->mi, NULL, ATTR_LIST, NULL, 0, NULL);
+ attr_list = mi_find_attr(ni, &ni->mi, NULL, ATTR_LIST, NULL, 0, NULL);
if (!attr_list)
return 0;
@@ -692,7 +695,7 @@ static int ni_try_remove_attr_list(struct ntfs_inode *ni)
if (!mi)
return 0;
- attr = mi_find_attr(mi, NULL, le->type, le_name(le),
+ attr = mi_find_attr(ni, mi, NULL, le->type, le_name(le),
le->name_len, &le->id);
if (!attr)
return 0;
@@ -728,7 +731,7 @@ static int ni_try_remove_attr_list(struct ntfs_inode *ni)
goto out;
}
- attr = mi_find_attr(mi, NULL, le->type, le_name(le),
+ attr = mi_find_attr(ni, mi, NULL, le->type, le_name(le),
le->name_len, &le->id);
if (!attr) {
/* Should never happened, 'cause already checked. */
@@ -737,7 +740,7 @@ static int ni_try_remove_attr_list(struct ntfs_inode *ni)
asize = le32_to_cpu(attr->size);
/* Insert into primary record. */
- attr_ins = mi_insert_attr(&ni->mi, le->type, le_name(le),
+ attr_ins = mi_insert_attr(ni, &ni->mi, le->type, le_name(le),
le->name_len, asize,
le16_to_cpu(attr->name_off));
if (!attr_ins) {
@@ -765,7 +768,7 @@ static int ni_try_remove_attr_list(struct ntfs_inode *ni)
if (!mi)
continue;
- attr = mi_find_attr(mi, NULL, le->type, le_name(le),
+ attr = mi_find_attr(ni, mi, NULL, le->type, le_name(le),
le->name_len, &le->id);
if (!attr)
continue;
@@ -813,10 +816,23 @@ int ni_create_attr_list(struct ntfs_inode *ni)
rs = sbi->record_size;
/*
- * Skip estimating exact memory requirement.
- * Looks like one record_size is always enough.
+ * Compute the exact size of the attribute list. Each attribute in the
+ * record yields one ATTR_LIST_ENTRY of le_size(name_len) bytes. The
+ * minimum on-disk attribute is SIZEOF_RESIDENT (0x18) bytes, but an
+ * unnamed one expands to le_size(0) (0x20) here, so a record crafted
+ * with many such attributes needs more than a single record_size; the
+ * previous fixed kzalloc(record_size) could therefore be overflowed by
+ * an attacker-controlled record.
*/
- le = kmalloc(al_aligned(rs), GFP_NOFS);
+ lsize = 0;
+ attr = NULL;
+ while ((attr = mi_enum_attr(ni, &ni->mi, attr)))
+ lsize += le_size(attr->name_len);
+
+ if (!lsize)
+ return -EINVAL;
+
+ le = kzalloc(al_aligned(lsize), GFP_NOFS);
if (!le) {
err = -ENOMEM;
goto out;
@@ -828,9 +844,8 @@ int ni_create_attr_list(struct ntfs_inode *ni)
attr = NULL;
nb = 0;
free_b = 0;
- attr = NULL;
- for (; (attr = mi_enum_attr(&ni->mi, attr)); le = Add2Ptr(le, sz)) {
+ for (; (attr = mi_enum_attr(ni, &ni->mi, attr)); le = Add2Ptr(le, sz)) {
sz = le_size(attr->name_len);
le->type = attr->type;
le->size = cpu_to_le16(sz);
@@ -885,7 +900,7 @@ int ni_create_attr_list(struct ntfs_inode *ni)
u32 asize = le32_to_cpu(b->size);
u16 name_off = le16_to_cpu(b->name_off);
- attr = mi_insert_attr(mi, b->type, Add2Ptr(b, name_off),
+ attr = mi_insert_attr(ni, mi, b->type, Add2Ptr(b, name_off),
b->name_len, asize, name_off);
if (!attr)
goto out1;
@@ -908,7 +923,7 @@ int ni_create_attr_list(struct ntfs_inode *ni)
goto out1;
}
- attr = mi_insert_attr(&ni->mi, ATTR_LIST, NULL, 0,
+ attr = mi_insert_attr(ni, &ni->mi, ATTR_LIST, NULL, 0,
lsize + SIZEOF_RESIDENT, SIZEOF_RESIDENT);
if (!attr)
goto out1;
@@ -995,13 +1010,13 @@ static int ni_ins_attr_ext(struct ntfs_inode *ni, struct ATTR_LIST_ENTRY *le,
mi = rb_entry(node, struct mft_inode, node);
if (is_mft_data &&
- (mi_enum_attr(mi, NULL) ||
+ (mi_enum_attr(ni, mi, NULL) ||
vbo <= ((u64)mi->rno << sbi->record_bits))) {
/* We can't accept this record 'cause MFT's bootstrapping. */
continue;
}
if (is_mft &&
- mi_find_attr(mi, NULL, ATTR_DATA, NULL, 0, NULL)) {
+ mi_find_attr(ni, mi, NULL, ATTR_DATA, NULL, 0, NULL)) {
/*
* This child record already has a ATTR_DATA.
* So it can't accept any other records.
@@ -1010,7 +1025,7 @@ static int ni_ins_attr_ext(struct ntfs_inode *ni, struct ATTR_LIST_ENTRY *le,
}
if ((type != ATTR_NAME || name_len) &&
- mi_find_attr(mi, NULL, type, name, name_len, NULL)) {
+ mi_find_attr(ni, mi, NULL, type, name, name_len, NULL)) {
/* Only indexed attributes can share same record. */
continue;
}
@@ -1159,7 +1174,7 @@ static int ni_insert_attr(struct ntfs_inode *ni, enum ATTR_TYPE type,
/* Estimate the result of moving all possible attributes away. */
attr = NULL;
- while ((attr = mi_enum_attr(&ni->mi, attr))) {
+ while ((attr = mi_enum_attr(ni, &ni->mi, attr))) {
if (attr->type == ATTR_STD)
continue;
if (attr->type == ATTR_LIST)
@@ -1177,7 +1192,7 @@ static int ni_insert_attr(struct ntfs_inode *ni, enum ATTR_TYPE type,
attr = NULL;
for (;;) {
- attr = mi_enum_attr(&ni->mi, attr);
+ attr = mi_enum_attr(ni, &ni->mi, attr);
if (!attr) {
/* We should never be here 'cause we have already check this case. */
err = -EINVAL;
@@ -1261,7 +1276,7 @@ static int ni_expand_mft_list(struct ntfs_inode *ni)
for (node = rb_first(&ni->mi_tree); node; node = rb_next(node)) {
mi = rb_entry(node, struct mft_inode, node);
- attr = mi_enum_attr(mi, NULL);
+ attr = mi_enum_attr(ni, mi, NULL);
if (!attr) {
mft_min = mi->rno;
@@ -1282,7 +1297,7 @@ static int ni_expand_mft_list(struct ntfs_inode *ni)
ni_remove_mi(ni, mi_new);
}
- attr = mi_find_attr(&ni->mi, NULL, ATTR_DATA, NULL, 0, NULL);
+ attr = mi_find_attr(ni, &ni->mi, NULL, ATTR_DATA, NULL, 0, NULL);
if (!attr) {
err = -EINVAL;
goto out;
@@ -1399,7 +1414,7 @@ int ni_expand_list(struct ntfs_inode *ni)
continue;
/* Find attribute in primary record. */
- attr = rec_find_attr_le(&ni->mi, le);
+ attr = rec_find_attr_le(ni, &ni->mi, le);
if (!attr) {
err = -EINVAL;
goto out;
@@ -3155,8 +3170,7 @@ int ni_add_name(struct ntfs_inode *dir_ni, struct ntfs_inode *ni,
* ni_rename - Remove one name and insert new name.
*/
int ni_rename(struct ntfs_inode *dir_ni, struct ntfs_inode *new_dir_ni,
- struct ntfs_inode *ni, struct NTFS_DE *de, struct NTFS_DE *new_de,
- bool *is_bad)
+ struct ntfs_inode *ni, struct NTFS_DE *de, struct NTFS_DE *new_de)
{
int err;
struct NTFS_DE *de2 = NULL;
@@ -3179,8 +3193,8 @@ int ni_rename(struct ntfs_inode *dir_ni, struct ntfs_inode *new_dir_ni,
err = ni_add_name(new_dir_ni, ni, new_de);
if (!err) {
err = ni_remove_name(dir_ni, ni, de, &de2, &undo);
- if (err && ni_remove_name(new_dir_ni, ni, new_de, &de2, &undo))
- *is_bad = true;
+ WARN_ON(err && ni_remove_name(new_dir_ni, ni, new_de, &de2,
+ &undo));
}
/*
@@ -3436,7 +3450,7 @@ int ni_write_inode(struct inode *inode, int sync, const char *hint)
if (!mi->dirty)
continue;
- is_empty = !mi_enum_attr(mi, NULL);
+ is_empty = !mi_enum_attr(ni, mi, NULL);
if (is_empty)
clear_rec_inuse(mi->mrec);
diff --git a/fs/ntfs3/fslog.c b/fs/ntfs3/fslog.c
index c7762a24f8fe..df80905fe980 100644
--- a/fs/ntfs3/fslog.c
+++ b/fs/ntfs3/fslog.c
@@ -4061,6 +4061,22 @@ int log_replay(struct ntfs_inode *ni, bool *initialized)
l_size = orig_file_size;
page_size =
norm_file_page(t32, &l_size, t32 == DefaultLogPageSize);
+
+ /*
+ * If the adopted on-disk page size is larger than the size used
+ * to allocate one_page_buf above, grow the scratch buffer so a
+ * later read_log_page() cannot overflow it.
+ */
+ if (page_size > log->page_size) {
+ void *buf;
+
+ buf = krealloc(log->one_page_buf, page_size, GFP_NOFS);
+ if (!buf) {
+ err = -ENOMEM;
+ goto out;
+ }
+ log->one_page_buf = buf;
+ }
}
if (page_size != t32 ||
diff --git a/fs/ntfs3/index.c b/fs/ntfs3/index.c
index 82feda419c0a..857245369389 100644
--- a/fs/ntfs3/index.c
+++ b/fs/ntfs3/index.c
@@ -1552,6 +1552,16 @@ static int indx_add_allocate(struct ntfs_index *indx, struct ntfs_inode *ni,
bmp_size = bmp_size_v = le32_to_cpu(bmp->res.data_size);
}
+ /*
+ * Index blocks exist, but $BITMAP has zero valid bits.
+ * This implies an on-disk corruption and must be rejected.
+ */
+ if (in->name == I30_NAME &&
+ unlikely(bmp_size_v == 0 && indx->alloc_run.count)) {
+ err = -EINVAL;
+ goto out1;
+ }
+
bit = bmp_size << 3;
}
diff --git a/fs/ntfs3/inode.c b/fs/ntfs3/inode.c
index b8ac0943e932..e38f8fac1130 100644
--- a/fs/ntfs3/inode.c
+++ b/fs/ntfs3/inode.c
@@ -580,13 +580,18 @@ static noinline int ntfs_get_block_vbo(struct inode *inode, u64 vbo,
}
if (is_resident(ni)) {
- ni_lock(ni);
- err = attr_data_read_resident(ni, page);
- ni_unlock(ni);
-
- if (!err)
- set_buffer_uptodate(bh);
+ bh->b_blocknr = RESIDENT_LCN;
bh->b_size = block_size;
+ if (!page) {
+ err = 0;
+ } else {
+ ni_lock(ni);
+ err = attr_data_read_resident(ni, page);
+ ni_unlock(ni);
+
+ if (!err)
+ set_buffer_uptodate(bh);
+ }
return err;
}
diff --git a/fs/ntfs3/namei.c b/fs/ntfs3/namei.c
index 7b7634b38eea..1adbc4af6aba 100644
--- a/fs/ntfs3/namei.c
+++ b/fs/ntfs3/namei.c
@@ -250,7 +250,7 @@ static int ntfs_rename(struct user_namespace *mnt_userns, struct inode *dir,
struct ntfs_inode *ni = ntfs_i(inode);
struct inode *new_inode = d_inode(new_dentry);
struct NTFS_DE *de, *new_de;
- bool is_same, is_bad;
+ bool is_same;
/*
* de - memory of PATH_MAX bytes:
* [0-1024) - original name (dentry->d_name)
@@ -314,12 +314,8 @@ static int ntfs_rename(struct user_namespace *mnt_userns, struct inode *dir,
ni_lock_dir(dir_ni);
ni_lock(ni);
- is_bad = false;
- err = ni_rename(dir_ni, new_dir_ni, ni, de, new_de, &is_bad);
- if (is_bad) {
- /* Restore after failed rename failed too. */
- _ntfs_bad_inode(inode);
- } else if (!err) {
+ err = ni_rename(dir_ni, new_dir_ni, ni, de, new_de);
+ if (!err) {
inode->i_ctime = dir->i_ctime = dir->i_mtime =
current_time(dir);
mark_inode_dirty(inode);
diff --git a/fs/ntfs3/ntfs_fs.h b/fs/ntfs3/ntfs_fs.h
index 482958c01644..34179fba57db 100644
--- a/fs/ntfs3/ntfs_fs.h
+++ b/fs/ntfs3/ntfs_fs.h
@@ -573,8 +573,7 @@ int ni_add_name(struct ntfs_inode *dir_ni, struct ntfs_inode *ni,
struct NTFS_DE *de);
int ni_rename(struct ntfs_inode *dir_ni, struct ntfs_inode *new_dir_ni,
- struct ntfs_inode *ni, struct NTFS_DE *de, struct NTFS_DE *new_de,
- bool *is_bad);
+ struct ntfs_inode *ni, struct NTFS_DE *de, struct NTFS_DE *new_de);
bool ni_is_dirty(struct inode *inode);
@@ -733,23 +732,24 @@ int mi_get(struct ntfs_sb_info *sbi, CLST rno, struct mft_inode **mi);
void mi_put(struct mft_inode *mi);
int mi_init(struct mft_inode *mi, struct ntfs_sb_info *sbi, CLST rno);
int mi_read(struct mft_inode *mi, bool is_mft);
-struct ATTRIB *mi_enum_attr(struct mft_inode *mi, struct ATTRIB *attr);
-// TODO: id?
-struct ATTRIB *mi_find_attr(struct mft_inode *mi, struct ATTRIB *attr,
- enum ATTR_TYPE type, const __le16 *name,
- size_t name_len, const __le16 *id);
-static inline struct ATTRIB *rec_find_attr_le(struct mft_inode *rec,
+struct ATTRIB *mi_enum_attr(struct ntfs_inode *ni, struct mft_inode *mi,
+ struct ATTRIB *attr);
+struct ATTRIB *mi_find_attr(struct ntfs_inode *ni, struct mft_inode *mi,
+ struct ATTRIB *attr, enum ATTR_TYPE type,
+ const __le16 *name, size_t name_len, const __le16 *id);
+static inline struct ATTRIB *rec_find_attr_le(struct ntfs_inode *ni,
+ struct mft_inode *rec,
struct ATTR_LIST_ENTRY *le)
{
- return mi_find_attr(rec, NULL, le->type, le_name(le), le->name_len,
+ return mi_find_attr(ni, rec, NULL, le->type, le_name(le), le->name_len,
&le->id);
}
int mi_write(struct mft_inode *mi, int wait);
int mi_format_new(struct mft_inode *mi, struct ntfs_sb_info *sbi, CLST rno,
__le16 flags, bool is_mft);
-struct ATTRIB *mi_insert_attr(struct mft_inode *mi, enum ATTR_TYPE type,
- const __le16 *name, u8 name_len, u32 asize,
- u16 name_off);
+struct ATTRIB *mi_insert_attr(struct ntfs_inode *ni, struct mft_inode *mi,
+ enum ATTR_TYPE type, const __le16 *name,
+ u8 name_len, u32 asize, u16 name_off);
bool mi_remove_attr(struct ntfs_inode *ni, struct mft_inode *mi,
struct ATTRIB *attr);
diff --git a/fs/ntfs3/record.c b/fs/ntfs3/record.c
index bfb1f4c2f271..1393e93fef13 100644
--- a/fs/ntfs3/record.c
+++ b/fs/ntfs3/record.c
@@ -31,7 +31,7 @@ static inline int compare_attr(const struct ATTRIB *left, enum ATTR_TYPE type,
*
* Return: Unused attribute id that is less than mrec->next_attr_id.
*/
-static __le16 mi_new_attt_id(struct mft_inode *mi)
+static __le16 mi_new_attt_id(struct ntfs_inode *ni, struct mft_inode *mi)
{
u16 free_id, max_id, t16;
struct MFT_REC *rec = mi->mrec;
@@ -52,7 +52,7 @@ static __le16 mi_new_attt_id(struct mft_inode *mi)
attr = NULL;
for (;;) {
- attr = mi_enum_attr(mi, attr);
+ attr = mi_enum_attr(ni, mi, attr);
if (!attr) {
rec->next_attr_id = cpu_to_le16(max_id + 1);
mi->dirty = true;
@@ -189,7 +189,8 @@ int mi_read(struct mft_inode *mi, bool is_mft)
return err;
}
-struct ATTRIB *mi_enum_attr(struct mft_inode *mi, struct ATTRIB *attr)
+struct ATTRIB *mi_enum_attr(struct ntfs_inode *ni, struct mft_inode *mi,
+ struct ATTRIB *attr)
{
const struct MFT_REC *rec = mi->mrec;
u32 used = le32_to_cpu(rec->used);
@@ -203,11 +204,11 @@ struct ATTRIB *mi_enum_attr(struct mft_inode *mi, struct ATTRIB *attr)
off = le16_to_cpu(rec->attr_off);
if (used > total)
- return NULL;
+ goto out;
if (off >= used || off < MFTRECORD_FIXUP_OFFSET_1 ||
- !IS_ALIGNED(off, 4)) {
- return NULL;
+ !IS_ALIGNED(off, 8)) {
+ goto out;
}
/* Skip non-resident records. */
@@ -230,11 +231,14 @@ struct ATTRIB *mi_enum_attr(struct mft_inode *mi, struct ATTRIB *attr)
off += asize;
}
- /* Can we use the first field (attr->type). */
- /* NOTE: this code also checks attr->size availability. */
+ /*
+ * Can we use the first fields:
+ * attr->type,
+ * attr->size
+ */
if (off + 8 > used) {
static_assert(ALIGN(sizeof(enum ATTR_TYPE), 8) == 8);
- return NULL;
+ goto out;
}
if (attr->type == ATTR_END) {
@@ -245,34 +249,41 @@ struct ATTRIB *mi_enum_attr(struct mft_inode *mi, struct ATTRIB *attr)
/* 0x100 is last known attribute for now. */
t32 = le32_to_cpu(attr->type);
if (!t32 || (t32 & 0xf) || (t32 > 0x100))
- return NULL;
+ goto out;
/* attributes in record must be ordered by type */
if (t32 < prev_type)
- return NULL;
+ goto out;
asize = le32_to_cpu(attr->size);
+ if (!IS_ALIGNED(asize, 8))
+ goto out;
+
/* Check overflow and boundary. */
if (off + asize < off || off + asize > used)
- return NULL;
+ goto out;
+
+ /* Can we use the field attr->non_res. */
+ if (off + 9 > used)
+ goto out;
/* Check size of attribute. */
if (!attr->non_res) {
/* Check resident fields. */
if (asize < SIZEOF_RESIDENT)
- return NULL;
+ goto out;
t16 = le16_to_cpu(attr->res.data_off);
if (t16 > asize)
- return NULL;
+ goto out;
if (le32_to_cpu(attr->res.data_size) > asize - t16)
- return NULL;
+ goto out;
if (attr->name_len &&
le16_to_cpu(attr->name_off) + sizeof(short) * attr->name_len > t16) {
- return NULL;
+ goto out;
}
return attr;
@@ -280,71 +291,75 @@ struct ATTRIB *mi_enum_attr(struct mft_inode *mi, struct ATTRIB *attr)
/* Check nonresident fields. */
if (attr->non_res != 1)
- return NULL;
+ goto out;
/* Can we use memory including attr->nres.valid_size? */
if (asize < SIZEOF_NONRESIDENT)
- return NULL;
+ goto out;
t16 = le16_to_cpu(attr->nres.run_off);
if (t16 > asize)
- return NULL;
+ goto out;
t32 = sizeof(short) * attr->name_len;
if (t32 && le16_to_cpu(attr->name_off) + t32 > t16)
- return NULL;
+ goto out;
/* Check start/end vcn. */
if (le64_to_cpu(attr->nres.svcn) > le64_to_cpu(attr->nres.evcn) + 1)
- return NULL;
+ goto out;
data_size = le64_to_cpu(attr->nres.data_size);
if (le64_to_cpu(attr->nres.valid_size) > data_size)
- return NULL;
+ goto out;
alloc_size = le64_to_cpu(attr->nres.alloc_size);
if (data_size > alloc_size)
- return NULL;
+ goto out;
t32 = mi->sbi->cluster_mask;
if (alloc_size & t32)
- return NULL;
+ goto out;
if (!attr->nres.svcn && is_attr_ext(attr)) {
/* First segment of sparse/compressed attribute */
/* Can we use memory including attr->nres.total_size? */
if (asize < SIZEOF_NONRESIDENT_EX)
- return NULL;
+ goto out;
tot_size = le64_to_cpu(attr->nres.total_size);
if (tot_size & t32)
- return NULL;
+ goto out;
if (tot_size > alloc_size)
- return NULL;
+ goto out;
} else {
if (attr->nres.c_unit)
- return NULL;
+ goto out;
if (alloc_size > mi->sbi->volume.size)
- return NULL;
+ goto out;
}
return attr;
+
+out:
+ _ntfs_bad_inode(&ni->vfs_inode);
+ return NULL;
}
/*
* mi_find_attr - Find the attribute by type and name and id.
*/
-struct ATTRIB *mi_find_attr(struct mft_inode *mi, struct ATTRIB *attr,
- enum ATTR_TYPE type, const __le16 *name,
- size_t name_len, const __le16 *id)
+struct ATTRIB *mi_find_attr(struct ntfs_inode *ni, struct mft_inode *mi,
+ struct ATTRIB *attr, enum ATTR_TYPE type,
+ const __le16 *name, size_t name_len, const __le16 *id)
{
u32 type_in = le32_to_cpu(type);
u32 atype;
next_attr:
- attr = mi_enum_attr(mi, attr);
+ attr = mi_enum_attr(ni, mi, attr);
if (!attr)
return NULL;
@@ -450,9 +465,9 @@ int mi_format_new(struct mft_inode *mi, struct ntfs_sb_info *sbi, CLST rno,
*
* Return: Not full constructed attribute or NULL if not possible to create.
*/
-struct ATTRIB *mi_insert_attr(struct mft_inode *mi, enum ATTR_TYPE type,
- const __le16 *name, u8 name_len, u32 asize,
- u16 name_off)
+struct ATTRIB *mi_insert_attr(struct ntfs_inode *ni, struct mft_inode *mi,
+ enum ATTR_TYPE type, const __le16 *name,
+ u8 name_len, u32 asize, u16 name_off)
{
size_t tail;
struct ATTRIB *attr;
@@ -472,7 +487,7 @@ struct ATTRIB *mi_insert_attr(struct mft_inode *mi, enum ATTR_TYPE type,
* at which we should insert it.
*/
attr = NULL;
- while ((attr = mi_enum_attr(mi, attr))) {
+ while ((attr = mi_enum_attr(ni, mi, attr))) {
diff = compare_attr(attr, type, name, name_len, upcase);
if (diff < 0)
@@ -490,7 +505,7 @@ struct ATTRIB *mi_insert_attr(struct mft_inode *mi, enum ATTR_TYPE type,
tail = used - PtrOffset(rec, attr);
}
- id = mi_new_attt_id(mi);
+ id = mi_new_attt_id(ni, mi);
memmove(Add2Ptr(attr, asize), attr, tail);
memset(attr, 0, asize);
diff --git a/fs/ntfs3/super.c b/fs/ntfs3/super.c
index 35dd0018e2bd..9bf327a7cc7b 100644
--- a/fs/ntfs3/super.c
+++ b/fs/ntfs3/super.c
@@ -33,6 +33,7 @@
#include <linux/minmax.h>
#include <linux/module.h>
#include <linux/nls.h>
+#include <linux/overflow.h>
#include <linux/seq_file.h>
#include <linux/statfs.h>
@@ -685,7 +686,7 @@ static int ntfs_init_from_boot(struct super_block *sb, u32 sector_size,
struct ntfs_sb_info *sbi = sb->s_fs_info;
int err;
u32 mb, gb, boot_sector_size, sct_per_clst, record_size;
- u64 sectors, clusters, mlcn, mlcn2;
+ u64 sectors, clusters, mlcn, mlcn2, mft_pos, mft2_pos;
struct NTFS_BOOT *boot;
struct buffer_head *bh;
struct MFT_REC *rec;
@@ -730,10 +731,15 @@ static int ntfs_init_from_boot(struct super_block *sb, u32 sector_size,
mlcn2 = le64_to_cpu(boot->mft2_clst);
sectors = le64_to_cpu(boot->sectors_per_volume);
- if (mlcn * sct_per_clst >= sectors)
- goto out;
-
- if (mlcn2 * sct_per_clst >= sectors)
+ /*
+ * Convert mlcn/mlcn2 to sector positions before comparing with
+ * 'sectors'. All three are u64 values that come from the boot
+ * sector, so use check_mul_overflow() to keep a wraparound from
+ * silently bypassing the comparison.
+ */
+ if (check_mul_overflow(mlcn, (u64)sct_per_clst, &mft_pos) ||
+ check_mul_overflow(mlcn2, (u64)sct_per_clst, &mft2_pos) ||
+ mft_pos >= sectors || mft2_pos >= sectors)
goto out;
/* Check MFT record size. */
diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c
index a350633bf722..77d65f3d3d60 100644
--- a/fs/ocfs2/cluster/heartbeat.c
+++ b/fs/ocfs2/cluster/heartbeat.c
@@ -14,6 +14,7 @@
#include <linux/file.h>
#include <linux/kthread.h>
#include <linux/configfs.h>
+#include <linux/mutex.h>
#include <linux/random.h>
#include <linux/crc32.h>
#include <linux/time.h>
@@ -210,6 +211,7 @@ struct o2hb_region {
/* protected by the hr_callback_sem */
struct task_struct *hr_task;
+ u8 hr_node_num;
unsigned int hr_blocks;
unsigned long long hr_start_block;
@@ -264,6 +266,9 @@ struct o2hb_region {
/* Message key for negotiate timeout message. */
unsigned int hr_key;
struct list_head hr_handler_list;
+ /* Serializes timeout arming against failed-start and teardown. */
+ struct mutex hr_arming_mutex;
+ bool hr_stopping;
/* last hb status, 0 for success, other value for error. */
int hr_last_hb_status;
@@ -323,9 +328,14 @@ static void o2hb_write_timeout(struct work_struct *work)
static void o2hb_arm_timeout(struct o2hb_region *reg)
{
+ mutex_lock(®->hr_arming_mutex);
+
+ if (reg->hr_stopping)
+ goto out_unlock;
+
/* Arm writeout only after thread reaches steady state */
if (atomic_read(®->hr_steady_iterations) != 0)
- return;
+ goto out_unlock;
mlog(ML_HEARTBEAT, "Queue write timeout for %u ms\n",
O2HB_MAX_WRITE_TIMEOUT_MS);
@@ -344,6 +354,18 @@ static void o2hb_arm_timeout(struct o2hb_region *reg)
schedule_delayed_work(®->hr_nego_timeout_work,
msecs_to_jiffies(O2HB_NEGO_TIMEOUT_MS));
memset(reg->hr_nego_node_bitmap, 0, sizeof(reg->hr_nego_node_bitmap));
+
+out_unlock:
+ mutex_unlock(®->hr_arming_mutex);
+}
+
+static void o2hb_queue_nego_timeout(struct o2hb_region *reg,
+ unsigned long delay)
+{
+ mutex_lock(®->hr_arming_mutex);
+ if (!reg->hr_stopping)
+ schedule_delayed_work(®->hr_nego_timeout_work, delay);
+ mutex_unlock(®->hr_arming_mutex);
}
static void o2hb_disarm_timeout(struct o2hb_region *reg)
@@ -352,12 +374,25 @@ static void o2hb_disarm_timeout(struct o2hb_region *reg)
cancel_delayed_work_sync(®->hr_nego_timeout_work);
}
-static int o2hb_send_nego_msg(int key, int type, u8 target)
+static void o2hb_set_region_stopping(struct o2hb_region *reg, bool stopping)
+{
+ mutex_lock(®->hr_arming_mutex);
+ reg->hr_stopping = stopping;
+ mutex_unlock(®->hr_arming_mutex);
+}
+
+static void o2hb_quiesce_timeout(struct o2hb_region *reg)
+{
+ o2hb_set_region_stopping(reg, true);
+ o2hb_disarm_timeout(reg);
+}
+
+static int o2hb_send_nego_msg(int key, int type, u8 target, u8 node_num)
{
struct o2hb_nego_msg msg;
int status, ret;
- msg.node_num = o2nm_this_node();
+ msg.node_num = node_num;
again:
ret = o2net_send_message(type, key, &msg, sizeof(msg),
target, &status);
@@ -375,8 +410,10 @@ static void o2hb_nego_timeout(struct work_struct *work)
unsigned long live_node_bitmap[BITS_TO_LONGS(O2NM_MAX_NODES)];
int master_node, i, ret;
struct o2hb_region *reg;
+ u8 node_num;
reg = container_of(work, struct o2hb_region, hr_nego_timeout_work.work);
+ node_num = reg->hr_node_num;
/* don't negotiate timeout if last hb failed since it is very
* possible io failed. Should let write timeout fence self.
*/
@@ -387,10 +424,10 @@ static void o2hb_nego_timeout(struct work_struct *work)
/* lowest node as master node to make negotiate decision. */
master_node = find_first_bit(live_node_bitmap, O2NM_MAX_NODES);
- if (master_node == o2nm_this_node()) {
+ if (master_node == node_num) {
if (!test_bit(master_node, reg->hr_nego_node_bitmap)) {
printk(KERN_NOTICE "o2hb: node %d hb write hung for %ds on region %s (%pg).\n",
- o2nm_this_node(), O2HB_NEGO_TIMEOUT_MS/1000,
+ node_num, O2HB_NEGO_TIMEOUT_MS / 1000,
config_item_name(®->hr_item), reg->hr_bdev);
set_bit(master_node, reg->hr_nego_node_bitmap);
}
@@ -399,8 +436,7 @@ static void o2hb_nego_timeout(struct work_struct *work)
/* check negotiate bitmap every second to do timeout
* approve decision.
*/
- schedule_delayed_work(®->hr_nego_timeout_work,
- msecs_to_jiffies(1000));
+ o2hb_queue_nego_timeout(reg, msecs_to_jiffies(1000));
return;
}
@@ -418,7 +454,7 @@ static void o2hb_nego_timeout(struct work_struct *work)
mlog(ML_HEARTBEAT, "send NEGO_APPROVE msg to node %d\n", i);
ret = o2hb_send_nego_msg(reg->hr_key,
- O2HB_NEGO_APPROVE_MSG, i);
+ O2HB_NEGO_APPROVE_MSG, i, node_num);
if (ret)
mlog(ML_ERROR, "send NEGO_APPROVE msg to node %d fail %d\n",
i, ret);
@@ -426,10 +462,10 @@ static void o2hb_nego_timeout(struct work_struct *work)
} else {
/* negotiate timeout with master node. */
printk(KERN_NOTICE "o2hb: node %d hb write hung for %ds on region %s (%pg), negotiate timeout with node %d.\n",
- o2nm_this_node(), O2HB_NEGO_TIMEOUT_MS/1000, config_item_name(®->hr_item),
+ node_num, O2HB_NEGO_TIMEOUT_MS / 1000, config_item_name(®->hr_item),
reg->hr_bdev, master_node);
ret = o2hb_send_nego_msg(reg->hr_key, O2HB_NEGO_TIMEOUT_MSG,
- master_node);
+ master_node, node_num);
if (ret)
mlog(ML_ERROR, "send NEGO_TIMEOUT msg to node %d fail %d\n",
master_node, ret);
@@ -601,7 +637,9 @@ static int o2hb_issue_node_write(struct o2hb_region *reg,
o2hb_bio_wait_init(write_wc);
- slot = o2nm_this_node();
+ slot = reg->hr_node_num;
+ if (slot >= O2NM_MAX_NODES)
+ return -EINVAL;
bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1,
REQ_OP_WRITE | REQ_SYNC);
@@ -670,8 +708,12 @@ static int o2hb_check_own_slot(struct o2hb_region *reg)
struct o2hb_disk_slot *slot;
struct o2hb_disk_heartbeat_block *hb_block;
char *errstr;
+ u8 node_num = reg->hr_node_num;
+
+ if (node_num >= O2NM_MAX_NODES)
+ return 0;
- slot = ®->hr_slots[o2nm_this_node()];
+ slot = ®->hr_slots[node_num];
/* Don't check on our 1st timestamp */
if (!slot->ds_last_time)
return 0;
@@ -712,7 +754,10 @@ static inline void o2hb_prepare_block(struct o2hb_region *reg,
struct o2hb_disk_slot *slot;
struct o2hb_disk_heartbeat_block *hb_block;
- node_num = o2nm_this_node();
+ node_num = reg->hr_node_num;
+ if (node_num >= O2NM_MAX_NODES)
+ return;
+
slot = ®->hr_slots[node_num];
hb_block = (struct o2hb_disk_heartbeat_block *)slot->ds_raw_block;
@@ -1206,7 +1251,7 @@ static int o2hb_thread(void *data)
set_user_nice(current, MIN_NICE);
/* Pin node */
- ret = o2nm_depend_this_node();
+ ret = o2nm_depend_node(reg->hr_node_num);
if (ret) {
mlog(ML_ERROR, "Node has been deleted, ret = %d\n", ret);
reg->hr_node_deleted = 1;
@@ -1215,7 +1260,8 @@ static int o2hb_thread(void *data)
}
while (!kthread_should_stop() &&
- !reg->hr_unclean_stop && !reg->hr_aborted_start) {
+ !reg->hr_unclean_stop && !reg->hr_aborted_start &&
+ o2nm_this_node() == reg->hr_node_num) {
/* We track the time spent inside
* o2hb_do_disk_heartbeat so that we avoid more than
* hr_timeout_ms between disk writes. On busy systems
@@ -1264,7 +1310,7 @@ static int o2hb_thread(void *data)
}
/* Unpin node */
- o2nm_undepend_this_node();
+ o2nm_undepend_node(reg->hr_node_num);
mlog(ML_HEARTBEAT|ML_KTHREAD, "o2hb thread exiting\n");
@@ -1511,33 +1557,47 @@ static struct o2hb_region *to_o2hb_region(struct config_item *item)
return item ? container_of(item, struct o2hb_region, hr_item) : NULL;
}
-/* drop_item only drops its ref after killing the thread, nothing should
- * be using the region anymore. this has to clean up any state that
- * attributes might have built up. */
-static void o2hb_region_release(struct config_item *item)
+static void o2hb_unmap_slot_data(struct o2hb_region *reg)
{
int i;
struct page *page;
- struct o2hb_region *reg = to_o2hb_region(item);
-
- mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg->hr_bdev);
-
- kfree(reg->hr_tmp_block);
if (reg->hr_slot_data) {
for (i = 0; i < reg->hr_num_pages; i++) {
page = reg->hr_slot_data[i];
- if (page)
+ if (page) {
__free_page(page);
+ reg->hr_slot_data[i] = NULL;
+ }
}
kfree(reg->hr_slot_data);
+ reg->hr_slot_data = NULL;
}
+ kfree(reg->hr_slots);
+ reg->hr_slots = NULL;
+
+ kfree(reg->hr_tmp_block);
+ reg->hr_tmp_block = NULL;
+}
+
+/* drop_item only drops its ref after killing the thread, nothing should
+ * be using the region anymore. this has to clean up any state that
+ * attributes might have built up.
+ */
+static void o2hb_region_release(struct config_item *item)
+{
+ struct o2hb_region *reg = to_o2hb_region(item);
+
+ mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg->hr_bdev);
+
+ o2hb_quiesce_timeout(reg);
+ o2net_unregister_and_flush_handler_list(®->hr_handler_list);
+ o2hb_unmap_slot_data(reg);
+
if (reg->hr_bdev)
blkdev_put(reg->hr_bdev, FMODE_READ|FMODE_WRITE);
- kfree(reg->hr_slots);
-
debugfs_remove_recursive(reg->hr_debug_dir);
kfree(reg->hr_db_livenodes);
kfree(reg->hr_db_regnum);
@@ -1548,7 +1608,6 @@ static void o2hb_region_release(struct config_item *item)
list_del(®->hr_all_item);
spin_unlock(&o2hb_live_lock);
- o2net_unregister_handler_list(®->hr_handler_list);
kfree(reg);
}
@@ -1688,6 +1747,7 @@ static void o2hb_init_region_params(struct o2hb_region *reg)
static int o2hb_map_slot_data(struct o2hb_region *reg)
{
int i, j;
+ int ret = -ENOMEM;
unsigned int last_slot;
unsigned int spp = reg->hr_slots_per_page;
struct page *page;
@@ -1695,15 +1755,15 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
struct o2hb_disk_slot *slot;
reg->hr_tmp_block = kmalloc(reg->hr_block_bytes, GFP_KERNEL);
- if (reg->hr_tmp_block == NULL)
- return -ENOMEM;
+ if (!reg->hr_tmp_block)
+ goto out;
reg->hr_slots = kcalloc(reg->hr_blocks,
sizeof(struct o2hb_disk_slot), GFP_KERNEL);
- if (reg->hr_slots == NULL)
- return -ENOMEM;
+ if (!reg->hr_slots)
+ goto out;
- for(i = 0; i < reg->hr_blocks; i++) {
+ for (i = 0; i < reg->hr_blocks; i++) {
slot = ®->hr_slots[i];
slot->ds_node_num = i;
INIT_LIST_HEAD(&slot->ds_live_item);
@@ -1718,12 +1778,12 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
reg->hr_slot_data = kcalloc(reg->hr_num_pages, sizeof(struct page *),
GFP_KERNEL);
if (!reg->hr_slot_data)
- return -ENOMEM;
+ goto out;
- for(i = 0; i < reg->hr_num_pages; i++) {
+ for (i = 0; i < reg->hr_num_pages; i++) {
page = alloc_page(GFP_KERNEL);
if (!page)
- return -ENOMEM;
+ goto out;
reg->hr_slot_data[i] = page;
@@ -1743,6 +1803,10 @@ static int o2hb_map_slot_data(struct o2hb_region *reg)
}
return 0;
+
+out:
+ o2hb_unmap_slot_data(reg);
+ return ret;
}
/* Read in all the slots available and populate the tracking
@@ -1795,7 +1859,8 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
/* We can't heartbeat without having had our node number
* configured yet. */
- if (o2nm_this_node() == O2NM_MAX_NODES)
+ reg->hr_node_num = o2nm_this_node();
+ if (reg->hr_node_num == O2NM_MAX_NODES)
goto out;
fd = simple_strtol(p, &p, 0);
@@ -1833,6 +1898,8 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
goto out3;
}
+ reg->hr_aborted_start = 0;
+ reg->hr_node_deleted = 0;
o2hb_init_region_params(reg);
/* Generation of zero is invalid */
@@ -1853,9 +1920,6 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
goto out3;
}
- INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout);
- INIT_DELAYED_WORK(®->hr_nego_timeout_work, o2hb_nego_timeout);
-
/*
* A node is considered live after it has beat LIVE_THRESHOLD
* times. We're not steady until we've given them a chance
@@ -1875,6 +1939,7 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
atomic_set(®->hr_steady_iterations, live_threshold);
/* unsteady_iterations is triple the steady_iterations */
atomic_set(®->hr_unsteady_iterations, (live_threshold * 3));
+ o2hb_set_region_stopping(reg, false);
hb_task = kthread_run(o2hb_thread, reg, "o2hb-%s",
reg->hr_item.ci_name);
@@ -1924,6 +1989,18 @@ static ssize_t o2hb_region_dev_store(struct config_item *item,
out3:
if (ret < 0) {
+ o2hb_quiesce_timeout(reg);
+
+ spin_lock(&o2hb_live_lock);
+ hb_task = reg->hr_task;
+ reg->hr_task = NULL;
+ spin_unlock(&o2hb_live_lock);
+
+ if (hb_task)
+ kthread_stop(hb_task);
+
+ o2hb_unmap_slot_data(reg);
+
blkdev_put(reg->hr_bdev, FMODE_READ | FMODE_WRITE);
reg->hr_bdev = NULL;
}
@@ -2031,6 +2108,7 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g
ret = -ENAMETOOLONG;
goto free;
}
+ reg->hr_node_num = O2NM_MAX_NODES;
spin_lock(&o2hb_live_lock);
reg->hr_region_num = 0;
@@ -2055,6 +2133,10 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g
*/
reg->hr_key = crc32_le(reg->hr_region_num + O2NM_MAX_REGIONS,
name, strlen(name));
+ mutex_init(®->hr_arming_mutex);
+ reg->hr_stopping = true;
+ INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout);
+ INIT_DELAYED_WORK(®->hr_nego_timeout_work, o2hb_nego_timeout);
INIT_LIST_HEAD(®->hr_handler_list);
ret = o2net_register_handler(O2HB_NEGO_TIMEOUT_MSG, reg->hr_key,
sizeof(struct o2hb_nego_msg),
@@ -2075,7 +2157,7 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g
return ®->hr_item;
unregister_handler:
- o2net_unregister_handler_list(®->hr_handler_list);
+ o2net_unregister_and_flush_handler_list(®->hr_handler_list);
remove_item:
spin_lock(&o2hb_live_lock);
list_del(®->hr_all_item);
@@ -2094,6 +2176,8 @@ static void o2hb_heartbeat_group_drop_item(struct config_group *group,
struct o2hb_region *reg = to_o2hb_region(item);
int quorum_region = 0;
+ o2hb_quiesce_timeout(reg);
+
/* stop the thread when the user removes the region dir */
spin_lock(&o2hb_live_lock);
hb_task = reg->hr_task;
diff --git a/fs/ocfs2/cluster/nodemanager.c b/fs/ocfs2/cluster/nodemanager.c
index d244def401ea..adb60bc6e9bc 100644
--- a/fs/ocfs2/cluster/nodemanager.c
+++ b/fs/ocfs2/cluster/nodemanager.c
@@ -368,6 +368,7 @@ static ssize_t o2nm_node_local_store(struct config_item *item, const char *page,
if (!tmp && cluster->cl_has_local &&
cluster->cl_local_node == node->nd_num) {
o2net_stop_listening(node);
+ cluster->cl_has_local = 0;
cluster->cl_local_node = O2NM_INVALID_NODE_NUM;
}
@@ -791,12 +792,12 @@ void o2nm_undepend_item(struct config_item *item)
configfs_undepend_item(item);
}
-int o2nm_depend_this_node(void)
+int o2nm_depend_node(u8 node_num)
{
int ret = 0;
struct o2nm_node *local_node;
- local_node = o2nm_get_node_by_num(o2nm_this_node());
+ local_node = o2nm_get_node_by_num(node_num);
if (!local_node) {
ret = -EINVAL;
goto out;
@@ -809,17 +810,27 @@ int o2nm_depend_this_node(void)
return ret;
}
-void o2nm_undepend_this_node(void)
+void o2nm_undepend_node(u8 node_num)
{
struct o2nm_node *local_node;
- local_node = o2nm_get_node_by_num(o2nm_this_node());
+ local_node = o2nm_get_node_by_num(node_num);
BUG_ON(!local_node);
o2nm_undepend_item(&local_node->nd_item);
o2nm_node_put(local_node);
}
+int o2nm_depend_this_node(void)
+{
+ return o2nm_depend_node(o2nm_this_node());
+}
+
+void o2nm_undepend_this_node(void)
+{
+ o2nm_undepend_node(o2nm_this_node());
+}
+
static void __exit exit_o2nm(void)
{
diff --git a/fs/ocfs2/cluster/nodemanager.h b/fs/ocfs2/cluster/nodemanager.h
index 2f72f56996bd..ca3483fb5450 100644
--- a/fs/ocfs2/cluster/nodemanager.h
+++ b/fs/ocfs2/cluster/nodemanager.h
@@ -66,6 +66,8 @@ void o2nm_node_put(struct o2nm_node *node);
int o2nm_depend_item(struct config_item *item);
int o2nm_depend_item_unlocked(struct config_item *item);
void o2nm_undepend_item(struct config_item *item);
+int o2nm_depend_node(u8 node_num);
+void o2nm_undepend_node(u8 node_num);
int o2nm_depend_this_node(void);
void o2nm_undepend_this_node(void);
diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c
index 0f6c2e3e7925..a992a5fe2fa6 100644
--- a/fs/ocfs2/cluster/tcp.c
+++ b/fs/ocfs2/cluster/tcp.c
@@ -38,6 +38,8 @@
*/
#include <linux/kernel.h>
+#include <linux/completion.h>
+#include <linux/mutex.h>
#include <linux/sched/mm.h>
#include <linux/jiffies.h>
#include <linux/slab.h>
@@ -101,9 +103,14 @@ static struct socket *o2net_listen_sock;
* quorum work is queued as sock containers are shutdown.. stop_listening
* tears down all the node's sock containers, preventing future shutdowns
* and queued quroum work, before canceling delayed quorum work and
- * destroying the work queue.
+ * destroying the work queue. Handler teardown can also race local listener
+ * shutdown, so keep a waitable destroying pointer until the old ordered
+ * queue has finished draining.
*/
static struct workqueue_struct *o2net_wq;
+static struct workqueue_struct *o2net_wq_destroying;
+static DEFINE_MUTEX(o2net_wq_mutex);
+static DECLARE_COMPLETION(o2net_wq_destroyed);
/* Heartbeat callbacks stay registered across local-node off/on. */
static bool o2net_listening;
static struct work_struct o2net_listen_work;
@@ -883,6 +890,27 @@ void o2net_unregister_handler_list(struct list_head *list)
}
EXPORT_SYMBOL_GPL(o2net_unregister_handler_list);
+static void o2net_flush_wq(void)
+{
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ wait_for_completion(&o2net_wq_destroyed);
+ return;
+ }
+
+ if (o2net_wq)
+ flush_workqueue(o2net_wq);
+ mutex_unlock(&o2net_wq_mutex);
+}
+
+void o2net_unregister_and_flush_handler_list(struct list_head *list)
+{
+ o2net_unregister_handler_list(list);
+ o2net_flush_wq();
+}
+EXPORT_SYMBOL_GPL(o2net_unregister_and_flush_handler_list);
+
static struct o2net_msg_handler *o2net_handler_get(u32 msg_type, u32 key)
{
struct o2net_msg_handler *nmh;
@@ -1604,6 +1632,7 @@ static void o2net_start_connect(struct work_struct *work)
sc->sc_sock = sock; /* freed by sc_kref_release */
sock->sk->sk_allocation = GFP_ATOMIC;
+ sock->sk->sk_use_task_frag = false;
myaddr.sin_family = AF_INET;
myaddr.sin_addr.s_addr = mynode->nd_ipv4_address;
@@ -1709,12 +1738,10 @@ void o2net_disconnect_node(struct o2nm_node *node)
o2net_set_nn_state(nn, NULL, 0, -ENOTCONN);
spin_unlock(&nn->nn_lock);
- if (o2net_wq) {
- cancel_delayed_work(&nn->nn_connect_expired);
- cancel_delayed_work(&nn->nn_connect_work);
- cancel_delayed_work(&nn->nn_still_up);
- flush_workqueue(o2net_wq);
- }
+ cancel_delayed_work(&nn->nn_connect_expired);
+ cancel_delayed_work(&nn->nn_connect_work);
+ cancel_delayed_work(&nn->nn_still_up);
+ o2net_flush_wq();
}
static void o2net_hb_node_down_cb(struct o2nm_node *node, int node_num,
@@ -2054,6 +2081,36 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port)
return ret;
}
+static void o2net_destroy_wq(void)
+{
+ struct workqueue_struct *wq;
+
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ wait_for_completion(&o2net_wq_destroyed);
+ return;
+ }
+
+ wq = o2net_wq;
+ if (!wq) {
+ mutex_unlock(&o2net_wq_mutex);
+ return;
+ }
+
+ reinit_completion(&o2net_wq_destroyed);
+ o2net_wq_destroying = wq;
+ mutex_unlock(&o2net_wq_mutex);
+
+ destroy_workqueue(wq);
+
+ mutex_lock(&o2net_wq_mutex);
+ o2net_wq = NULL;
+ o2net_wq_destroying = NULL;
+ complete_all(&o2net_wq_destroyed);
+ mutex_unlock(&o2net_wq_mutex);
+}
+
/*
* called from node manager when we should bring up our network listening
* socket. node manager handles all the serialization to only call this
@@ -2064,24 +2121,44 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port)
int o2net_start_listening(struct o2nm_node *node)
{
int ret = 0;
+ struct workqueue_struct *wq;
if (WARN_ON_ONCE(READ_ONCE(o2net_listening)))
return -EBUSY;
- BUG_ON(o2net_wq != NULL);
+
+ mutex_lock(&o2net_wq_mutex);
+ if (o2net_wq_destroying) {
+ mutex_unlock(&o2net_wq_mutex);
+ return -EBUSY;
+ }
+ if (WARN_ON_ONCE(o2net_wq)) {
+ mutex_unlock(&o2net_wq_mutex);
+ return -EBUSY;
+ }
+ mutex_unlock(&o2net_wq_mutex);
+
BUG_ON(o2net_listen_sock != NULL);
mlog(ML_KTHREAD, "starting o2net thread...\n");
- o2net_wq = alloc_ordered_workqueue("o2net", WQ_MEM_RECLAIM);
- if (o2net_wq == NULL) {
+ wq = alloc_ordered_workqueue("o2net", WQ_MEM_RECLAIM);
+ if (!wq) {
mlog(ML_ERROR, "unable to launch o2net thread\n");
return -ENOMEM; /* ? */
}
+ mutex_lock(&o2net_wq_mutex);
+ if (unlikely(o2net_wq_destroying || o2net_wq)) {
+ mutex_unlock(&o2net_wq_mutex);
+ destroy_workqueue(wq);
+ return -EBUSY;
+ }
+ o2net_wq = wq;
+ mutex_unlock(&o2net_wq_mutex);
+
ret = o2net_open_listening_sock(node->nd_ipv4_address,
node->nd_ipv4_port);
if (ret) {
- destroy_workqueue(o2net_wq);
- o2net_wq = NULL;
+ o2net_destroy_wq();
} else
o2quo_conn_up(node->nd_num);
@@ -2117,8 +2194,7 @@ void o2net_stop_listening(struct o2nm_node *node)
/* finish all work and tear down the work queue */
mlog(ML_KTHREAD, "waiting for o2net thread to exit....\n");
- destroy_workqueue(o2net_wq);
- o2net_wq = NULL;
+ o2net_destroy_wq();
sock_release(o2net_listen_sock);
o2net_listen_sock = NULL;
diff --git a/fs/ocfs2/cluster/tcp.h b/fs/ocfs2/cluster/tcp.h
index 2e86d42b5faf..a11bcee28947 100644
--- a/fs/ocfs2/cluster/tcp.h
+++ b/fs/ocfs2/cluster/tcp.h
@@ -89,6 +89,7 @@ int o2net_register_handler(u32 msg_type, u32 key, u32 max_len,
o2net_post_msg_handler_func *post_func,
struct list_head *unreg_list);
void o2net_unregister_handler_list(struct list_head *list);
+void o2net_unregister_and_flush_handler_list(struct list_head *list);
void o2net_fill_node_map(unsigned long *map, unsigned bytes);
diff --git a/fs/ocfs2/dir.c b/fs/ocfs2/dir.c
index c85ad5999e84..bb3a67b9edff 100644
--- a/fs/ocfs2/dir.c
+++ b/fs/ocfs2/dir.c
@@ -580,7 +580,7 @@ static int ocfs2_validate_dx_root(struct super_block *sb,
mlog(ML_ERROR,
"Checksum failed for dir index root block %llu\n",
(unsigned long long)bh->b_blocknr);
- return ret;
+ goto bail;
}
if (!OCFS2_IS_VALID_DX_ROOT(dx_root)) {
@@ -588,8 +588,54 @@ static int ocfs2_validate_dx_root(struct super_block *sb,
"Dir Index Root # %llu has bad signature %.*s\n",
(unsigned long long)le64_to_cpu(dx_root->dr_blkno),
7, dx_root->dr_signature);
+ goto bail;
}
+ if (!(dx_root->dr_flags & OCFS2_DX_FLAG_INLINE)) {
+ struct ocfs2_extent_list *el = &dx_root->dr_list;
+
+ if (le16_to_cpu(el->l_count) != ocfs2_extent_recs_per_dx_root(sb)) {
+ ret = ocfs2_error(sb,
+ "Dir Index Root # %llu has invalid l_count %u (expected %u)\n",
+ (unsigned long long)le64_to_cpu(dx_root->dr_blkno),
+ le16_to_cpu(el->l_count),
+ ocfs2_extent_recs_per_dx_root(sb));
+ goto bail;
+ }
+
+ if (le16_to_cpu(el->l_next_free_rec) > le16_to_cpu(el->l_count)) {
+ ret = ocfs2_error(sb,
+ "Dir Index Root # %llu has invalid l_next_free_rec %u (l_count %u)\n",
+ (unsigned long long)le64_to_cpu(dx_root->dr_blkno),
+ le16_to_cpu(el->l_next_free_rec),
+ le16_to_cpu(el->l_count));
+ goto bail;
+ }
+ } else {
+ struct ocfs2_dx_entry_list *dl_list = &dx_root->dr_entries;
+
+ if (le16_to_cpu(dl_list->de_count) !=
+ ocfs2_dx_entries_per_root(sb)) {
+ ret = ocfs2_error(sb,
+ "Dir Index Root # %llu has invalid de_count %u (expected %u)\n",
+ (unsigned long long)le64_to_cpu(dx_root->dr_blkno),
+ le16_to_cpu(dl_list->de_count),
+ ocfs2_dx_entries_per_root(sb));
+ goto bail;
+ }
+
+ if (le16_to_cpu(dl_list->de_num_used) >
+ le16_to_cpu(dl_list->de_count)) {
+ ret = ocfs2_error(sb,
+ "Dir Index Root # %llu has invalid de_num_used %u (de_count %u)\n",
+ (unsigned long long)le64_to_cpu(dx_root->dr_blkno),
+ le16_to_cpu(dl_list->de_num_used),
+ le16_to_cpu(dl_list->de_count));
+ goto bail;
+ }
+ }
+
+bail:
return ret;
}
@@ -626,10 +672,25 @@ static int ocfs2_validate_dx_leaf(struct super_block *sb,
return ret;
}
- if (!OCFS2_IS_VALID_DX_LEAF(dx_leaf)) {
- ret = ocfs2_error(sb, "Dir Index Leaf has bad signature %.*s\n",
- 7, dx_leaf->dl_signature);
- }
+ if (!OCFS2_IS_VALID_DX_LEAF(dx_leaf))
+ return ocfs2_error(sb, "Dir Index Leaf has bad signature %.*s\n",
+ 7, dx_leaf->dl_signature);
+
+ if (le16_to_cpu(dx_leaf->dl_list.de_count) !=
+ ocfs2_dx_entries_per_leaf(sb))
+ return ocfs2_error(sb,
+ "Dir Index Leaf # %llu has invalid de_count %u (expected %u)\n",
+ (unsigned long long)le64_to_cpu(dx_leaf->dl_blkno),
+ le16_to_cpu(dx_leaf->dl_list.de_count),
+ ocfs2_dx_entries_per_leaf(sb));
+
+ if (le16_to_cpu(dx_leaf->dl_list.de_num_used) >
+ le16_to_cpu(dx_leaf->dl_list.de_count))
+ return ocfs2_error(sb,
+ "Dir Index Leaf # %llu has invalid de_num_used %u (de_count %u)\n",
+ (unsigned long long)le64_to_cpu(dx_leaf->dl_blkno),
+ le16_to_cpu(dx_leaf->dl_list.de_num_used),
+ le16_to_cpu(dx_leaf->dl_list.de_count));
return ret;
}
diff --git a/fs/ocfs2/dlm/dlmcommon.h b/fs/ocfs2/dlm/dlmcommon.h
index fd2022712167..2ec4586f0bbd 100644
--- a/fs/ocfs2/dlm/dlmcommon.h
+++ b/fs/ocfs2/dlm/dlmcommon.h
@@ -435,6 +435,8 @@ enum {
DLM_DEREF_LOCKRES_DONE = 522,
};
+extern spinlock_t dlm_reco_state_lock;
+
struct dlm_reco_node_data
{
int state;
diff --git a/fs/ocfs2/dlm/dlmdebug.c b/fs/ocfs2/dlm/dlmdebug.c
index e62ed98df7de..afaea6396b6f 100644
--- a/fs/ocfs2/dlm/dlmdebug.c
+++ b/fs/ocfs2/dlm/dlmdebug.c
@@ -792,8 +792,11 @@ static int debug_state_print(struct dlm_ctxt *dlm, char *buf, int len)
buf + out, len - out);
out += scnprintf(buf + out, len - out, "\n");
+ spin_unlock(&dlm->spinlock);
+
/* Recovery Node State: */
out += scnprintf(buf + out, len - out, "Recovery Node State:\n");
+ spin_lock(&dlm_reco_state_lock);
list_for_each_entry(node, &dlm->reco.node_data, list) {
switch (node->state) {
case DLM_RECO_NODE_DATA_INIT:
@@ -825,7 +828,7 @@ static int debug_state_print(struct dlm_ctxt *dlm, char *buf, int len)
node->node_num, state);
}
- spin_unlock(&dlm->spinlock);
+ spin_unlock(&dlm_reco_state_lock);
return out;
}
diff --git a/fs/ocfs2/dlm/dlmrecovery.c b/fs/ocfs2/dlm/dlmrecovery.c
index 8ae9b8d7b1de..a458ab62cd7a 100644
--- a/fs/ocfs2/dlm/dlmrecovery.c
+++ b/fs/ocfs2/dlm/dlmrecovery.c
@@ -80,7 +80,7 @@ static int dlm_lockres_master_requery(struct dlm_ctxt *dlm,
static u64 dlm_get_next_mig_cookie(void);
-static DEFINE_SPINLOCK(dlm_reco_state_lock);
+DEFINE_SPINLOCK(dlm_reco_state_lock);
static DEFINE_SPINLOCK(dlm_mig_cookie_lock);
static u64 dlm_mig_cookie = 1;
diff --git a/fs/ocfs2/inode.c b/fs/ocfs2/inode.c
index 38170b4f4647..3d96cc7fc507 100644
--- a/fs/ocfs2/inode.c
+++ b/fs/ocfs2/inode.c
@@ -233,6 +233,22 @@ struct inode *ocfs2_iget(struct ocfs2_super *osb, u64 blkno, unsigned flags,
return inode;
}
+static int ocfs2_dinode_has_extents(struct ocfs2_dinode *di)
+{
+ /* inodes flagged with other stuff in id2 */
+ if (le32_to_cpu(di->i_flags) &
+ (OCFS2_SUPER_BLOCK_FL | OCFS2_LOCAL_ALLOC_FL | OCFS2_CHAIN_FL |
+ OCFS2_DEALLOC_FL))
+ return 0;
+ /* i_flags doesn't indicate when id2 is a fast symlink */
+ if (S_ISLNK(le16_to_cpu(di->i_mode)) && le64_to_cpu(di->i_size) &&
+ !le32_to_cpu(di->i_clusters))
+ return 0;
+ if (le16_to_cpu(di->i_dyn_features) & OCFS2_INLINE_DATA_FL)
+ return 0;
+
+ return 1;
+}
/*
* here's how inodes get read from disk:
@@ -1563,6 +1579,109 @@ int ocfs2_validate_inode_block(struct super_block *sb,
}
}
+ if (le16_to_cpu(di->i_suballoc_slot) != (u16)OCFS2_INVALID_SLOT &&
+ (u32)le16_to_cpu(di->i_suballoc_slot) > OCFS2_SB(sb)->max_slots - 1) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: suballoc slot %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(di->i_suballoc_slot));
+ goto bail;
+ }
+
+ if ((le32_to_cpu(di->i_flags) & OCFS2_ORPHANED_FL) &&
+ le16_to_cpu(di->i_orphaned_slot) >= OCFS2_SB(sb)->max_slots) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: orphaned slot %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(di->i_orphaned_slot));
+ goto bail;
+ }
+
+ if ((le32_to_cpu(di->i_flags) & OCFS2_DIO_ORPHANED_FL) &&
+ le16_to_cpu(di->i_dio_orphaned_slot) >= OCFS2_SB(sb)->max_slots) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: DIO orphaned slot %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(di->i_dio_orphaned_slot));
+ goto bail;
+ }
+
+ if (S_ISLNK(le16_to_cpu(di->i_mode)) &&
+ !le32_to_cpu(di->i_clusters)) {
+ int max_inline = ocfs2_fast_symlink_chars(sb);
+ u64 i_size = le64_to_cpu(di->i_size);
+
+ if (i_size >= max_inline) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode #%llu: fast symlink i_size %llu exceeds max %d\n",
+ (unsigned long long)bh->b_blocknr,
+ (unsigned long long)i_size,
+ max_inline - 1);
+ goto bail;
+ }
+
+ if (strnlen((char *)di->id2.i_symlink, i_size + 1) != i_size) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode #%llu: fast symlink is not NUL-terminated at i_size %llu\n",
+ (unsigned long long)bh->b_blocknr,
+ (unsigned long long)i_size);
+ goto bail;
+ }
+ }
+
+ if (le32_to_cpu(di->i_flags) & OCFS2_CHAIN_FL) {
+ struct ocfs2_chain_list *cl = &di->id2.i_chain;
+
+ if (le16_to_cpu(cl->cl_count) != ocfs2_chain_recs_per_inode(sb)) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: chain list count %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(cl->cl_count));
+ goto bail;
+ }
+ if (le16_to_cpu(cl->cl_next_free_rec) > le16_to_cpu(cl->cl_count)) {
+ rc = ocfs2_error(sb, "Invalid dinode %llu: chain list index %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(cl->cl_next_free_rec));
+ goto bail;
+ }
+ }
+
+ if ((le16_to_cpu(di->i_dyn_features) & OCFS2_HAS_REFCOUNT_FL) &&
+ !di->i_refcount_loc) {
+ rc = ocfs2_error(sb, "Inode #%llu has refcount flag but no i_refcount_loc\n",
+ (unsigned long long)bh->b_blocknr);
+ goto bail;
+ }
+
+ if (ocfs2_dinode_has_extents(di)) {
+ struct ocfs2_extent_list *el = &di->id2.i_list;
+ u16 count = le16_to_cpu(el->l_count);
+ u16 next_free = le16_to_cpu(el->l_next_free_rec);
+
+ if (count == 0) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode %llu: extent list l_count is zero\n",
+ (unsigned long long)bh->b_blocknr);
+ goto bail;
+ }
+ /*
+ * The exact capacity depends on i_xattr_inline_size, another
+ * unvalidated on-disk field. Inline xattrs only shrink the
+ * list, so the no-xattr maximum is a safe upper bound that a
+ * valid l_count never exceeds.
+ */
+ if (count > ocfs2_extent_recs_per_inode(sb)) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode %llu: extent list l_count %u exceeds max %u\n",
+ (unsigned long long)bh->b_blocknr, count,
+ ocfs2_extent_recs_per_inode(sb));
+ goto bail;
+ }
+ if (next_free > count) {
+ rc = ocfs2_error(sb,
+ "Invalid dinode %llu: extent list l_next_free_rec %u exceeds l_count %u\n",
+ (unsigned long long)bh->b_blocknr, next_free, count);
+ goto bail;
+ }
+ }
+
rc = 0;
bail:
@@ -1728,6 +1847,16 @@ static int ocfs2_filecheck_repair_inode_block(struct super_block *sb,
le32_to_cpu(di->i_fs_generation));
}
+ if (ocfs2_dinode_has_extents(di) &&
+ le16_to_cpu(di->id2.i_list.l_next_free_rec) > le16_to_cpu(di->id2.i_list.l_count)) {
+ di->id2.i_list.l_next_free_rec = di->id2.i_list.l_count;
+ changed = 1;
+ mlog(ML_ERROR,
+ "Filecheck: reset dinode #%llu: l_next_free_rec to %u\n",
+ (unsigned long long)bh->b_blocknr,
+ le16_to_cpu(di->id2.i_list.l_next_free_rec));
+ }
+
if (changed || ocfs2_validate_meta_ecc(sb, bh->b_data, &di->i_check)) {
ocfs2_compute_meta_ecc(sb, bh->b_data, &di->i_check);
mark_buffer_dirty(bh);
diff --git a/fs/ocfs2/namei.c b/fs/ocfs2/namei.c
index 62f9d976c8fb..60770904b243 100644
--- a/fs/ocfs2/namei.c
+++ b/fs/ocfs2/namei.c
@@ -1340,7 +1340,7 @@ static int ocfs2_rename(struct user_namespace *mnt_userns,
goto bail;
}
- if (S_ISDIR(old_inode->i_mode)) {
+ if (S_ISDIR(old_inode->i_mode) && new_dir != old_dir) {
u64 old_inode_parent;
update_dot_dot = 1;
@@ -1357,8 +1357,7 @@ static int ocfs2_rename(struct user_namespace *mnt_userns,
goto bail;
}
- if (!new_inode && new_dir != old_dir &&
- new_dir->i_nlink >= ocfs2_link_max(osb)) {
+ if (!new_inode && new_dir->i_nlink >= ocfs2_link_max(osb)) {
status = -EMLINK;
goto bail;
}
@@ -1601,6 +1600,9 @@ static int ocfs2_rename(struct user_namespace *mnt_userns,
if (update_dot_dot) {
status = ocfs2_update_entry(old_inode, handle,
&old_inode_dot_dot_res, new_dir);
+ }
+
+ if (S_ISDIR(old_inode->i_mode)) {
drop_nlink(old_dir);
if (new_inode) {
drop_nlink(new_inode);
diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c
index cc0c077165bd..3eec30bb6406 100644
--- a/fs/overlayfs/file.c
+++ b/fs/overlayfs/file.c
@@ -570,6 +570,7 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in,
struct file *file_out, loff_t pos_out,
loff_t len, unsigned int flags, enum ovl_copyop op)
{
+ struct inode *inode_in = file_inode(file_in);
struct inode *inode_out = file_inode(file_out);
struct fd real_in, real_out;
const struct cred *old_cred;
@@ -594,7 +595,23 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in,
goto out_unlock;
}
- old_cred = ovl_override_creds(file_inode(file_out)->i_sb);
+ /*
+ * For cross-sb copy, vfs_copy_file_range() will verify read access with
+ * the mounter creds of the dest fs mounter, so we need to explicitly
+ * verify read access with the source mounter creds.
+ */
+ if (unlikely(inode_in->i_sb != inode_out->i_sb)) {
+ old_cred = ovl_override_creds(inode_in->i_sb);
+ ret = rw_verify_area(READ, real_in.file, &pos_in, len);
+ revert_creds(old_cred);
+ if (unlikely(ret)) {
+ fdput(real_in);
+ fdput(real_out);
+ goto out_unlock;
+ }
+ }
+
+ old_cred = ovl_override_creds(inode_out->i_sb);
switch (op) {
case OVL_COPY:
ret = vfs_copy_file_range(real_in.file, pos_in,
diff --git a/fs/pipe.c b/fs/pipe.c
index aa8e6ffe1cb5..2c49e74afeca 100644
--- a/fs/pipe.c
+++ b/fs/pipe.c
@@ -750,9 +750,11 @@ pipe_fasync(int fd, struct file *filp, int on)
if (filp->f_mode & FMODE_READ)
retval = fasync_helper(fd, filp, on, &pipe->fasync_readers);
if ((filp->f_mode & FMODE_WRITE) && retval >= 0) {
+ int read_ret = retval;
+
retval = fasync_helper(fd, filp, on, &pipe->fasync_writers);
- if (retval < 0 && (filp->f_mode & FMODE_READ))
- /* this can happen only if on == T */
+ /* Undo only a reader entry added by this call. */
+ if (retval < 0 && read_ret > 0)
fasync_helper(-1, filp, 0, &pipe->fasync_readers);
}
__pipe_unlock(pipe);
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 484886cdd272..eb002adf9f59 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -727,7 +727,9 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
struct mem_size_stats *mss = walk->private;
struct vm_area_struct *vma = walk->vma;
struct page *page = NULL;
+ spinlock_t *ptl;
+ ptl = huge_pte_lock(hstate_vma(vma), walk->mm, pte);
if (pte_present(*pte)) {
page = vm_normal_page(vma, addr, *pte);
} else if (is_swap_pte(*pte)) {
@@ -742,6 +744,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask,
else
mss->private_hugetlb += huge_page_size(hstate_vma(vma));
}
+ spin_unlock(ptl);
return 0;
}
#else
diff --git a/fs/reiserfs/namei.c b/fs/reiserfs/namei.c
index b916859992ec..0e6bf4f6a80f 100644
--- a/fs/reiserfs/namei.c
+++ b/fs/reiserfs/namei.c
@@ -1325,9 +1325,8 @@ static int reiserfs_rename(struct user_namespace *mnt_userns,
struct inode *old_inode, *new_dentry_inode;
struct reiserfs_transaction_handle th;
int jbegin_count;
- umode_t old_inode_mode;
unsigned long savelink = 1;
- struct timespec64 ctime;
+ bool update_dir_parent = false;
if (flags & ~RENAME_NOREPLACE)
return -EINVAL;
@@ -1377,8 +1376,7 @@ static int reiserfs_rename(struct user_namespace *mnt_userns,
return -ENOENT;
}
- old_inode_mode = old_inode->i_mode;
- if (S_ISDIR(old_inode_mode)) {
+ if (S_ISDIR(old_inode->i_mode)) {
/*
* make sure that directory being renamed has correct ".."
* and that its new parent directory has not too many links
@@ -1391,24 +1389,28 @@ static int reiserfs_rename(struct user_namespace *mnt_userns,
}
}
- /*
- * directory is renamed, its parent directory will be changed,
- * so find ".." entry
- */
- dot_dot_de.de_gen_number_bit_string = NULL;
- retval =
- reiserfs_find_entry(old_inode, "..", 2, &dot_dot_entry_path,
+ if (old_dir != new_dir) {
+ /*
+ * directory is renamed, its parent directory will be
+ * changed, so find ".." entry
+ */
+ dot_dot_de.de_gen_number_bit_string = NULL;
+ retval =
+ reiserfs_find_entry(old_inode, "..", 2,
+ &dot_dot_entry_path,
&dot_dot_de);
- pathrelse(&dot_dot_entry_path);
- if (retval != NAME_FOUND) {
- reiserfs_write_unlock(old_dir->i_sb);
- return -EIO;
- }
+ pathrelse(&dot_dot_entry_path);
+ if (retval != NAME_FOUND) {
+ reiserfs_write_unlock(old_dir->i_sb);
+ return -EIO;
+ }
- /* inode number of .. must equal old_dir->i_ino */
- if (dot_dot_de.de_objectid != old_dir->i_ino) {
- reiserfs_write_unlock(old_dir->i_sb);
- return -EIO;
+ /* inode number of .. must equal old_dir->i_ino */
+ if (dot_dot_de.de_objectid != old_dir->i_ino) {
+ reiserfs_write_unlock(old_dir->i_sb);
+ return -EIO;
+ }
+ update_dir_parent = true;
}
}
@@ -1488,7 +1490,7 @@ static int reiserfs_rename(struct user_namespace *mnt_userns,
reiserfs_prepare_for_journal(old_inode->i_sb, new_de.de_bh, 1);
- if (S_ISDIR(old_inode->i_mode)) {
+ if (update_dir_parent) {
if ((retval =
search_by_entry_key(new_dir->i_sb,
&dot_dot_de.de_entry_key,
@@ -1536,14 +1538,14 @@ static int reiserfs_rename(struct user_namespace *mnt_userns,
new_de.de_bh);
reiserfs_restore_prepared_buffer(old_inode->i_sb,
old_de.de_bh);
- if (S_ISDIR(old_inode_mode))
+ if (update_dir_parent)
reiserfs_restore_prepared_buffer(old_inode->
i_sb,
dot_dot_de.
de_bh);
continue;
}
- if (S_ISDIR(old_inode_mode)) {
+ if (update_dir_parent) {
if (item_moved(&dot_dot_ih, &dot_dot_entry_path) ||
!entry_points_to_object("..", 2, &dot_dot_de,
old_dir)) {
@@ -1561,7 +1563,7 @@ static int reiserfs_rename(struct user_namespace *mnt_userns,
}
}
- RFALSE(S_ISDIR(old_inode_mode) &&
+ RFALSE(update_dir_parent &&
!buffer_journal_prepared(dot_dot_de.de_bh), "");
break;
@@ -1578,14 +1580,11 @@ static int reiserfs_rename(struct user_namespace *mnt_userns,
mark_de_hidden(old_de.de_deh + old_de.de_entry_num);
journal_mark_dirty(&th, old_de.de_bh);
- ctime = current_time(old_dir);
- old_dir->i_ctime = old_dir->i_mtime = ctime;
- new_dir->i_ctime = new_dir->i_mtime = ctime;
/*
* thanks to Alex Adriaanse <alex_a@caltech.edu> for patch
* which adds ctime update of renamed object
*/
- old_inode->i_ctime = ctime;
+ simple_rename_timestamp(old_dir, old_dentry, new_dir, new_dentry);
if (new_dentry_inode) {
/* adjust link number of the victim */
@@ -1594,15 +1593,15 @@ static int reiserfs_rename(struct user_namespace *mnt_userns,
} else {
drop_nlink(new_dentry_inode);
}
- new_dentry_inode->i_ctime = ctime;
savelink = new_dentry_inode->i_nlink;
}
- if (S_ISDIR(old_inode_mode)) {
+ if (update_dir_parent) {
/* adjust ".." of renamed directory */
set_ino_in_dir_entry(&dot_dot_de, INODE_PKEY(new_dir));
journal_mark_dirty(&th, dot_dot_de.de_bh);
-
+ }
+ if (S_ISDIR(old_inode->i_mode)) {
/*
* there (in new_dir) was no directory, so it got new link
* (".." of renamed directory)
diff --git a/fs/smb/client/cached_dir.c b/fs/smb/client/cached_dir.c
index fca865692939..3ff56321b7a1 100644
--- a/fs/smb/client/cached_dir.c
+++ b/fs/smb/client/cached_dir.c
@@ -30,7 +30,7 @@ static struct cached_fid *find_or_create_cached_dir(struct cached_fids *cfids,
* fully cached or it may be in the process of
* being deleted due to a lease break.
*/
- if (!cfid->has_lease) {
+ if (!cfid->time || !cfid->has_lease) {
spin_unlock(&cfids->cfid_list_lock);
return NULL;
}
@@ -57,6 +57,16 @@ static struct cached_fid *find_or_create_cached_dir(struct cached_fids *cfids,
list_add(&cfid->entry, &cfids->entries);
cfid->on_list = true;
kref_get(&cfid->refcount);
+ /*
+ * Set @cfid->has_lease to true during construction so that the lease
+ * reference can be put in cached_dir_lease_break() due to a potential
+ * lease break right after the request is sent or while @cfid is still
+ * being cached, or if a reconnection is triggered during construction.
+ * Concurrent processes won't be to use it yet due to @cfid->time being
+ * zero.
+ */
+ cfid->has_lease = true;
+
spin_unlock(&cfids->cfid_list_lock);
return cfid;
}
@@ -168,11 +178,11 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon,
return -ENOENT;
}
/*
- * At this point we either have a lease already and we can just
- * return it. If not we are guaranteed to be the only thread accessing
- * this cfid.
+ * Return cached fid if it is valid (has a lease and has a time).
+ * Otherwise, it is either a new entry or laundromat worker removed it
+ * from @cfids->entries. Caller will put last reference if the latter.
*/
- if (cfid->has_lease) {
+ if (cfid->has_lease && cfid->time) {
*ret_cfid = cfid;
kfree(utf16_path);
return 0;
@@ -188,9 +198,20 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon,
npath = path_no_prefix(cifs_sb, path);
if (IS_ERR(npath)) {
rc = PTR_ERR(npath);
- kfree(utf16_path);
- return rc;
+ goto out;
+ }
+
+ if (!npath[0]) {
+ dentry = dget(cifs_sb->root);
+ } else {
+ dentry = path_to_dentry(cifs_sb, npath);
+ if (IS_ERR(dentry)) {
+ rc = -ENOENT;
+ goto out;
+ }
}
+ cfid->dentry = dentry;
+ cfid->tcon = tcon;
/*
* We do not hold the lock for the open because in case
@@ -264,9 +285,10 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon,
}
goto oshr_free;
}
- cfid->tcon = tcon;
cfid->is_open = true;
+ spin_lock(&cfids->cfid_list_lock);
+
o_rsp = (struct smb2_create_rsp *)rsp_iov[0].iov_base;
oparms.fid->persistent_fid = o_rsp->PersistentFileId;
oparms.fid->volatile_fid = o_rsp->VolatileFileId;
@@ -274,20 +296,30 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon,
oparms.fid->mid = le64_to_cpu(o_rsp->hdr.MessageId);
#endif /* CIFS_DEBUG2 */
- if (o_rsp->OplockLevel != SMB2_OPLOCK_LEVEL_LEASE)
+ rc = -EINVAL;
+ if (o_rsp->OplockLevel != SMB2_OPLOCK_LEVEL_LEASE) {
+ spin_unlock(&cfids->cfid_list_lock);
goto oshr_free;
+ }
rc = smb2_parse_contexts(server, rsp_iov,
&oparms.fid->epoch,
oparms.fid->lease_key,
&oplock, NULL, NULL);
- if (rc)
+ if (rc) {
+ spin_unlock(&cfids->cfid_list_lock);
goto oshr_free;
- if (!(oplock & SMB2_LEASE_READ_CACHING_HE))
+ }
+ rc = -EINVAL;
+ if (!(oplock & SMB2_LEASE_READ_CACHING_HE)) {
+ spin_unlock(&cfids->cfid_list_lock);
goto oshr_free;
+ }
qi_rsp = (struct smb2_query_info_rsp *)rsp_iov[1].iov_base;
- if (le32_to_cpu(qi_rsp->OutputBufferLength) < sizeof(struct smb2_file_all_info))
+ if (le32_to_cpu(qi_rsp->OutputBufferLength) < sizeof(struct smb2_file_all_info)) {
+ spin_unlock(&cfids->cfid_list_lock);
goto oshr_free;
+ }
if (!smb2_validate_and_copy_iov(
le16_to_cpu(qi_rsp->OutputBufferOffset),
sizeof(struct smb2_file_all_info),
@@ -295,57 +327,49 @@ int open_cached_dir(unsigned int xid, struct cifs_tcon *tcon,
(char *)&cfid->file_all_info))
cfid->file_all_info_is_valid = true;
- if (!npath[0])
- dentry = dget(cifs_sb->root);
- else {
- dentry = path_to_dentry(cifs_sb, npath);
- if (IS_ERR(dentry)) {
- rc = -ENOENT;
- goto oshr_free;
- }
- }
- cfid->dentry = dentry;
cfid->time = jiffies;
- cfid->has_lease = true;
+ spin_unlock(&cfids->cfid_list_lock);
+ /* At this point the directory handle is fully cached */
+ rc = 0;
oshr_free:
- kfree(utf16_path);
SMB2_open_free(&rqst[0]);
SMB2_query_info_free(&rqst[1]);
free_rsp_buf(resp_buftype[0], rsp_iov[0].iov_base);
free_rsp_buf(resp_buftype[1], rsp_iov[1].iov_base);
- spin_lock(&cfids->cfid_list_lock);
- if (rc && !cfid->has_lease) {
+ if (rc) {
+ spin_lock(&cfids->cfid_list_lock);
if (cfid->on_list) {
list_del(&cfid->entry);
cfid->on_list = false;
cfids->num_entries--;
}
- rc = -ENOENT;
- }
- spin_unlock(&cfids->cfid_list_lock);
- if (!rc && !cfid->has_lease) {
- /*
- * We are guaranteed to have two references at this point.
- * One for the caller and one for a potential lease.
- * Release the Lease-ref so that the directory will be closed
- * when the caller closes the cached handle.
- */
- kref_put(&cfid->refcount, smb2_close_cached_fid);
+ if (cfid->has_lease) {
+ /*
+ * We are guaranteed to have two references at this
+ * point. One for the caller and one for a potential
+ * lease. Release the Lease-ref so that the directory
+ * will be closed when the caller closes the cached
+ * handle.
+ */
+ cfid->has_lease = false;
+ spin_unlock(&cfids->cfid_list_lock);
+ close_cached_dir(cfid);
+ goto out;
+ }
+ spin_unlock(&cfids->cfid_list_lock);
}
+out:
if (rc) {
if (cfid->is_open)
SMB2_close(0, cfid->tcon, cfid->fid.persistent_fid,
cfid->fid.volatile_fid);
free_cached_dir(cfid);
- cfid = NULL;
- }
-
- if (rc == 0) {
+ } else {
*ret_cfid = cfid;
atomic_inc(&tcon->num_remote_opens);
}
-
+ kfree(utf16_path);
return rc;
}
@@ -375,12 +399,14 @@ int open_cached_dir_by_dentry(struct cifs_tcon *tcon,
static void
smb2_close_cached_fid(struct kref *ref)
+__releases(&cfid->cfids->cfid_list_lock)
{
struct cached_fid *cfid = container_of(ref, struct cached_fid,
refcount);
int rc;
- spin_lock(&cfid->cfids->cfid_list_lock);
+ lockdep_assert_held(&cfid->cfids->cfid_list_lock);
+
if (cfid->on_list) {
list_del(&cfid->entry);
cfid->on_list = false;
@@ -415,7 +441,7 @@ void drop_cached_dir_by_name(const unsigned int xid, struct cifs_tcon *tcon,
spin_lock(&cfid->cfids->cfid_list_lock);
if (cfid->has_lease) {
cfid->has_lease = false;
- kref_put(&cfid->refcount, smb2_close_cached_fid);
+ close_cached_dir(cfid);
}
spin_unlock(&cfid->cfids->cfid_list_lock);
close_cached_dir(cfid);
@@ -424,7 +450,7 @@ void drop_cached_dir_by_name(const unsigned int xid, struct cifs_tcon *tcon,
void close_cached_dir(struct cached_fid *cfid)
{
- kref_put(&cfid->refcount, smb2_close_cached_fid);
+ kref_put_lock(&cfid->refcount, smb2_close_cached_fid, &cfid->cfids->cfid_list_lock);
}
/*
@@ -486,10 +512,10 @@ void invalidate_all_cached_dirs(struct cifs_tcon *tcon)
spin_lock(&cfids->cfid_list_lock);
cfid->has_lease = false;
spin_unlock(&cfids->cfid_list_lock);
- kref_put(&cfid->refcount, smb2_close_cached_fid);
+ close_cached_dir(cfid);
}
/* Drop the extra reference opened above*/
- kref_put(&cfid->refcount, smb2_close_cached_fid);
+ close_cached_dir(cfid);
}
}
@@ -502,7 +528,7 @@ smb2_cached_lease_break(struct work_struct *work)
spin_lock(&cfid->cfids->cfid_list_lock);
cfid->has_lease = false;
spin_unlock(&cfid->cfids->cfid_list_lock);
- kref_put(&cfid->refcount, smb2_close_cached_fid);
+ close_cached_dir(cfid);
}
int cached_dir_lease_break(struct cifs_tcon *tcon, __u8 lease_key[16])
diff --git a/fs/smb/client/cifsacl.c b/fs/smb/client/cifsacl.c
index 93e3f0bcecac..7bb35cdbc4d6 100644
--- a/fs/smb/client/cifsacl.c
+++ b/fs/smb/client/cifsacl.c
@@ -1016,12 +1016,12 @@ unsigned int setup_special_user_owner_ACE(struct cifs_ace *pntace)
static void populate_new_aces(char *nacl_base,
struct cifs_sid *pownersid,
struct cifs_sid *pgrpsid,
- __u64 *pnmode, u32 *pnum_aces, u16 *pnsize,
+ __u64 *pnmode, u32 *pnum_aces, u32 *pnsize,
bool modefromsid)
{
__u64 nmode;
u32 num_aces = 0;
- u16 nsize = 0;
+ u32 nsize = 0;
__u64 user_mode;
__u64 group_mode;
__u64 other_mode;
@@ -1118,16 +1118,16 @@ static void populate_new_aces(char *nacl_base,
*pnsize = nsize;
}
-static __u16 replace_sids_and_copy_aces(struct cifs_acl *pdacl, struct cifs_acl *pndacl,
+static int replace_sids_and_copy_aces(struct cifs_acl *pdacl, struct cifs_acl *pndacl,
struct cifs_sid *pownersid, struct cifs_sid *pgrpsid,
- struct cifs_sid *pnownersid, struct cifs_sid *pngrpsid)
+ struct cifs_sid *pnownersid, struct cifs_sid *pngrpsid, u16 *pnsize)
{
int i;
u16 size = 0;
struct cifs_ace *pntace = NULL;
char *acl_base = NULL;
u32 src_num_aces = 0;
- u16 nsize = 0;
+ u32 nsize = 0;
struct cifs_ace *pnntace = NULL;
char *nacl_base = NULL;
u16 ace_size = 0;
@@ -1153,9 +1153,12 @@ static __u16 replace_sids_and_copy_aces(struct cifs_acl *pdacl, struct cifs_acl
size += le16_to_cpu(pntace->size);
nsize += ace_size;
+ if (nsize > U16_MAX)
+ return -EOVERFLOW;
}
- return nsize;
+ *pnsize = nsize;
+ return 0;
}
static int set_chmod_dacl(struct cifs_acl *pdacl, struct cifs_acl *pndacl,
@@ -1167,7 +1170,7 @@ static int set_chmod_dacl(struct cifs_acl *pdacl, struct cifs_acl *pndacl,
struct cifs_ace *pntace = NULL;
char *acl_base = NULL;
u32 src_num_aces = 0;
- u16 nsize = 0;
+ u32 nsize = 0;
struct cifs_ace *pnntace = NULL;
char *nacl_base = NULL;
u32 num_aces = 0;
@@ -1218,6 +1221,8 @@ static int set_chmod_dacl(struct cifs_acl *pdacl, struct cifs_acl *pndacl,
nsize += cifs_copy_ace(pnntace, pntace, NULL);
num_aces++;
+ if (nsize > U16_MAX)
+ return -EOVERFLOW;
next_ace:
size += le16_to_cpu(pntace->size);
@@ -1234,6 +1239,10 @@ static int set_chmod_dacl(struct cifs_acl *pdacl, struct cifs_acl *pndacl,
}
finalize_dacl:
+ /* The DACL size field is 16-bit on the wire, see MS-DTYP 2.4.5 */
+ if (nsize > U16_MAX)
+ return -EOVERFLOW;
+
pndacl->num_aces = cpu_to_le32(num_aces);
pndacl->size = cpu_to_le16(nsize);
@@ -1291,6 +1300,7 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
{
int rc = 0;
struct cifs_sid *owner_sid_ptr, *group_sid_ptr;
+ unsigned int sbflags = cifs_sb->mnt_cifs_flags;
struct cifs_acl *dacl_ptr; /* no need for SACL ptr */
char *end_of_acl = ((char *)pntsd) + acl_len;
__u32 dacloffset;
@@ -1307,17 +1317,22 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
pntsd->revision, pntsd->type, le32_to_cpu(pntsd->osidoffset),
le32_to_cpu(pntsd->gsidoffset),
le32_to_cpu(pntsd->sacloffset), dacloffset);
+ fattr->cf_uid = cifs_sb->ctx->linux_uid;
+ fattr->cf_gid = cifs_sb->ctx->linux_gid;
+
/* cifs_dump_mem("owner_sid: ", owner_sid_ptr, 64); */
rc = parse_sid(owner_sid_ptr, end_of_acl);
if (rc) {
cifs_dbg(FYI, "%s: Error %d parsing Owner SID\n", __func__, rc);
return rc;
}
- rc = sid_to_id(cifs_sb, owner_sid_ptr, fattr, SIDOWNER);
- if (rc) {
- cifs_dbg(FYI, "%s: Error %d mapping Owner SID to uid\n",
- __func__, rc);
- return rc;
+ if (!(sbflags & CIFS_MOUNT_OVERR_UID)) {
+ rc = sid_to_id(cifs_sb, owner_sid_ptr, fattr, SIDOWNER);
+ if (rc) {
+ cifs_dbg(FYI, "%s: Error %d mapping Owner SID to uid\n",
+ __func__, rc);
+ return rc;
+ }
}
rc = parse_sid(group_sid_ptr, end_of_acl);
@@ -1326,11 +1341,13 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb,
__func__, rc);
return rc;
}
- rc = sid_to_id(cifs_sb, group_sid_ptr, fattr, SIDGROUP);
- if (rc) {
- cifs_dbg(FYI, "%s: Error %d mapping Group SID to gid\n",
- __func__, rc);
- return rc;
+ if (!(sbflags & CIFS_MOUNT_OVERR_GID)) {
+ rc = sid_to_id(cifs_sb, group_sid_ptr, fattr, SIDGROUP);
+ if (rc) {
+ cifs_dbg(FYI, "%s: Error %d mapping Group SID to gid\n",
+ __func__, rc);
+ return rc;
+ }
}
if (dacloffset) {
@@ -1394,6 +1411,8 @@ static int build_sec_desc(struct cifs_ntsd *pntsd, struct cifs_ntsd *pnntsd,
rc = set_chmod_dacl(dacl_ptr, ndacl_ptr, owner_sid_ptr, group_sid_ptr,
pnmode, mode_from_sid);
+ if (rc)
+ return rc;
sidsoffset = ndacloffset + le16_to_cpu(ndacl_ptr->size);
/* copy the non-dacl portion of secdesc */
@@ -1469,9 +1488,11 @@ static int build_sec_desc(struct cifs_ntsd *pntsd, struct cifs_ntsd *pnntsd,
if (dacloffset) {
/* Replace ACEs for old owner with new one */
- size = replace_sids_and_copy_aces(dacl_ptr, ndacl_ptr,
+ rc = replace_sids_and_copy_aces(dacl_ptr, ndacl_ptr,
owner_sid_ptr, group_sid_ptr,
- nowner_sid_ptr, ngroup_sid_ptr);
+ nowner_sid_ptr, ngroup_sid_ptr, &size);
+ if (rc)
+ goto chown_chgrp_exit;
ndacl_ptr->size = cpu_to_le16(size);
}
@@ -1751,11 +1772,13 @@ id_mode_to_cifs_acl(struct inode *inode, const char *path, __u64 *pnmode,
cifs_put_tlink(tlink);
return rc;
}
- if (mode_from_sid)
- nsecdesclen +=
- le32_to_cpu(dacl_ptr->num_aces) * sizeof(struct cifs_ace);
- else /* cifsacl */
- nsecdesclen += le16_to_cpu(dacl_ptr->size);
+ /*
+ * Worst case: every ACE is rewritten with a new SID of
+ * SID_MAX_SUB_AUTHORITIES sub-auths -> sizeof(cifs_ace) each,
+ * plus the cifs_acl header replace_sids_and_copy_aces() emits.
+ */
+ nsecdesclen += sizeof(struct cifs_acl) +
+ le32_to_cpu(dacl_ptr->num_aces) * sizeof(struct cifs_ace);
}
}
diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c
index 8046b54976aa..c43c937b3f57 100644
--- a/fs/smb/client/cifsfs.c
+++ b/fs/smb/client/cifsfs.c
@@ -399,6 +399,7 @@ cifs_alloc_inode(struct super_block *sb)
return NULL;
cifs_inode->cifsAttrs = 0x20; /* default */
cifs_inode->time = 0;
+ cifs_inode->time_last_write = 0;
/*
* Until the file is open and we have gotten oplock info back from the
* server, can not assume caching of file data or metadata.
diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h
index a5e9f79c10f9..a9dce6fe2705 100644
--- a/fs/smb/client/cifsglob.h
+++ b/fs/smb/client/cifsglob.h
@@ -80,10 +80,6 @@
#define SMB_ECHO_INTERVAL_MAX 600
#define SMB_ECHO_INTERVAL_DEFAULT 60
-/* dns resolution intervals in seconds */
-#define SMB_DNS_RESOLVE_INTERVAL_MIN 120
-#define SMB_DNS_RESOLVE_INTERVAL_DEFAULT 600
-
/* smb multichannel query server interfaces interval in seconds */
#define SMB_INTERFACE_POLL_INTERVAL 600
@@ -589,6 +585,7 @@ get_rfc1002_length(void *buf)
struct TCP_Server_Info {
struct list_head tcp_ses_list;
struct list_head smb_ses_list;
+ struct list_head rlist; /* reconnect list */
spinlock_t srv_lock; /* protect anything here that is not protected */
__u64 conn_id; /* connection identifier (useful for debugging) */
int srv_count; /* reference counter */
@@ -671,7 +668,6 @@ struct TCP_Server_Info {
/* point to the SMBD connection if RDMA is used instead of socket */
struct smbd_connection *smbd_conn;
struct delayed_work echo; /* echo ping workqueue job */
- struct delayed_work resolve; /* dns resolution workqueue job */
char *smallbuf; /* pointer to current "small" buffer */
char *bigbuf; /* pointer to current "big" buffer */
/* Total size of this PDU. Only valid from cifs_demultiplex_thread */
@@ -1337,10 +1333,10 @@ struct cifsFileInfo {
struct dentry *dentry;
struct tcon_link *tlink;
unsigned int f_flags;
- bool invalidHandle:1; /* file closed via session abend */
- bool swapfile:1;
- bool oplock_break_cancelled:1;
- bool offload:1; /* offload final part of _put to a wq */
+ bool invalidHandle; /* file closed via session abend */
+ bool swapfile;
+ bool oplock_break_cancelled;
+ bool offload; /* offload final part of _put to a wq */
__u16 oplock_epoch; /* epoch from the lease break */
__u32 oplock_level; /* oplock/lease level from the lease break */
int count;
@@ -1504,6 +1500,7 @@ struct cifsInodeInfo {
spinlock_t writers_lock;
unsigned int writers; /* Number of writers on this inode */
unsigned long time; /* jiffies of last update of inode */
+ unsigned long time_last_write; /* jiffies of last writable close or truncate */
u64 server_eof; /* current file size on server -- protected by i_lock */
u64 uniqueid; /* server inode number */
u64 createtime; /* creation time on server */
diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c
index 32c57b040e7c..e94a031c19d3 100644
--- a/fs/smb/client/connect.c
+++ b/fs/smb/client/connect.c
@@ -91,8 +91,6 @@ static int reconn_set_ipaddr_from_hostname(struct TCP_Server_Info *server)
int rc;
int len;
char *unc, *ipaddr = NULL;
- time64_t expiry, now;
- unsigned long ttl = SMB_DNS_RESOLVE_INTERVAL_DEFAULT;
if (!server->hostname)
return -EINVAL;
@@ -110,13 +108,13 @@ static int reconn_set_ipaddr_from_hostname(struct TCP_Server_Info *server)
}
scnprintf(unc, len, "\\\\%s", server->hostname);
- rc = dns_resolve_server_name_to_ip(unc, &ipaddr, &expiry);
+ rc = dns_resolve_server_name_to_ip(unc, &ipaddr, NULL);
kfree(unc);
if (rc < 0) {
cifs_dbg(FYI, "%s: failed to resolve server part of %s to IP: %d\n",
__func__, server->hostname, rc);
- goto requeue_resolve;
+ return rc;
}
spin_lock(&server->srv_lock);
@@ -125,23 +123,8 @@ static int reconn_set_ipaddr_from_hostname(struct TCP_Server_Info *server)
spin_unlock(&server->srv_lock);
kfree(ipaddr);
- /* rc == 1 means success here */
- if (rc) {
- now = ktime_get_real_seconds();
- if (expiry && expiry > now)
- /*
- * To make sure we don't use the cached entry, retry 1s
- * after expiry.
- */
- ttl = max_t(unsigned long, expiry - now, SMB_DNS_RESOLVE_INTERVAL_MIN) + 1;
- }
rc = !rc ? -1 : 0;
-requeue_resolve:
- cifs_dbg(FYI, "%s: next dns resolution scheduled for %lu seconds in the future\n",
- __func__, ttl);
- mod_delayed_work(cifsiod_wq, &server->resolve, (ttl * HZ));
-
return rc;
}
@@ -165,25 +148,13 @@ static void smb2_query_server_interfaces(struct work_struct *work)
(SMB_INTERFACE_POLL_INTERVAL * HZ));
}
-static void cifs_resolve_server(struct work_struct *work)
-{
- int rc;
- struct TCP_Server_Info *server = container_of(work,
- struct TCP_Server_Info, resolve.work);
-
- cifs_server_lock(server);
-
- /*
- * Resolve the hostname again to make sure that IP address is up-to-date.
- */
- rc = reconn_set_ipaddr_from_hostname(server);
- if (rc) {
- cifs_dbg(FYI, "%s: failed to resolve hostname: %d\n",
- __func__, rc);
- }
-
- cifs_server_unlock(server);
-}
+#define set_need_reco(server) \
+do { \
+ spin_lock(&server->srv_lock); \
+ if (server->tcpStatus != CifsExiting) \
+ server->tcpStatus = CifsNeedReconnect; \
+ spin_unlock(&server->srv_lock); \
+} while (0)
/*
* Update the tcpStatus for the server.
@@ -198,35 +169,45 @@ void
cifs_signal_cifsd_for_reconnect(struct TCP_Server_Info *server,
bool all_channels)
{
- struct TCP_Server_Info *pserver;
+ struct TCP_Server_Info *nserver;
struct cifs_ses *ses;
+ LIST_HEAD(reco);
int i;
- /* If server is a channel, select the primary channel */
- pserver = CIFS_SERVER_IS_CHAN(server) ? server->primary_server : server;
-
/* if we need to signal just this channel */
if (!all_channels) {
- spin_lock(&server->srv_lock);
- if (server->tcpStatus != CifsExiting)
- server->tcpStatus = CifsNeedReconnect;
- spin_unlock(&server->srv_lock);
+ set_need_reco(server);
return;
}
- spin_lock(&cifs_tcp_ses_lock);
- list_for_each_entry(ses, &pserver->smb_ses_list, smb_ses_list) {
- if (cifs_ses_exiting(ses))
- continue;
- spin_lock(&ses->chan_lock);
- for (i = 0; i < ses->chan_count; i++) {
- spin_lock(&ses->chans[i].server->srv_lock);
- ses->chans[i].server->tcpStatus = CifsNeedReconnect;
- spin_unlock(&ses->chans[i].server->srv_lock);
+ if (CIFS_SERVER_IS_CHAN(server))
+ server = server->primary_server;
+ scoped_guard(spinlock, &cifs_tcp_ses_lock) {
+ set_need_reco(server);
+ list_for_each_entry(ses, &server->smb_ses_list, smb_ses_list) {
+ spin_lock(&ses->ses_lock);
+ if (ses->ses_status == SES_EXITING) {
+ spin_unlock(&ses->ses_lock);
+ continue;
+ }
+ spin_lock(&ses->chan_lock);
+ for (i = 1; i < ses->chan_count; i++) {
+ nserver = ses->chans[i].server;
+ if (!nserver)
+ continue;
+ nserver->srv_count++;
+ list_add(&nserver->rlist, &reco);
+ }
+ spin_unlock(&ses->chan_lock);
+ spin_unlock(&ses->ses_lock);
}
- spin_unlock(&ses->chan_lock);
}
- spin_unlock(&cifs_tcp_ses_lock);
+
+ list_for_each_entry_safe(server, nserver, &reco, rlist) {
+ list_del_init(&server->rlist);
+ set_need_reco(server);
+ cifs_put_tcp_session(server, 0);
+ }
}
/*
@@ -1008,7 +989,6 @@ static void clean_demultiplex_info(struct TCP_Server_Info *server)
spin_unlock(&server->srv_lock);
cancel_delayed_work_sync(&server->echo);
- cancel_delayed_work_sync(&server->resolve);
spin_lock(&server->srv_lock);
server->tcpStatus = CifsExiting;
@@ -1083,10 +1063,12 @@ static void clean_demultiplex_info(struct TCP_Server_Info *server)
*/
}
+ put_net(cifs_net_ns(server));
#ifdef CONFIG_CIFS_DFS_UPCALL
kfree(server->origin_fullpath);
kfree(server->leaf_fullpath);
#endif
+ kfree(server->hostname);
kfree(server);
length = atomic_dec_return(&tcpSesAllocCount);
@@ -1677,8 +1659,6 @@ cifs_put_tcp_session(struct TCP_Server_Info *server, int from_reconnect)
/* srv_count can never go negative */
WARN_ON(server->srv_count < 0);
- put_net(cifs_net_ns(server));
-
list_del_init(&server->tcp_ses_list);
spin_unlock(&cifs_tcp_ses_lock);
@@ -1687,7 +1667,6 @@ cifs_put_tcp_session(struct TCP_Server_Info *server, int from_reconnect)
cifs_put_tcp_session(server->primary_server, from_reconnect);
cancel_delayed_work_sync(&server->echo);
- cancel_delayed_work_sync(&server->resolve);
if (from_reconnect)
/*
@@ -1709,8 +1688,6 @@ cifs_put_tcp_session(struct TCP_Server_Info *server, int from_reconnect)
kfree_sensitive(server->session_key.response);
server->session_key.response = NULL;
server->session_key.len = 0;
- kfree(server->hostname);
- server->hostname = NULL;
task = xchg(&server->tsk, NULL);
if (task)
@@ -1785,7 +1762,6 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
INIT_LIST_HEAD(&tcp_ses->tcp_ses_list);
INIT_LIST_HEAD(&tcp_ses->smb_ses_list);
INIT_DELAYED_WORK(&tcp_ses->echo, cifs_echo_request);
- INIT_DELAYED_WORK(&tcp_ses->resolve, cifs_resolve_server);
INIT_DELAYED_WORK(&tcp_ses->reconnect, smb2_reconnect_server);
mutex_init(&tcp_ses->reconnect_mutex);
#ifdef CONFIG_CIFS_DFS_UPCALL
@@ -1841,23 +1817,13 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
* this will succeed. No need for try_module_get().
*/
__module_get(THIS_MODULE);
- tcp_ses->tsk = kthread_run(cifs_demultiplex_thread,
- tcp_ses, "cifsd");
- if (IS_ERR(tcp_ses->tsk)) {
- rc = PTR_ERR(tcp_ses->tsk);
- cifs_dbg(VFS, "error %d create cifsd thread\n", rc);
- module_put(THIS_MODULE);
- goto out_err_crypto_release;
- }
tcp_ses->min_offload = ctx->min_offload;
/*
* at this point we are the only ones with the pointer
* to the struct since the kernel thread not created yet
* no need to spinlock this update of tcpStatus
*/
- spin_lock(&tcp_ses->srv_lock);
tcp_ses->tcpStatus = CifsNeedNegotiate;
- spin_unlock(&tcp_ses->srv_lock);
if ((ctx->max_credits < 20) || (ctx->max_credits > 60000))
tcp_ses->max_credits = SMB2_MAX_CREDITS_AVAILABLE;
@@ -1866,7 +1832,16 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
tcp_ses->nr_targets = 1;
tcp_ses->ignore_signature = ctx->ignore_signature;
- /* thread spawned, put it on the list */
+
+ tcp_ses->tsk = kthread_create(cifs_demultiplex_thread,
+ tcp_ses, "cifsd");
+ if (IS_ERR(tcp_ses->tsk)) {
+ rc = PTR_ERR(tcp_ses->tsk);
+ cifs_dbg(VFS, "error %d create cifsd thread\n", rc);
+ module_put(THIS_MODULE);
+ goto out_err_crypto_release;
+ }
+ /* thread created, put it on the list */
spin_lock(&cifs_tcp_ses_lock);
list_add(&tcp_ses->tcp_ses_list, &cifs_tcp_ses_list);
spin_unlock(&cifs_tcp_ses_lock);
@@ -1874,11 +1849,11 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
/* queue echo request delayed work */
queue_delayed_work(cifsiod_wq, &tcp_ses->echo, tcp_ses->echo_interval);
- /* queue dns resolution delayed work */
- cifs_dbg(FYI, "%s: next dns resolution scheduled for %d seconds in the future\n",
- __func__, SMB_DNS_RESOLVE_INTERVAL_DEFAULT);
-
- queue_delayed_work(cifsiod_wq, &tcp_ses->resolve, (SMB_DNS_RESOLVE_INTERVAL_DEFAULT * HZ));
+ /*
+ * Use split create/wake logic to ensure that tcp_ses is fully populated
+ * and tcp_ses->tsk is valid
+ */
+ wake_up_process(tcp_ses->tsk);
return tcp_ses;
@@ -3079,7 +3054,10 @@ generic_ip_connect(struct TCP_Server_Info *server)
}
if (socket == NULL) {
- rc = __sock_create(cifs_net_ns(server), sfamily, SOCK_STREAM,
+ struct net *net = cifs_net_ns(server);
+ struct sock *sk;
+
+ rc = __sock_create(net, sfamily, SOCK_STREAM,
IPPROTO_TCP, &socket, 1);
if (rc < 0) {
cifs_server_dbg(VFS, "Error %d creating socket\n", rc);
@@ -3087,10 +3065,16 @@ generic_ip_connect(struct TCP_Server_Info *server)
return rc;
}
+ sk = socket->sk;
+ sk->sk_net_refcnt = 1;
+ get_net_track(net, &sk->ns_tracker, GFP_KERNEL);
+ sock_inuse_add(net, 1);
+
/* BB other socket options to set KEEPALIVE, NODELAY? */
cifs_dbg(FYI, "Socket created\n");
server->ssocket = socket;
socket->sk->sk_allocation = GFP_NOFS;
+ socket->sk->sk_use_task_frag = false;
if (sfamily == AF_INET6)
cifs_reclassify_socket6(socket);
else
@@ -4114,6 +4098,10 @@ CIFSTCon(const unsigned int xid, struct cifs_ses *ses,
tcon->tid = smb_buffer_response->Tid;
bcc_ptr = pByteArea(smb_buffer_response);
bytes_left = get_bcc(smb_buffer_response);
+ if (bytes_left < 2) {
+ rc = -EIO;
+ goto out;
+ }
length = strnlen(bcc_ptr, bytes_left - 2);
if (smb_buffer->Flags2 & SMBFLG2_UNICODE)
is_unicode = true;
@@ -4169,6 +4157,7 @@ CIFSTCon(const unsigned int xid, struct cifs_ses *ses,
reset_cifs_unix_caps(xid, tcon, NULL, NULL);
}
}
+out:
cifs_buf_release(smb_buffer);
return rc;
}
@@ -4368,14 +4357,25 @@ cifs_setup_session(const unsigned int xid, struct cifs_ses *ses,
return rc;
}
-static int
-cifs_set_vol_auth(struct smb3_fs_context *ctx, struct cifs_ses *ses)
+static int set_fs_context_auth(struct smb3_fs_context *ctx,
+ struct cifs_ses *ses)
{
ctx->sectype = ses->sectype;
- /* krb5 is special, since we don't need username or pw */
- if (ctx->sectype == Kerberos)
+ /*
+ * krb5 is special as we might need to pass username (passwordless) down
+ * to cifs.upcall(8) for keytab.
+ */
+ if (ctx->sectype == Kerberos) {
+ if (ses->user_name && ses->user_name[0]) {
+ ctx->username = kstrndup(ses->user_name,
+ CIFS_MAX_USERNAME_LEN,
+ GFP_KERNEL);
+ if (!ctx->username)
+ return -ENOMEM;
+ }
return 0;
+ }
return cifs_set_cifscreds(ctx, ses);
}
@@ -4412,7 +4412,7 @@ cifs_construct_tcon(struct cifs_sb_info *cifs_sb, kuid_t fsuid)
ctx->seal = master_tcon->seal;
ctx->witness = master_tcon->use_witness;
- rc = cifs_set_vol_auth(ctx, master_tcon->ses);
+ rc = set_fs_context_auth(ctx, master_tcon->ses);
if (rc) {
tcon = ERR_PTR(rc);
goto out;
diff --git a/fs/smb/client/dir.c b/fs/smb/client/dir.c
index b5b64b061b71..d34230b8e124 100644
--- a/fs/smb/client/dir.c
+++ b/fs/smb/client/dir.c
@@ -176,7 +176,8 @@ check_name(struct dentry *direntry, struct cifs_tcon *tcon)
static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned int xid,
struct tcon_link *tlink, unsigned int oflags, umode_t mode, __u32 *oplock,
- struct cifs_fid *fid, struct cifs_open_info_data *buf)
+ struct cifs_fid *fid, struct cifs_open_info_data *buf,
+ bool *opened)
{
int rc = -ENOENT;
int create_options = CREATE_NOT_DIR;
@@ -191,6 +192,7 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned
struct cifs_open_parms oparms;
int rdwr_for_fscache = 0;
+ *opened = false;
*oplock = 0;
if (tcon->ses->server->oplocks)
*oplock = REQ_OPLOCK;
@@ -213,6 +215,7 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned
oflags, oplock, &fid->netfid, xid);
switch (rc) {
case 0:
+ *opened = true;
if (newinode == NULL) {
/* query inode info */
goto cifs_create_get_file_info;
@@ -229,11 +232,9 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned
/*
* The server may allow us to open things like
* FIFOs, but the client isn't set up to deal
- * with that. If it's not a regular file, just
- * close it and proceed as if it were a normal
- * lookup.
+ * with that. Keep the handle until the caller
+ * can finish the lookup.
*/
- CIFSSMBClose(xid, tcon, fid->netfid);
goto cifs_create_get_file_info;
}
/* success, no need to query */
@@ -333,6 +334,7 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned
}
goto out;
}
+ *opened = true;
if (rdwr_for_fscache == 2)
cifs_invalidate_cache(inode, FSCACHE_INVAL_DIO_WRITE);
@@ -423,7 +425,7 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned
return rc;
out_err:
- if (server->ops->close)
+ if (*opened && server->ops->close)
server->ops->close(xid, tcon, fid);
if (newinode)
iput(newinode);
@@ -442,6 +444,8 @@ cifs_atomic_open(struct inode *inode, struct dentry *direntry,
struct cifs_fid fid = {};
struct cifs_pending_open open;
__u32 oplock;
+ bool is_regular;
+ bool opened;
struct cifsFileInfo *file_info;
struct cifs_open_info_data buf = {};
@@ -501,15 +505,28 @@ cifs_atomic_open(struct inode *inode, struct dentry *direntry,
cifs_add_pending_open(&fid, tlink, &open);
rc = cifs_do_create(inode, direntry, xid, tlink, oflags, mode,
- &oplock, &fid, &buf);
+ &oplock, &fid, &buf, &opened);
if (rc) {
cifs_del_pending_open(&open);
goto out;
}
- if ((oflags & (O_CREAT | O_EXCL)) == (O_CREAT | O_EXCL))
+ is_regular = d_is_reg(direntry);
+ if (!is_regular || !opened) {
+ if (opened && server->ops->close)
+ server->ops->close(xid, tcon, &fid);
+ cifs_del_pending_open(&open);
+ }
+
+ if (is_regular && opened &&
+ (oflags & (O_CREAT | O_EXCL)) == (O_CREAT | O_EXCL))
file->f_mode |= FMODE_CREATED;
+ if (!is_regular || !opened) {
+ rc = finish_no_open(file, NULL);
+ goto out;
+ }
+
rc = finish_open(file, direntry, generic_file_open);
if (rc) {
if (server->ops->close)
@@ -564,6 +581,7 @@ int cifs_create(struct user_namespace *mnt_userns, struct inode *inode,
struct TCP_Server_Info *server;
struct cifs_fid fid;
__u32 oplock;
+ bool opened;
struct cifs_open_info_data buf = {};
cifs_dbg(FYI, "cifs_create parent inode = 0x%p name is: %pd and dentry = 0x%p\n",
@@ -585,8 +603,9 @@ int cifs_create(struct user_namespace *mnt_userns, struct inode *inode,
if (server->ops->new_lease_key)
server->ops->new_lease_key(&fid);
- rc = cifs_do_create(inode, direntry, xid, tlink, oflags, mode, &oplock, &fid, &buf);
- if (!rc && server->ops->close)
+ rc = cifs_do_create(inode, direntry, xid, tlink, oflags, mode,
+ &oplock, &fid, &buf, &opened);
+ if (!rc && opened && server->ops->close)
server->ops->close(xid, tcon, &fid);
cifs_free_open_info(&buf);
diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c
index 466d25c90a10..af2e918313a2 100644
--- a/fs/smb/client/file.c
+++ b/fs/smb/client/file.c
@@ -1001,11 +1001,21 @@ void smb2_deferred_work_close(struct work_struct *work)
{
struct cifsFileInfo *cfile = container_of(work,
struct cifsFileInfo, deferred.work);
+ struct cifsInodeInfo *cinode = CIFS_I(d_inode(cfile->dentry));
- spin_lock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
+ spin_lock(&cinode->deferred_lock);
cifs_del_deferred_close(cfile);
cfile->deferred_close_scheduled = false;
- spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
+ spin_unlock(&cinode->deferred_lock);
+ /*
+ * Refresh time_last_write immediately before the actual server close
+ * so the protection window is anchored to the real close time, not
+ * the earlier userspace close time stored by cifs_close().
+ */
+ if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cinode->time_last_write, jiffies);
+ }
_cifsFileInfo_put(cfile, true, false);
}
@@ -1021,6 +1031,10 @@ int cifs_close(struct inode *inode, struct file *file)
if (file->private_data != NULL) {
cfile = file->private_data;
file->private_data = NULL;
+ if (file->f_mode & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cinode->time_last_write, jiffies);
+ }
dclose = kmalloc(sizeof(struct cifs_deferred_close), GFP_KERNEL);
if ((cifs_sb->ctx->closetimeo && cinode->oplock == CIFS_CACHE_RHW_FLG)
&& cinode->lease_granted &&
@@ -1042,11 +1056,18 @@ int cifs_close(struct inode *inode, struct file *file)
cifsFileInfo_get(cfile);
} else {
/* Deferred close for files */
- queue_delayed_work(deferredclose_wq,
- &cfile->deferred, cifs_sb->ctx->closetimeo);
- cfile->deferred_close_scheduled = true;
- spin_unlock(&cinode->deferred_lock);
- return 0;
+ /*
+ * Each queued execution owns one reference.
+ * If nothing was queued, the reference of
+ * the closing file is dropped below.
+ */
+ if (queue_delayed_work(deferredclose_wq,
+ &cfile->deferred,
+ cifs_sb->ctx->closetimeo)) {
+ cfile->deferred_close_scheduled = true;
+ spin_unlock(&cinode->deferred_lock);
+ return 0;
+ }
}
spin_unlock(&cinode->deferred_lock);
_cifsFileInfo_put(cfile, true, false);
@@ -5173,8 +5194,6 @@ void cifs_oplock_break(struct work_struct *work)
struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo,
oplock_break);
struct inode *inode = d_inode(cfile->dentry);
- struct super_block *sb = inode->i_sb;
- struct cifs_sb_info *cifs_sb = CIFS_SB(sb);
struct cifsInodeInfo *cinode = CIFS_I(inode);
struct cifs_tcon *tcon;
struct TCP_Server_Info *server;
@@ -5187,8 +5206,8 @@ void cifs_oplock_break(struct work_struct *work)
wait_on_bit(&cinode->flags, CIFS_INODE_PENDING_WRITERS,
TASK_UNINTERRUPTIBLE);
- tlink = cifs_sb_tlink(cifs_sb);
- if (IS_ERR(tlink)) {
+ tlink = cifs_get_tlink(cfile->tlink);
+ if (IS_ERR_OR_NULL(tlink)) {
/* drop the reference taken when the break was queued */
_cifsFileInfo_put(cfile, false /* do not wait for ourself */, false);
goto out;
diff --git a/fs/smb/client/fs_context.c b/fs/smb/client/fs_context.c
index 35f2c94aafd1..bd19aab1f3c6 100644
--- a/fs/smb/client/fs_context.c
+++ b/fs/smb/client/fs_context.c
@@ -1167,11 +1167,13 @@ static int smb3_fs_context_parse_param(struct fs_context *fc,
cifs_errorf(fc, "Unknown error parsing devname\n");
goto cifs_parse_mount_err;
}
+ kfree(ctx->source);
ctx->source = kstrdup(param->string, GFP_KERNEL);
if (ctx->source == NULL) {
cifs_errorf(fc, "OOM when copying UNC string\n");
goto cifs_parse_mount_err;
}
+ kfree(fc->source);
fc->source = kstrdup(param->string, GFP_KERNEL);
if (fc->source == NULL) {
cifs_errorf(fc, "OOM when copying UNC string\n");
@@ -1509,6 +1511,10 @@ static int smb3_fs_context_parse_param(struct fs_context *fc,
cifs_parse_mount_err:
kfree_sensitive(ctx->password);
ctx->password = NULL;
+ kfree(ctx->source);
+ ctx->source = NULL;
+ kfree(fc->source);
+ fc->source = NULL;
return -EINVAL;
}
diff --git a/fs/smb/client/inode.c b/fs/smb/client/inode.c
index 51d38449ca85..f87a0a6d68c9 100644
--- a/fs/smb/client/inode.c
+++ b/fs/smb/client/inode.c
@@ -672,6 +672,7 @@ static void smb311_posix_info_to_fattr(struct cifs_fattr *fattr, struct cifs_ope
/* fattr->cf_rdev = le32_to_cpu(info->DeviceId); */
if (symlink) {
+ fattr->cf_mode &= ~S_IFMT;
fattr->cf_mode |= S_IFLNK;
fattr->cf_dtype = DT_LNK;
fattr->cf_symlink_target = data->symlink_target;
@@ -2668,13 +2669,17 @@ int cifs_truncate_page(struct address_space *mapping, loff_t from)
void cifs_setsize(struct inode *inode, loff_t offset)
{
struct cifsInodeInfo *cifs_i = CIFS_I(inode);
+ loff_t old_size;
spin_lock(&inode->i_lock);
+ old_size = i_size_read(inode);
i_size_write(inode, offset);
spin_unlock(&inode->i_lock);
/* Cached inode must be refreshed on truncate */
cifs_i->time = 0;
+ if (offset > old_size)
+ pagecache_isize_extended(inode, old_size, offset);
truncate_pagecache(inode, offset);
}
@@ -2826,9 +2831,13 @@ cifs_setattr_unix(struct dentry *direntry, struct iattr *attrs)
goto out;
}
- /* skip mode change if it's just for clearing setuid/setgid */
- if (attrs->ia_valid & (ATTR_KILL_SUID|ATTR_KILL_SGID))
- attrs->ia_valid &= ~ATTR_MODE;
+ /*
+ * This function is only called when Unix extensions are in effect,
+ * so the mode is always sent to and stored on the server. Do not
+ * skip the mode change when clearing setuid/setgid bits: dropping
+ * ATTR_MODE here would leave those bits set on the server after a
+ * write, which is a security issue.
+ */
args = kmalloc(sizeof(*args), GFP_KERNEL);
if (args == NULL) {
@@ -2893,6 +2902,8 @@ cifs_setattr_unix(struct dentry *direntry, struct iattr *attrs)
if ((attrs->ia_valid & ATTR_SIZE) &&
attrs->ia_size != i_size_read(inode)) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cifsInode->time_last_write, jiffies);
truncate_setsize(inode, attrs->ia_size);
fscache_resize_cookie(cifs_inode_cookie(inode), attrs->ia_size);
}
@@ -2999,7 +3010,8 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs)
gid = attrs->ia_gid;
if ((cifs_sb->mnt_cifs_flags & CIFS_MOUNT_CIFS_ACL) ||
- (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MODE_FROM_SID)) {
+ (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MODE_FROM_SID) ||
+ cifs_sb_master_tcon(cifs_sb)->posix_extensions) {
if (uid_valid(uid) || gid_valid(gid)) {
mode = NO_CHANGE_64;
rc = id_mode_to_cifs_acl(inode, full_path, &mode,
@@ -3014,8 +3026,22 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs)
if (!(cifs_sb->mnt_cifs_flags & CIFS_MOUNT_SET_UID))
attrs->ia_valid &= ~(ATTR_UID | ATTR_GID);
- /* skip mode change if it's just for clearing setuid/setgid */
- if (attrs->ia_valid & (ATTR_KILL_SUID|ATTR_KILL_SGID))
+ /*
+ * Skip the mode change if it is only being done to clear the
+ * setuid/setgid bits *and* the mode is emulated via the DOS
+ * read-only attribute (the default, non-ACL case), which cannot
+ * represent the setuid/setgid bits anyway.
+ *
+ * When the mode is instead stored on the server - i.e. with the
+ * cifsacl or modefromsid mount options (via an ACL) - the cleared
+ * mode must be pushed to the server. Dropping ATTR_MODE here
+ * would leave the setuid/setgid bit set on the server after a
+ * write, which is a security issue (the bits are not stripped as
+ * they are on local filesystems).
+ */
+ if ((attrs->ia_valid & (ATTR_KILL_SUID|ATTR_KILL_SGID)) &&
+ !(cifs_sb->mnt_cifs_flags &
+ (CIFS_MOUNT_CIFS_ACL | CIFS_MOUNT_MODE_FROM_SID)))
attrs->ia_valid &= ~ATTR_MODE;
if (attrs->ia_valid & ATTR_MODE) {
@@ -3092,6 +3118,8 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs)
if ((attrs->ia_valid & ATTR_SIZE) &&
attrs->ia_size != i_size_read(inode)) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&cifsInode->time_last_write, jiffies);
truncate_setsize(inode, attrs->ia_size);
fscache_resize_cookie(cifs_inode_cookie(inode), attrs->ia_size);
}
diff --git a/fs/smb/client/link.c b/fs/smb/client/link.c
index d71feb3fdbd2..cf90ef040471 100644
--- a/fs/smb/client/link.c
+++ b/fs/smb/client/link.c
@@ -355,7 +355,7 @@ smb3_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon,
int buf_type = CIFS_NO_BUFFER;
__le16 *utf16_path;
__u8 oplock = SMB2_OPLOCK_LEVEL_NONE;
- struct smb2_file_all_info *pfile_info = NULL;
+ struct cifs_open_info_data data = {};
oparms = (struct cifs_open_parms) {
.tcon = tcon,
@@ -371,20 +371,12 @@ smb3_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon,
if (utf16_path == NULL)
return -ENOMEM;
- pfile_info = kzalloc(sizeof(struct smb2_file_all_info) + PATH_MAX * 2,
- GFP_KERNEL);
-
- if (pfile_info == NULL) {
- kfree(utf16_path);
- return -ENOMEM;
- }
-
- rc = SMB2_open(xid, &oparms, utf16_path, &oplock, pfile_info, NULL,
+ rc = SMB2_open(xid, &oparms, utf16_path, &oplock, &data, NULL,
NULL, NULL);
if (rc)
goto qmf_out_open_fail;
- if (pfile_info->EndOfFile != cpu_to_le64(CIFS_MF_SYMLINK_FILE_SIZE)) {
+ if (data.fi.EndOfFile != cpu_to_le64(CIFS_MF_SYMLINK_FILE_SIZE)) {
/* it's not a symlink */
rc = -ENOENT; /* Is there a better rc to return? */
goto qmf_out;
@@ -402,7 +394,6 @@ smb3_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon,
SMB2_close(xid, tcon, fid.persistent_fid, fid.volatile_fid);
qmf_out_open_fail:
kfree(utf16_path);
- kfree(pfile_info);
return rc;
}
diff --git a/fs/smb/client/misc.c b/fs/smb/client/misc.c
index fc6fe4c38634..fb841235617c 100644
--- a/fs/smb/client/misc.c
+++ b/fs/smb/client/misc.c
@@ -757,7 +757,7 @@ cifs_del_deferred_close(struct cifsFileInfo *cfile)
void
cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
{
- struct cifsFileInfo *cfile = NULL;
+ struct cifsFileInfo *cfile = NULL, *failed_cfile = NULL;
struct file_list *tmp_list, *tmp_next_list;
struct list_head file_head;
@@ -774,8 +774,10 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
spin_unlock(&cifs_inode->deferred_lock);
tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
@@ -783,8 +785,24 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
}
spin_unlock(&cifs_inode->open_file_lock);
+ if (failed_cfile) {
+ if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(failed_cfile, false, false);
+ }
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
- _cifsFileInfo_put(tmp_list->cfile, false, false);
+ struct cifsFileInfo *cfile = tmp_list->cfile;
+
+ if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(cfile, false, false);
list_del(&tmp_list->list);
kfree(tmp_list);
}
@@ -793,7 +811,7 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode)
void
cifs_close_all_deferred_files(struct cifs_tcon *tcon)
{
- struct cifsFileInfo *cfile;
+ struct cifsFileInfo *cfile, *failed_cfile = NULL;
struct file_list *tmp_list, *tmp_next_list;
struct list_head file_head;
@@ -807,8 +825,10 @@ cifs_close_all_deferred_files(struct cifs_tcon *tcon)
spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
@@ -816,8 +836,24 @@ cifs_close_all_deferred_files(struct cifs_tcon *tcon)
}
spin_unlock(&tcon->open_file_lock);
+ if (failed_cfile) {
+ if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(failed_cfile, true, false);
+ }
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
- _cifsFileInfo_put(tmp_list->cfile, true, false);
+ struct cifsFileInfo *cfile = tmp_list->cfile;
+
+ if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(cfile, true, false);
list_del(&tmp_list->list);
kfree(tmp_list);
}
@@ -861,7 +897,7 @@ void cifs_close_all_deferred_files_sb(struct cifs_sb_info *cifs_sb)
void
cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon, const char *path)
{
- struct cifsFileInfo *cfile;
+ struct cifsFileInfo *cfile, *failed_cfile = NULL;
struct file_list *tmp_list, *tmp_next_list;
struct list_head file_head;
void *page;
@@ -880,8 +916,10 @@ cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon, const char *path)
spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock);
tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC);
- if (tmp_list == NULL)
+ if (tmp_list == NULL) {
+ failed_cfile = cfile;
break;
+ }
tmp_list->cfile = cfile;
list_add_tail(&tmp_list->list, &file_head);
}
@@ -890,8 +928,24 @@ cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon, const char *path)
}
spin_unlock(&tcon->open_file_lock);
+ if (failed_cfile) {
+ if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(failed_cfile, true, false);
+ }
+
list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) {
- _cifsFileInfo_put(tmp_list->cfile, true, false);
+ struct cifsFileInfo *cfile = tmp_list->cfile;
+
+ if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) {
+ /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */
+ smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write,
+ jiffies);
+ }
+ _cifsFileInfo_put(cfile, true, false);
list_del(&tmp_list->list);
kfree(tmp_list);
}
diff --git a/fs/smb/client/readdir.c b/fs/smb/client/readdir.c
index 7564927e2c42..1ba176a690cc 100644
--- a/fs/smb/client/readdir.c
+++ b/fs/smb/client/readdir.c
@@ -233,9 +233,11 @@ static void
cifs_posix_to_fattr(struct cifs_fattr *fattr, struct smb2_posix_info *info,
struct cifs_sb_info *cifs_sb)
{
+ unsigned int sbflags = cifs_sb->mnt_cifs_flags;
struct smb2_posix_info_parsed parsed;
+ int rc;
- posix_info_parse(info, NULL, &parsed);
+ rc = posix_info_parse(info, NULL, &parsed);
memset(fattr, 0, sizeof(*fattr));
fattr->cf_uniqueid = le64_to_cpu(info->Inode);
@@ -277,8 +279,17 @@ cifs_posix_to_fattr(struct cifs_fattr *fattr, struct smb2_posix_info *info,
if (reparse_file_needs_reval(fattr))
fattr->cf_flags |= CIFS_FATTR_NEED_REVAL;
- sid_to_id(cifs_sb, &parsed.owner, fattr, SIDOWNER);
- sid_to_id(cifs_sb, &parsed.group, fattr, SIDGROUP);
+ fattr->cf_uid = cifs_sb->ctx->linux_uid;
+ fattr->cf_gid = cifs_sb->ctx->linux_gid;
+ if (rc < 0) {
+ cifs_dbg(VFS, "%s: failed to parse SIDs: %d\n",
+ __func__, rc);
+ } else {
+ if (!(sbflags & CIFS_MOUNT_OVERR_UID))
+ sid_to_id(cifs_sb, &parsed.owner, fattr, SIDOWNER);
+ if (!(sbflags & CIFS_MOUNT_OVERR_GID))
+ sid_to_id(cifs_sb, &parsed.group, fattr, SIDGROUP);
+ }
}
static void __dir_info_to_fattr(struct cifs_fattr *fattr, const void *info)
diff --git a/fs/smb/client/sess.c b/fs/smb/client/sess.c
index ffeb3e7d34d5..1f1043e47c5c 100644
--- a/fs/smb/client/sess.c
+++ b/fs/smb/client/sess.c
@@ -163,9 +163,9 @@ int cifs_try_adding_channels(struct cifs_sb_info *cifs_sb, struct cifs_ses *ses)
int old_chan_count, new_chan_count;
int left;
int rc = 0;
- int tries = 0;
+ int tries = 0, attempts;
size_t iface_weight = 0, iface_min_speed = 0;
- struct cifs_server_iface *iface = NULL, *niface = NULL;
+ struct cifs_server_iface *iface = NULL, *candidate = NULL;
struct cifs_server_iface *last_iface = NULL;
spin_lock(&ses->chan_lock);
@@ -211,67 +211,89 @@ int cifs_try_adding_channels(struct cifs_sb_info *cifs_sb, struct cifs_ses *ses)
break;
}
- if (!iface)
- iface = list_first_entry(&ses->iface_list, struct cifs_server_iface,
- iface_head);
last_iface = list_last_entry(&ses->iface_list, struct cifs_server_iface,
iface_head);
iface_min_speed = last_iface->speed;
+ spin_unlock(&ses->iface_lock);
- list_for_each_entry_safe_from(iface, niface, &ses->iface_list,
- iface_head) {
- /* do not mix rdma and non-rdma interfaces */
- if (iface->rdma_capable != ses->server->rdma)
- continue;
-
- /* skip ifaces that are unusable */
- if (!iface->is_active ||
- (is_ses_using_iface(ses, iface) &&
- !iface->rss_capable))
- continue;
+ attempts = 0;
+ while (left > 0) {
+ spin_lock(&ses->iface_lock);
- /* check if we already allocated enough channels */
- iface_weight = iface->speed / iface_min_speed;
+ /*
+ * iface_lock must be dropped while opening a channel,
+ * and a concurrent interface refresh may remove and
+ * free entries during that window, so no list entry
+ * may be kept across it without a reference. Scan
+ * the list from the beginning each time and only pass
+ * a referenced candidate to cifs_ses_add_channel();
+ * weight_fulfilled tracks the progress so that no
+ * iface is selected beyond its weight.
+ */
+ candidate = NULL;
+ list_for_each_entry(iface, &ses->iface_list, iface_head) {
+ /* do not mix rdma and non-rdma interfaces */
+ if (iface->rdma_capable != ses->server->rdma)
+ continue;
+
+ /* skip ifaces that are unusable */
+ if (!iface->is_active ||
+ (is_ses_using_iface(ses, iface) &&
+ !iface->rss_capable))
+ continue;
+
+ /* check if we already allocated enough channels */
+ iface_weight = iface->speed / iface_min_speed;
+
+ if (iface->weight_fulfilled >= iface_weight)
+ continue;
+
+ /* take ref before unlock */
+ kref_get(&iface->refcount);
+ candidate = iface;
+ break;
+ }
- if (iface->weight_fulfilled >= iface_weight)
- continue;
+ if (!candidate) {
+ /* no usable iface. reset weight_fulfilled and start over */
+ list_for_each_entry(iface, &ses->iface_list, iface_head)
+ iface->weight_fulfilled = 0;
+ spin_unlock(&ses->iface_lock);
+ break;
+ }
- /* take ref before unlock */
- kref_get(&iface->refcount);
+ attempts++;
+ if (attempts > 3 * ses->chan_max) {
+ kref_put(&candidate->refcount, release_iface);
+ spin_unlock(&ses->iface_lock);
+ break;
+ }
spin_unlock(&ses->iface_lock);
- rc = cifs_ses_add_channel(cifs_sb, ses, iface);
+ rc = cifs_ses_add_channel(cifs_sb, ses, candidate);
spin_lock(&ses->iface_lock);
if (rc) {
cifs_dbg(VFS, "failed to open extra channel on iface:%pIS rc=%d\n",
- &iface->sockaddr,
+ &candidate->sockaddr,
rc);
/* failure to add chan should increase weight */
- iface->weight_fulfilled++;
- kref_put(&iface->refcount, release_iface);
+ candidate->weight_fulfilled++;
+ kref_put(&candidate->refcount, release_iface);
+ spin_unlock(&ses->iface_lock);
continue;
}
- iface->num_channels++;
- iface->weight_fulfilled++;
+ candidate->num_channels++;
+ candidate->weight_fulfilled++;
cifs_dbg(VFS, "successfully opened new channel on iface:%pIS\n",
- &iface->sockaddr);
- break;
- }
-
- /* reached end of list. reset weight_fulfilled and start over */
- if (list_entry_is_head(iface, &ses->iface_list, iface_head)) {
- list_for_each_entry(iface, &ses->iface_list, iface_head)
- iface->weight_fulfilled = 0;
+ &candidate->sockaddr);
spin_unlock(&ses->iface_lock);
- iface = NULL;
- continue;
- }
- spin_unlock(&ses->iface_lock);
- left--;
- new_chan_count++;
+ left--;
+ new_chan_count++;
+ break;
+ }
}
return new_chan_count - old_chan_count;
diff --git a/fs/smb/client/smb1ops.c b/fs/smb/client/smb1ops.c
index 1489b9d21b60..5c570cd23884 100644
--- a/fs/smb/client/smb1ops.c
+++ b/fs/smb/client/smb1ops.c
@@ -325,12 +325,31 @@ coalesce_t2(char *second_buf, struct smb_hdr *target_hdr)
data_area_of_tgt = (char *)&pSMBt->hdr.Protocol +
get_unaligned_le16(&pSMBt->t2_rsp.DataOffset);
- /* validate target area */
data_area_of_src = (char *)&pSMBs->hdr.Protocol +
get_unaligned_le16(&pSMBs->t2_rsp.DataOffset);
data_area_of_tgt += total_in_tgt;
+ /*
+ * DataOffset fields are server-supplied and not validated against
+ * buffer bounds; check both data pointers before mutating the
+ * target header.
+ */
+ if (data_area_of_tgt < (char *)target_hdr +
+ sizeof(struct smb_t2_rsp) + sizeof(__le16) ||
+ data_area_of_tgt + total_in_src >
+ (char *)target_hdr + CIFSMaxBufSize + MAX_CIFS_HDR_SIZE) {
+ cifs_dbg(VFS, "%s: target data area out of bounds\n", __func__);
+ return -EPROTO;
+ }
+ if (data_area_of_src < second_buf +
+ sizeof(struct smb_t2_rsp) + sizeof(__le16) ||
+ data_area_of_src + total_in_src >
+ second_buf + smbCalcSize((struct smb_hdr *)second_buf)) {
+ cifs_dbg(VFS, "%s: secondary data area out of bounds\n", __func__);
+ return -EPROTO;
+ }
+
total_in_tgt += total_in_src;
/* is the result too big for the field? */
if (total_in_tgt > USHRT_MAX) {
diff --git a/fs/smb/client/smb2file.c b/fs/smb/client/smb2file.c
index 2d0dd87dae0d..e3da8107bda4 100644
--- a/fs/smb/client/smb2file.c
+++ b/fs/smb/client/smb2file.c
@@ -112,8 +112,6 @@ int smb2_open_file(const unsigned int xid, struct cifs_open_parms *oparms, __u32
__le16 *smb2_path;
__u8 smb2_oplock;
struct cifs_open_info_data *data = buf;
- struct smb2_file_all_info file_info = {};
- struct smb2_file_all_info *smb2_data = data ? &file_info : NULL;
struct kvec err_iov = {};
int err_buftype = CIFS_NO_BUFFER;
struct cifs_fid *fid = oparms->fid;
@@ -130,14 +128,14 @@ int smb2_open_file(const unsigned int xid, struct cifs_open_parms *oparms, __u32
}
smb2_oplock = SMB2_OPLOCK_LEVEL_BATCH;
- rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data, NULL, &err_iov,
+ rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data, NULL, &err_iov,
&err_buftype);
if (rc == -EACCES && retry_without_read_attributes) {
free_rsp_buf(err_buftype, err_iov.iov_base);
memset(&err_iov, 0, sizeof(err_iov));
err_buftype = CIFS_NO_BUFFER;
oparms->desired_access &= ~FILE_READ_ATTRIBUTES;
- rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data, NULL, &err_iov,
+ rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data, NULL, &err_iov,
&err_buftype);
}
if (rc && data) {
@@ -149,9 +147,9 @@ int smb2_open_file(const unsigned int xid, struct cifs_open_parms *oparms, __u32
rc = smb2_parse_symlink_response(oparms->cifs_sb, &err_iov,
&data->symlink_target);
if (!rc) {
- memset(smb2_data, 0, sizeof(*smb2_data));
+ memset(&data->fi, 0, sizeof(data->fi));
oparms->create_options |= OPEN_REPARSE_POINT;
- rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data,
+ rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data,
NULL, NULL, NULL);
oparms->create_options &= ~OPEN_REPARSE_POINT;
}
@@ -180,23 +178,22 @@ int smb2_open_file(const unsigned int xid, struct cifs_open_parms *oparms, __u32
rc = 0;
}
- if (smb2_data) {
+ if (data) {
/* if open response does not have IndexNumber field - get it */
- if (smb2_data->IndexNumber == 0) {
+ if (data->fi.IndexNumber == 0) {
rc = SMB2_get_srv_num(xid, oparms->tcon,
fid->persistent_fid,
fid->volatile_fid,
- &smb2_data->IndexNumber);
+ &data->fi.IndexNumber);
if (rc) {
/*
* let get_inode_info disable server inode
* numbers
*/
- smb2_data->IndexNumber = 0;
+ data->fi.IndexNumber = 0;
rc = 0;
}
}
- memcpy(&data->fi, smb2_data, sizeof(data->fi));
}
*oplock = smb2_oplock;
diff --git a/fs/smb/client/smb2misc.c b/fs/smb/client/smb2misc.c
index cd00bf48c9a5..27d0fc56cd82 100644
--- a/fs/smb/client/smb2misc.c
+++ b/fs/smb/client/smb2misc.c
@@ -818,7 +818,8 @@ smb2_cancelled_close_fid(struct work_struct *work)
*/
static int
__smb2_handle_cancelled_cmd(struct cifs_tcon *tcon, __u16 cmd, __u64 mid,
- __u64 persistent_fid, __u64 volatile_fid)
+ __u64 persistent_fid, __u64 volatile_fid,
+ bool account_remote_open)
{
struct close_cancelled_open *cancelled;
@@ -832,6 +833,8 @@ __smb2_handle_cancelled_cmd(struct cifs_tcon *tcon, __u16 cmd, __u64 mid,
cancelled->cmd = cmd;
cancelled->mid = mid;
INIT_WORK(&cancelled->work, smb2_cancelled_close_fid);
+ if (account_remote_open)
+ atomic_inc(&tcon->num_remote_opens);
WARN_ON(queue_work(cifsiod_wq, &cancelled->work) == false);
return 0;
@@ -844,14 +847,14 @@ smb2_handle_cancelled_close(struct cifs_tcon *tcon, __u64 persistent_fid,
int rc;
cifs_dbg(FYI, "%s: tc_count=%d\n", __func__, tcon->tc_count);
- spin_lock(&cifs_tcp_ses_lock);
+ spin_lock(&tcon->tc_lock);
if (tcon->tc_count <= 0) {
struct TCP_Server_Info *server = NULL;
trace_smb3_tcon_ref(tcon->debug_id, tcon->tc_count,
netfs_trace_tcon_ref_see_cancelled_close);
WARN_ONCE(tcon->tc_count < 0, "tcon refcount is negative");
- spin_unlock(&cifs_tcp_ses_lock);
+ spin_unlock(&tcon->tc_lock);
if (tcon->ses) {
server = tcon->ses->server;
@@ -865,10 +868,10 @@ smb2_handle_cancelled_close(struct cifs_tcon *tcon, __u64 persistent_fid,
tcon->tc_count++;
trace_smb3_tcon_ref(tcon->debug_id, tcon->tc_count,
netfs_trace_tcon_ref_get_cancelled_close);
- spin_unlock(&cifs_tcp_ses_lock);
+ spin_unlock(&tcon->tc_lock);
rc = __smb2_handle_cancelled_cmd(tcon, SMB2_CLOSE_HE, 0,
- persistent_fid, volatile_fid);
+ persistent_fid, volatile_fid, false);
if (rc)
cifs_put_tcon(tcon, netfs_trace_tcon_ref_put_cancelled_close);
@@ -896,7 +899,7 @@ smb2_handle_cancelled_mid(struct mid_q_entry *mid, struct TCP_Server_Info *serve
le16_to_cpu(hdr->Command),
le64_to_cpu(hdr->MessageId),
rsp->PersistentFileId,
- rsp->VolatileFileId);
+ rsp->VolatileFileId, true);
if (rc)
cifs_put_tcon(tcon, netfs_trace_tcon_ref_put_cancelled_mid);
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index 3de0f2f385b5..5a07d411a00a 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -590,13 +590,6 @@ parse_server_interfaces(struct network_interface_info_ioctl_rsp *buf,
p = buf;
spin_lock(&ses->iface_lock);
- /* do not query too frequently, this time with lock held */
- if (ses->iface_last_update &&
- time_before(jiffies, ses->iface_last_update +
- (SMB_INTERFACE_POLL_INTERVAL * HZ))) {
- spin_unlock(&ses->iface_lock);
- return 0;
- }
/*
* Go through iface_list and mark them as inactive
@@ -727,7 +720,6 @@ parse_server_interfaces(struct network_interface_info_ioctl_rsp *buf,
ses->iface_count++;
spin_unlock(&ses->iface_lock);
- ses->iface_last_update = jiffies;
next_iface:
nb_iface++;
next = le32_to_cpu(p->Next);
@@ -792,10 +784,17 @@ SMB3_request_interfaces(const unsigned int xid, struct cifs_tcon *tcon, bool in_
struct TCP_Server_Info *pserver;
/* do not query too frequently */
+ spin_lock(&ses->iface_lock);
if (ses->iface_last_update &&
time_before(jiffies, ses->iface_last_update +
- (SMB_INTERFACE_POLL_INTERVAL * HZ)))
+ (SMB_INTERFACE_POLL_INTERVAL * HZ))) {
+ spin_unlock(&ses->iface_lock);
return 0;
+ }
+
+ ses->iface_last_update = jiffies;
+
+ spin_unlock(&ses->iface_lock);
rc = SMB2_ioctl(xid, tcon, NO_FILE_ID, NO_FILE_ID,
FSCTL_QUERY_NETWORK_INTERFACE_INFO,
@@ -1005,8 +1004,9 @@ move_smb2_ea_to_cifs(char *dst, size_t dst_size,
char *name, *value;
size_t buf_size = dst_size;
size_t name_len, value_len, user_name_len;
+ u32 next_off;
- while (src_size > 0) {
+ while (src_size >= sizeof(*src)) {
name_len = (size_t)src->ea_name_length;
value_len = (size_t)le16_to_cpu(src->ea_value_length);
@@ -1061,14 +1061,21 @@ move_smb2_ea_to_cifs(char *dst, size_t dst_size,
if (!src->next_entry_offset)
break;
- if (src_size < le32_to_cpu(src->next_entry_offset)) {
- /* stop before overrun buffer */
- rc = -ERANGE;
- break;
+ next_off = le32_to_cpu(src->next_entry_offset);
+ if (next_off < sizeof(*src) || src_size < next_off) {
+ cifs_dbg(FYI, "EA next_entry_offset %u out of range [%zu, %zu]\n",
+ next_off, sizeof(*src), src_size);
+ rc = -EIO;
+ goto out;
+ }
+ src_size -= next_off;
+ src = (void *)((char *)src + next_off);
+ if (src_size > 0 && src_size < sizeof(*src)) {
+ cifs_dbg(FYI, "EA next_entry_offset %u left truncated entry (%zu bytes)\n",
+ next_off, src_size);
+ rc = -EIO;
+ goto out;
}
- src_size -= le32_to_cpu(src->next_entry_offset);
- src = (void *)((char *)src +
- le32_to_cpu(src->next_entry_offset));
}
/* didn't find the named attribute */
@@ -2020,7 +2027,7 @@ smb2_duplicate_extents(const unsigned int xid,
src_off, dest_off, len);
inode = d_inode(trgtfile->dentry);
- if (inode->i_size < dest_off + len) {
+ if (i_size_read(inode) < dest_off + len) {
rc = smb2_set_file_size(xid, tcon, trgtfile, dest_off + len, false);
if (rc)
goto duplicate_extents_out;
@@ -2045,6 +2052,11 @@ smb2_duplicate_extents(const unsigned int xid,
if (ret_data_len > 0)
cifs_dbg(FYI, "Non-zero response length in duplicate extents\n");
+ if (rc) {
+ CIFS_I(inode)->time = 0; /* force reval */
+ cifs_invalidate_cache(inode, 0);
+ }
+
duplicate_extents_out:
return rc;
}
@@ -2855,7 +2867,9 @@ smb2_get_dfs_refer(const unsigned int xid, struct cifs_ses *ses,
struct cifs_tcon,
tcon_list);
if (tcon) {
+ spin_lock(&tcon->tc_lock);
tcon->tc_count++;
+ spin_unlock(&tcon->tc_lock);
trace_smb3_tcon_ref(tcon->debug_id, tcon->tc_count,
netfs_trace_tcon_ref_get_dfs_refer);
}
@@ -2920,13 +2934,7 @@ smb2_get_dfs_refer(const unsigned int xid, struct cifs_ses *ses,
out:
if (tcon && !tcon->ipc) {
/* ipc tcons are not refcounted */
- spin_lock(&cifs_tcp_ses_lock);
- tcon->tc_count--;
- trace_smb3_tcon_ref(tcon->debug_id, tcon->tc_count,
- netfs_trace_tcon_ref_dec_dfs_refer);
- /* tc_count can never go negative */
- WARN_ON(tcon->tc_count < 0);
- spin_unlock(&cifs_tcp_ses_lock);
+ cifs_put_tcon(tcon, netfs_trace_tcon_ref_put_dfs_refer);
}
kfree(utf16_path);
kfree(dfs_req);
@@ -3534,8 +3542,11 @@ static long smb3_zero_range(struct file *file, struct cifs_tcon *tcon,
/*
* We zero the range through ioctl, so we need remove the page caches
* first, otherwise the data may be inconsistent with the server.
+ *
+ * Start at the old EOF when extending so the folio straddling it, which
+ * may hold data written past EOF through an mmap, is dropped too.
*/
- truncate_pagecache_range(inode, offset, offset + len - 1);
+ truncate_pagecache_range(inode, min(offset, i_size), offset + len - 1);
/* if file not oplocked can't be sure whether asking to extend size */
rc = -EOPNOTSUPP;
@@ -3593,6 +3604,15 @@ static long smb3_punch_hole(struct file *file, struct cifs_tcon *tcon,
goto out;
filemap_invalidate_lock(inode->i_mapping);
+ /*
+ * Flush dirty data first, otherwise a dirty folio spanning the punched
+ * range may be written back after the ioctl and refill the hole.
+ */
+ rc = filemap_write_and_wait_range(inode->i_mapping, offset,
+ offset + len - 1);
+ if (rc < 0)
+ goto unlock;
+
/*
* We implement the punch hole through ioctl, so we need remove the page
* caches first, otherwise the data may be inconsistent with the server.
@@ -3609,6 +3629,7 @@ static long smb3_punch_hole(struct file *file, struct cifs_tcon *tcon,
(char *)&fsctl_buf,
sizeof(struct file_zero_data_information),
CIFSMaxBufSize, NULL, NULL);
+unlock:
filemap_invalidate_unlock(inode->i_mapping);
out:
inode_unlock(inode);
@@ -4383,25 +4404,37 @@ smb3_create_lease_buf(u8 *lease_key, u8 oplock)
static __u8
smb2_parse_lease_buf(void *buf, __u16 *epoch, char *lease_key)
{
- struct create_lease *lc = (struct create_lease *)buf;
+ struct create_context *cc = buf;
+ struct lease_context lc;
*epoch = 0; /* not used */
- if (lc->lcontext.LeaseFlags & SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE)
+ if (le32_to_cpu(cc->DataLength) != sizeof(lc))
+ return 0;
+
+ memcpy(&lc, (u8 *)cc + le16_to_cpu(cc->DataOffset), sizeof(lc));
+ if (lc.LeaseFlags & SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE)
return SMB2_OPLOCK_LEVEL_NOCHANGE;
- return le32_to_cpu(lc->lcontext.LeaseState);
+ return le32_to_cpu(lc.LeaseState);
}
static __u8
smb3_parse_lease_buf(void *buf, __u16 *epoch, char *lease_key)
{
- struct create_lease_v2 *lc = (struct create_lease_v2 *)buf;
+ struct create_context *cc = buf;
+ struct lease_context_v2 lc;
- *epoch = le16_to_cpu(lc->lcontext.Epoch);
- if (lc->lcontext.LeaseFlags & SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE)
+ if (le32_to_cpu(cc->DataLength) != sizeof(lc)) {
+ *epoch = 0;
+ return 0;
+ }
+
+ memcpy(&lc, (u8 *)cc + le16_to_cpu(cc->DataOffset), sizeof(lc));
+ *epoch = le16_to_cpu(lc.Epoch);
+ if (lc.LeaseFlags & SMB2_LEASE_FLAG_BREAK_IN_PROGRESS_LE)
return SMB2_OPLOCK_LEVEL_NOCHANGE;
if (lease_key)
- memcpy(lease_key, &lc->lcontext.LeaseKey, SMB2_LEASE_KEY_SIZE);
- return le32_to_cpu(lc->lcontext.LeaseState);
+ memcpy(lease_key, lc.LeaseKey, SMB2_LEASE_KEY_SIZE);
+ return le32_to_cpu(lc.LeaseState);
}
static unsigned int
@@ -4758,9 +4791,9 @@ decrypt_raw_data(struct TCP_Server_Info *server, char *buf,
return rc;
}
} else {
- if (unlikely(!server->secmech.dec))
- return -EIO;
-
+ rc = smb3_crypto_aead_allocate(server);
+ if (unlikely(rc))
+ return rc;
tfm = server->secmech.dec;
}
diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c
index e23060ed57da..e69b4bb9dc91 100644
--- a/fs/smb/client/smb2pdu.c
+++ b/fs/smb/client/smb2pdu.c
@@ -2179,11 +2179,17 @@ create_reconnect_durable_buf(struct cifs_fid *fid)
static void
parse_query_id_ctxt(struct create_context *cc, struct smb2_file_all_info *buf)
{
- struct create_on_disk_id *pdisk_id = (struct create_on_disk_id *)cc;
+ u16 doff = le16_to_cpu(cc->DataOffset);
+ u32 dlen = le32_to_cpu(cc->DataLength);
+ u8 *beg;
+
+ if (dlen < sizeof(__le64))
+ return;
- cifs_dbg(FYI, "parse query id context 0x%llx 0x%llx\n",
- pdisk_id->DiskFileId, pdisk_id->VolumeId);
- buf->IndexNumber = pdisk_id->DiskFileId;
+ beg = (u8 *)cc + doff;
+ memcpy(&buf->IndexNumber, beg, sizeof(__le64));
+ cifs_dbg(FYI, "parse query id context 0x%llx\n",
+ le64_to_cpu(buf->IndexNumber));
}
static void
@@ -2234,6 +2240,7 @@ int smb2_parse_contexts(struct TCP_Server_Info *server,
struct smb2_create_rsp *rsp = rsp_iov->iov_base;
struct create_context *cc;
size_t rem, off, len;
+ size_t cc_len;
size_t doff, dlen;
size_t noff, nlen;
char *name;
@@ -2256,29 +2263,41 @@ int smb2_parse_contexts(struct TCP_Server_Info *server,
buf->IndexNumber = 0;
while (rem >= sizeof(*cc)) {
+ off = le32_to_cpu(cc->Next);
+ if (off) {
+ if ((off & 0x7) || off >= rem || off < sizeof(*cc))
+ return -EINVAL;
+ cc_len = off;
+ } else {
+ cc_len = rem;
+ }
+
doff = le16_to_cpu(cc->DataOffset);
dlen = le32_to_cpu(cc->DataLength);
- if (check_add_overflow(doff, dlen, &len) || len > rem)
+ if (doff < sizeof(*cc) ||
+ check_add_overflow(doff, dlen, &len) || len > cc_len)
return -EINVAL;
noff = le16_to_cpu(cc->NameOffset);
nlen = le16_to_cpu(cc->NameLength);
- if (noff + nlen > doff)
+ if (noff < sizeof(*cc) ||
+ check_add_overflow(noff, nlen, &len) || len > cc_len ||
+ (dlen && len > doff))
return -EINVAL;
name = (char *)cc + noff;
switch (nlen) {
case 4:
- if (!strncmp(name, SMB2_CREATE_REQUEST_LEASE, 4)) {
+ if (dlen && !strncmp(name, SMB2_CREATE_REQUEST_LEASE, 4)) {
*oplock = server->ops->parse_lease_buf(cc, epoch,
lease_key);
- } else if (buf &&
+ } else if (dlen && buf &&
!strncmp(name, SMB2_CREATE_QUERY_ON_DISK_ID, 4)) {
parse_query_id_ctxt(cc, buf);
}
break;
case 16:
- if (posix && !memcmp(name, smb3_create_tag_posix, 16))
+ if (dlen && posix && !memcmp(name, smb3_create_tag_posix, 16))
parse_posix_ctxt(cc, buf, posix);
break;
default:
@@ -2290,13 +2309,18 @@ int smb2_parse_contexts(struct TCP_Server_Info *server,
}
off = le32_to_cpu(cc->Next);
- if (!off)
+ if (!off) {
+ rem = 0;
break;
+ }
if (check_sub_overflow(rem, off, &rem))
return -EINVAL;
cc = (struct create_context *)((u8 *)cc + off);
}
+ if (rem)
+ return -EINVAL;
+
if (rsp->OplockLevel != SMB2_OPLOCK_LEVEL_LEASE)
*oplock = rsp->OplockLevel;
@@ -3097,7 +3121,7 @@ SMB2_open_free(struct smb_rqst *rqst)
int
SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path,
- __u8 *oplock, struct smb2_file_all_info *buf,
+ __u8 *oplock, struct cifs_open_info_data *buf,
struct create_posix_rsp *posix,
struct kvec *err_iov, int *buftype)
{
@@ -3111,6 +3135,7 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path,
int resp_buftype = CIFS_NO_BUFFER;
int rc = 0;
int flags = 0;
+ struct smb2_file_all_info *file_info = buf ? &buf->fi : NULL;
cifs_dbg(FYI, "create/open\n");
if (!ses || !server)
@@ -3155,9 +3180,6 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path,
goto creat_exit;
} else if (rsp == NULL) /* unlikely to happen, but safer to check */
goto creat_exit;
- else
- trace_smb3_open_done(xid, rsp->PersistentFileId, tcon->tid, ses->Suid,
- oparms->create_options, oparms->desired_access);
atomic_inc(&tcon->num_remote_opens);
oparms->fid->persistent_fid = rsp->PersistentFileId;
@@ -3167,21 +3189,27 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path,
oparms->fid->mid = le64_to_cpu(rsp->hdr.MessageId);
#endif /* CIFS_DEBUG2 */
- if (buf) {
- buf->CreationTime = rsp->CreationTime;
- buf->LastAccessTime = rsp->LastAccessTime;
- buf->LastWriteTime = rsp->LastWriteTime;
- buf->ChangeTime = rsp->ChangeTime;
- buf->AllocationSize = rsp->AllocationSize;
- buf->EndOfFile = rsp->EndofFile;
- buf->Attributes = rsp->FileAttributes;
- buf->NumberOfLinks = cpu_to_le32(1);
- buf->DeletePending = 0;
+ if (file_info) {
+ file_info->CreationTime = rsp->CreationTime;
+ file_info->LastAccessTime = rsp->LastAccessTime;
+ file_info->LastWriteTime = rsp->LastWriteTime;
+ file_info->ChangeTime = rsp->ChangeTime;
+ file_info->AllocationSize = rsp->AllocationSize;
+ file_info->EndOfFile = rsp->EndofFile;
+ file_info->Attributes = rsp->FileAttributes;
+ file_info->NumberOfLinks = cpu_to_le32(1);
+ file_info->DeletePending = 0;
}
rc = smb2_parse_contexts(server, &rsp_iov, &oparms->fid->epoch,
- oparms->fid->lease_key, oplock, buf, posix);
+ oparms->fid->lease_key, oplock, file_info, posix);
+ if (rc)
+ SMB2_close(xid, tcon, oparms->fid->persistent_fid,
+ oparms->fid->volatile_fid);
+
+ trace_smb3_open_done(xid, rsp->PersistentFileId, tcon->tid, ses->Suid,
+ oparms->create_options, oparms->desired_access);
creat_exit:
SMB2_open_free(&rqst);
free_rsp_buf(resp_buftype, rsp);
@@ -3997,7 +4025,9 @@ void smb2_reconnect_server(struct work_struct *work)
list_for_each_entry(tcon, &ses->tcon_list, tcon_list) {
if (tcon->need_reconnect || tcon->need_reopen_files) {
+ spin_lock(&tcon->tc_lock);
tcon->tc_count++;
+ spin_unlock(&tcon->tc_lock);
trace_smb3_tcon_ref(tcon->debug_id, tcon->tc_count,
netfs_trace_tcon_ref_get_reconnect_server);
list_add_tail(&tcon->rlist, &tmp_list);
@@ -4257,8 +4287,10 @@ smb2_new_read_req(void **buf, unsigned int *total_len,
if (rc)
return rc;
- if (server == NULL)
- return -ECONNABORTED;
+ if (!server) {
+ rc = -ECONNABORTED;
+ goto free_req;
+ }
shdr = &req->hdr;
shdr->Id.SyncId.ProcessId = cpu_to_le32(io_parms->pid);
@@ -4289,8 +4321,10 @@ smb2_new_read_req(void **buf, unsigned int *total_len,
server->smbd_conn, rdata->pages,
rdata->nr_pages, rdata->page_offset,
rdata->tailsz, true, need_invalidate);
- if (!rdata->mr)
- return -EAGAIN;
+ if (!rdata->mr) {
+ rc = -EAGAIN;
+ goto free_req;
+ }
req->Channel = SMB2_CHANNEL_RDMA_V1_INVALIDATE;
if (need_invalidate)
@@ -4333,6 +4367,10 @@ smb2_new_read_req(void **buf, unsigned int *total_len,
*buf = req;
return rc;
+
+free_req:
+ cifs_small_buf_release(req);
+ return rc;
}
static void
diff --git a/fs/smb/client/smb2pdu.h b/fs/smb/client/smb2pdu.h
index d12ca9c7e62b..2c2a2177b553 100644
--- a/fs/smb/client/smb2pdu.h
+++ b/fs/smb/client/smb2pdu.h
@@ -170,15 +170,6 @@ struct durable_reconnect_context_v2 {
__le32 Flags; /* see above DHANDLE_FLAG_PERSISTENT */
} __packed;
-/* See MS-SMB2 2.2.14.2.9 */
-struct create_on_disk_id {
- struct create_context ccontext;
- __u8 Name[8];
- __le64 DiskFileId;
- __le64 VolumeId;
- __u32 Reserved[4];
-} __packed;
-
/* See MS-SMB2 2.2.14.2.12 */
struct durable_reconnect_context_v2_rsp {
__le32 Timeout;
diff --git a/fs/smb/client/smb2proto.h b/fs/smb/client/smb2proto.h
index 79e5e7764c3d..e166f49bdf69 100644
--- a/fs/smb/client/smb2proto.h
+++ b/fs/smb/client/smb2proto.h
@@ -37,8 +37,6 @@ extern struct mid_q_entry *smb2_setup_request(struct cifs_ses *ses,
struct smb_rqst *rqst);
extern struct mid_q_entry *smb2_setup_async_request(
struct TCP_Server_Info *server, struct smb_rqst *rqst);
-extern struct cifs_ses *smb2_find_smb_ses(struct TCP_Server_Info *server,
- __u64 ses_id);
extern struct cifs_tcon *smb2_find_smb_tcon(struct TCP_Server_Info *server,
__u64 ses_id, __u32 tid);
extern int smb2_calc_signature(struct smb_rqst *rqst,
@@ -122,7 +120,7 @@ extern int SMB2_tcon(const unsigned int xid, struct cifs_ses *ses,
extern int SMB2_tdis(const unsigned int xid, struct cifs_tcon *tcon);
extern int SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms,
__le16 *path, __u8 *oplock,
- struct smb2_file_all_info *buf,
+ struct cifs_open_info_data *buf,
struct create_posix_rsp *posix,
struct kvec *err_iov, int *resp_buftype);
extern int SMB2_open_init(struct cifs_tcon *tcon,
diff --git a/fs/smb/client/smb2transport.c b/fs/smb/client/smb2transport.c
index 3a69f3500acd..b8441ffcff76 100644
--- a/fs/smb/client/smb2transport.c
+++ b/fs/smb/client/smb2transport.c
@@ -75,7 +75,7 @@ smb311_crypto_shash_allocate(struct TCP_Server_Info *server)
static
-int smb2_get_sign_key(__u64 ses_id, struct TCP_Server_Info *server, u8 *key)
+int smb3_get_sign_key(__u64 ses_id, struct TCP_Server_Info *server, u8 *key)
{
struct cifs_chan *chan;
struct TCP_Server_Info *pserver;
@@ -168,16 +168,41 @@ smb2_find_smb_ses_unlocked(struct TCP_Server_Info *server, __u64 ses_id)
return NULL;
}
-struct cifs_ses *
-smb2_find_smb_ses(struct TCP_Server_Info *server, __u64 ses_id)
+static int smb2_get_sign_key(struct TCP_Server_Info *server,
+ __u64 ses_id, u8 *key)
{
struct cifs_ses *ses;
+ int rc = -ENOENT;
+
+ if (CIFS_SERVER_IS_CHAN(server))
+ server = server->primary_server;
spin_lock(&cifs_tcp_ses_lock);
- ses = smb2_find_smb_ses_unlocked(server, ses_id);
- spin_unlock(&cifs_tcp_ses_lock);
+ list_for_each_entry(ses, &server->smb_ses_list, smb_ses_list) {
+ if (ses->Suid != ses_id)
+ continue;
- return ses;
+ rc = 0;
+ spin_lock(&ses->ses_lock);
+ switch (ses->ses_status) {
+ case SES_EXITING: /* SMB2_LOGOFF */
+ case SES_GOOD:
+ if (likely(ses->auth_key.response)) {
+ memcpy(key, ses->auth_key.response,
+ SMB2_NTLMV2_SESSKEY_SIZE);
+ } else {
+ rc = -EIO;
+ }
+ break;
+ default:
+ rc = -EAGAIN;
+ break;
+ }
+ spin_unlock(&ses->ses_lock);
+ break;
+ }
+ spin_unlock(&cifs_tcp_ses_lock);
+ return rc;
}
static struct cifs_tcon *
@@ -188,7 +213,9 @@ smb2_find_smb_sess_tcon_unlocked(struct cifs_ses *ses, __u32 tid)
list_for_each_entry(tcon, &ses->tcon_list, tcon_list) {
if (tcon->tid != tid)
continue;
+ spin_lock(&tcon->tc_lock);
++tcon->tc_count;
+ spin_unlock(&tcon->tc_lock);
trace_smb3_tcon_ref(tcon->debug_id, tcon->tc_count,
netfs_trace_tcon_ref_get_find_sess_tcon);
return tcon;
@@ -236,14 +263,16 @@ smb2_calc_signature(struct smb_rqst *rqst, struct TCP_Server_Info *server,
unsigned char *sigptr = smb2_signature;
struct kvec *iov = rqst->rq_iov;
struct smb2_hdr *shdr = (struct smb2_hdr *)iov[0].iov_base;
- struct cifs_ses *ses;
struct shash_desc *shash = NULL;
struct smb_rqst drqst;
+ __u64 sid = le64_to_cpu(shdr->SessionId);
+ u8 key[SMB2_NTLMV2_SESSKEY_SIZE];
- ses = smb2_find_smb_ses(server, le64_to_cpu(shdr->SessionId));
- if (unlikely(!ses)) {
- cifs_server_dbg(VFS, "%s: Could not find session\n", __func__);
- return -ENOENT;
+ rc = smb2_get_sign_key(server, sid, key);
+ if (unlikely(rc)) {
+ cifs_server_dbg(FYI, "%s: [sesid=0x%llx] couldn't find signing key: %d\n",
+ __func__, sid, rc);
+ return rc;
}
memset(smb2_signature, 0x0, SMB2_HMACSHA256_SIZE);
@@ -260,8 +289,7 @@ smb2_calc_signature(struct smb_rqst *rqst, struct TCP_Server_Info *server,
shash = server->secmech.hmacsha256;
}
- rc = crypto_shash_setkey(shash->tfm, ses->auth_key.response,
- SMB2_NTLMV2_SESSKEY_SIZE);
+ rc = crypto_shash_setkey(shash->tfm, key, sizeof(key));
if (rc) {
cifs_server_dbg(VFS,
"%s: Could not update with response\n",
@@ -303,8 +331,6 @@ smb2_calc_signature(struct smb_rqst *rqst, struct TCP_Server_Info *server,
out:
if (allocate_crypto)
cifs_free_hash(&shash);
- if (ses)
- cifs_put_smb_ses(ses);
return rc;
}
@@ -582,7 +608,7 @@ smb3_calc_signature(struct smb_rqst *rqst, struct TCP_Server_Info *server,
struct smb_rqst drqst;
u8 key[SMB3_SIGN_KEY_SIZE];
- rc = smb2_get_sign_key(le64_to_cpu(shdr->SessionId), server, key);
+ rc = smb3_get_sign_key(le64_to_cpu(shdr->SessionId), server, key);
if (unlikely(rc)) {
cifs_server_dbg(VFS, "%s: Could not get signing key\n", __func__);
return rc;
diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c
index 9b16840952c0..894f5ffca909 100644
--- a/fs/smb/client/smbdirect.c
+++ b/fs/smb/client/smbdirect.c
@@ -461,7 +461,9 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
struct smbd_response *response =
container_of(wc->wr_cqe, struct smbd_response, cqe);
struct smbd_connection *info = response->info;
- int data_length = 0;
+ u32 data_offset = 0;
+ u32 data_length = 0;
+ u32 remaining_data_length = 0;
log_rdma_recv(INFO, "response=0x%p type=%d wc status=%d wc opcode %d byte_len=%d pkey_index=%u\n",
response, response->type, wc->status, wc->opcode,
@@ -493,7 +495,22 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc)
/* SMBD data transfer packet */
case SMBD_TRANSFER_DATA:
data_transfer = smbd_response_payload(response);
+
+ if (wc->byte_len <
+ offsetof(struct smbd_data_transfer, padding))
+ goto error;
+
+ remaining_data_length = le32_to_cpu(data_transfer->remaining_data_length);
+ data_offset = le32_to_cpu(data_transfer->data_offset);
data_length = le32_to_cpu(data_transfer->data_length);
+ if (wc->byte_len < data_offset ||
+ (u64)wc->byte_len < (u64)data_offset + data_length)
+ goto error;
+
+ if (remaining_data_length > info->max_fragmented_recv_size ||
+ data_length > info->max_fragmented_recv_size ||
+ (u64)remaining_data_length + (u64)data_length > (u64)info->max_fragmented_recv_size)
+ goto error;
/*
* If this is a packet with data playload place the data in
@@ -1358,6 +1375,9 @@ void smbd_destroy(struct TCP_Server_Info *server)
info->transport_status == SMBD_DISCONNECTED);
}
+ log_rdma_event(INFO, "cancelling post_send_credits_work\n");
+ cancel_work_sync(&info->post_send_credits_work);
+
log_rdma_event(INFO, "destroying qp\n");
ib_drain_qp(info->id->qp);
rdma_destroy_qp(info->id);
diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c
index 59063b26afe4..c5c81b426198 100644
--- a/fs/smb/client/transport.c
+++ b/fs/smb/client/transport.c
@@ -1100,6 +1100,7 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses,
int *resp_buf_type, struct kvec *resp_iov)
{
int i, j, optype, rc = 0;
+ int num_processed = 0;
struct mid_q_entry *midQ[MAX_COMPOUND];
bool cancelled_mid[MAX_COMPOUND] = {false};
struct cifs_credits credits[MAX_COMPOUND] = {
@@ -1242,6 +1243,17 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses,
break;
}
if (rc != 0) {
+ /*
+ * A completed CREATE earlier in the compound chain may have
+ * opened a remote handle even though a later wait was
+ * interrupted. Mark it cancelled so __release_mid() invokes
+ * the existing unmatched-open cleanup.
+ */
+ spin_lock(&server->mid_lock);
+ for (j = 0; j < i; j++)
+ midQ[j]->mid_flags |= MID_WAIT_CANCELLED;
+ spin_unlock(&server->mid_lock);
+
for (; i < num_rqst; i++) {
cifs_server_dbg(FYI, "Cancelling wait for mid %llu cmd: %d\n",
midQ[i]->mid, le16_to_cpu(midQ[i]->command));
@@ -1264,6 +1276,17 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses,
rc = cifs_sync_mid_result(midQ[i], server);
if (rc != 0) {
+ /*
+ * A previous CREATE may have completed before this
+ * response failed. Mark it cancelled so its remote
+ * handle is closed when the mid is released.
+ */
+ spin_lock(&server->mid_lock);
+ for (j = 0; j < i; j++)
+ midQ[j]->mid_flags |= MID_WAIT_CANCELLED;
+ spin_unlock(&server->mid_lock);
+ /* Keep their response buffers for cancelled-mid cleanup. */
+ num_processed = 0;
/* mark this mid as cancelled to not free it below */
cancelled_mid[i] = true;
goto out;
@@ -1273,9 +1296,26 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses,
midQ[i]->mid_state != MID_RESPONSE_READY) {
rc = -EIO;
cifs_dbg(FYI, "Bad MID state?\n");
+ spin_lock(&server->mid_lock);
+ for (j = 0; j < i; j++)
+ midQ[j]->mid_flags |= MID_WAIT_CANCELLED;
+ spin_unlock(&server->mid_lock);
+ num_processed = 0;
goto out;
}
+ rc = server->ops->check_receive(midQ[i], server,
+ flags & CIFS_LOG_ERROR);
+ num_processed = i + 1;
+ }
+
+out:
+ /*
+ * Delay moving response buffers out of their mids until response
+ * synchronization completes. This lets cancelled-mid cleanup inspect
+ * an earlier CREATE response if a later MID fails.
+ */
+ for (i = 0; i < num_processed; i++) {
buf = (char *)midQ[i]->resp_buf;
resp_iov[i].iov_base = buf;
resp_iov[i].iov_len = midQ[i]->resp_buf_size +
@@ -1286,33 +1326,30 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses,
else
resp_buf_type[i] = CIFS_SMALL_BUFFER;
- rc = server->ops->check_receive(midQ[i], server,
- flags & CIFS_LOG_ERROR);
-
/* mark it so buf will not be freed by delete_mid */
if ((flags & CIFS_NO_RSP_BUF) == 0)
midQ[i]->resp_buf = NULL;
-
}
/*
* Compounding is never used during session establish.
*/
- spin_lock(&ses->ses_lock);
- if ((ses->ses_status == SES_NEW) || (optype & CIFS_NEG_OP) || (optype & CIFS_SESS_OP)) {
- struct kvec iov = {
- .iov_base = resp_iov[0].iov_base,
- .iov_len = resp_iov[0].iov_len
- };
- spin_unlock(&ses->ses_lock);
- cifs_server_lock(server);
- smb311_update_preauth_hash(ses, server, &iov, 1);
- cifs_server_unlock(server);
+ if (num_processed == num_rqst) {
spin_lock(&ses->ses_lock);
+ if ((ses->ses_status == SES_NEW) || (optype & CIFS_NEG_OP) || (optype & CIFS_SESS_OP)) {
+ struct kvec iov = {
+ .iov_base = resp_iov[0].iov_base,
+ .iov_len = resp_iov[0].iov_len
+ };
+ spin_unlock(&ses->ses_lock);
+ cifs_server_lock(server);
+ smb311_update_preauth_hash(ses, server, &iov, 1);
+ cifs_server_unlock(server);
+ spin_lock(&ses->ses_lock);
+ }
+ spin_unlock(&ses->ses_lock);
}
- spin_unlock(&ses->ses_lock);
-out:
/*
* This will dequeue all mids. After this it is important that the
* demultiplex_thread will not process any of these mids any futher.
diff --git a/fs/smb/common/smb2pdu.h b/fs/smb/common/smb2pdu.h
index 7d69a2a1d3ba..430b65512228 100644
--- a/fs/smb/common/smb2pdu.h
+++ b/fs/smb/common/smb2pdu.h
@@ -1240,6 +1240,7 @@ struct create_posix {
#define SMB2_LEASE_KEY_SIZE 16
+/* See MS-SMB2 2.2.13.2.8 */
struct lease_context {
__u8 LeaseKey[SMB2_LEASE_KEY_SIZE];
__le32 LeaseState;
@@ -1247,6 +1248,7 @@ struct lease_context {
__le64 LeaseDuration;
} __packed;
+/* See MS-SMB2 2.2.13.2.10 */
struct lease_context_v2 {
__u8 LeaseKey[SMB2_LEASE_KEY_SIZE];
__le32 LeaseState;
@@ -1270,6 +1272,15 @@ struct create_lease_v2 {
__u8 Pad[4];
} __packed;
+/* See MS-SMB2 2.2.14.2.9 */
+struct create_disk_id_rsp {
+ struct create_context ccontext;
+ __u8 Name[8];
+ __le64 DiskFileId;
+ __le64 VolumeId;
+ __u8 Reserved[16];
+} __packed;
+
/* See MS-SMB2 2.2.31 and 2.2.32 */
struct smb2_ioctl_req {
struct smb2_hdr hdr;
diff --git a/fs/smb/server/auth.c b/fs/smb/server/auth.c
index 650981477093..57be9520f879 100644
--- a/fs/smb/server/auth.c
+++ b/fs/smb/server/auth.c
@@ -72,13 +72,12 @@ void ksmbd_copy_gss_neg_header(void *buf)
/**
* ksmbd_gen_sess_key() - function to generate session key
- * @sess: session of connection
+ * @sess_key: output buffer for the session key
* @hash: source hash value to be used for find session key
* @hmac: source hmac value to be used for finding session key
*
*/
-static int ksmbd_gen_sess_key(struct ksmbd_session *sess, char *hash,
- char *hmac)
+static int ksmbd_gen_sess_key(char *sess_key, char *hash, char *hmac)
{
struct ksmbd_crypto_ctx *ctx;
int rc;
@@ -111,7 +110,7 @@ static int ksmbd_gen_sess_key(struct ksmbd_session *sess, char *hash,
goto out;
}
- rc = crypto_shash_final(CRYPTO_HMACMD5(ctx), sess->sess_key);
+ rc = crypto_shash_final(CRYPTO_HMACMD5(ctx), sess_key);
if (rc) {
ksmbd_debug(AUTH, "Could not generate hmacmd5 hash error %d\n", rc);
goto out;
@@ -222,6 +221,7 @@ int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess,
{
char ntlmv2_hash[CIFS_ENCPWD_SIZE];
char ntlmv2_rsp[CIFS_HMAC_MD5_HASH_SIZE];
+ char sess_key[SMB2_NTLMV2_SESSKEY_SIZE];
struct ksmbd_crypto_ctx *ctx = NULL;
char *construct = NULL;
int rc, len;
@@ -276,19 +276,27 @@ int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess,
ksmbd_release_crypto_ctx(ctx);
ctx = NULL;
- rc = ksmbd_gen_sess_key(sess, ntlmv2_hash, ntlmv2_rsp);
+ rc = ksmbd_gen_sess_key(sess_key, ntlmv2_hash, ntlmv2_rsp);
if (rc) {
ksmbd_debug(AUTH, "Could not generate sess key\n");
goto out;
}
if (crypto_memneq(ntlmv2->ntlmv2_hash, ntlmv2_rsp,
- CIFS_HMAC_MD5_HASH_SIZE))
+ CIFS_HMAC_MD5_HASH_SIZE)) {
rc = -EINVAL;
+ goto out;
+ }
+
+ memcpy(sess->sess_key, sess_key, sizeof(sess_key));
+ rc = 0;
out:
if (ctx)
ksmbd_release_crypto_ctx(ctx);
kfree(construct);
+ memzero_explicit(ntlmv2_hash, sizeof(ntlmv2_hash));
+ memzero_explicit(ntlmv2_rsp, sizeof(ntlmv2_rsp));
+ memzero_explicit(sess_key, sizeof(sess_key));
return rc;
}
@@ -345,6 +353,8 @@ int ksmbd_decode_ntlmssp_auth_blob(struct authenticate_message *authblob,
nt_len - CIFS_ENCPWD_SIZE,
domain_name, conn->ntlmssp.cryptkey);
kfree(domain_name);
+ if (ret)
+ return ret;
/* The recovered secondary session key */
if (conn->ntlmssp.client_flags & NTLMSSP_NEGOTIATE_KEY_XCH) {
diff --git a/fs/smb/server/connection.c b/fs/smb/server/connection.c
index 0d27651aa4cc..f1184ee9b224 100644
--- a/fs/smb/server/connection.c
+++ b/fs/smb/server/connection.c
@@ -11,6 +11,7 @@
#include "server.h"
#include "smb_common.h"
#include "mgmt/ksmbd_ida.h"
+#include "mgmt/user_session.h"
#include "connection.h"
#include "transport_tcp.h"
#include "transport_rdma.h"
@@ -85,6 +86,7 @@ void ksmbd_conn_free(struct ksmbd_conn *conn)
kvfree(conn->request_buf);
kfree(conn->preauth_info);
kfree(conn->mechToken);
+ ksmbd_preauth_session_destroy(conn);
ksmbd_conn_put(conn);
}
@@ -115,6 +117,7 @@ struct ksmbd_conn *ksmbd_conn_alloc(void)
atomic_set(&conn->req_running, 0);
atomic_set(&conn->r_count, 0);
atomic_set(&conn->refcnt, 1);
+ atomic_set(&conn->mux_smb_requests, 0);
conn->total_credits = 1;
conn->outstanding_credits = 0;
@@ -123,6 +126,7 @@ struct ksmbd_conn *ksmbd_conn_alloc(void)
INIT_LIST_HEAD(&conn->conns_list);
INIT_LIST_HEAD(&conn->requests);
INIT_LIST_HEAD(&conn->async_requests);
+ INIT_LIST_HEAD(&conn->preauth_sess_table);
spin_lock_init(&conn->request_lock);
spin_lock_init(&conn->credits_lock);
ida_init(&conn->async_ida);
@@ -441,6 +445,25 @@ void ksmbd_conn_init_server_callbacks(struct ksmbd_conn_ops *ops)
default_conn_ops.terminate_fn = ops->terminate_fn;
}
+void ksmbd_conn_r_count_inc(struct ksmbd_conn *conn)
+{
+ atomic_inc(&conn->r_count);
+}
+
+void ksmbd_conn_r_count_dec(struct ksmbd_conn *conn)
+{
+ /*
+ * Checking waitqueue to dropping pending requests on
+ * disconnection. waitqueue_active is safe because it
+ * uses atomic operation for condition.
+ */
+ atomic_inc(&conn->refcnt);
+ if (!atomic_dec_return(&conn->r_count) && waitqueue_active(&conn->r_count_q))
+ wake_up(&conn->r_count_q);
+
+ ksmbd_conn_put(conn);
+}
+
int ksmbd_conn_transport_init(void)
{
int ret;
diff --git a/fs/smb/server/connection.h b/fs/smb/server/connection.h
index d01f137223b4..29b32486242c 100644
--- a/fs/smb/server/connection.h
+++ b/fs/smb/server/connection.h
@@ -27,6 +27,7 @@ enum {
KSMBD_SESS_EXITING,
KSMBD_SESS_NEED_RECONNECT,
KSMBD_SESS_NEED_NEGOTIATE,
+ KSMBD_SESS_NEED_SETUP,
KSMBD_SESS_RELEASING
};
@@ -113,6 +114,7 @@ struct ksmbd_conn {
__le16 signing_algorithm;
bool binding;
atomic_t refcnt;
+ atomic_t mux_smb_requests;
};
struct ksmbd_conn_ops {
@@ -175,6 +177,8 @@ int ksmbd_conn_transport_init(void);
void ksmbd_conn_transport_destroy(void);
void ksmbd_conn_lock(struct ksmbd_conn *conn);
void ksmbd_conn_unlock(struct ksmbd_conn *conn);
+void ksmbd_conn_r_count_inc(struct ksmbd_conn *conn);
+void ksmbd_conn_r_count_dec(struct ksmbd_conn *conn);
/*
* WARNING
@@ -197,6 +201,11 @@ static inline bool ksmbd_conn_need_negotiate(struct ksmbd_conn *conn)
return READ_ONCE(conn->status) == KSMBD_SESS_NEED_NEGOTIATE;
}
+static inline bool ksmbd_conn_need_setup(struct ksmbd_conn *conn)
+{
+ return READ_ONCE(conn->status) == KSMBD_SESS_NEED_SETUP;
+}
+
static inline bool ksmbd_conn_need_reconnect(struct ksmbd_conn *conn)
{
return READ_ONCE(conn->status) == KSMBD_SESS_NEED_RECONNECT;
@@ -227,6 +236,11 @@ static inline void ksmbd_conn_set_need_negotiate(struct ksmbd_conn *conn)
WRITE_ONCE(conn->status, KSMBD_SESS_NEED_NEGOTIATE);
}
+static inline void ksmbd_conn_set_need_setup(struct ksmbd_conn *conn)
+{
+ WRITE_ONCE(conn->status, KSMBD_SESS_NEED_SETUP);
+}
+
static inline void ksmbd_conn_set_need_reconnect(struct ksmbd_conn *conn)
{
WRITE_ONCE(conn->status, KSMBD_SESS_NEED_RECONNECT);
diff --git a/fs/smb/server/ksmbd_work.c b/fs/smb/server/ksmbd_work.c
index d7c676c151e2..544d8ccd29b0 100644
--- a/fs/smb/server/ksmbd_work.c
+++ b/fs/smb/server/ksmbd_work.c
@@ -26,7 +26,6 @@ struct ksmbd_work *ksmbd_alloc_work_struct(void)
INIT_LIST_HEAD(&work->request_entry);
INIT_LIST_HEAD(&work->async_request_entry);
INIT_LIST_HEAD(&work->fp_entry);
- INIT_LIST_HEAD(&work->interim_entry);
INIT_LIST_HEAD(&work->aux_read_list);
work->iov_alloc_cnt = 4;
work->iov = kcalloc(work->iov_alloc_cnt, sizeof(struct kvec),
@@ -56,8 +55,6 @@ void ksmbd_free_work_struct(struct ksmbd_work *work)
kfree(work->tr_buf);
kvfree(work->request_buf);
kfree(work->iov);
- if (!list_empty(&work->interim_entry))
- list_del(&work->interim_entry);
if (work->async_id)
ksmbd_release_id(&work->conn->async_ida, work->async_id);
diff --git a/fs/smb/server/ksmbd_work.h b/fs/smb/server/ksmbd_work.h
index 9c3f834901e1..6372ba368e14 100644
--- a/fs/smb/server/ksmbd_work.h
+++ b/fs/smb/server/ksmbd_work.h
@@ -96,7 +96,6 @@ struct ksmbd_work {
/* List head at conn->async_requests */
struct list_head async_request_entry;
struct list_head fp_entry;
- struct list_head interim_entry;
};
/**
diff --git a/fs/smb/server/mgmt/share_config.c b/fs/smb/server/mgmt/share_config.c
index 7db7345f1af9..cb89e54c4da8 100644
--- a/fs/smb/server/mgmt/share_config.c
+++ b/fs/smb/server/mgmt/share_config.c
@@ -161,6 +161,11 @@ static struct ksmbd_share_config *share_config_request(struct ksmbd_work *work,
atomic_set(&share->refcount, 1);
INIT_LIST_HEAD(&share->veto_list);
share->name = kstrdup(name, GFP_KERNEL);
+ if (!share->name) {
+ kill_share(share);
+ share = NULL;
+ goto out;
+ }
if (!test_share_config_flag(share, KSMBD_SHARE_FLAG_PIPE)) {
size_t path_len;
@@ -217,7 +222,7 @@ static struct ksmbd_share_config *share_config_request(struct ksmbd_work *work,
share->path = NULL;
}
}
- if (ret || !share->name) {
+ if (ret) {
kill_share(share);
share = NULL;
goto out;
diff --git a/fs/smb/server/mgmt/tree_connect.c b/fs/smb/server/mgmt/tree_connect.c
index 9bde1b58f9c4..02c46e07aa87 100644
--- a/fs/smb/server/mgmt/tree_connect.c
+++ b/fs/smb/server/mgmt/tree_connect.c
@@ -76,7 +76,7 @@ ksmbd_tree_conn_connect(struct ksmbd_work *work, const char *share_name)
tree_conn->share_conf = sc;
tree_conn->t_state = TREE_NEW;
status.tree_conn = tree_conn;
- atomic_set(&tree_conn->refcount, 1);
+ atomic_set(&tree_conn->refcount, 2);
ret = xa_err(xa_store(&sess->tree_conns, tree_conn->id, tree_conn,
GFP_KERNEL));
@@ -98,8 +98,10 @@ ksmbd_tree_conn_connect(struct ksmbd_work *work, const char *share_name)
void ksmbd_tree_connect_put(struct ksmbd_tree_connect *tcon)
{
- if (atomic_dec_and_test(&tcon->refcount))
+ if (atomic_dec_and_test(&tcon->refcount)) {
+ ksmbd_share_config_put(tcon->share_conf);
kfree(tcon);
+ }
}
int ksmbd_tree_conn_disconnect(struct ksmbd_session *sess,
@@ -108,14 +110,18 @@ int ksmbd_tree_conn_disconnect(struct ksmbd_session *sess,
int ret;
write_lock(&sess->tree_conns_lock);
+ if (tree_conn->t_state == TREE_DISCONNECTED ||
+ xa_load(&sess->tree_conns, tree_conn->id) != tree_conn) {
+ write_unlock(&sess->tree_conns_lock);
+ return -ENOENT;
+ }
+ tree_conn->t_state = TREE_DISCONNECTED;
xa_erase(&sess->tree_conns, tree_conn->id);
write_unlock(&sess->tree_conns_lock);
ret = ksmbd_ipc_tree_disconnect_request(sess->id, tree_conn->id);
ksmbd_release_tree_conn_id(sess, tree_conn->id);
- ksmbd_share_config_put(tree_conn->share_conf);
- if (atomic_dec_and_test(&tree_conn->refcount))
- kfree(tree_conn);
+ ksmbd_tree_connect_put(tree_conn);
return ret;
}
@@ -146,18 +152,8 @@ int ksmbd_tree_conn_session_logoff(struct ksmbd_session *sess)
if (!sess)
return -EINVAL;
- xa_for_each(&sess->tree_conns, id, tc) {
- write_lock(&sess->tree_conns_lock);
- if (tc->t_state == TREE_DISCONNECTED) {
- write_unlock(&sess->tree_conns_lock);
- ret = -ENOENT;
- continue;
- }
- tc->t_state = TREE_DISCONNECTED;
- write_unlock(&sess->tree_conns_lock);
-
+ xa_for_each(&sess->tree_conns, id, tc)
ret |= ksmbd_tree_conn_disconnect(sess, tc);
- }
xa_destroy(&sess->tree_conns);
return ret;
}
diff --git a/fs/smb/server/mgmt/user_config.c b/fs/smb/server/mgmt/user_config.c
index 279d00feff21..a9c0d80dc368 100644
--- a/fs/smb/server/mgmt/user_config.c
+++ b/fs/smb/server/mgmt/user_config.c
@@ -31,6 +31,17 @@ struct ksmbd_user *ksmbd_alloc_user(struct ksmbd_login_response *resp)
{
struct ksmbd_user *user = NULL;
+ /*
+ * resp->hash_sz is a __u16 taken from the mountd IPC login response but
+ * resp->hash[] is only KSMBD_REQ_MAX_HASH_SZ bytes. A malformed or
+ * malicious response can set hash_sz far beyond that (up to 65535),
+ * making the memcpy() below read past the response object
+ * (slab-out-of-bounds in ksmbd_alloc_user()). Reject any oversized
+ * hash rather than trust the length.
+ */
+ if (resp->hash_sz > sizeof(resp->hash))
+ return NULL;
+
user = kmalloc(sizeof(struct ksmbd_user), GFP_KERNEL);
if (!user)
return NULL;
diff --git a/fs/smb/server/mgmt/user_session.c b/fs/smb/server/mgmt/user_session.c
index 605e17c7b2a1..09c19d1f71e3 100644
--- a/fs/smb/server/mgmt/user_session.c
+++ b/fs/smb/server/mgmt/user_session.c
@@ -32,12 +32,14 @@ static void free_channel_list(struct ksmbd_session *sess)
struct channel *chann;
unsigned long index;
+ down_write(&sess->chann_lock);
xa_for_each(&sess->ksmbd_chann_list, index, chann) {
xa_erase(&sess->ksmbd_chann_list, index);
kfree(chann);
}
xa_destroy(&sess->ksmbd_chann_list);
+ up_write(&sess->chann_lock);
}
static void __session_rpc_close(struct ksmbd_session *sess,
@@ -210,17 +212,30 @@ static void ksmbd_expire_session(struct ksmbd_conn *conn)
int ksmbd_session_register(struct ksmbd_conn *conn,
struct ksmbd_session *sess)
{
+ int ret;
+
sess->dialect = conn->dialect;
memcpy(sess->ClientGUID, conn->ClientGUID, SMB2_CLIENT_GUID_SIZE);
ksmbd_expire_session(conn);
- return xa_err(xa_store(&conn->sessions, sess->id, sess, GFP_KERNEL));
+ ret = xa_err(xa_store(&conn->sessions, sess->id, sess,
+ GFP_KERNEL));
+ if (ret) {
+ down_write(&sessions_table_lock);
+ hash_del(&sess->hlist);
+ up_write(&sessions_table_lock);
+ ksmbd_user_session_put(sess);
+ }
+
+ return ret;
}
static int ksmbd_chann_del(struct ksmbd_conn *conn, struct ksmbd_session *sess)
{
struct channel *chann;
+ down_write(&sess->chann_lock);
chann = xa_erase(&sess->ksmbd_chann_list, (long)conn);
+ up_write(&sess->chann_lock);
if (!chann)
return -ENOENT;
@@ -367,6 +382,17 @@ struct preauth_session *ksmbd_preauth_session_alloc(struct ksmbd_conn *conn,
return sess;
}
+void ksmbd_preauth_session_destroy(struct ksmbd_conn *conn)
+{
+ struct preauth_session *sess, *tmp;
+
+ list_for_each_entry_safe(sess, tmp, &conn->preauth_sess_table,
+ preauth_entry) {
+ list_del(&sess->preauth_entry);
+ kfree(sess);
+ }
+}
+
static bool ksmbd_preauth_session_id_match(struct preauth_session *sess,
unsigned long long id)
{
@@ -422,6 +448,7 @@ static struct ksmbd_session *__session_create(int protocol)
rwlock_init(&sess->tree_conns_lock);
atomic_set(&sess->refcnt, 2);
init_rwsem(&sess->rpc_lock);
+ init_rwsem(&sess->chann_lock);
ret = __init_smb2_session(sess);
if (ret)
diff --git a/fs/smb/server/mgmt/user_session.h b/fs/smb/server/mgmt/user_session.h
index 011362e975b0..03b92c3667f6 100644
--- a/fs/smb/server/mgmt/user_session.h
+++ b/fs/smb/server/mgmt/user_session.h
@@ -49,6 +49,7 @@ struct ksmbd_session {
char sess_key[CIFS_KEY_SIZE];
struct hlist_node hlist;
+ struct rw_semaphore chann_lock;
struct xarray ksmbd_chann_list;
struct xarray tree_conns;
struct ida tree_conn_ida;
@@ -97,6 +98,7 @@ struct ksmbd_session *ksmbd_session_lookup_all(struct ksmbd_conn *conn,
unsigned long long id);
struct preauth_session *ksmbd_preauth_session_alloc(struct ksmbd_conn *conn,
u64 sess_id);
+void ksmbd_preauth_session_destroy(struct ksmbd_conn *conn);
struct preauth_session *ksmbd_preauth_session_lookup(struct ksmbd_conn *conn,
unsigned long long id);
diff --git a/fs/smb/server/misc.c b/fs/smb/server/misc.c
index 11314f7c622a..98da57bb5fd2 100644
--- a/fs/smb/server/misc.c
+++ b/fs/smb/server/misc.c
@@ -284,39 +284,6 @@ char *ksmbd_extract_sharename(struct unicode_map *um, const char *treename)
return ksmbd_casefold_sharename(um, name);
}
-/**
- * convert_to_unix_name() - convert windows name to unix format
- * @share: ksmbd_share_config pointer
- * @name: file name that is relative to share
- *
- * Return: converted name on success, otherwise NULL
- */
-char *convert_to_unix_name(struct ksmbd_share_config *share, const char *name)
-{
- int no_slash = 0, name_len, path_len;
- char *new_name;
-
- if (name[0] == '/')
- name++;
-
- path_len = share->path_sz;
- name_len = strlen(name);
- new_name = kmalloc(path_len + name_len + 2, GFP_KERNEL);
- if (!new_name)
- return new_name;
-
- memcpy(new_name, share->path, path_len);
- if (new_name[path_len - 1] != '/') {
- new_name[path_len] = '/';
- no_slash = 1;
- }
-
- memcpy(new_name + path_len + no_slash, name, name_len);
- path_len += name_len + no_slash;
- new_name[path_len] = 0x00;
- return new_name;
-}
-
char *ksmbd_convert_dir_info_name(struct ksmbd_dir_info *d_info,
const struct nls_table *local_nls,
int *conv_len)
diff --git a/fs/smb/server/misc.h b/fs/smb/server/misc.h
index 1facfcd21200..53282a6d2436 100644
--- a/fs/smb/server/misc.h
+++ b/fs/smb/server/misc.h
@@ -22,7 +22,6 @@ void ksmbd_strip_last_slash(char *path);
void ksmbd_conv_path_to_windows(char *path);
char *ksmbd_casefold_sharename(struct unicode_map *um, const char *name);
char *ksmbd_extract_sharename(struct unicode_map *um, const char *treename);
-char *convert_to_unix_name(struct ksmbd_share_config *share, const char *name);
#define KSMBD_DIR_INFO_ALIGNMENT 8
struct ksmbd_dir_info;
diff --git a/fs/smb/server/oplock.c b/fs/smb/server/oplock.c
index 0b71399af50c..932e31a20216 100644
--- a/fs/smb/server/oplock.c
+++ b/fs/smb/server/oplock.c
@@ -41,11 +41,11 @@ static struct oplock_info *alloc_opinfo(struct ksmbd_work *work,
opinfo->conn = ksmbd_conn_get(work->conn);
opinfo->level = SMB2_OPLOCK_LEVEL_NONE;
opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock_init(&opinfo->state_lock);
opinfo->pending_break = 0;
opinfo->fid = id;
opinfo->Tid = Tid;
INIT_LIST_HEAD(&opinfo->op_entry);
- INIT_LIST_HEAD(&opinfo->interim_list);
init_waitqueue_head(&opinfo->oplock_q);
init_waitqueue_head(&opinfo->oplock_brk);
atomic_set(&opinfo->refcount, 1);
@@ -406,14 +406,23 @@ void close_id_del_oplock(struct ksmbd_file *fp)
opinfo_del(opinfo);
rcu_assign_pointer(fp->f_opinfo, NULL);
- if (opinfo->op_state == OPLOCK_ACK_WAIT) {
- opinfo->op_state = OPLOCK_CLOSING;
- wake_up_interruptible_all(&opinfo->oplock_q);
- if (opinfo->is_lease) {
- atomic_set(&opinfo->breaking_cnt, 0);
- wake_up_interruptible_all(&opinfo->oplock_brk);
- }
- }
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state == OPLOCK_ACK_WAIT && opinfo->is_lease)
+ atomic_set(&opinfo->breaking_cnt, 0);
+ /*
+ * An opinfo that has been removed from the inode list is terminal. Keep
+ * this transition and releasing pending_break under state_lock. a breaker
+ * takes the same lock before it acquires pending_break or sets ACK_WAIT.
+ */
+ opinfo->op_state = OPLOCK_CLOSING;
+ clear_bit_unlock(0, &opinfo->pending_break);
+ spin_unlock(&opinfo->state_lock);
+ wake_up_interruptible_all(&opinfo->oplock_q);
+ if (opinfo->is_lease)
+ wake_up_interruptible_all(&opinfo->oplock_brk);
+ /* memory barrier is needed for wake_up_bit() */
+ smp_mb__after_atomic();
+ wake_up_bit(&opinfo->pending_break, 0);
opinfo_count_dec(fp);
atomic_dec(&opinfo->refcount);
@@ -592,10 +601,16 @@ static void wait_for_break_ack(struct oplock_info *opinfo)
/* is this a timeout ? */
if (!rc) {
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state == OPLOCK_CLOSING) {
+ spin_unlock(&opinfo->state_lock);
+ return;
+ }
if (opinfo->is_lease)
opinfo->o_lease->state = SMB2_LEASE_NONE_LE;
opinfo->level = SMB2_OPLOCK_LEVEL_NONE;
opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
}
}
@@ -607,17 +622,46 @@ static void wake_up_oplock_break(struct oplock_info *opinfo)
wake_up_bit(&opinfo->pending_break, 0);
}
+static bool oplock_break_set_ack_wait(struct oplock_info *opinfo)
+{
+ bool ret = false;
+
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING) {
+ opinfo->op_state = OPLOCK_ACK_WAIT;
+ ret = true;
+ }
+ spin_unlock(&opinfo->state_lock);
+
+ return ret;
+}
+
static int oplock_break_pending(struct oplock_info *opinfo, int req_op_level)
{
- while (test_and_set_bit(0, &opinfo->pending_break)) {
+ for (;;) {
+ bool closing;
+
+ spin_lock(&opinfo->state_lock);
+ closing = opinfo->op_state == OPLOCK_CLOSING;
+ if (!closing && !test_and_set_bit(0, &opinfo->pending_break)) {
+ spin_unlock(&opinfo->state_lock);
+ break;
+ }
+ spin_unlock(&opinfo->state_lock);
+ if (closing)
+ return -ENOENT;
+
wait_on_bit(&opinfo->pending_break, 0, TASK_UNINTERRUPTIBLE);
/* Not immediately break to none. */
opinfo->open_trunc = 0;
- if (opinfo->op_state == OPLOCK_CLOSING)
+ spin_lock(&opinfo->state_lock);
+ closing = opinfo->op_state == OPLOCK_CLOSING;
+ spin_unlock(&opinfo->state_lock);
+ if (closing)
return -ENOENT;
- else if (opinfo->level <= req_op_level) {
+ if (opinfo->level <= req_op_level) {
if (opinfo->is_lease == false)
return 1;
@@ -657,6 +701,7 @@ static void __smb2_oplock_break_noti(struct work_struct *wk)
{
struct smb2_oplock_break *rsp = NULL;
struct ksmbd_work *work = container_of(wk, struct ksmbd_work, work);
+ struct ksmbd_conn *conn = work->conn;
struct oplock_break_info *br_info = work->request_buf;
struct smb2_hdr *rsp_hdr;
struct ksmbd_file *fp;
@@ -712,6 +757,8 @@ static void __smb2_oplock_break_noti(struct work_struct *wk)
out:
ksmbd_free_work_struct(work);
+ ksmbd_conn_r_count_dec(conn);
+ ksmbd_conn_put(conn);
}
/**
@@ -742,9 +789,10 @@ static int smb2_oplock_break_noti(struct oplock_info *opinfo)
br_info->open_trunc = opinfo->open_trunc;
work->request_buf = (char *)br_info;
- work->conn = conn;
+ work->conn = ksmbd_conn_get(conn);
work->sess = opinfo->sess;
+ ksmbd_conn_r_count_inc(conn);
if (opinfo->op_state == OPLOCK_ACK_WAIT) {
INIT_WORK(&work->work, __smb2_oplock_break_noti);
ksmbd_queue_work(work);
@@ -767,6 +815,7 @@ static void __smb2_lease_break_noti(struct work_struct *wk)
{
struct smb2_lease_break *rsp = NULL;
struct ksmbd_work *work = container_of(wk, struct ksmbd_work, work);
+ struct ksmbd_conn *conn = work->conn;
struct lease_break_info *br_info = work->request_buf;
struct smb2_hdr *rsp_hdr;
@@ -813,6 +862,8 @@ static void __smb2_lease_break_noti(struct work_struct *wk)
out:
ksmbd_free_work_struct(work);
+ ksmbd_conn_r_count_dec(conn);
+ ksmbd_conn_put(conn);
}
/**
@@ -825,7 +876,6 @@ static void __smb2_lease_break_noti(struct work_struct *wk)
static int smb2_lease_break_noti(struct oplock_info *opinfo)
{
struct ksmbd_conn *conn = opinfo->conn;
- struct list_head *tmp, *t;
struct ksmbd_work *work;
struct lease_break_info *br_info;
struct lease *lease = opinfo->o_lease;
@@ -849,20 +899,11 @@ static int smb2_lease_break_noti(struct oplock_info *opinfo)
memcpy(br_info->lease_key, lease->lease_key, SMB2_LEASE_KEY_SIZE);
work->request_buf = (char *)br_info;
- work->conn = conn;
+ work->conn = ksmbd_conn_get(conn);
work->sess = opinfo->sess;
+ ksmbd_conn_r_count_inc(conn);
if (opinfo->op_state == OPLOCK_ACK_WAIT) {
- list_for_each_safe(tmp, t, &opinfo->interim_list) {
- struct ksmbd_work *in_work;
-
- in_work = list_entry(tmp, struct ksmbd_work,
- interim_entry);
- setup_async_work(in_work, NULL, NULL);
- smb2_send_interim_resp(in_work, STATUS_PENDING);
- list_del_init(&in_work->interim_entry);
- release_async_work(in_work);
- }
INIT_WORK(&work->work, __smb2_lease_break_noti);
ksmbd_queue_work(work);
wait_for_break_ack(opinfo);
@@ -893,7 +934,8 @@ static void wait_lease_breaking(struct oplock_info *opinfo)
}
}
-static int oplock_break(struct oplock_info *brk_opinfo, int req_op_level)
+static int oplock_break(struct oplock_info *brk_opinfo, int req_op_level,
+ struct ksmbd_work *in_work)
{
int err = 0;
@@ -936,9 +978,19 @@ static int oplock_break(struct oplock_info *brk_opinfo, int req_op_level)
}
if (lease->state & (SMB2_LEASE_WRITE_CACHING_LE |
- SMB2_LEASE_HANDLE_CACHING_LE))
- brk_opinfo->op_state = OPLOCK_ACK_WAIT;
- else
+ SMB2_LEASE_HANDLE_CACHING_LE)) {
+ if (in_work) {
+ setup_async_work(in_work, NULL, NULL);
+ smb2_send_interim_resp(in_work, STATUS_PENDING);
+ release_async_work(in_work);
+ }
+
+ if (!oplock_break_set_ack_wait(brk_opinfo)) {
+ atomic_dec_if_positive(&brk_opinfo->breaking_cnt);
+ wake_up_oplock_break(brk_opinfo);
+ return -ENOENT;
+ }
+ } else
atomic_dec(&brk_opinfo->breaking_cnt);
} else {
err = oplock_break_pending(brk_opinfo, req_op_level);
@@ -946,8 +998,12 @@ static int oplock_break(struct oplock_info *brk_opinfo, int req_op_level)
return err < 0 ? err : 0;
if (brk_opinfo->level == SMB2_OPLOCK_LEVEL_BATCH ||
- brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE)
- brk_opinfo->op_state = OPLOCK_ACK_WAIT;
+ brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) {
+ if (!oplock_break_set_ack_wait(brk_opinfo)) {
+ wake_up_oplock_break(brk_opinfo);
+ return -ENOENT;
+ }
+ }
}
if (brk_opinfo->is_lease)
@@ -1141,11 +1197,12 @@ void smb_send_parent_lease_break_noti(struct ksmbd_file *fp,
atomic_inc(&opinfo->conn->r_count);
if (ksmbd_conn_releasing(opinfo->conn)) {
atomic_dec(&opinfo->conn->r_count);
+ opinfo_put(opinfo);
continue;
}
read_unlock(&p_ci->m_lock);
- oplock_break(opinfo, SMB2_OPLOCK_LEVEL_NONE);
+ oplock_break(opinfo, SMB2_OPLOCK_LEVEL_NONE, NULL);
opinfo_conn_put(opinfo);
read_lock(&p_ci->m_lock);
}
@@ -1183,10 +1240,11 @@ void smb_lazy_parent_lease_break_close(struct ksmbd_file *fp)
atomic_inc(&opinfo->conn->r_count);
if (ksmbd_conn_releasing(opinfo->conn)) {
atomic_dec(&opinfo->conn->r_count);
+ opinfo_put(opinfo);
continue;
}
read_unlock(&p_ci->m_lock);
- oplock_break(opinfo, SMB2_OPLOCK_LEVEL_NONE);
+ oplock_break(opinfo, SMB2_OPLOCK_LEVEL_NONE, NULL);
opinfo_conn_put(opinfo);
read_lock(&p_ci->m_lock);
}
@@ -1287,8 +1345,7 @@ int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid,
goto op_break_not_needed;
}
- list_add(&work->interim_entry, &prev_opinfo->interim_list);
- err = oplock_break(prev_opinfo, SMB2_OPLOCK_LEVEL_II);
+ err = oplock_break(prev_opinfo, SMB2_OPLOCK_LEVEL_II, work);
opinfo_conn_put(prev_opinfo);
if (err == -ENOENT)
goto set_lev;
@@ -1357,8 +1414,7 @@ static void smb_break_all_write_oplock(struct ksmbd_work *work,
}
brk_opinfo->open_trunc = is_trunc;
- list_add(&work->interim_entry, &brk_opinfo->interim_list);
- oplock_break(brk_opinfo, SMB2_OPLOCK_LEVEL_II);
+ oplock_break(brk_opinfo, SMB2_OPLOCK_LEVEL_II, work);
opinfo_conn_put(brk_opinfo);
}
@@ -1391,6 +1447,7 @@ void smb_break_all_levII_oplock(struct ksmbd_work *work, struct ksmbd_file *fp,
atomic_inc(&brk_op->conn->r_count);
if (ksmbd_conn_releasing(brk_op->conn)) {
atomic_dec(&brk_op->conn->r_count);
+ opinfo_put(brk_op);
continue;
}
@@ -1421,7 +1478,7 @@ void smb_break_all_levII_oplock(struct ksmbd_work *work, struct ksmbd_file *fp,
SMB2_LEASE_KEY_SIZE))
goto next;
brk_op->open_trunc = is_trunc;
- oplock_break(brk_op, SMB2_OPLOCK_LEVEL_NONE);
+ oplock_break(brk_op, SMB2_OPLOCK_LEVEL_NONE, NULL);
next:
opinfo_conn_put(brk_op);
rcu_read_lock();
diff --git a/fs/smb/server/oplock.h b/fs/smb/server/oplock.h
index 0e2b1d63ca26..a631006fc50d 100644
--- a/fs/smb/server/oplock.h
+++ b/fs/smb/server/oplock.h
@@ -66,6 +66,7 @@ struct oplock_info {
struct ksmbd_file *o_fp;
int level;
int op_state;
+ spinlock_t state_lock;
unsigned long pending_break;
u64 fid;
atomic_t breaking_cnt;
@@ -74,7 +75,6 @@ struct oplock_info {
bool is_lease;
bool open_trunc; /* truncate on open */
struct lease *o_lease;
- struct list_head interim_list;
struct list_head op_entry;
struct list_head lease_entry;
wait_queue_head_t oplock_q; /* Other server threads */
diff --git a/fs/smb/server/server.c b/fs/smb/server/server.c
index a5e9bb40e219..08fe608ac75a 100644
--- a/fs/smb/server/server.c
+++ b/fs/smb/server/server.c
@@ -290,13 +290,8 @@ static void handle_ksmbd_work(struct work_struct *wk)
ksmbd_conn_try_dequeue_request(work);
ksmbd_free_work_struct(work);
- /*
- * Checking waitqueue to dropping pending requests on
- * disconnection. waitqueue_active is safe because it
- * uses atomic operation for condition.
- */
- if (!atomic_dec_return(&conn->r_count) && waitqueue_active(&conn->r_count_q))
- wake_up(&conn->r_count_q);
+ atomic_dec(&conn->mux_smb_requests);
+ ksmbd_conn_r_count_dec(conn);
}
/**
@@ -311,6 +306,15 @@ static int queue_ksmbd_work(struct ksmbd_conn *conn)
struct ksmbd_work *work;
int err;
+ err = ksmbd_init_smb_server(conn);
+ if (err)
+ return 0;
+
+ if (atomic_inc_return(&conn->mux_smb_requests) >= conn->vals->max_credits) {
+ atomic_dec_return(&conn->mux_smb_requests);
+ return -ENOSPC;
+ }
+
work = ksmbd_alloc_work_struct();
if (!work) {
pr_err("allocation for work failed\n");
@@ -321,14 +325,8 @@ static int queue_ksmbd_work(struct ksmbd_conn *conn)
work->request_buf = conn->request_buf;
conn->request_buf = NULL;
- err = ksmbd_init_smb_server(work);
- if (err) {
- ksmbd_free_work_struct(work);
- return 0;
- }
-
ksmbd_conn_enqueue_request(work);
- atomic_inc(&conn->r_count);
+ ksmbd_conn_r_count_inc(conn);
/* update activity on connection */
conn->last_active = jiffies;
INIT_WORK(&work->work, handle_ksmbd_work);
diff --git a/fs/smb/server/smb2ops.c b/fs/smb/server/smb2ops.c
index c1ed62cc8296..e5accc715348 100644
--- a/fs/smb/server/smb2ops.c
+++ b/fs/smb/server/smb2ops.c
@@ -283,7 +283,6 @@ int init_smb3_11_server(struct ksmbd_conn *conn)
if (server_conf.flags & KSMBD_GLOBAL_FLAG_SMB3_MULTICHANNEL)
conn->vals->req_capabilities |= SMB2_GLOBAL_CAP_MULTI_CHANNEL;
- INIT_LIST_HEAD(&conn->preauth_sess_table);
return 0;
}
diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c
index f30866efe2f4..bead3fb3e2e3 100644
--- a/fs/smb/server/smb2pdu.c
+++ b/fs/smb/server/smb2pdu.c
@@ -78,7 +78,13 @@ static inline bool check_session_id(struct ksmbd_conn *conn, u64 id)
struct channel *lookup_chann_list(struct ksmbd_session *sess, struct ksmbd_conn *conn)
{
- return xa_load(&sess->ksmbd_chann_list, (long)conn);
+ struct channel *chann;
+
+ down_read(&sess->chann_lock);
+ chann = xa_load(&sess->ksmbd_chann_list, (long)conn);
+ up_read(&sess->chann_lock);
+
+ return chann;
}
/**
@@ -665,6 +671,11 @@ smb2_get_name(const char *src, const int maxlen, struct nls_table *local_nls)
return name;
}
+ if (*name == '\0') {
+ kfree(name);
+ return ERR_PTR(-EINVAL);
+ }
+
ksmbd_conv_path_to_unix(name);
ksmbd_strip_last_slash(name);
return name;
@@ -1269,7 +1280,7 @@ int smb2_handle_negotiate(struct ksmbd_work *work)
}
conn->srv_sec_mode = le16_to_cpu(rsp->SecurityMode);
- ksmbd_conn_set_need_negotiate(conn);
+ ksmbd_conn_set_need_setup(conn);
err_out:
if (rc && rsp->hdr.Status == STATUS_SUCCESS)
@@ -1290,6 +1301,9 @@ static int alloc_preauth_hash(struct ksmbd_session *sess,
if (sess->Preauth_HashValue)
return 0;
+ if (!conn->preauth_info)
+ return -ENOMEM;
+
sess->Preauth_HashValue = kmemdup(conn->preauth_info->Preauth_HashValue,
PREAUTH_HASHVALUE_SIZE, GFP_KERNEL);
if (!sess->Preauth_HashValue)
@@ -1572,7 +1586,9 @@ static int ntlm_authenticate(struct ksmbd_work *work,
return -ENOMEM;
chann->conn = conn;
+ down_write(&sess->chann_lock);
xa_store(&sess->ksmbd_chann_list, (long)conn, chann, GFP_KERNEL);
+ up_write(&sess->chann_lock);
}
}
@@ -1609,7 +1625,7 @@ static int krb5_authenticate(struct ksmbd_work *work,
in_len = le16_to_cpu(req->SecurityBufferLength);
out_blob = (char *)&rsp->hdr.ProtocolId +
le16_to_cpu(rsp->SecurityBufferOffset);
- out_len = work->response_sz -
+ out_len = work->response_sz - work->next_smb2_rsp_hdr_off -
(le16_to_cpu(rsp->SecurityBufferOffset) + 4);
retval = ksmbd_krb5_authenticate(sess, in_blob, in_len,
@@ -1665,7 +1681,9 @@ static int krb5_authenticate(struct ksmbd_work *work,
return -ENOMEM;
chann->conn = conn;
+ down_write(&sess->chann_lock);
xa_store(&sess->ksmbd_chann_list, (long)conn, chann, GFP_KERNEL);
+ up_write(&sess->chann_lock);
}
}
@@ -1704,6 +1722,11 @@ int smb2_sess_setup(struct ksmbd_work *work)
ksmbd_debug(SMB, "Received request for session setup\n");
+ if (!ksmbd_conn_need_setup(conn) && !ksmbd_conn_good(conn)) {
+ work->send_no_response = 1;
+ return rc;
+ }
+
WORK_BUFFERS(work, req, rsp);
rsp->StructureSize = cpu_to_le16(9);
@@ -1949,7 +1972,7 @@ int smb2_sess_setup(struct ksmbd_work *work)
if (try_delay) {
ksmbd_conn_set_need_reconnect(conn);
ssleep(5);
- ksmbd_conn_set_need_negotiate(conn);
+ ksmbd_conn_set_need_setup(conn);
}
}
smb2_set_err_rsp(work);
@@ -1985,6 +2008,7 @@ int smb2_tree_connect(struct ksmbd_work *work)
struct ksmbd_session *sess = work->sess;
char *treename = NULL, *name = NULL;
struct ksmbd_tree_conn_status status;
+ struct ksmbd_tree_connect *tree_conn = NULL;
struct ksmbd_share_config *share;
int rc = -EINVAL;
@@ -2009,9 +2033,10 @@ int smb2_tree_connect(struct ksmbd_work *work)
name, treename);
status = ksmbd_tree_conn_connect(work, name);
- if (status.ret == KSMBD_TREE_CONN_STATUS_OK)
+ if (status.ret == KSMBD_TREE_CONN_STATUS_OK) {
+ tree_conn = status.tree_conn;
rsp->hdr.Id.SyncId.TreeId = cpu_to_le32(status.tree_conn->id);
- else
+ } else
goto out_err1;
share = status.tree_conn->share_conf;
@@ -2043,8 +2068,13 @@ int smb2_tree_connect(struct ksmbd_work *work)
status.tree_conn->posix_extensions = true;
write_lock(&sess->tree_conns_lock);
- status.tree_conn->t_state = TREE_CONNECTED;
+ if (status.tree_conn->t_state == TREE_DISCONNECTED)
+ status.ret = KSMBD_TREE_CONN_STATUS_ERROR;
+ else
+ status.tree_conn->t_state = TREE_CONNECTED;
write_unlock(&sess->tree_conns_lock);
+ if (status.ret != KSMBD_TREE_CONN_STATUS_OK)
+ goto out_err1;
rsp->StructureSize = cpu_to_le16(16);
out_err1:
rsp->Capabilities = 0;
@@ -2053,8 +2083,13 @@ int smb2_tree_connect(struct ksmbd_work *work)
rsp->ShareFlags = SMB2_SHAREFLAG_MANUAL_CACHING;
rc = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_tree_connect_rsp));
- if (rc)
+ if (rc) {
+ if (status.ret == KSMBD_TREE_CONN_STATUS_OK) {
+ ksmbd_tree_conn_disconnect(sess, status.tree_conn);
+ status.tree_conn = NULL;
+ }
status.ret = KSMBD_TREE_CONN_STATUS_NOMEM;
+ }
if (!IS_ERR(treename))
kfree(treename);
@@ -2090,6 +2125,9 @@ int smb2_tree_connect(struct ksmbd_work *work)
if (status.ret != KSMBD_TREE_CONN_STATUS_OK)
smb2_set_err_rsp(work);
+ if (tree_conn)
+ ksmbd_tree_connect_put(tree_conn);
+
return rc;
}
@@ -2191,17 +2229,6 @@ int smb2_tree_disconnect(struct ksmbd_work *work)
ksmbd_close_tree_conn_fds(work);
- write_lock(&sess->tree_conns_lock);
- if (tcon->t_state == TREE_DISCONNECTED) {
- write_unlock(&sess->tree_conns_lock);
- rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED;
- err = -ENOENT;
- goto err_out;
- }
-
- tcon->t_state = TREE_DISCONNECTED;
- write_unlock(&sess->tree_conns_lock);
-
err = ksmbd_tree_conn_disconnect(sess, tcon);
if (err) {
rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED;
@@ -2268,7 +2295,7 @@ int smb2_session_logoff(struct ksmbd_work *work)
sess->state = SMB2_SESSION_EXPIRED;
up_write(&conn->session_lock);
- ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_NEGOTIATE);
+ ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_SETUP);
rsp->StructureSize = cpu_to_le16(4);
err = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_logoff_rsp));
@@ -3003,6 +3030,7 @@ int smb2_open(struct ksmbd_work *work)
if (file_present && !(req->CreateOptions & FILE_DELETE_ON_CLOSE_LE)) {
rc = smb_check_perm_dacl(conn, &path, &daccess,
+ req->DesiredAccess,
sess->user->uid);
if (rc)
goto err_out;
@@ -3168,20 +3196,24 @@ int smb2_open(struct ksmbd_work *work)
KSMBD_SHARE_FLAG_ACL_XATTR)) {
struct smb_fattr fattr;
struct smb_ntsd *pntsd;
- int pntsd_size, ace_num = 0;
+ int pntsd_size;
+ size_t scratch_len;
ksmbd_acls_fattr(&fattr, user_ns, inode);
- if (fattr.cf_acls)
- ace_num = fattr.cf_acls->a_count;
- if (fattr.cf_dacls)
- ace_num += fattr.cf_dacls->a_count;
-
- pntsd = kmalloc(sizeof(struct smb_ntsd) +
- sizeof(struct smb_sid) * 3 +
- sizeof(struct smb_acl) +
- sizeof(struct smb_ace) * ace_num * 2,
- GFP_KERNEL);
+ scratch_len = smb_acl_sec_desc_scratch_len(&fattr,
+ NULL, 0,
+ OWNER_SECINFO | GROUP_SECINFO |
+ DACL_SECINFO);
+ if (!scratch_len || scratch_len == SIZE_MAX) {
+ rc = -EFBIG;
+ posix_acl_release(fattr.cf_acls);
+ posix_acl_release(fattr.cf_dacls);
+ goto err_out;
+ }
+
+ pntsd = kvzalloc(scratch_len, GFP_KERNEL);
if (!pntsd) {
+ rc = -ENOMEM;
posix_acl_release(fattr.cf_acls);
posix_acl_release(fattr.cf_dacls);
goto err_out;
@@ -3196,7 +3228,7 @@ int smb2_open(struct ksmbd_work *work)
posix_acl_release(fattr.cf_acls);
posix_acl_release(fattr.cf_dacls);
if (rc) {
- kfree(pntsd);
+ kvfree(pntsd);
goto err_out;
}
@@ -3206,7 +3238,7 @@ int smb2_open(struct ksmbd_work *work)
pntsd,
pntsd_size,
false);
- kfree(pntsd);
+ kvfree(pntsd);
if (rc)
pr_err("failed to store ntacl in xattr : %d\n",
rc);
@@ -3483,8 +3515,12 @@ int smb2_open(struct ksmbd_work *work)
rsp->hdr.Status = STATUS_INVALID_PARAMETER;
else if (rc == -EOPNOTSUPP)
rsp->hdr.Status = STATUS_NOT_SUPPORTED;
- else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV)
- rsp->hdr.Status = STATUS_ACCESS_DENIED;
+ else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV) {
+ if (req->DesiredAccess & FILE_ACCESS_SYSTEM_SECURITY_LE)
+ rsp->hdr.Status = STATUS_PRIVILEGE_NOT_HELD;
+ else
+ rsp->hdr.Status = STATUS_ACCESS_DENIED;
+ }
else if (rc == -ENOENT)
rsp->hdr.Status = STATUS_OBJECT_NAME_INVALID;
else if (rc == -EPERM)
@@ -4988,8 +5024,9 @@ static int smb2_get_info_file(struct ksmbd_work *work,
if (test_share_config_flag(work->tcon->share_conf,
KSMBD_SHARE_FLAG_PIPE)) {
/* smb2 info file called for pipe */
- return smb2_get_info_file_pipe(work->sess, req, rsp,
+ rc = smb2_get_info_file_pipe(work->sess, req, rsp,
work->response_buf);
+ goto iov_pin_out;
}
if (work->next_smb2_rcv_hdr_off) {
@@ -5089,6 +5126,12 @@ static int smb2_get_info_file(struct ksmbd_work *work,
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
rsp, work->response_buf);
ksmbd_fd_put(work, fp);
+
+iov_pin_out:
+ if (!rc)
+ rc = ksmbd_iov_pin_rsp(work, (void *)rsp,
+ offsetof(struct smb2_query_info_rsp, Buffer) +
+ le32_to_cpu(rsp->OutputBufferLength));
return rc;
}
@@ -5096,7 +5139,6 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
struct smb2_query_info_req *req,
struct smb2_query_info_rsp *rsp)
{
- struct ksmbd_session *sess = work->sess;
struct ksmbd_conn *conn = work->conn;
struct ksmbd_share_config *share = work->tcon->share_conf;
int fsinfoclass = 0;
@@ -5225,11 +5267,15 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
struct object_id_info *info;
info = (struct object_id_info *)(rsp->Buffer);
+ memset(info, 0, sizeof(*info));
- if (!user_guest(sess->user))
- memcpy(info->objid, user_passkey(sess->user), 16);
- else
- memset(info->objid, 0, 16);
+ if (!uuid_is_null(&path.mnt->mnt_sb->s_uuid))
+ memcpy(info->objid, path.mnt->mnt_sb->s_uuid.b,
+ sizeof(path.mnt->mnt_sb->s_uuid));
+ else {
+ memset(info->objid, 0, sizeof(info->objid));
+ memcpy(info->objid, &stfs.f_fsid, sizeof(stfs.f_fsid));
+ }
info->extended_info.magic = cpu_to_le32(EXTENDED_INFO_MAGIC);
info->extended_info.version = cpu_to_le32(1);
@@ -5276,6 +5322,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->FreeSpaceStopFiltering = 0;
info->DefaultQuotaThreshold = cpu_to_le64(SMB2_NO_FID);
info->DefaultQuotaLimit = cpu_to_le64(SMB2_NO_FID);
+ info->FileSystemControlFlags = 0;
info->Padding = 0;
rsp->OutputBufferLength = cpu_to_le32(48);
break;
@@ -5297,6 +5344,9 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
info->UserBlocksAvail = cpu_to_le64(stfs.f_bavail);
info->TotalFileNodes = cpu_to_le64(stfs.f_files);
info->FreeFileNodes = cpu_to_le64(stfs.f_ffree);
+ info->FileSysIdentifier =
+ cpu_to_le64((u64)(u32)stfs.f_fsid.val[1] << 32 |
+ (u32)stfs.f_fsid.val[0]);
rsp->OutputBufferLength = cpu_to_le32(56);
}
break;
@@ -5308,6 +5358,11 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work,
rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
rsp, work->response_buf);
path_put(&path);
+
+ if (!rc)
+ rc = ksmbd_iov_pin_rsp(work, (void *)rsp,
+ offsetof(struct smb2_query_info_rsp, Buffer) +
+ le32_to_cpu(rsp->OutputBufferLength));
return rc;
}
@@ -5317,13 +5372,14 @@ static int smb2_get_info_sec(struct ksmbd_work *work,
{
struct ksmbd_file *fp;
struct user_namespace *user_ns;
- struct smb_ntsd *pntsd = (struct smb_ntsd *)rsp->Buffer, *ppntsd = NULL;
+ struct smb_ntsd *pntsd = NULL, *ppntsd = NULL;
struct smb_fattr fattr = {{0}};
struct inode *inode;
__u32 secdesclen = 0;
unsigned int id = KSMBD_NO_FID, pid = KSMBD_NO_FID;
int addition_info = le32_to_cpu(req->AdditionalInformation);
- int rc = 0, ppntsd_size = 0;
+ int rc = 0, ppntsd_size = 0, max_len;
+ size_t scratch_len = 0;
if (addition_info & ~(OWNER_SECINFO | GROUP_SECINFO | DACL_SECINFO |
PROTECTED_DACL_SECINFO |
@@ -5331,6 +5387,11 @@ static int smb2_get_info_sec(struct ksmbd_work *work,
ksmbd_debug(SMB, "Unsupported addition info: 0x%x)\n",
addition_info);
+ pntsd = kzalloc(ALIGN(sizeof(struct smb_ntsd), 8),
+ GFP_KERNEL);
+ if (!pntsd)
+ return -ENOMEM;
+
pntsd->revision = cpu_to_le16(1);
pntsd->type = cpu_to_le16(SELF_RELATIVE | DACL_PROTECTED);
pntsd->osidoffset = 0;
@@ -5339,9 +5400,7 @@ static int smb2_get_info_sec(struct ksmbd_work *work,
pntsd->dacloffset = 0;
secdesclen = sizeof(struct smb_ntsd);
- rsp->OutputBufferLength = cpu_to_le32(secdesclen);
-
- return 0;
+ goto iov_pin;
}
if (work->next_smb2_rcv_hdr_off) {
@@ -5373,18 +5432,58 @@ static int smb2_get_info_sec(struct ksmbd_work *work,
&ppntsd);
/* Check if sd buffer size exceeds response buffer size */
- if (smb2_resp_buf_len(work, 8) > ppntsd_size)
- rc = build_sec_desc(user_ns, pntsd, ppntsd, ppntsd_size,
- addition_info, &secdesclen, &fattr);
+ max_len = smb2_calc_max_out_buf_len(work,
+ offsetof(struct smb2_query_info_rsp, Buffer),
+ le32_to_cpu(req->OutputBufferLength));
+ if (max_len < 0) {
+ rc = -EINVAL;
+ goto release_acl;
+ }
+
+ scratch_len = smb_acl_sec_desc_scratch_len(&fattr, ppntsd,
+ ppntsd_size, addition_info);
+ if (!scratch_len || scratch_len == SIZE_MAX) {
+ rc = -EFBIG;
+ goto release_acl;
+ }
+
+ pntsd = kvzalloc(scratch_len, GFP_KERNEL);
+ if (!pntsd) {
+ rc = -ENOMEM;
+ goto release_acl;
+ }
+
+ rc = build_sec_desc(user_ns, pntsd, ppntsd, ppntsd_size,
+ addition_info, &secdesclen, &fattr);
+
+release_acl:
posix_acl_release(fattr.cf_acls);
posix_acl_release(fattr.cf_dacls);
kfree(ppntsd);
ksmbd_fd_put(work, fp);
+
+ if (!rc && ALIGN(secdesclen, 8) > scratch_len)
+ rc = -EFBIG;
if (rc)
- return rc;
+ goto err_out;
+iov_pin:
rsp->OutputBufferLength = cpu_to_le32(secdesclen);
- return 0;
+ rc = buffer_check_err(le32_to_cpu(req->OutputBufferLength),
+ rsp, work->response_buf);
+ if (rc)
+ goto err_out;
+
+ rc = ksmbd_iov_pin_rsp_read(work, (void *)rsp,
+ offsetof(struct smb2_query_info_rsp, Buffer),
+ pntsd, secdesclen);
+err_out:
+ if (rc) {
+ rsp->OutputBufferLength = 0;
+ kvfree(pntsd);
+ }
+
+ return rc;
}
/**
@@ -5408,6 +5507,9 @@ int smb2_query_info(struct ksmbd_work *work)
goto err_out;
}
+ rsp->StructureSize = cpu_to_le16(9);
+ rsp->OutputBufferOffset = cpu_to_le16(72);
+
switch (req->InfoType) {
case SMB2_O_INFO_FILE:
ksmbd_debug(SMB, "GOT SMB2_O_INFO_FILE\n");
@@ -5428,14 +5530,6 @@ int smb2_query_info(struct ksmbd_work *work)
}
ksmbd_revert_fsids(work);
- if (!rc) {
- rsp->StructureSize = cpu_to_le16(9);
- rsp->OutputBufferOffset = cpu_to_le16(72);
- rc = ksmbd_iov_pin_rsp(work, (void *)rsp,
- offsetof(struct smb2_query_info_rsp, Buffer) +
- le32_to_cpu(rsp->OutputBufferLength));
- }
-
err_out:
if (rc < 0) {
if (rc == -EACCES)
@@ -5446,6 +5540,8 @@ int smb2_query_info(struct ksmbd_work *work)
rsp->hdr.Status = STATUS_UNEXPECTED_IO_ERROR;
else if (rc == -ENOMEM)
rsp->hdr.Status = STATUS_INSUFFICIENT_RESOURCES;
+ else if (rc == -EINVAL && rsp->hdr.Status == 0)
+ rsp->hdr.Status = STATUS_INVALID_PARAMETER;
else if (rc == -EOPNOTSUPP || rsp->hdr.Status == 0)
rsp->hdr.Status = STATUS_INVALID_INFO_CLASS;
smb2_set_err_rsp(work);
@@ -6285,13 +6381,18 @@ static noinline int smb2_read_pipe(struct ksmbd_work *work)
}
aux_payload_buf =
- kvmalloc(rpc_resp->payload_sz, GFP_KERNEL);
+ kvmalloc(ALIGN(rpc_resp->payload_sz, 8),
+ GFP_KERNEL);
if (!aux_payload_buf) {
err = -ENOMEM;
goto out;
}
memcpy(aux_payload_buf, rpc_resp->payload, rpc_resp->payload_sz);
+ if (rpc_resp->payload_sz & 7)
+ memset(aux_payload_buf + rpc_resp->payload_sz, 0,
+ ALIGN(rpc_resp->payload_sz, 8) -
+ rpc_resp->payload_sz);
nbytes = rpc_resp->payload_sz;
err = ksmbd_iov_pin_rsp_read(work, (void *)rsp,
@@ -6990,6 +7091,19 @@ static void smb2_remove_blocked_lock(void **argv)
wake_up(&flock->fl_wait);
}
+static void smb2_free_lock(struct file_lock *flock)
+{
+ ksmbd_vfs_posix_lock_unblock(flock);
+ locks_free_lock(flock);
+}
+
+static void smb2_free_blocked_lock(struct file_lock *flock)
+{
+ ksmbd_vfs_posix_lock_unblock(flock);
+ wake_up(&flock->fl_wait);
+ locks_free_lock(flock);
+}
+
static inline bool lock_defer_pending(struct file_lock *fl)
{
/* check pending lock waiters */
@@ -7143,14 +7257,14 @@ int smb2_lock(struct ksmbd_work *work)
cmp_lock->end == smb_lock->end &&
!lock_defer_pending(cmp_lock->fl)) {
nolock = 0;
- list_del(&cmp_lock->flist);
- list_del(&cmp_lock->clist);
+ list_del_init(&cmp_lock->flist);
+ list_del_init(&cmp_lock->clist);
cmp_lock->conn = NULL;
spin_unlock(&conn->llist_lock);
up_read(&conn_list_lock);
ksmbd_conn_put(conn);
- locks_free_lock(cmp_lock->fl);
+ smb2_free_lock(cmp_lock->fl);
kfree(cmp_lock);
goto out_check_cl;
}
@@ -7232,11 +7346,12 @@ int smb2_lock(struct ksmbd_work *work)
ksmbd_debug(SMB,
"would have to wait for getting lock\n");
- list_add(&smb_lock->llist, &rollback_list);
argv = kmalloc(sizeof(void *), GFP_KERNEL);
if (!argv) {
err = -ENOMEM;
+ smb2_free_blocked_lock(flock);
+ kfree(smb_lock);
goto out;
}
argv[0] = flock;
@@ -7247,8 +7362,11 @@ int smb2_lock(struct ksmbd_work *work)
if (rc) {
kfree(argv);
err = -ENOMEM;
+ smb2_free_blocked_lock(flock);
+ kfree(smb_lock);
goto out;
}
+ list_add(&smb_lock->llist, &rollback_list);
spin_lock(&fp->f_lock);
list_add(&work->fp_entry, &fp->blocked_works);
spin_unlock(&fp->f_lock);
@@ -7281,16 +7399,10 @@ int smb2_lock(struct ksmbd_work *work)
rsp->hdr.Status =
STATUS_RANGE_NOT_LOCKED;
kfree(smb_lock);
- goto out2;
+ /* rollback_list may still hold earlier grants */
+ goto out;
} else if (!rc) {
list_add(&smb_lock->llist, &rollback_list);
- smb_lock->conn = ksmbd_conn_get(work->conn);
- spin_lock(&work->conn->llist_lock);
- list_add_tail(&smb_lock->clist,
- &work->conn->lock_list);
- list_add_tail(&smb_lock->flist,
- &fp->lock_list);
- spin_unlock(&work->conn->llist_lock);
ksmbd_debug(SMB, "successful in taking lock\n");
} else {
locks_free_lock(flock);
@@ -7312,6 +7424,19 @@ int smb2_lock(struct ksmbd_work *work)
if (err)
goto out;
+ /* publish only once the whole batch has committed */
+ if (!list_empty(&rollback_list)) {
+ spin_lock(&work->conn->llist_lock);
+ list_for_each_entry(smb_lock, &rollback_list, llist) {
+ smb_lock->conn = ksmbd_conn_get(work->conn);
+ list_add_tail(&smb_lock->clist,
+ &work->conn->lock_list);
+ list_add_tail(&smb_lock->flist,
+ &fp->lock_list);
+ }
+ spin_unlock(&work->conn->llist_lock);
+ }
+
ksmbd_fd_put(work, fp);
return 0;
@@ -7339,16 +7464,7 @@ int smb2_lock(struct ksmbd_work *work)
}
list_del(&smb_lock->llist);
- conn = smb_lock->conn;
- spin_lock(&conn->llist_lock);
- if (!list_empty(&smb_lock->flist))
- list_del(&smb_lock->flist);
- list_del(&smb_lock->clist);
- smb_lock->conn = NULL;
- spin_unlock(&conn->llist_lock);
- ksmbd_conn_put(conn);
-
- locks_free_lock(smb_lock->fl);
+ smb2_free_lock(smb_lock->fl);
if (rlock)
locks_free_lock(rlock);
kfree(smb_lock);
@@ -8286,7 +8402,10 @@ static void smb20_oplock_break_ack(struct ksmbd_work *work)
smb2_set_err_rsp(work);
out:
- opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING)
+ opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
wake_up_interruptible_all(&opinfo->oplock_q);
opinfo_put(opinfo);
ksmbd_fd_put(work, fp);
@@ -8432,9 +8551,12 @@ static void smb21_lease_break_ack(struct ksmbd_work *work)
smb2_set_err_rsp(work);
}
- opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_lock(&opinfo->state_lock);
+ if (opinfo->op_state != OPLOCK_CLOSING)
+ opinfo->op_state = OPLOCK_STATE_NONE;
+ spin_unlock(&opinfo->state_lock);
wake_up_interruptible_all(&opinfo->oplock_q);
- atomic_dec(&opinfo->breaking_cnt);
+ atomic_dec_if_positive(&opinfo->breaking_cnt);
wake_up_interruptible_all(&opinfo->oplock_brk);
opinfo_put(opinfo);
}
diff --git a/fs/smb/server/smb2pdu.h b/fs/smb/server/smb2pdu.h
index 6011fca98045..7518aa42f11e 100644
--- a/fs/smb/server/smb2pdu.h
+++ b/fs/smb/server/smb2pdu.h
@@ -146,14 +146,6 @@ struct create_mxac_rsp {
__le32 MaximalAccess;
} __packed;
-struct create_disk_id_rsp {
- struct create_context ccontext;
- __u8 Name[8];
- __le64 DiskFileId;
- __le64 VolumeId;
- __u8 Reserved[16];
-} __packed;
-
/* equivalent of the contents of SMB3.1.1 POSIX open context response */
struct create_posix_rsp {
struct create_context ccontext;
diff --git a/fs/smb/server/smb_common.c b/fs/smb/server/smb_common.c
index bcecbd67189a..0350697c9093 100644
--- a/fs/smb/server/smb_common.c
+++ b/fs/smb/server/smb_common.c
@@ -388,6 +388,10 @@ static struct smb_version_ops smb1_server_ops = {
.set_rsp_status = set_smb1_rsp_status,
};
+static struct smb_version_values smb1_server_values = {
+ .max_credits = SMB2_MAX_CREDITS,
+};
+
static int smb1_negotiate(struct ksmbd_work *work)
{
return ksmbd_smb_negotiate_common(work, SMB_COM_NEGOTIATE);
@@ -399,18 +403,18 @@ static struct smb_version_cmds smb1_server_cmds[1] = {
static int init_smb1_server(struct ksmbd_conn *conn)
{
+ conn->vals = &smb1_server_values;
conn->ops = &smb1_server_ops;
conn->cmds = smb1_server_cmds;
conn->max_cmds = ARRAY_SIZE(smb1_server_cmds);
return 0;
}
-int ksmbd_init_smb_server(struct ksmbd_work *work)
+int ksmbd_init_smb_server(struct ksmbd_conn *conn)
{
- struct ksmbd_conn *conn = work->conn;
__le32 proto;
- proto = *(__le32 *)((struct smb_hdr *)work->request_buf)->Protocol;
+ proto = *(__le32 *)((struct smb_hdr *)conn->request_buf)->Protocol;
if (conn->need_neg == false) {
if (proto == SMB1_PROTO_NUMBER)
return -EINVAL;
diff --git a/fs/smb/server/smb_common.h b/fs/smb/server/smb_common.h
index d4291834b04f..75de404a9efc 100644
--- a/fs/smb/server/smb_common.h
+++ b/fs/smb/server/smb_common.h
@@ -391,7 +391,7 @@ bool ksmbd_smb_request(struct ksmbd_conn *conn);
int ksmbd_lookup_dialect_by_id(__le16 *cli_dialects, __le16 dialects_count);
-int ksmbd_init_smb_server(struct ksmbd_work *work);
+int ksmbd_init_smb_server(struct ksmbd_conn *conn);
struct ksmbd_kstat;
int ksmbd_populate_dot_dotdot_entries(struct ksmbd_work *work,
diff --git a/fs/smb/server/smbacl.c b/fs/smb/server/smbacl.c
index 5cb020fc0105..b0bf27cab2b3 100644
--- a/fs/smb/server/smbacl.c
+++ b/fs/smb/server/smbacl.c
@@ -387,6 +387,7 @@ static void parse_dacl(struct user_namespace *user_ns,
{
int i, ret;
int num_aces = 0;
+ u16 dacl_size;
unsigned int acl_size;
char *acl_base;
struct smb_ace **ppace;
@@ -416,7 +417,11 @@ static void parse_dacl(struct user_namespace *user_ns,
if (num_aces <= 0)
return;
- if (num_aces > (le16_to_cpu(pdacl->size) - sizeof(struct smb_acl)) /
+ dacl_size = le16_to_cpu(pdacl->size);
+ if (dacl_size < sizeof(struct smb_acl))
+ return;
+
+ if (num_aces > (dacl_size - sizeof(struct smb_acl)) /
(offsetof(struct smb_ace, sid) +
offsetof(struct smb_sid, sub_auth) + sizeof(__le16)))
return;
@@ -603,7 +608,8 @@ static void parse_dacl(struct user_namespace *user_ns,
static void set_posix_acl_entries_dacl(struct user_namespace *user_ns,
struct smb_ace *pndace,
struct smb_fattr *fattr, u32 *num_aces,
- u16 *size, u32 nt_aces_num)
+ u16 *size, u32 existing_nt_aces,
+ bool had_nt_aces)
{
struct posix_acl_entry *pace;
struct smb_sid *sid;
@@ -635,14 +641,14 @@ static void set_posix_acl_entries_dacl(struct user_namespace *user_ns,
gid = posix_acl_gid_translate(user_ns, pace);
id_to_sid(gid, SIDUNIX_GROUP, sid);
- } else if (pace->e_tag == ACL_OTHER && !nt_aces_num) {
+ } else if (pace->e_tag == ACL_OTHER && !had_nt_aces) {
smb_copy_sid(sid, &sid_everyone);
} else {
kfree(sid);
continue;
}
ntace = pndace;
- for (j = 0; j < nt_aces_num; j++) {
+ for (j = 0; j < existing_nt_aces; j++) {
if (ntace->sid.sub_auth[ntace->sid.num_subauth - 1] ==
sid->sub_auth[sid->num_subauth - 1])
goto pass_same_sid;
@@ -686,7 +692,7 @@ static void set_posix_acl_entries_dacl(struct user_namespace *user_ns,
kfree(sid);
}
- if (nt_aces_num)
+ if (had_nt_aces)
return;
posix_default_acl:
@@ -740,6 +746,7 @@ static void set_ntacl_dacl(struct user_namespace *user_ns,
{
struct smb_ace *ntace, *pndace;
int nt_num_aces = le32_to_cpu(nt_dacl->num_aces), num_aces = 0;
+ u32 copied_nt_aces;
unsigned short size = 0;
int i;
@@ -780,8 +787,10 @@ static void set_ntacl_dacl(struct user_namespace *user_ns,
}
}
+ copied_nt_aces = num_aces;
set_posix_acl_entries_dacl(user_ns, pndace, fattr,
- &num_aces, &size, nt_num_aces);
+ &num_aces, &size, copied_nt_aces,
+ nt_num_aces != 0);
pndacl->num_aces = cpu_to_le32(num_aces);
pndacl->size = cpu_to_le16(le16_to_cpu(pndacl->size) + size);
}
@@ -799,7 +808,7 @@ static void set_mode_dacl(struct user_namespace *user_ns,
if (fattr->cf_acls) {
set_posix_acl_entries_dacl(user_ns, pndace, fattr,
- &num_aces, &size, num_aces);
+ &num_aces, &size, num_aces, false);
goto out;
}
@@ -961,6 +970,49 @@ int parse_sec_desc(struct user_namespace *user_ns, struct smb_ntsd *pntsd,
return 0;
}
+size_t smb_acl_sec_desc_scratch_len(struct smb_fattr *fattr,
+ struct smb_ntsd *ppntsd, int ppntsd_size, int addition_info)
+{
+ size_t len = sizeof(struct smb_ntsd);
+ size_t tmp;
+
+ if (addition_info & OWNER_SECINFO)
+ len += sizeof(struct smb_sid);
+ if (addition_info & GROUP_SECINFO)
+ len += sizeof(struct smb_sid);
+ if (!(addition_info & DACL_SECINFO))
+ return len;
+
+ len += sizeof(struct smb_acl);
+ if (ppntsd && ppntsd_size > 0) {
+ unsigned int dacl_offset = le32_to_cpu(ppntsd->dacloffset);
+
+ if (dacl_offset < ppntsd_size &&
+ check_add_overflow(len, ppntsd_size - dacl_offset, &len))
+ return 0;
+ }
+
+ if (fattr->cf_acls) {
+ if (check_mul_overflow((size_t)fattr->cf_acls->a_count,
+ 2 * sizeof(struct smb_ace), &tmp) ||
+ check_add_overflow(len, tmp, &len))
+ return 0;
+ } else {
+ /* default/minimum DACL */
+ if (check_add_overflow(len, 5 * sizeof(struct smb_ace), &len))
+ return 0;
+ }
+
+ if (fattr->cf_dacls) {
+ if (check_mul_overflow((size_t)fattr->cf_dacls->a_count,
+ sizeof(struct smb_ace), &tmp) ||
+ check_add_overflow(len, tmp, &len))
+ return 0;
+ }
+
+ return len;
+}
+
/* Convert permission bits from mode to equivalent CIFS ACL */
int build_sec_desc(struct user_namespace *user_ns,
struct smb_ntsd *pntsd, struct smb_ntsd *ppntsd,
@@ -1062,7 +1114,60 @@ static void smb_set_ace(struct smb_ace *ace, const struct smb_sid *sid, u8 type,
ace->flags = flags;
ace->access_req = access_req;
smb_copy_sid(&ace->sid, sid);
- ace->size = cpu_to_le16(1 + 1 + 2 + 4 + 1 + 1 + 6 + (sid->num_subauth * 4));
+ ace->size = cpu_to_le16(1 + 1 + 2 + 4 + 1 + 1 + 6 +
+ (ace->sid.num_subauth * 4));
+}
+
+static int smb_append_inherited_ace(struct smb_ace **ace, int *nt_size,
+ int *ace_cnt, const struct smb_sid *sid,
+ u8 type, u8 flags, __le32 access_req)
+{
+ int ace_size;
+
+ smb_set_ace(*ace, sid, type, flags, access_req);
+ ace_size = le16_to_cpu((*ace)->size);
+ /* pdacl->size is __le16 and includes struct smb_acl. */
+ if (check_add_overflow(*nt_size, ace_size, nt_size) ||
+ *nt_size > U16_MAX - (int)sizeof(struct smb_acl))
+ return -EINVAL;
+
+ (*ace_cnt)++;
+ *ace = (struct smb_ace *)((char *)*ace + ace_size);
+ return 0;
+}
+
+static int smb_validate_ntsd_sid(struct smb_ntsd *pntsd, size_t pntsd_size,
+ unsigned int sid_offset, struct smb_sid **sid,
+ size_t *sid_size)
+{
+ size_t sid_end;
+
+ *sid = NULL;
+ *sid_size = 0;
+
+ if (!sid_offset)
+ return 0;
+
+ if (sid_offset < sizeof(struct smb_ntsd) ||
+ check_add_overflow(sid_offset, (size_t)CIFS_SID_BASE_SIZE,
+ &sid_end) ||
+ sid_end > pntsd_size)
+ return -EINVAL;
+
+ *sid = (struct smb_sid *)((char *)pntsd + sid_offset);
+ if ((*sid)->num_subauth > SID_MAX_SUB_AUTHORITIES)
+ return -EINVAL;
+
+ if (check_add_overflow((size_t)CIFS_SID_BASE_SIZE,
+ sizeof(__le32) * (size_t)(*sid)->num_subauth,
+ &sid_end))
+ return -EINVAL;
+
+ if (sid_offset > pntsd_size || sid_end > pntsd_size - sid_offset)
+ return -EINVAL;
+
+ *sid_size = sid_end;
+ return 0;
}
int smb_inherit_dacl(struct ksmbd_conn *conn,
@@ -1077,27 +1182,27 @@ int smb_inherit_dacl(struct ksmbd_conn *conn,
struct dentry *parent = path->dentry->d_parent;
struct user_namespace *user_ns = mnt_user_ns(path->mnt);
int inherited_flags = 0, flags = 0, i, ace_cnt = 0, nt_size = 0, pdacl_size;
- int rc = 0, num_aces, pntsd_type, pntsd_size, acl_len, aces_size;
+ int rc = 0, num_aces, pntsd_type, ppntsd_size, acl_len, aces_size;
unsigned int dacloffset;
size_t dacl_struct_end;
char *aces_base;
bool is_dir = S_ISDIR(d_inode(path->dentry)->i_mode);
- pntsd_size = ksmbd_vfs_get_sd_xattr(conn, user_ns,
+ ppntsd_size = ksmbd_vfs_get_sd_xattr(conn, user_ns,
parent, &parent_pntsd);
- if (pntsd_size <= 0)
+ if (ppntsd_size <= 0)
return -ENOENT;
dacloffset = le32_to_cpu(parent_pntsd->dacloffset);
if (!dacloffset ||
check_add_overflow(dacloffset, sizeof(struct smb_acl), &dacl_struct_end) ||
- dacl_struct_end > (size_t)pntsd_size) {
+ dacl_struct_end > (size_t)ppntsd_size) {
rc = -EINVAL;
goto free_parent_pntsd;
}
parent_pdacl = (struct smb_acl *)((char *)parent_pntsd + dacloffset);
- acl_len = pntsd_size - dacloffset;
+ acl_len = ppntsd_size - dacloffset;
num_aces = le32_to_cpu(parent_pdacl->num_aces);
pntsd_type = le16_to_cpu(parent_pntsd->type);
pdacl_size = le16_to_cpu(parent_pdacl->size);
@@ -1107,7 +1212,24 @@ int smb_inherit_dacl(struct ksmbd_conn *conn,
goto free_parent_pntsd;
}
- aces_base = kmalloc(sizeof(struct smb_ace) * num_aces * 2, GFP_KERNEL);
+ aces_size = pdacl_size - sizeof(struct smb_acl);
+
+ /*
+ * Validate num_aces against the DACL payload before allocating.
+ * Each ACE must be at least as large as its fixed-size header
+ * (up to the SID base), so num_aces cannot exceed the payload
+ * divided by the minimum ACE size. This mirrors the existing
+ * check in parse_dacl().
+ */
+ if (num_aces > aces_size / (offsetof(struct smb_ace, sid) +
+ offsetof(struct smb_sid, sub_auth) +
+ sizeof(__le16))) {
+ rc = -EINVAL;
+ goto free_parent_pntsd;
+ }
+
+ aces_base = kmalloc_array(num_aces * 2, sizeof(struct smb_ace),
+ GFP_KERNEL);
if (!aces_base) {
rc = -ENOMEM;
goto free_parent_pntsd;
@@ -1116,7 +1238,6 @@ int smb_inherit_dacl(struct ksmbd_conn *conn,
aces = (struct smb_ace *)aces_base;
parent_aces = (struct smb_ace *)((char *)parent_pdacl +
sizeof(struct smb_acl));
- aces_size = acl_len - sizeof(struct smb_acl);
if (pntsd_type & DACL_AUTO_INHERITED)
inherited_flags = INHERITED_ACE;
@@ -1124,11 +1245,20 @@ int smb_inherit_dacl(struct ksmbd_conn *conn,
for (i = 0; i < num_aces; i++) {
int pace_size;
- if (offsetof(struct smb_ace, access_req) > aces_size)
+ if (aces_size < offsetof(struct smb_ace, sid) +
+ CIFS_SID_BASE_SIZE)
break;
pace_size = le16_to_cpu(parent_aces->size);
- if (pace_size > aces_size)
+ if (pace_size > aces_size ||
+ pace_size < offsetof(struct smb_ace, sid) +
+ CIFS_SID_BASE_SIZE)
+ break;
+
+ if (parent_aces->sid.num_subauth > SID_MAX_SUB_AUTHORITIES ||
+ pace_size < offsetof(struct smb_ace, sid) +
+ CIFS_SID_BASE_SIZE +
+ sizeof(__le32) * parent_aces->sid.num_subauth)
break;
aces_size -= pace_size;
@@ -1160,22 +1290,24 @@ int smb_inherit_dacl(struct ksmbd_conn *conn,
}
if (is_dir && creator && flags & CONTAINER_INHERIT_ACE) {
- smb_set_ace(aces, psid, parent_aces->type, inherited_flags,
- parent_aces->access_req);
- nt_size += le16_to_cpu(aces->size);
- ace_cnt++;
- aces = (struct smb_ace *)((char *)aces + le16_to_cpu(aces->size));
+ rc = smb_append_inherited_ace(&aces, &nt_size, &ace_cnt,
+ psid, parent_aces->type,
+ inherited_flags,
+ parent_aces->access_req);
+ if (rc)
+ goto free_aces_base;
flags |= INHERIT_ONLY_ACE;
psid = creator;
} else if (is_dir && !(parent_aces->flags & NO_PROPAGATE_INHERIT_ACE)) {
psid = &parent_aces->sid;
}
- smb_set_ace(aces, psid, parent_aces->type, flags | inherited_flags,
- parent_aces->access_req);
- nt_size += le16_to_cpu(aces->size);
- aces = (struct smb_ace *)((char *)aces + le16_to_cpu(aces->size));
- ace_cnt++;
+ rc = smb_append_inherited_ace(&aces, &nt_size, &ace_cnt, psid,
+ parent_aces->type,
+ flags | inherited_flags,
+ parent_aces->access_req);
+ if (rc)
+ goto free_aces_base;
pass:
parent_aces = (struct smb_ace *)((char *)parent_aces + pace_size);
}
@@ -1184,22 +1316,33 @@ int smb_inherit_dacl(struct ksmbd_conn *conn,
struct smb_ntsd *pntsd;
struct smb_acl *pdacl;
struct smb_sid *powner_sid = NULL, *pgroup_sid = NULL;
- int powner_sid_size = 0, pgroup_sid_size = 0, pntsd_size;
- int pntsd_alloc_size;
+ size_t powner_sid_size = 0, pgroup_sid_size = 0, pntsd_size;
+ size_t pntsd_alloc_size;
- if (parent_pntsd->osidoffset) {
- powner_sid = (struct smb_sid *)((char *)parent_pntsd +
- le32_to_cpu(parent_pntsd->osidoffset));
- powner_sid_size = 1 + 1 + 6 + (powner_sid->num_subauth * 4);
- }
- if (parent_pntsd->gsidoffset) {
- pgroup_sid = (struct smb_sid *)((char *)parent_pntsd +
- le32_to_cpu(parent_pntsd->gsidoffset));
- pgroup_sid_size = 1 + 1 + 6 + (pgroup_sid->num_subauth * 4);
- }
+ rc = smb_validate_ntsd_sid(parent_pntsd, ppntsd_size,
+ le32_to_cpu(parent_pntsd->osidoffset),
+ &powner_sid, &powner_sid_size);
+ if (rc)
+ goto free_aces_base;
+ rc = smb_validate_ntsd_sid(parent_pntsd, ppntsd_size,
+ le32_to_cpu(parent_pntsd->gsidoffset),
+ &pgroup_sid, &pgroup_sid_size);
+ if (rc)
+ goto free_aces_base;
- pntsd_alloc_size = sizeof(struct smb_ntsd) + powner_sid_size +
- pgroup_sid_size + sizeof(struct smb_acl) + nt_size;
+ if (check_add_overflow(sizeof(struct smb_ntsd),
+ (size_t)powner_sid_size,
+ &pntsd_alloc_size) ||
+ check_add_overflow(pntsd_alloc_size,
+ (size_t)pgroup_sid_size,
+ &pntsd_alloc_size) ||
+ check_add_overflow(pntsd_alloc_size, sizeof(struct smb_acl),
+ &pntsd_alloc_size) ||
+ check_add_overflow(pntsd_alloc_size, (size_t)nt_size,
+ &pntsd_alloc_size)) {
+ rc = -EINVAL;
+ goto free_aces_base;
+ }
pntsd = kzalloc(pntsd_alloc_size, GFP_KERNEL);
if (!pntsd) {
@@ -1288,7 +1431,7 @@ bool smb_inherit_flags(int flags, bool is_dir)
}
int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
- __le32 *pdaccess, int uid)
+ __le32 *pdaccess, __le32 raw_daccess, int uid)
{
struct user_namespace *user_ns = mnt_user_ns(path->mnt);
struct smb_ntsd *pntsd = NULL;
@@ -1298,10 +1441,11 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
unsigned int dacl_offset;
size_t dacl_struct_end;
struct smb_sid sid;
- int granted = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE);
+ int requested = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE);
+ int granted = requested;
struct smb_ace *ace;
int i, found = 0;
- unsigned int access_bits = 0;
+ unsigned int access_bits = 0, denied = 0;
struct smb_ace *others_ace = NULL;
struct posix_acl_entry *pa_entry;
unsigned int sid_type = SIDOWNER;
@@ -1335,12 +1479,15 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
goto err_out;
}
+ if (!uid)
+ sid_type = SIDUNIX_USER;
+ id_to_sid(uid, sid_type, &sid);
+
if (*pdaccess & FILE_MAXIMAL_ACCESS_LE) {
- granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
+ access_bits = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
DELETE;
-
ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl));
- aces_size = acl_size - sizeof(struct smb_acl);
+ aces_size = pdacl_size - sizeof(struct smb_acl);
for (i = 0; i < le32_to_cpu(pdacl->num_aces); i++) {
if (aces_size < offsetof(struct smb_ace, sid) +
CIFS_SID_BASE_SIZE)
@@ -1351,20 +1498,43 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
CIFS_SID_BASE_SIZE)
break;
aces_size -= ace_size;
- granted |= le32_to_cpu(ace->access_req);
+
+ if (ace->sid.num_subauth > SID_MAX_SUB_AUTHORITIES ||
+ ace_size < offsetof(struct smb_ace, sid) +
+ CIFS_SID_BASE_SIZE +
+ sizeof(__le32) * ace->sid.num_subauth)
+ break;
+
+ if (ace->flags & INHERIT_ONLY_ACE ||
+ (compare_sids(&sid, &ace->sid) &&
+ compare_sids(&sid_unix_NFS_mode, &ace->sid) &&
+ compare_sids(&sid_everyone, &ace->sid) &&
+ compare_sids(&sid_authusers, &ace->sid)))
+ goto next_ace;
+
+ switch (ace->type) {
+ case ACCESS_ALLOWED_ACE_TYPE:
+ access_bits |= le32_to_cpu(ace->access_req);
+ break;
+ case ACCESS_DENIED_ACE_TYPE:
+ case ACCESS_DENIED_CALLBACK_ACE_TYPE:
+ denied |= ~access_bits &
+ le32_to_cpu(ace->access_req);
+ break;
+ }
+next_ace:
ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size));
}
-
- if (!pdacl->num_aces)
- granted = GENERIC_ALL_FLAGS;
+ access_bits &= ~denied;
+ if ((raw_daccess & FILE_GENERIC_EXECUTE_LE) &&
+ S_ISREG(d_inode(path->dentry)->i_mode) &&
+ (access_bits & GENERIC_READ_FLAGS) == GENERIC_READ_FLAGS)
+ access_bits |= FILE_EXECUTE;
+ granted = requested | access_bits;
}
- if (!uid)
- sid_type = SIDUNIX_USER;
- id_to_sid(uid, sid_type, &sid);
-
ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl));
- aces_size = acl_size - sizeof(struct smb_acl);
+ aces_size = pdacl_size - sizeof(struct smb_acl);
for (i = 0; i < le32_to_cpu(pdacl->num_aces); i++) {
if (aces_size < offsetof(struct smb_ace, sid) +
CIFS_SID_BASE_SIZE)
@@ -1386,25 +1556,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
found = 1;
break;
}
- if (!compare_sids(&sid_everyone, &ace->sid))
+ if (!compare_sids(&sid_everyone, &ace->sid) ||
+ !compare_sids(&sid_authusers, &ace->sid))
others_ace = ace;
ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size));
}
- if (*pdaccess & FILE_MAXIMAL_ACCESS_LE && found) {
- granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES |
- DELETE;
-
- granted |= le32_to_cpu(ace->access_req);
-
- if (!pdacl->num_aces)
- granted = GENERIC_ALL_FLAGS;
- }
-
if (IS_ENABLED(CONFIG_FS_POSIX_ACL)) {
posix_acls = get_acl(d_inode(path->dentry), ACL_TYPE_ACCESS);
- if (!IS_ERR_OR_NULL(posix_acls) && !found) {
+ if (!IS_ERR_OR_NULL(posix_acls) && !found && !others_ace) {
unsigned int id = -1;
pa_entry = posix_acls->a_entries;
@@ -1442,14 +1603,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
}
}
- switch (ace->type) {
- case ACCESS_ALLOWED_ACE_TYPE:
- access_bits = le32_to_cpu(ace->access_req);
- break;
- case ACCESS_DENIED_ACE_TYPE:
- case ACCESS_DENIED_CALLBACK_ACE_TYPE:
- access_bits = le32_to_cpu(~ace->access_req);
- break;
+ if (!(*pdaccess & FILE_MAXIMAL_ACCESS_LE)) {
+ switch (ace->type) {
+ case ACCESS_ALLOWED_ACE_TYPE:
+ access_bits = le32_to_cpu(ace->access_req);
+ break;
+ case ACCESS_DENIED_ACE_TYPE:
+ case ACCESS_DENIED_CALLBACK_ACE_TYPE:
+ access_bits = le32_to_cpu(~ace->access_req);
+ break;
+ }
}
check_access_bits:
diff --git a/fs/smb/server/smbacl.h b/fs/smb/server/smbacl.h
index 9651a2551888..7f074aac45cf 100644
--- a/fs/smb/server/smbacl.h
+++ b/fs/smb/server/smbacl.h
@@ -204,12 +204,14 @@ bool smb_inherit_flags(int flags, bool is_dir);
int smb_inherit_dacl(struct ksmbd_conn *conn, const struct path *path,
unsigned int uid, unsigned int gid);
int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path,
- __le32 *pdaccess, int uid);
+ __le32 *pdaccess, __le32 raw_daccess, int uid);
int set_info_sec(struct ksmbd_conn *conn, struct ksmbd_tree_connect *tcon,
const struct path *path, struct smb_ntsd *pntsd, int ntsd_len,
bool type_check, bool get_write);
void id_to_sid(unsigned int cid, uint sidtype, struct smb_sid *ssid);
void ksmbd_init_domain(u32 *sub_auth);
+size_t smb_acl_sec_desc_scratch_len(struct smb_fattr *fattr,
+ struct smb_ntsd *ppntsd, int ppntsd_size, int addition_info);
static inline uid_t posix_acl_uid_translate(struct user_namespace *mnt_userns,
struct posix_acl_entry *pace)
diff --git a/fs/smb/server/transport_ipc.c b/fs/smb/server/transport_ipc.c
index 89fae5e1886a..63d0e8bf39aa 100644
--- a/fs/smb/server/transport_ipc.c
+++ b/fs/smb/server/transport_ipc.c
@@ -13,6 +13,7 @@
#include <net/genetlink.h>
#include <linux/socket.h>
#include <linux/workqueue.h>
+#include <linux/overflow.h>
#include "vfs_cache.h"
#include "transport_ipc.h"
@@ -478,7 +479,9 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry)
if (entry->type == KSMBD_EVENT_RPC_REQUEST) {
struct ksmbd_rpc_command *resp = entry->response;
- msg_sz = sizeof(struct ksmbd_rpc_command) + resp->payload_sz;
+ if (check_add_overflow(sizeof(struct ksmbd_rpc_command),
+ resp->payload_sz, &msg_sz))
+ return -EINVAL;
} else if (entry->type == KSMBD_EVENT_SPNEGO_AUTHEN_REQUEST) {
struct ksmbd_spnego_authen_response *resp = entry->response;
diff --git a/fs/smb/server/transport_rdma.c b/fs/smb/server/transport_rdma.c
index 1290c0fa7dd1..bcf0631ce10a 100644
--- a/fs/smb/server/transport_rdma.c
+++ b/fs/smb/server/transport_rdma.c
@@ -1967,9 +1967,9 @@ static int smb_direct_prepare(struct ksmbd_transport *t)
goto out;
req = (struct smb_direct_negotiate_req *)recvmsg->packet;
- st->max_recv_size = min_t(int, st->max_recv_size,
+ st->max_recv_size = min_t(u32, st->max_recv_size,
le32_to_cpu(req->preferred_send_size));
- st->max_send_size = min_t(int, st->max_send_size,
+ st->max_send_size = min_t(u32, st->max_send_size,
le32_to_cpu(req->max_receive_size));
st->max_fragmented_send_size =
le32_to_cpu(req->max_fragmented_size);
diff --git a/fs/smb/server/vfs.c b/fs/smb/server/vfs.c
index 1da397999941..8631e57df599 100644
--- a/fs/smb/server/vfs.c
+++ b/fs/smb/server/vfs.c
@@ -285,17 +285,20 @@ static ssize_t ksmbd_vfs_getcasexattr(struct user_namespace *user_ns,
static int ksmbd_vfs_stream_read(struct ksmbd_file *fp, char *buf, loff_t *pos,
size_t count)
{
+ const struct cred *saved_cred;
ssize_t v_len;
char *stream_buf = NULL;
ksmbd_debug(VFS, "read stream data pos : %llu, count : %zd\n",
*pos, count);
+ saved_cred = override_creds(fp->filp->f_cred);
v_len = ksmbd_vfs_getcasexattr(file_mnt_user_ns(fp->filp),
fp->filp->f_path.dentry,
fp->stream.name,
fp->stream.size,
&stream_buf);
+ revert_creds(saved_cred);
if ((int)v_len <= 0)
return (int)v_len;
@@ -414,6 +417,7 @@ int ksmbd_vfs_read(struct ksmbd_work *work, struct ksmbd_file *fp, size_t count,
static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos,
size_t count)
{
+ const struct cred *saved_cred;
char *stream_buf = NULL, *wbuf;
struct user_namespace *user_ns = file_mnt_user_ns(fp->filp);
size_t size;
@@ -434,6 +438,7 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos,
count = XATTR_SIZE_MAX - *pos;
}
+ saved_cred = override_creds(fp->filp->f_cred);
v_len = ksmbd_vfs_getcasexattr(user_ns,
fp->filp->f_path.dentry,
fp->stream.name,
@@ -442,14 +447,14 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos,
if (v_len < 0) {
pr_err("not found stream in xattr : %zd\n", v_len);
err = v_len;
- goto out;
+ goto out_revert;
}
if (v_len < size) {
wbuf = kvzalloc(size, GFP_KERNEL);
if (!wbuf) {
err = -ENOMEM;
- goto out;
+ goto out_revert;
}
if (v_len > 0)
@@ -467,6 +472,8 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos,
size,
0,
true);
+out_revert:
+ revert_creds(saved_cred);
if (err < 0)
goto out;
@@ -1079,13 +1086,15 @@ int ksmbd_vfs_remove_xattr(struct user_namespace *user_ns,
int ksmbd_vfs_unlink(struct file *filp)
{
+ const struct cred *saved_cred;
int err = 0;
struct dentry *dir, *dentry = filp->f_path.dentry;
struct user_namespace *user_ns = file_mnt_user_ns(filp);
+ saved_cred = override_creds(filp->f_cred);
err = mnt_want_write(filp->f_path.mnt);
if (err)
- return err;
+ goto out_revert;
dir = dget_parent(dentry);
err = ksmbd_vfs_lock_parent(dir, dentry);
@@ -1105,7 +1114,8 @@ int ksmbd_vfs_unlink(struct file *filp)
out:
dput(dir);
mnt_drop_write(filp->f_path.mnt);
-
+out_revert:
+ revert_creds(saved_cred);
return err;
}
@@ -1229,7 +1239,7 @@ int ksmbd_vfs_kern_path_locked(struct ksmbd_work *work, char *name,
if (!err)
return 0;
- if (caseless) {
+ if (caseless && err == -ENOENT) {
char *filepath;
size_t path_len, remain_len;
@@ -1260,7 +1270,7 @@ int ksmbd_vfs_kern_path_locked(struct ksmbd_work *work, char *name,
err = vfs_path_lookup(share_conf->vfs_path.dentry,
share_conf->vfs_path.mnt,
filepath,
- flags,
+ flags | LOOKUP_BENEATH,
path);
if (!is_last)
next[0] = '/';
@@ -1311,15 +1321,50 @@ struct dentry *ksmbd_vfs_kern_path_create(struct ksmbd_work *work,
unsigned int flags,
struct path *path)
{
- char *abs_name;
+ struct ksmbd_share_config *share_conf = work->tcon->share_conf;
+ struct filename *filename;
+ struct qstr last;
struct dentry *dent;
+ int err, type;
+
+ /* resolve the name beneath the share root so ".." cannot escape */
+ filename = getname_kernel(name);
+ if (IS_ERR(filename))
+ return ERR_CAST(filename);
+
+ err = vfs_path_parent_lookup(filename, flags | LOOKUP_BENEATH,
+ path, &last, &type, &share_conf->vfs_path);
+ if (err) {
+ putname(filename);
+ return ERR_PTR(err);
+ }
+
+ if (unlikely(type != LAST_NORM)) {
+ path_put(path);
+ putname(filename);
+ return ERR_PTR(-EEXIST);
+ }
- abs_name = convert_to_unix_name(work->tcon->share_conf, name);
- if (!abs_name)
- return ERR_PTR(-ENOMEM);
+ err = mnt_want_write(path->mnt);
+ if (err) {
+ path_put(path);
+ putname(filename);
+ return ERR_PTR(err);
+ }
- dent = kern_path_create(AT_FDCWD, abs_name, path, flags);
- kfree(abs_name);
+ inode_lock_nested(path->dentry->d_inode, I_MUTEX_PARENT);
+ dent = lookup_one_qstr_excl(&last, path->dentry,
+ LOOKUP_CREATE | LOOKUP_EXCL);
+ if (!IS_ERR(dent) && d_is_positive(dent)) {
+ dput(dent);
+ dent = ERR_PTR(-EEXIST);
+ }
+ if (IS_ERR(dent)) {
+ inode_unlock(path->dentry->d_inode);
+ mnt_drop_write(path->mnt);
+ path_put(path);
+ }
+ putname(filename);
return dent;
}
@@ -1908,9 +1953,6 @@ int ksmbd_vfs_inherit_posix_acl(struct user_namespace *user_ns,
struct path *path, struct inode *parent_inode)
{
struct posix_acl *acls;
- struct posix_acl_entry *pace;
- struct inode *inode = d_inode(path->dentry);
- int rc, i;
if (!IS_ENABLED(CONFIG_FS_POSIX_ACL))
return -EOPNOTSUPP;
@@ -1918,27 +1960,7 @@ int ksmbd_vfs_inherit_posix_acl(struct user_namespace *user_ns,
acls = get_acl(parent_inode, ACL_TYPE_DEFAULT);
if (IS_ERR_OR_NULL(acls))
return -ENOENT;
- pace = acls->a_entries;
-
- for (i = 0; i < acls->a_count; i++, pace++) {
- if (pace->e_tag == ACL_MASK) {
- pace->e_perm = 0x07;
- break;
- }
- }
-
- rc = set_posix_acl(user_ns, inode, ACL_TYPE_ACCESS, acls);
- if (rc < 0)
- ksmbd_debug(SMB, "Set posix acl(ACL_TYPE_ACCESS) failed, rc : %d\n",
- rc);
- if (S_ISDIR(inode->i_mode)) {
- rc = set_posix_acl(user_ns, inode, ACL_TYPE_DEFAULT,
- acls);
- if (rc < 0)
- ksmbd_debug(SMB, "Set posix acl(ACL_TYPE_DEFAULT) failed, rc : %d\n",
- rc);
- }
posix_acl_release(acls);
- return rc;
+ return 0;
}
diff --git a/fs/smb/server/vfs_cache.c b/fs/smb/server/vfs_cache.c
index 9eb9f892ac2b..3eff8b7df9a2 100644
--- a/fs/smb/server/vfs_cache.c
+++ b/fs/smb/server/vfs_cache.c
@@ -104,40 +104,62 @@ int ksmbd_query_inode_status(struct dentry *dentry)
read_lock(&inode_hash_lock);
ci = __ksmbd_inode_lookup(dentry);
- if (ci) {
- ret = KSMBD_INODE_STATUS_OK;
- if (ci->m_flags & (S_DEL_PENDING | S_DEL_ON_CLS))
- ret = KSMBD_INODE_STATUS_PENDING_DELETE;
- atomic_dec(&ci->m_count);
- }
read_unlock(&inode_hash_lock);
+ if (!ci)
+ return ret;
+
+ read_lock(&ci->m_lock);
+ if (ci->m_flags & (S_DEL_PENDING | S_DEL_ON_CLS))
+ ret = KSMBD_INODE_STATUS_PENDING_DELETE;
+ else
+ ret = KSMBD_INODE_STATUS_OK;
+ read_unlock(&ci->m_lock);
+
+ atomic_dec(&ci->m_count);
return ret;
}
bool ksmbd_inode_pending_delete(struct ksmbd_file *fp)
{
- return (fp->f_ci->m_flags & (S_DEL_PENDING | S_DEL_ON_CLS));
+ struct ksmbd_inode *ci = fp->f_ci;
+ int ret;
+
+ read_lock(&ci->m_lock);
+ ret = (ci->m_flags & (S_DEL_PENDING | S_DEL_ON_CLS));
+ read_unlock(&ci->m_lock);
+
+ return ret;
}
void ksmbd_set_inode_pending_delete(struct ksmbd_file *fp)
{
- fp->f_ci->m_flags |= S_DEL_PENDING;
+ struct ksmbd_inode *ci = fp->f_ci;
+
+ write_lock(&ci->m_lock);
+ ci->m_flags |= S_DEL_PENDING;
+ write_unlock(&ci->m_lock);
}
void ksmbd_clear_inode_pending_delete(struct ksmbd_file *fp)
{
- fp->f_ci->m_flags &= ~S_DEL_PENDING;
+ struct ksmbd_inode *ci = fp->f_ci;
+
+ write_lock(&ci->m_lock);
+ ci->m_flags &= ~S_DEL_PENDING;
+ write_unlock(&ci->m_lock);
}
void ksmbd_fd_set_delete_on_close(struct ksmbd_file *fp,
int file_info)
{
- if (ksmbd_stream_fd(fp)) {
- fp->f_ci->m_flags |= S_DEL_ON_CLS_STREAM;
- return;
- }
+ struct ksmbd_inode *ci = fp->f_ci;
- fp->f_ci->m_flags |= S_DEL_ON_CLS;
+ write_lock(&ci->m_lock);
+ if (ksmbd_stream_fd(fp))
+ ci->m_flags |= S_DEL_ON_CLS_STREAM;
+ else
+ ci->m_flags |= S_DEL_ON_CLS;
+ write_unlock(&ci->m_lock);
}
static void ksmbd_inode_hash(struct ksmbd_inode *ci)
@@ -249,26 +271,44 @@ static void __ksmbd_inode_close(struct ksmbd_file *fp)
struct file *filp;
filp = fp->filp;
- if (ksmbd_stream_fd(fp) && (ci->m_flags & S_DEL_ON_CLS_STREAM)) {
- ci->m_flags &= ~S_DEL_ON_CLS_STREAM;
- err = ksmbd_vfs_remove_xattr(file_mnt_user_ns(filp),
- &filp->f_path,
- fp->stream.name);
- if (err)
- pr_err("remove xattr failed : %s\n",
- fp->stream.name);
+
+ if (ksmbd_stream_fd(fp)) {
+ bool remove_stream_xattr = false;
+
+ write_lock(&ci->m_lock);
+ if (ci->m_flags & S_DEL_ON_CLS_STREAM) {
+ ci->m_flags &= ~S_DEL_ON_CLS_STREAM;
+ remove_stream_xattr = true;
+ }
+ write_unlock(&ci->m_lock);
+
+ if (remove_stream_xattr) {
+ const struct cred *saved_cred;
+
+ saved_cred = override_creds(filp->f_cred);
+ err = ksmbd_vfs_remove_xattr(file_mnt_user_ns(filp),
+ &filp->f_path,
+ fp->stream.name);
+ revert_creds(saved_cred);
+ if (err)
+ pr_err("remove xattr failed : %s\n",
+ fp->stream.name);
+ }
}
if (atomic_dec_and_test(&ci->m_count)) {
+ bool do_unlink = false;
+
write_lock(&ci->m_lock);
if (ci->m_flags & (S_DEL_ON_CLS | S_DEL_PENDING)) {
ci->m_flags &= ~(S_DEL_ON_CLS | S_DEL_PENDING);
- write_unlock(&ci->m_lock);
- ksmbd_vfs_unlink(filp);
- write_lock(&ci->m_lock);
+ do_unlink = true;
}
write_unlock(&ci->m_lock);
+ if (do_unlink)
+ ksmbd_vfs_unlink(filp);
+
ksmbd_inode_free(ci);
}
}
@@ -313,8 +353,9 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp)
if (!IS_ERR_OR_NULL(filp))
fput(filp);
- /* because the reference count of fp is 0, it is guaranteed that
- * there are not accesses to fp->lock_list.
+ /*
+ * The zero fp reference count serializes access to fp->lock_list, but
+ * the VFS may still have blocked requests chained below these locks.
*/
list_for_each_entry_safe(smb_lock, tmp_lock, &fp->lock_list, flist) {
struct ksmbd_conn *conn = smb_lock->conn;
@@ -327,7 +368,8 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp)
ksmbd_conn_put(conn);
}
- list_del(&smb_lock->flist);
+ list_del_init(&smb_lock->flist);
+ ksmbd_vfs_posix_lock_unblock(smb_lock->fl);
locks_free_lock(smb_lock->fl);
kfree(smb_lock);
}
@@ -648,21 +690,46 @@ __close_file_table_ids(struct ksmbd_file_table *ft,
bool (*skip)(struct ksmbd_tree_connect *tcon,
struct ksmbd_file *fp))
{
- unsigned int id;
- struct ksmbd_file *fp;
- int num = 0;
+ struct ksmbd_file *fp;
+ unsigned int id = 0;
+ int num = 0;
+
+ while (1) {
+ write_lock(&ft->lock);
+ fp = idr_get_next(ft->idr, &id);
+ if (!fp) {
+ write_unlock(&ft->lock);
+ break;
+ }
- idr_for_each_entry(ft->idr, fp, id) {
- if (skip(tcon, fp))
+ if (skip(tcon, fp)) {
+ id++;
+ write_unlock(&ft->lock);
continue;
+ }
set_close_state_blocked_works(fp);
- if (!atomic_dec_and_test(&fp->refcount))
+ if (!atomic_dec_and_test(&fp->refcount)) {
+ id++;
+ write_unlock(&ft->lock);
continue;
+ }
+
+ idr_remove(ft->idr, fp->volatile_id);
+ fp->volatile_id = KSMBD_NO_FID;
+ write_unlock(&ft->lock);
+
+ write_lock(&fp->f_ci->m_lock);
+ list_del_init(&fp->node);
+ write_unlock(&fp->f_ci->m_lock);
+
__ksmbd_close_fd(ft, fp);
+
num++;
+ id++;
}
+
return num;
}
diff --git a/fs/unicode/utf8-core.c b/fs/unicode/utf8-core.c
index c887edc4f527..48793e5e3cd4 100644
--- a/fs/unicode/utf8-core.c
+++ b/fs/unicode/utf8-core.c
@@ -3,6 +3,7 @@
#include <linux/kernel.h>
#include <linux/string.h>
#include <linux/slab.h>
+#include <linux/rcupdate.h>
#include <linux/parser.h>
#include <linux/errno.h>
#include <linux/stringhash.h>
@@ -205,12 +206,24 @@ struct unicode_map *utf8_load(unsigned int version)
}
EXPORT_SYMBOL(utf8_load);
+static void utf8_unload_rcu(struct rcu_head *head)
+{
+ struct unicode_map *um = container_of(head, struct unicode_map, rcu);
+
+ symbol_put(utf8_data_table);
+ kfree(um);
+}
+
+/*
+ * A pathwalk in rcu mode may still be in ->d_hash() or ->d_compare() of a
+ * filesystem that is being torn down and read the map through its
+ * superblock, so the map and the tables it points to have to outlive the
+ * grace period.
+ */
void utf8_unload(struct unicode_map *um)
{
- if (um) {
- symbol_put(utf8_data_table);
- kfree(um);
- }
+ if (um)
+ call_rcu(&um->rcu, utf8_unload_rcu);
}
EXPORT_SYMBOL(utf8_unload);
diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c
index 6ebee2b3a5a8..59c4fd41a35a 100644
--- a/fs/userfaultfd.c
+++ b/fs/userfaultfd.c
@@ -108,6 +108,13 @@ static bool userfaultfd_is_initialized(struct userfaultfd_ctx *ctx)
return ctx->features & UFFD_FEATURE_INITIALIZED;
}
+bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
+{
+ struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx;
+
+ return uffd_ctx && (uffd_ctx->features & UFFD_FEATURE_EVENT_REMAP) == 0;
+}
+
static void userfaultfd_set_vm_flags(struct vm_area_struct *vma,
vm_flags_t flags)
{
@@ -908,6 +915,10 @@ static int userfaultfd_release(struct inode *inode, struct file *file)
prev = vma;
continue;
}
+ /* Reset ptes for the whole vma range if wr-protected */
+ if (userfaultfd_wp(vma))
+ uffd_wp_range(mm, vma, vma->vm_start,
+ vma->vm_end - vma->vm_start, false);
new_flags = vma->vm_flags & ~__VM_UFFD_FLAGS;
prev = vma_merge(mm, prev, vma->vm_start, vma->vm_end,
new_flags, vma->anon_vma,
diff --git a/fs/verity/Kconfig b/fs/verity/Kconfig
index aad1f1d998b9..6882f85e0a11 100644
--- a/fs/verity/Kconfig
+++ b/fs/verity/Kconfig
@@ -2,6 +2,9 @@
config FS_VERITY
bool "FS Verity (read-only file-based authenticity protection)"
+ # Filesystems cache the Merkle tree at a 64K aligned offset in the
+ # pagecache. That approach assumes the page size is at most 64K.
+ depends on !PAGE_SIZE_256KB
select CRYPTO
select CRYPTO_HASH_INFO
# SHA-256 is implied as it's intended to be the default hash algorithm.
diff --git a/fs/xfs/libxfs/xfs_alloc.c b/fs/xfs/libxfs/xfs_alloc.c
index cd5b197d7046..5bc37f984b00 100644
--- a/fs/xfs/libxfs/xfs_alloc.c
+++ b/fs/xfs/libxfs/xfs_alloc.c
@@ -26,6 +26,7 @@
#include "xfs_ag.h"
#include "xfs_ag_resv.h"
#include "xfs_bmap.h"
+#include "xfs_health.h"
struct kmem_cache *xfs_extfree_item_cache;
@@ -3167,18 +3168,43 @@ xfs_alloc_read_agf(
atomic64_add(allocbt_blks,
&pag->pag_mount->m_allocbt_blks);
}
+
#ifdef DEBUG
- else if (!xfs_is_shutdown(pag->pag_mount)) {
- ASSERT(pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks));
- ASSERT(pag->pagf_btreeblks == be32_to_cpu(agf->agf_btreeblks));
- ASSERT(pag->pagf_flcount == be32_to_cpu(agf->agf_flcount));
- ASSERT(pag->pagf_longest == be32_to_cpu(agf->agf_longest));
- ASSERT(pag->pagf_levels[XFS_BTNUM_BNOi] ==
- be32_to_cpu(agf->agf_levels[XFS_BTNUM_BNOi]));
- ASSERT(pag->pagf_levels[XFS_BTNUM_CNTi] ==
- be32_to_cpu(agf->agf_levels[XFS_BTNUM_CNTi]));
+ /*
+ * It's possible for the AGF to be out of sync if the block device is
+ * silently dropping writes. This can happen in fstests with dmflakey
+ * enabled, which allows the buffer to be cleaned and reclaimed by
+ * memory pressure and then re-read from disk here. We will get a
+ * stale version of the AGF from disk, and nothing good can happen from
+ * here. Hence if we detect this situation, immediately shut down the
+ * filesystem.
+ *
+ * This can also happen if we are already in the middle of a forced
+ * shutdown, so don't bother checking if we are already shut down.
+ */
+ if (!xfs_is_shutdown(pag->pag_mount)) {
+ bool ok = true;
+
+ ok &= pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks);
+ ok &= pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks);
+ ok &= pag->pagf_btreeblks == be32_to_cpu(agf->agf_btreeblks);
+ ok &= pag->pagf_flcount == be32_to_cpu(agf->agf_flcount);
+ ok &= pag->pagf_longest == be32_to_cpu(agf->agf_longest);
+ ok &= pag->pagf_levels[XFS_BTNUM_BNOi] ==
+ be32_to_cpu(agf->agf_levels[XFS_BTNUM_BNOi]);
+ ok &= pag->pagf_levels[XFS_BTNUM_CNTi] ==
+ be32_to_cpu(agf->agf_levels[XFS_BTNUM_CNTi]);
+
+ if (XFS_IS_CORRUPT(pag->pag_mount, !ok)) {
+ xfs_ag_mark_sick(pag, XFS_SICK_AG_AGF);
+ xfs_trans_brelse(tp, agfbp);
+ xfs_force_shutdown(pag->pag_mount,
+ SHUTDOWN_CORRUPT_ONDISK);
+ return -EFSCORRUPTED;
+ }
}
-#endif
+#endif /* DEBUG */
+
if (agfbpp)
*agfbpp = agfbp;
else
diff --git a/fs/xfs/libxfs/xfs_attr.c b/fs/xfs/libxfs/xfs_attr.c
index 32d350e97e0f..236291dbb427 100644
--- a/fs/xfs/libxfs/xfs_attr.c
+++ b/fs/xfs/libxfs/xfs_attr.c
@@ -49,7 +49,6 @@ STATIC int xfs_attr_shortform_addname(xfs_da_args_t *args);
*/
STATIC int xfs_attr_leaf_get(xfs_da_args_t *args);
STATIC int xfs_attr_leaf_removename(xfs_da_args_t *args);
-STATIC int xfs_attr_leaf_hasname(struct xfs_da_args *args, struct xfs_buf **bp);
STATIC int xfs_attr_leaf_try_add(struct xfs_da_args *args);
/*
@@ -866,11 +865,11 @@ xfs_attr_lookup(
return xfs_attr_sf_findname(args, NULL, NULL);
if (xfs_attr_is_leaf(dp)) {
- error = xfs_attr_leaf_hasname(args, &bp);
-
- if (bp)
- xfs_trans_brelse(args->trans, bp);
-
+ error = xfs_attr3_leaf_read(args->trans, args->dp, 0, &bp);
+ if (error)
+ return error;
+ error = xfs_attr3_leaf_lookup_int(bp, args);
+ xfs_trans_brelse(args->trans, bp);
return error;
}
@@ -1246,27 +1245,6 @@ xfs_attr_leaf_try_add(
return error;
}
-/*
- * Return EEXIST if attr is found, or ENOATTR if not
- */
-STATIC int
-xfs_attr_leaf_hasname(
- struct xfs_da_args *args,
- struct xfs_buf **bp)
-{
- int error = 0;
-
- error = xfs_attr3_leaf_read(args->trans, args->dp, 0, bp);
- if (error)
- return error;
-
- error = xfs_attr3_leaf_lookup_int(*bp, args);
- if (error != -ENOATTR && error != -EEXIST)
- xfs_trans_brelse(args->trans, *bp);
-
- return error;
-}
-
/*
* Remove a name from the leaf attribute list structure
*
@@ -1277,25 +1255,22 @@ STATIC int
xfs_attr_leaf_removename(
struct xfs_da_args *args)
{
- struct xfs_inode *dp;
- struct xfs_buf *bp;
+ struct xfs_inode *dp = args->dp;
int error, forkoff;
+ struct xfs_buf *bp;
trace_xfs_attr_leaf_removename(args);
- /*
- * Remove the attribute.
- */
- dp = args->dp;
-
- error = xfs_attr_leaf_hasname(args, &bp);
- if (error == -ENOATTR) {
+ error = xfs_attr3_leaf_read(args->trans, args->dp, 0, &bp);
+ if (error)
+ return error;
+ error = xfs_attr3_leaf_lookup_int(bp, args);
+ if (error != -EEXIST) {
xfs_trans_brelse(args->trans, bp);
- if (args->op_flags & XFS_DA_OP_RECOVERY)
+ if (error == -ENOATTR && (args->op_flags & XFS_DA_OP_RECOVERY))
return 0;
return error;
- } else if (error != -EEXIST)
- return error;
+ }
xfs_attr3_leaf_remove(bp, args);
@@ -1319,23 +1294,20 @@ xfs_attr_leaf_removename(
* Returns 0 on successful retrieval, otherwise an error.
*/
STATIC int
-xfs_attr_leaf_get(xfs_da_args_t *args)
+xfs_attr_leaf_get(
+ struct xfs_da_args *args)
{
- struct xfs_buf *bp;
- int error;
+ struct xfs_buf *bp;
+ int error;
trace_xfs_attr_leaf_get(args);
- error = xfs_attr_leaf_hasname(args, &bp);
-
- if (error == -ENOATTR) {
- xfs_trans_brelse(args->trans, bp);
- return error;
- } else if (error != -EEXIST)
+ error = xfs_attr3_leaf_read(args->trans, args->dp, 0, &bp);
+ if (error)
return error;
-
-
- error = xfs_attr3_leaf_getvalue(bp, args);
+ error = xfs_attr3_leaf_lookup_int(bp, args);
+ if (error == -EEXIST)
+ error = xfs_attr3_leaf_getvalue(bp, args);
xfs_trans_brelse(args->trans, bp);
return error;
}
diff --git a/fs/xfs/libxfs/xfs_ialloc.c b/fs/xfs/libxfs/xfs_ialloc.c
index 4b2894c85cea..1d3c07506c59 100644
--- a/fs/xfs/libxfs/xfs_ialloc.c
+++ b/fs/xfs/libxfs/xfs_ialloc.c
@@ -27,6 +27,7 @@
#include "xfs_log.h"
#include "xfs_rmap.h"
#include "xfs_ag.h"
+#include "xfs_health.h"
/*
* Lookup a record by ino in the btree given by cur.
@@ -2645,12 +2646,35 @@ xfs_ialloc_read_agi(
pag->pagi_init = 1;
}
+#ifdef DEBUG
/*
- * It's possible for these to be out of sync if
- * we are in the middle of a forced shutdown.
+ * It's possible for the AGF to be out of sync if the block device is
+ * silently dropping writes. This can happen in fstests with dmflakey
+ * enabled, which allows the buffer to be cleaned and reclaimed by
+ * memory pressure and then re-read from disk here. We will get a
+ * stale version of the AGF from disk, and nothing good can happen from
+ * here. Hence if we detect this situation, immediately shut down the
+ * filesystem.
+ *
+ * This can also happen if we are already in the middle of a forced
+ * shutdown, so don't bother checking if we are already shut down.
*/
- ASSERT(pag->pagi_freecount == be32_to_cpu(agi->agi_freecount) ||
- xfs_is_shutdown(pag->pag_mount));
+ if (!xfs_is_shutdown(pag->pag_mount)) {
+ bool ok = true;
+
+ ok &= pag->pagi_freecount == be32_to_cpu(agi->agi_freecount);
+ ok &= pag->pagi_count == be32_to_cpu(agi->agi_count);
+
+ if (XFS_IS_CORRUPT(pag->pag_mount, !ok)) {
+ xfs_ag_mark_sick(pag, XFS_SICK_AG_AGI);
+ xfs_trans_brelse(tp, agibp);
+ xfs_force_shutdown(pag->pag_mount,
+ SHUTDOWN_CORRUPT_ONDISK);
+ return -EFSCORRUPTED;
+ }
+ }
+#endif /* DEBUG */
+
if (agibpp)
*agibpp = agibp;
else
diff --git a/fs/xfs/libxfs/xfs_sb.c b/fs/xfs/libxfs/xfs_sb.c
index b5f081993cd4..6c5c2b40056d 100644
--- a/fs/xfs/libxfs/xfs_sb.c
+++ b/fs/xfs/libxfs/xfs_sb.c
@@ -1148,26 +1148,23 @@ xfs_sync_sb_buf(
struct xfs_mount *mp)
{
struct xfs_trans *tp;
- struct xfs_buf *bp;
int error;
error = xfs_trans_alloc(mp, &M_RES(mp)->tr_sb, 0, 0, 0, &tp);
if (error)
return error;
- bp = xfs_trans_getsb(tp);
xfs_log_sb(tp);
- xfs_trans_bhold(tp, bp);
xfs_trans_set_sync(tp);
error = xfs_trans_commit(tp);
if (error)
- goto out;
- /*
- * write out the sb buffer to get the changes to disk
- */
- error = xfs_bwrite(bp);
-out:
- xfs_buf_relse(bp);
+ return error;
+
+ /* Re-acquire and write the sb to disk. */
+ xfs_buf_lock(mp->m_sb_bp);
+ error = xfs_bwrite(mp->m_sb_bp);
+ xfs_buf_unlock(mp->m_sb_bp);
+
return error;
}
diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c
index 72d6e2dd5ef5..a5f3408e213d 100644
--- a/fs/xfs/xfs_buf_item.c
+++ b/fs/xfs/xfs_buf_item.c
@@ -32,6 +32,63 @@ static inline struct xfs_buf_log_item *BUF_ITEM(struct xfs_log_item *lip)
return container_of(lip, struct xfs_buf_log_item, bli_item);
}
+static void
+xfs_buf_item_get_format(
+ struct xfs_buf_log_item *bip,
+ int count)
+{
+ ASSERT(bip->bli_formats == NULL);
+ bip->bli_format_count = count;
+
+ if (count == 1) {
+ bip->bli_formats = &bip->__bli_format;
+ return;
+ }
+
+ bip->bli_formats = kmem_zalloc(count * sizeof(struct xfs_buf_log_format),
+ 0);
+}
+
+static void
+xfs_buf_item_free_format(
+ struct xfs_buf_log_item *bip)
+{
+ if (bip->bli_formats != &bip->__bli_format) {
+ kmem_free(bip->bli_formats);
+ bip->bli_formats = NULL;
+ }
+}
+
+static void
+xfs_buf_item_free(
+ struct xfs_buf_log_item *bip)
+{
+ xfs_buf_item_free_format(bip);
+ kmem_free(bip->bli_item.li_lv_shadow);
+ kmem_cache_free(xfs_buf_item_cache, bip);
+}
+
+/*
+ * xfs_buf_item_relse() is called when the buf log item is no longer needed.
+ */
+static void
+xfs_buf_item_relse(
+ struct xfs_buf_log_item *bip)
+{
+ struct xfs_buf *bp = bip->bli_buf;
+
+ trace_xfs_buf_item_relse(bp, _RET_IP_);
+
+ ASSERT(!test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags));
+
+ if (atomic_read(&bip->bli_refcount))
+ return;
+
+ bp->b_log_item = NULL;
+ xfs_buf_rele(bp);
+ xfs_buf_item_free(bip);
+}
+
/* Is this log iovec plausibly large enough to contain the buffer log format? */
bool
xfs_buf_log_check_iovec(
@@ -484,6 +541,42 @@ xfs_buf_item_pin(
atomic_inc(&bip->bli_buf->b_pin_count);
}
+/*
+ * For a stale BLI, process all the necessary completions that must be
+ * performed when the final BLI reference goes away. The buffer will be
+ * referenced and locked here - we return to the caller with the buffer still
+ * referenced and locked for them to finalise processing of the buffer.
+ */
+static void
+xfs_buf_item_finish_stale(
+ struct xfs_buf_log_item *bip)
+{
+ struct xfs_buf *bp = bip->bli_buf;
+ struct xfs_log_item *lip = &bip->bli_item;
+
+ ASSERT(bip->bli_flags & XFS_BLI_STALE);
+ ASSERT(xfs_buf_islocked(bp));
+ ASSERT(bp->b_flags & XBF_STALE);
+ ASSERT(bip->__bli_format.blf_flags & XFS_BLF_CANCEL);
+ ASSERT(list_empty(&lip->li_trans));
+ ASSERT(!bp->b_transp);
+
+ if (bip->bli_flags & XFS_BLI_STALE_INODE) {
+ xfs_buf_item_done(bp);
+ xfs_buf_inode_iodone(bp);
+ ASSERT(list_empty(&bp->b_li_list));
+ return;
+ }
+
+ /*
+ * We may or may not be on the AIL here, xfs_trans_ail_delete() will do
+ * the right thing regardless of the situation in which we are called.
+ */
+ xfs_trans_ail_delete(lip, SHUTDOWN_LOG_IO_ERROR);
+ xfs_buf_item_relse(bip);
+ ASSERT(bp->b_log_item == NULL);
+}
+
/*
* This is called to unpin the buffer associated with the buf log item which was
* previously pinned with a call to xfs_buf_item_pin(). We enter this function
@@ -533,13 +626,6 @@ xfs_buf_item_unpin(
}
if (stale) {
- ASSERT(bip->bli_flags & XFS_BLI_STALE);
- ASSERT(xfs_buf_islocked(bp));
- ASSERT(bp->b_flags & XBF_STALE);
- ASSERT(bip->__bli_format.blf_flags & XFS_BLF_CANCEL);
- ASSERT(list_empty(&lip->li_trans));
- ASSERT(!bp->b_transp);
-
trace_xfs_buf_item_unpin_stale(bip);
/*
@@ -550,22 +636,7 @@ xfs_buf_item_unpin(
* processing is complete.
*/
xfs_buf_rele(bp);
-
- /*
- * If we get called here because of an IO error, we may or may
- * not have the item on the AIL. xfs_trans_ail_delete() will
- * take care of that situation. xfs_trans_ail_delete() drops
- * the AIL lock.
- */
- if (bip->bli_flags & XFS_BLI_STALE_INODE) {
- xfs_buf_item_done(bp);
- xfs_buf_inode_iodone(bp);
- ASSERT(list_empty(&bp->b_li_list));
- } else {
- xfs_trans_ail_delete(lip, SHUTDOWN_LOG_IO_ERROR);
- xfs_buf_item_relse(bp);
- ASSERT(bp->b_log_item == NULL);
- }
+ xfs_buf_item_finish_stale(bip);
xfs_buf_relse(bp);
return;
}
@@ -638,43 +709,42 @@ xfs_buf_item_push(
* Drop the buffer log item refcount and take appropriate action. This helper
* determines whether the bli must be freed or not, since a decrement to zero
* does not necessarily mean the bli is unused.
- *
- * Return true if the bli is freed, false otherwise.
*/
-bool
+void
xfs_buf_item_put(
struct xfs_buf_log_item *bip)
{
- struct xfs_log_item *lip = &bip->bli_item;
- bool aborted;
- bool dirty;
+
+ ASSERT(xfs_buf_islocked(bip->bli_buf));
/* drop the bli ref and return if it wasn't the last one */
if (!atomic_dec_and_test(&bip->bli_refcount))
- return false;
+ return;
- /*
- * We dropped the last ref and must free the item if clean or aborted.
- * If the bli is dirty and non-aborted, the buffer was clean in the
- * transaction but still awaiting writeback from previous changes. In
- * that case, the bli is freed on buffer writeback completion.
- */
- aborted = test_bit(XFS_LI_ABORTED, &lip->li_flags) ||
- xlog_is_shutdown(lip->li_log);
- dirty = bip->bli_flags & XFS_BLI_DIRTY;
- if (dirty && !aborted)
- return false;
+ /* If the BLI is in the AIL, then it is still dirty and in use */
+ if (test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags)) {
+ ASSERT(bip->bli_flags & XFS_BLI_DIRTY);
+ return;
+ }
/*
- * The bli is aborted or clean. An aborted item may be in the AIL
- * regardless of dirty state. For example, consider an aborted
- * transaction that invalidated a dirty bli and cleared the dirty
- * state.
+ * In shutdown conditions, we can be asked to free a dirty BLI that
+ * isn't in the AIL. This can occur due to a checkpoint aborting a BLI
+ * instead of inserting it into the AIL at checkpoint IO completion. If
+ * there's another bli reference (e.g. a btree cursor holds a clean
+ * reference) and it is released via xfs_trans_brelse(), we can get here
+ * with that aborted, dirty BLI. In this case, it is safe to free the
+ * dirty BLI immediately, as it is not in the AIL and there are no
+ * other references to it.
+ *
+ * We should never get here with a stale BLI via that path as
+ * xfs_trans_brelse() specifically holds onto stale buffers rather than
+ * releasing them.
*/
- if (aborted)
- xfs_trans_ail_delete(lip, 0);
- xfs_buf_item_relse(bip->bli_buf);
- return true;
+ ASSERT(!(bip->bli_flags & XFS_BLI_DIRTY) ||
+ test_bit(XFS_LI_ABORTED, &bip->bli_item.li_flags));
+ ASSERT(!(bip->bli_flags & XFS_BLI_STALE));
+ xfs_buf_item_relse(bip);
}
/*
@@ -695,6 +765,15 @@ xfs_buf_item_put(
* if necessary but do not unlock the buffer. This is for support of
* xfs_trans_bhold(). Make sure the XFS_BLI_HOLD field is cleared if we don't
* free the item.
+ *
+ * If the XFS_BLI_STALE flag is set, the last reference to the BLI *must*
+ * perform a completion abort of any objects attached to the buffer for IO
+ * tracking purposes. This generally only happens in shutdown situations,
+ * normally xfs_buf_item_unpin() will drop the last BLI reference and perform
+ * completion processing. However, because transaction completion can race with
+ * checkpoint completion during a shutdown, this release context may end up
+ * being the last active reference to the BLI and so needs to perform this
+ * cleanup.
*/
STATIC void
xfs_buf_item_release(
@@ -702,18 +781,19 @@ xfs_buf_item_release(
{
struct xfs_buf_log_item *bip = BUF_ITEM(lip);
struct xfs_buf *bp = bip->bli_buf;
- bool released;
bool hold = bip->bli_flags & XFS_BLI_HOLD;
bool stale = bip->bli_flags & XFS_BLI_STALE;
-#if defined(DEBUG) || defined(XFS_WARN)
- bool ordered = bip->bli_flags & XFS_BLI_ORDERED;
- bool dirty = bip->bli_flags & XFS_BLI_DIRTY;
bool aborted = test_bit(XFS_LI_ABORTED,
&lip->li_flags);
+ bool dirty = bip->bli_flags & XFS_BLI_DIRTY;
+#if defined(DEBUG) || defined(XFS_WARN)
+ bool ordered = bip->bli_flags & XFS_BLI_ORDERED;
#endif
trace_xfs_buf_item_release(bip);
+ ASSERT(xfs_buf_islocked(bp));
+
/*
* The bli dirty state should match whether the blf has logged segments
* except for ordered buffers, where only the bli should be dirty.
@@ -729,16 +809,56 @@ xfs_buf_item_release(
bp->b_transp = NULL;
bip->bli_flags &= ~(XFS_BLI_LOGGED | XFS_BLI_HOLD | XFS_BLI_ORDERED);
+ /* If there are other references, then we have nothing to do. */
+ if (!atomic_dec_and_test(&bip->bli_refcount))
+ goto out_release;
+
+ /*
+ * Stale buffer completion frees the BLI, unlocks and releases the
+ * buffer. Neither the BLI or buffer are safe to reference after this
+ * call, so there's nothing more we need to do here.
+ *
+ * If we get here with a stale buffer and references to the BLI remain,
+ * we must not unlock the buffer as the last BLI reference owns lock
+ * context, not us.
+ */
+ if (stale) {
+ xfs_buf_item_finish_stale(bip);
+ xfs_buf_relse(bp);
+ ASSERT(!hold);
+ return;
+ }
+
+ /*
+ * Dirty or clean, aborted items are done and need to be removed from
+ * the AIL and released. This frees the BLI, but leaves the buffer
+ * locked and referenced.
+ */
+ if (aborted || xlog_is_shutdown(lip->li_log)) {
+ ASSERT(list_empty(&bip->bli_buf->b_li_list));
+ xfs_buf_item_done(bp);
+ goto out_release;
+ }
+
+ /*
+ * Clean, unreferenced BLIs can be immediately freed, leaving the buffer
+ * locked and referenced.
+ *
+ * Dirty, unreferenced BLIs *must* be in the AIL awaiting writeback.
+ */
+ if (!dirty)
+ xfs_buf_item_relse(bip);
+ else
+ ASSERT(test_bit(XFS_LI_IN_AIL, &lip->li_flags));
+
+ /* Not safe to reference the BLI from here */
+out_release:
/*
- * Unref the item and unlock the buffer unless held or stale. Stale
- * buffers remain locked until final unpin unless the bli is freed by
- * the unref call. The latter implies shutdown because buffer
- * invalidation dirties the bli and transaction.
+ * If we get here with a stale buffer, we must not unlock the
+ * buffer as the last BLI reference owns lock context, not us.
*/
- released = xfs_buf_item_put(bip);
- if (hold || (stale && !released))
+ if (stale || hold)
return;
- ASSERT(!stale || aborted);
xfs_buf_relse(bp);
}
@@ -824,33 +944,6 @@ static const struct xfs_item_ops xfs_buf_item_ops = {
.iop_push = xfs_buf_item_push,
};
-STATIC void
-xfs_buf_item_get_format(
- struct xfs_buf_log_item *bip,
- int count)
-{
- ASSERT(bip->bli_formats == NULL);
- bip->bli_format_count = count;
-
- if (count == 1) {
- bip->bli_formats = &bip->__bli_format;
- return;
- }
-
- bip->bli_formats = kmem_zalloc(count * sizeof(struct xfs_buf_log_format),
- 0);
-}
-
-STATIC void
-xfs_buf_item_free_format(
- struct xfs_buf_log_item *bip)
-{
- if (bip->bli_formats != &bip->__bli_format) {
- kmem_free(bip->bli_formats);
- bip->bli_formats = NULL;
- }
-}
-
/*
* Allocate a new buf log item to go with the given buffer.
* Set the buffer's b_log_item field to point to the new
@@ -1072,34 +1165,6 @@ xfs_buf_item_dirty_format(
return false;
}
-STATIC void
-xfs_buf_item_free(
- struct xfs_buf_log_item *bip)
-{
- xfs_buf_item_free_format(bip);
- kmem_free(bip->bli_item.li_lv_shadow);
- kmem_cache_free(xfs_buf_item_cache, bip);
-}
-
-/*
- * xfs_buf_item_relse() is called when the buf log item is no longer needed.
- */
-void
-xfs_buf_item_relse(
- struct xfs_buf *bp)
-{
- struct xfs_buf_log_item *bip = bp->b_log_item;
-
- trace_xfs_buf_item_relse(bp, _RET_IP_);
- ASSERT(!test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags));
-
- if (atomic_read(&bip->bli_refcount))
- return;
- bp->b_log_item = NULL;
- xfs_buf_rele(bp);
- xfs_buf_item_free(bip);
-}
-
void
xfs_buf_item_done(
struct xfs_buf *bp)
@@ -1119,5 +1184,5 @@ xfs_buf_item_done(
xfs_trans_ail_delete(&bp->b_log_item->bli_item,
(bp->b_flags & _XBF_LOGRECOVERY) ? 0 :
SHUTDOWN_CORRUPT_INCORE);
- xfs_buf_item_relse(bp);
+ xfs_buf_item_relse(bp->b_log_item);
}
diff --git a/fs/xfs/xfs_buf_item.h b/fs/xfs/xfs_buf_item.h
index 7ddda0aa9dac..141c3780a36b 100644
--- a/fs/xfs/xfs_buf_item.h
+++ b/fs/xfs/xfs_buf_item.h
@@ -49,8 +49,7 @@ struct xfs_buf_log_item {
int xfs_buf_item_init(struct xfs_buf *, struct xfs_mount *);
void xfs_buf_item_done(struct xfs_buf *bp);
-void xfs_buf_item_relse(struct xfs_buf *);
-bool xfs_buf_item_put(struct xfs_buf_log_item *);
+void xfs_buf_item_put(struct xfs_buf_log_item *bip);
void xfs_buf_item_log(struct xfs_buf_log_item *, uint, uint);
bool xfs_buf_item_dirty_format(struct xfs_buf_log_item *);
void xfs_buf_inode_iodone(struct xfs_buf *);
diff --git a/fs/xfs/xfs_fsmap.c b/fs/xfs/xfs_fsmap.c
index a0668a1ef100..b8c8a2f8a4aa 100644
--- a/fs/xfs/xfs_fsmap.c
+++ b/fs/xfs/xfs_fsmap.c
@@ -902,7 +902,7 @@ xfs_getfsmap(
return -EINVAL;
use_rmap = xfs_has_rmapbt(mp) &&
- has_capability_noaudit(current, CAP_SYS_ADMIN);
+ ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN);
head->fmh_entries = 0;
/* Set up our device handlers. */
diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c
index 5f6c2a1f3299..d4eb83e7934d 100644
--- a/fs/xfs/xfs_icache.c
+++ b/fs/xfs/xfs_icache.c
@@ -916,7 +916,15 @@ xfs_reclaim_inode(
*/
if (xlog_is_shutdown(ip->i_mount->m_log)) {
xfs_iunpin_wait(ip);
+ /*
+ * Avoid a ABBA deadlock on the inode cluster buffer vs
+ * concurrent xfs_ifree_cluster() trying to mark the inode
+ * stale. We don't need the inode locked to run the flush abort
+ * code, but the flush abort needs to lock the cluster buffer.
+ */
+ xfs_iunlock(ip, XFS_ILOCK_EXCL);
xfs_iflush_shutdown_abort(ip);
+ xfs_ilock(ip, XFS_ILOCK_EXCL);
goto reclaim;
}
if (xfs_ipincount(ip))
diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c
index 3ccbc31767b3..3139363d892b 100644
--- a/fs/xfs/xfs_inode.c
+++ b/fs/xfs/xfs_inode.c
@@ -2225,7 +2225,7 @@ xfs_ifree_mark_inode_stale(
iip = ip->i_itemp;
if (__xfs_iflags_test(ip, XFS_IFLUSHING)) {
ASSERT(!list_empty(&iip->ili_item.li_bio_list));
- ASSERT(iip->ili_last_fields);
+ ASSERT(iip->ili_last_fields || xlog_is_shutdown(mp->m_log));
goto out_iunlock;
}
diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c
index ef3dc0778566..c4afe6947294 100644
--- a/fs/xfs/xfs_ioctl.c
+++ b/fs/xfs/xfs_ioctl.c
@@ -1222,7 +1222,7 @@ xfs_ioctl_setattr_get_trans(
goto out_error;
error = xfs_trans_alloc_ichange(ip, NULL, NULL, pdqp,
- has_capability_noaudit(current, CAP_FOWNER), &tp);
+ ns_capable_noaudit(&init_user_ns, CAP_FOWNER), &tp);
if (error)
goto out_error;
diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c
index 9ca1b8bf1f05..4bf9de257b62 100644
--- a/fs/xfs/xfs_iops.c
+++ b/fs/xfs/xfs_iops.c
@@ -707,7 +707,7 @@ xfs_setattr_nonsize(
}
error = xfs_trans_alloc_ichange(ip, udqp, gdqp, NULL,
- has_capability_noaudit(current, CAP_FOWNER), &tp);
+ ns_capable_noaudit(&init_user_ns, CAP_FOWNER), &tp);
if (error)
goto out_dqrele;
diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c
index ffd52f5a4b89..bf510a6fb92e 100644
--- a/fs/xfs/xfs_log_recover.c
+++ b/fs/xfs/xfs_log_recover.c
@@ -2694,12 +2694,13 @@ xlog_recover_iunlink_bucket(
{
struct xfs_mount *mp = pag->pag_mount;
struct xfs_inode *prev_ip = NULL;
- struct xfs_inode *ip;
xfs_agino_t prev_agino, agino;
int error = 0;
agino = be32_to_cpu(agi->agi_unlinked[bucket]);
while (agino != NULLAGINO) {
+ struct xfs_inode *ip;
+
error = xfs_iget(mp, NULL,
XFS_AGINO_TO_INO(mp, pag->pag_agno, agino),
0, 0, &ip);
@@ -2709,11 +2710,11 @@ xlog_recover_iunlink_bucket(
ASSERT(VFS_I(ip)->i_nlink == 0);
ASSERT(VFS_I(ip)->i_mode != 0);
xfs_iflags_clear(ip, XFS_IRECOVERY);
- agino = ip->i_next_unlinked;
if (prev_ip) {
ip->i_prev_unlinked = prev_agino;
xfs_irele(prev_ip);
+ prev_ip = NULL;
/*
* Ensure the inode is removed from the unlinked list
@@ -2725,18 +2726,20 @@ xlog_recover_iunlink_bucket(
* complete.
*/
error = xfs_inodegc_flush(mp);
- if (error)
- break;
+ if (error) {
+ xfs_irele(ip);
+ return error;
+ }
}
prev_agino = agino;
+ agino = ip->i_next_unlinked;
prev_ip = ip;
}
if (prev_ip) {
int error2;
- ip->i_prev_unlinked = prev_agino;
xfs_irele(prev_ip);
error2 = xfs_inodegc_flush(mp);
diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h
index 6badc50ec4e6..147b560947c6 100644
--- a/include/acpi/acpi_bus.h
+++ b/include/acpi/acpi_bus.h
@@ -556,6 +556,7 @@ void acpi_bus_unregister_driver(struct acpi_driver *driver);
int acpi_bus_scan(acpi_handle handle);
void acpi_bus_trim(struct acpi_device *start);
acpi_status acpi_bus_get_ejd(acpi_handle handle, acpi_handle * ejd);
+struct device *acpi_bus_get_primary_device(struct acpi_device *adev);
int acpi_match_device_ids(struct acpi_device *device,
const struct acpi_device_id *ids);
void acpi_set_modalias(struct acpi_device *adev, const char *default_id,
@@ -797,6 +798,11 @@ static inline void acpi_put_acpi_dev(struct acpi_device *adev)
}
#else /* CONFIG_ACPI */
+static inline struct device *acpi_bus_get_primary_device(struct acpi_device *adev)
+{
+ return NULL;
+}
+
static inline int register_acpi_bus_type(void *bus) { return 0; }
static inline int unregister_acpi_bus_type(void *bus) { return 0; }
diff --git a/include/acpi/battery.h b/include/acpi/battery.h
index b8d56b702c7a..611a2561a014 100644
--- a/include/acpi/battery.h
+++ b/include/acpi/battery.h
@@ -12,8 +12,8 @@
struct acpi_battery_hook {
const char *name;
- int (*add_battery)(struct power_supply *battery);
- int (*remove_battery)(struct power_supply *battery);
+ int (*add_battery)(struct power_supply *battery, struct acpi_battery_hook *hook);
+ int (*remove_battery)(struct power_supply *battery, struct acpi_battery_hook *hook);
struct list_head list;
};
diff --git a/include/clocksource/samsung_pwm.h b/include/clocksource/samsung_pwm.h
index 9b435caa95fe..36f6f246e559 100644
--- a/include/clocksource/samsung_pwm.h
+++ b/include/clocksource/samsung_pwm.h
@@ -15,7 +15,7 @@
* spinlock is not shared between both drivers.
*/
#ifdef CONFIG_CLKSRC_SAMSUNG_PWM
-extern spinlock_t samsung_pwm_lock;
+extern raw_spinlock_t samsung_pwm_lock;
#endif
struct samsung_pwm_variant {
diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h
index 17e7e3145a05..8ee129d3796f 100644
--- a/include/drm/gpu_scheduler.h
+++ b/include/drm/gpu_scheduler.h
@@ -470,11 +470,32 @@ struct drm_gpu_scheduler {
struct device *dev;
};
+/**
+ * struct drm_sched_init_args - parameters for initializing a DRM GPU scheduler
+ *
+ * @ops: backend operations provided by the driver
+ * @credit_limit: the number of credits this scheduler can hold from all jobs
+ * @hang_limit: number of times to allow a job to hang before dropping it.
+ * This mechanism is DEPRECATED. Set it to 0.
+ * @timeout: timeout value in jiffies for submitted jobs.
+ * @timeout_wq: workqueue to use for timeout work. If NULL, the system_wq is used.
+ * @score: score atomic shared with other schedulers. May be NULL.
+ * @name: name (typically the driver's name). Used for debugging
+ * @dev: associated device. Used for debugging
+ */
+struct drm_sched_init_args {
+ const struct drm_sched_backend_ops *ops;
+ struct workqueue_struct *timeout_wq;
+ u32 credit_limit;
+ unsigned int hang_limit;
+ long timeout;
+ atomic_t *score;
+ const char *name;
+ struct device *dev;
+};
+
int drm_sched_init(struct drm_gpu_scheduler *sched,
- const struct drm_sched_backend_ops *ops,
- uint32_t hw_submission, unsigned hang_limit,
- long timeout, struct workqueue_struct *timeout_wq,
- atomic_t *score, const char *name, struct device *dev);
+ const struct drm_sched_init_args *args);
void drm_sched_fini(struct drm_gpu_scheduler *sched);
int drm_sched_job_init(struct drm_sched_job *job,
diff --git a/include/linux/blkdev.h b/include/linux/blkdev.h
index de013fc37ef8..c5d07c7609d4 100644
--- a/include/linux/blkdev.h
+++ b/include/linux/blkdev.h
@@ -397,6 +397,7 @@ struct request_queue {
struct blk_queue_stats *stats;
struct rq_qos *rq_qos;
+ struct mutex rq_qos_mutex;
const struct blk_mq_ops *mq_ops;
diff --git a/include/linux/bpf.h b/include/linux/bpf.h
index 92884bf344f8..0ce4f1eb92d1 100644
--- a/include/linux/bpf.h
+++ b/include/linux/bpf.h
@@ -354,7 +354,10 @@ static inline void __copy_map_value(struct bpf_map *map, void *dst, void *src, b
memcpy(dst + curr_off, src + curr_off, next_off - curr_off);
curr_off = next_off + map->off_arr->field_sz[i];
}
- memcpy(dst + curr_off, src + curr_off, map->value_size - curr_off);
+ if (long_memcpy)
+ memcpy(dst + curr_off, src + curr_off, round_up(map->value_size, 8) - curr_off);
+ else
+ memcpy(dst + curr_off, src + curr_off, map->value_size - curr_off);
}
static inline void copy_map_value(struct bpf_map *map, void *dst, void *src)
@@ -1014,13 +1017,19 @@ static __always_inline __nocfi unsigned int bpf_dispatcher_nop_func(
}
#ifdef CONFIG_BPF_JIT
-int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr);
-int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr);
+int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
+ struct bpf_trampoline *tr,
+ struct bpf_prog *tgt_prog);
+int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
+ struct bpf_trampoline *tr,
+ struct bpf_prog *tgt_prog);
struct bpf_trampoline *bpf_trampoline_get(u64 key,
struct bpf_attach_target_info *tgt_info);
void bpf_trampoline_put(struct bpf_trampoline *tr);
int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs);
+void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags);
+
/*
* When the architecture supports STATIC_CALL replace the bpf_dispatcher_fn
* indirection with a direct call to the bpf program. If the architecture does
@@ -1096,12 +1105,14 @@ void bpf_jit_uncharge_modmem(u32 size);
bool bpf_prog_has_trampoline(const struct bpf_prog *prog);
#else
static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
- struct bpf_trampoline *tr)
+ struct bpf_trampoline *tr,
+ struct bpf_prog *tgt_prog)
{
return -ENOTSUPP;
}
static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
- struct bpf_trampoline *tr)
+ struct bpf_trampoline *tr,
+ struct bpf_prog *tgt_prog)
{
return -ENOTSUPP;
}
@@ -1126,6 +1137,7 @@ static inline bool bpf_prog_has_trampoline(const struct bpf_prog *prog)
{
return false;
}
+static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) {}
#endif
struct bpf_func_info_aux {
@@ -1198,6 +1210,9 @@ struct bpf_prog_aux {
bool sleepable;
bool tail_call_reachable;
bool xdp_has_frags;
+ bool is_extended; /* true if extended by freplace program */
+ u64 prog_array_member_cnt; /* counts how many times as member of prog_array */
+ struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */
/* BTF_KIND_FUNC_PROTO for valid attach_btf_id */
const struct btf_type *attach_func_proto;
/* function name for valid attach_btf_id */
@@ -1305,12 +1320,26 @@ struct bpf_link {
enum bpf_link_type type;
const struct bpf_link_ops *ops;
struct bpf_prog *prog;
- struct work_struct work;
+ /* rcu is used before freeing, work can be used to schedule that
+ * RCU-based freeing before that, so they never overlap
+ */
+ union {
+ struct rcu_head rcu;
+ struct work_struct work;
+ };
};
struct bpf_link_ops {
void (*release)(struct bpf_link *link);
+ /* deallocate link resources callback, called without RCU grace period
+ * waiting
+ */
void (*dealloc)(struct bpf_link *link);
+ /* deallocate link resources callback, called after RCU grace period;
+ * if underlying BPF program is sleepable we go through tasks trace
+ * RCU GP and then "classic" RCU GP
+ */
+ void (*dealloc_deferred)(struct bpf_link *link);
int (*detach)(struct bpf_link *link);
int (*update_prog)(struct bpf_link *link, struct bpf_prog *new_prog,
struct bpf_prog *old_prog);
@@ -1790,6 +1819,7 @@ struct bpf_map *bpf_map_get_with_uref(u32 ufd);
struct bpf_map *__bpf_map_get(struct fd f);
void bpf_map_inc(struct bpf_map *map);
void bpf_map_inc_with_uref(struct bpf_map *map);
+struct bpf_map *__bpf_map_inc_not_zero(struct bpf_map *map, bool uref);
struct bpf_map * __must_check bpf_map_inc_not_zero(struct bpf_map *map);
void bpf_map_put_with_uref(struct bpf_map *map);
void bpf_map_put(struct bpf_map *map);
diff --git a/include/linux/can/skb.h b/include/linux/can/skb.h
index 869ea574a40a..27d0ca115c6c 100644
--- a/include/linux/can/skb.h
+++ b/include/linux/can/skb.h
@@ -49,11 +49,13 @@ bool can_dropped_invalid_skb(struct net_device *dev, struct sk_buff *skb);
/**
* struct can_skb_priv - private additional data inside CAN sk_buffs
* @ifindex: ifindex of the first interface the CAN frame appeared on
+ * @skbcnt: atomic counter to have an unique id together with skb pointer
* @frame_len: length of CAN frame in data link layer
* @cf: align to the following CAN frame at skb->data
*/
struct can_skb_priv {
int ifindex;
+ u32 skbcnt;
unsigned int frame_len;
struct can_frame cf[];
};
diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index 6ab81c06afbc..7b6f48490749 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -421,7 +421,10 @@ struct cgroup {
int nr_threaded_children; /* # of live threaded child cgroups */
- /* sequence number for cgroup.kill, serialized by css_set_lock. */
+ /*
+ * Sequence number for cgroup.kill. Incremented with both cgroup_mutex
+ * and css_set_lock held. Readers hold either one.
+ */
unsigned int kill_seq;
struct kernfs_node *kn; /* cgroup kernfs entry */
diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h
index 2b7d077de7ef..255b05e18cab 100644
--- a/include/linux/cgroup.h
+++ b/include/linux/cgroup.h
@@ -762,6 +762,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen)
/*
* cgroup scalable recursive statistics.
*/
+void __cgroup_rstat_updated(struct cgroup *cgrp, int cpu);
void cgroup_rstat_updated(struct cgroup *cgrp, int cpu);
void cgroup_rstat_flush(struct cgroup *cgrp);
void cgroup_rstat_flush_irqsafe(struct cgroup *cgrp);
diff --git a/include/linux/compiler_attributes.h b/include/linux/compiler_attributes.h
index b9778e13b0a0..b54961e97aef 100644
--- a/include/linux/compiler_attributes.h
+++ b/include/linux/compiler_attributes.h
@@ -107,6 +107,19 @@
# define __copy(symbol)
#endif
+/*
+ * Optional: only supported since gcc >= 14
+ * Optional: only supported since clang >= 18
+ *
+ * gcc: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108896
+ * clang: https://reviews.llvm.org/D148381
+ */
+#if __has_attribute(__counted_by__)
+# define __counted_by(member) __attribute__((__counted_by__(member)))
+#else
+# define __counted_by(member)
+#endif
+
/*
* Optional: not supported by gcc
* Optional: only supported since clang >= 14.0
diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h
index 201dd1ab7f1c..bda8f373c91b 100644
--- a/include/linux/device-mapper.h
+++ b/include/linux/device-mapper.h
@@ -357,6 +357,11 @@ struct dm_target {
*/
bool discards_supported:1;
+ /*
+ * Set if this target requires stable data during writeback.
+ */
+ bool needs_stable_writes:1;
+
/*
* Set if we need to limit the number of in-flight bios when swapping.
*/
@@ -433,10 +438,12 @@ const char *dm_shift_arg(struct dm_arg_set *as);
*/
void dm_consume_args(struct dm_arg_set *as, unsigned int num_args);
-/*-----------------------------------------------------------------
+/*
+ *----------------------------------------------------------------
* Functions for creating and manipulating mapped devices.
* Drop the reference with dm_put when you finish with the object.
- *---------------------------------------------------------------*/
+ *----------------------------------------------------------------
+ */
/*
* DM_ANY_MINOR chooses the next available minor number.
@@ -517,9 +524,11 @@ struct queue_limits *dm_get_queue_limits(struct mapped_device *md);
int dm_get_geometry(struct mapped_device *md, struct hd_geometry *geo);
int dm_set_geometry(struct mapped_device *md, struct hd_geometry *geo);
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Functions for manipulating device-mapper tables.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
/*
* First create an empty table.
@@ -593,9 +602,11 @@ struct dm_table *dm_swap_table(struct mapped_device *md,
*/
void dm_destroy_crypto_profile(struct blk_crypto_profile *profile);
-/*-----------------------------------------------------------------
+/*
+ *---------------------------------------------------------------
* Macros.
- *---------------------------------------------------------------*/
+ *---------------------------------------------------------------
+ */
#define DM_NAME "device-mapper"
#define DM_FMT(fmt) DM_NAME ": " DM_MSG_PREFIX ": " fmt "\n"
diff --git a/include/linux/device.h b/include/linux/device.h
index eae657af3a0e..0e83be1bccc4 100644
--- a/include/linux/device.h
+++ b/include/linux/device.h
@@ -957,6 +957,9 @@ int device_for_each_child(struct device *dev, void *data,
int (*fn)(struct device *dev, void *data));
int device_for_each_child_reverse(struct device *dev, void *data,
int (*fn)(struct device *dev, void *data));
+int device_for_each_child_reverse_from(struct device *parent,
+ struct device *from, const void *data,
+ int (*fn)(struct device *, const void *));
struct device *device_find_child(struct device *dev, void *data,
int (*match)(struct device *dev, void *data));
struct device *device_find_child_by_name(struct device *parent,
diff --git a/include/linux/dm-dirty-log.h b/include/linux/dm-dirty-log.h
index 843c857f07b0..36e72cb19813 100644
--- a/include/linux/dm-dirty-log.h
+++ b/include/linux/dm-dirty-log.h
@@ -96,7 +96,7 @@ struct dm_dirty_log_type {
* Do not confuse this function with 'in_sync()', one
* tells you if an area is synchronised, the other
* assigns recovery work.
- */
+ */
int (*get_resync_work)(struct dm_dirty_log *log, region_t *region);
/*
diff --git a/include/linux/dm-region-hash.h b/include/linux/dm-region-hash.h
index e8691539e1d7..3ab31308bdce 100644
--- a/include/linux/dm-region-hash.h
+++ b/include/linux/dm-region-hash.h
@@ -12,9 +12,11 @@
#include <linux/dm-dirty-log.h>
-/*-----------------------------------------------------------------
+/*
+ *----------------------------------------------------------------
* Region hash
- *----------------------------------------------------------------*/
+ *----------------------------------------------------------------
+ */
struct dm_region_hash;
struct dm_region;
diff --git a/include/linux/dma-buf.h b/include/linux/dma-buf.h
index 9c31f1f430d8..f4e2ec3320c4 100644
--- a/include/linux/dma-buf.h
+++ b/include/linux/dma-buf.h
@@ -615,6 +615,7 @@ void dma_buf_unpin(struct dma_buf_attachment *attach);
struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info);
int dma_buf_fd(struct dma_buf *dmabuf, int flags);
+void dma_buf_fd_install(struct dma_buf *dmabuf, int fd);
struct dma_buf *dma_buf_get(int fd);
void dma_buf_put(struct dma_buf *dmabuf);
diff --git a/include/linux/dma/k3-udma-glue.h b/include/linux/dma/k3-udma-glue.h
index e443be4d3b4b..a76f1c09a884 100644
--- a/include/linux/dma/k3-udma-glue.h
+++ b/include/linux/dma/k3-udma-glue.h
@@ -130,8 +130,7 @@ void k3_udma_glue_rx_put_irq(struct k3_udma_glue_rx_channel *rx_chn,
u32 flow_num);
void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn,
u32 flow_num, void *data,
- void (*cleanup)(void *data, dma_addr_t desc_dma),
- bool skip_fdq);
+ void (*cleanup)(void *data, dma_addr_t desc_dma));
int k3_udma_glue_rx_flow_enable(struct k3_udma_glue_rx_channel *rx_chn,
u32 flow_idx);
int k3_udma_glue_rx_flow_disable(struct k3_udma_glue_rx_channel *rx_chn,
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index 3576c6e89fea..368a03920c92 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -1489,6 +1489,7 @@ struct dma_chan *__dma_request_channel(const dma_cap_mask_t *mask,
struct dma_chan *dma_request_chan(struct device *dev, const char *name);
struct dma_chan *dma_request_chan_by_mask(const dma_cap_mask_t *mask);
+struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name);
void dma_release_channel(struct dma_chan *chan);
int dma_get_slave_caps(struct dma_chan *chan, struct dma_slave_caps *caps);
@@ -1525,6 +1526,12 @@ static inline struct dma_chan *dma_request_chan_by_mask(
{
return ERR_PTR(-ENODEV);
}
+
+static inline struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name)
+{
+ return ERR_PTR(-ENODEV);
+}
+
static inline void dma_release_channel(struct dma_chan *chan)
{
}
diff --git a/include/linux/f2fs_fs.h b/include/linux/f2fs_fs.h
index 1373c037255d..bf834b801b55 100644
--- a/include/linux/f2fs_fs.h
+++ b/include/linux/f2fs_fs.h
@@ -27,6 +27,7 @@
#define F2FS_BYTES_TO_BLK(bytes) ((bytes) >> F2FS_BLKSIZE_BITS)
#define F2FS_BLK_TO_BYTES(blk) ((blk) << F2FS_BLKSIZE_BITS)
+#define F2FS_BLK_END_BYTES(blk) (F2FS_BLK_TO_BYTES(blk + 1) - 1)
/* 0, 1(node nid), 2(meta nid) are reserved node id */
#define F2FS_RESERVED_NODE_NUM 3
diff --git a/include/linux/fb.h b/include/linux/fb.h
index 5f94c58c4672..10844193fb9d 100644
--- a/include/linux/fb.h
+++ b/include/linux/fb.h
@@ -601,12 +601,37 @@ static inline struct apertures_struct *alloc_apertures(unsigned int max_num) {
extern int fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var);
extern int fb_pan_display(struct fb_info *info, struct fb_var_screeninfo *var);
extern int fb_blank(struct fb_info *info, int blank);
+
+int fb_set_var_from_user(struct fb_info *info, struct fb_var_screeninfo *var);
+
extern void cfb_fillrect(struct fb_info *info, const struct fb_fillrect *rect);
extern void cfb_copyarea(struct fb_info *info, const struct fb_copyarea *area);
extern void cfb_imageblit(struct fb_info *info, const struct fb_image *image);
+/*
+ * Initializes struct fb_ops for framebuffers in I/O memory.
+ */
+
+#define __FB_DEFAULT_IO_OPS_RDWR \
+ .fb_read = fb_io_read, \
+ .fb_write = fb_io_write
+
+#define __FB_DEFAULT_IO_OPS_DRAW \
+ .fb_fillrect = cfb_fillrect, \
+ .fb_copyarea = cfb_copyarea, \
+ .fb_imageblit = cfb_imageblit
+
+#define __FB_DEFAULT_IO_OPS_MMAP \
+ .fb_mmap = NULL /* default implementation */
+
+#define FB_DEFAULT_IO_OPS \
+ __FB_DEFAULT_IO_OPS_RDWR, \
+ __FB_DEFAULT_IO_OPS_DRAW, \
+ __FB_DEFAULT_IO_OPS_MMAP
+
/*
* Drawing operations where framebuffer is in system RAM
*/
+
extern void sys_fillrect(struct fb_info *info, const struct fb_fillrect *rect);
extern void sys_copyarea(struct fb_info *info, const struct fb_copyarea *area);
extern void sys_imageblit(struct fb_info *info, const struct fb_image *image);
@@ -615,6 +640,27 @@ extern ssize_t fb_sys_read(struct fb_info *info, char __user *buf,
extern ssize_t fb_sys_write(struct fb_info *info, const char __user *buf,
size_t count, loff_t *ppos);
+/*
+ * Initializes struct fb_ops for framebuffers in system memory.
+ */
+
+#define __FB_DEFAULT_SYS_OPS_RDWR \
+ .fb_read = fb_sys_read, \
+ .fb_write = fb_sys_write
+
+#define __FB_DEFAULT_SYS_OPS_DRAW \
+ .fb_fillrect = sys_fillrect, \
+ .fb_copyarea = sys_copyarea, \
+ .fb_imageblit = sys_imageblit
+
+#define __FB_DEFAULT_SYS_OPS_MMAP \
+ .fb_mmap = NULL /* default implementation */
+
+#define FB_DEFAULT_SYS_OPS \
+ __FB_DEFAULT_SYS_OPS_RDWR, \
+ __FB_DEFAULT_SYS_OPS_DRAW, \
+ __FB_DEFAULT_SYS_OPS_MMAP
+
/* drivers/video/fbmem.c */
extern int register_framebuffer(struct fb_info *fb_info);
extern void unregister_framebuffer(struct fb_info *fb_info);
@@ -670,6 +716,75 @@ extern void fb_deferred_io_cleanup(struct fb_info *info);
extern int fb_deferred_io_fsync(struct file *file, loff_t start,
loff_t end, int datasync);
+/*
+ * Generate callbacks for deferred I/O
+ */
+
+#define __FB_GEN_DEFAULT_DEFERRED_OPS_RDWR(__prefix, __damage_range, __mode) \
+ static ssize_t __prefix ## _defio_read(struct fb_info *info, char __user *buf, \
+ size_t count, loff_t *ppos) \
+ { \
+ return fb_ ## __mode ## _read(info, buf, count, ppos); \
+ } \
+ static ssize_t __prefix ## _defio_write(struct fb_info *info, const char __user *buf, \
+ size_t count, loff_t *ppos) \
+ { \
+ unsigned long offset = *ppos; \
+ ssize_t ret = fb_ ## __mode ## _write(info, buf, count, ppos); \
+ if (ret > 0) \
+ __damage_range(info, offset, ret); \
+ return ret; \
+ }
+
+#define __FB_GEN_DEFAULT_DEFERRED_OPS_DRAW(__prefix, __damage_area, __mode) \
+ static void __prefix ## _defio_fillrect(struct fb_info *info, \
+ const struct fb_fillrect *rect) \
+ { \
+ __mode ## _fillrect(info, rect); \
+ __damage_area(info, rect->dx, rect->dy, rect->width, rect->height); \
+ } \
+ static void __prefix ## _defio_copyarea(struct fb_info *info, \
+ const struct fb_copyarea *area) \
+ { \
+ __mode ## _copyarea(info, area); \
+ __damage_area(info, area->dx, area->dy, area->width, area->height); \
+ } \
+ static void __prefix ## _defio_imageblit(struct fb_info *info, \
+ const struct fb_image *image) \
+ { \
+ __mode ## _imageblit(info, image); \
+ __damage_area(info, image->dx, image->dy, image->width, image->height); \
+ }
+
+#define FB_GEN_DEFAULT_DEFERRED_IO_OPS(__prefix, __damage_range, __damage_area) \
+ __FB_GEN_DEFAULT_DEFERRED_OPS_RDWR(__prefix, __damage_range, io) \
+ __FB_GEN_DEFAULT_DEFERRED_OPS_DRAW(__prefix, __damage_area, cfb)
+
+#define FB_GEN_DEFAULT_DEFERRED_SYS_OPS(__prefix, __damage_range, __damage_area) \
+ __FB_GEN_DEFAULT_DEFERRED_OPS_RDWR(__prefix, __damage_range, sys) \
+ __FB_GEN_DEFAULT_DEFERRED_OPS_DRAW(__prefix, __damage_area, sys)
+
+/*
+ * Initializes struct fb_ops for deferred I/O.
+ */
+
+#define __FB_DEFAULT_DEFERRED_OPS_RDWR(__prefix) \
+ .fb_read = __prefix ## _defio_read, \
+ .fb_write = __prefix ## _defio_write
+
+#define __FB_DEFAULT_DEFERRED_OPS_DRAW(__prefix) \
+ .fb_fillrect = __prefix ## _defio_fillrect, \
+ .fb_copyarea = __prefix ## _defio_copyarea, \
+ .fb_imageblit = __prefix ## _defio_imageblit
+
+#define __FB_DEFAULT_DEFERRED_OPS_MMAP(__prefix) \
+ .fb_mmap = fb_deferred_io_mmap
+
+#define FB_DEFAULT_DEFERRED_OPS(__prefix) \
+ __FB_DEFAULT_DEFERRED_OPS_RDWR(__prefix), \
+ __FB_DEFAULT_DEFERRED_OPS_DRAW(__prefix), \
+ __FB_DEFAULT_DEFERRED_OPS_MMAP(__prefix)
+
static inline bool fb_be_math(struct fb_info *info)
{
#ifdef CONFIG_FB_FOREIGN_ENDIAN
diff --git a/include/linux/fs.h b/include/linux/fs.h
index 264381ae06e2..0692c2e0fcf3 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -3483,6 +3483,8 @@ extern int simple_open(struct inode *inode, struct file *file);
extern int simple_link(struct dentry *, struct inode *, struct dentry *);
extern int simple_unlink(struct inode *, struct dentry *);
extern int simple_rmdir(struct inode *, struct dentry *);
+void simple_rename_timestamp(struct inode *old_dir, struct dentry *old_dentry,
+ struct inode *new_dir, struct dentry *new_dentry);
extern int simple_rename_exchange(struct inode *old_dir, struct dentry *old_dentry,
struct inode *new_dir, struct dentry *new_dentry);
extern int simple_rename(struct user_namespace *, struct inode *,
diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h
index 514faca8be42..3b4dd5aeb879 100644
--- a/include/linux/fsnotify_backend.h
+++ b/include/linux/fsnotify_backend.h
@@ -474,6 +474,7 @@ struct fsnotify_mark_connector {
unsigned short type; /* Type of object [lock] */
#define FSNOTIFY_CONN_FLAG_HAS_FSID 0x01
#define FSNOTIFY_CONN_FLAG_HAS_IREF 0x02
+#define FSNOTIFY_CONN_FLAG_DESTROYING 0x04
unsigned short flags; /* flags [lock] */
__kernel_fsid_t fsid; /* fsid of filesystem containing object */
union {
diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h
index d29b74e4e7c3..07003b68b7b2 100644
--- a/include/linux/ftrace.h
+++ b/include/linux/ftrace.h
@@ -644,8 +644,9 @@ unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec);
unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec);
extern ftrace_func_t ftrace_trace_function;
+struct trace_array;
-int ftrace_regex_open(struct ftrace_ops *ops, int flag,
+int ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag,
struct inode *inode, struct file *file);
ssize_t ftrace_filter_write(struct file *file, const char __user *ubuf,
size_t cnt, loff_t *ppos);
@@ -859,7 +860,7 @@ static inline unsigned long ftrace_location(unsigned long ip)
* have them defined when ftrace is not enabled, but these
* functions may still be called. Use a macro instead of inline.
*/
-#define ftrace_regex_open(ops, flag, inod, file) ({ -ENODEV; })
+#define ftrace_regex_open(tr, ops, flag, inode, file) ({ -ENODEV; })
#define ftrace_set_early_filter(ops, buf, enable) do { } while (0)
#define ftrace_set_filter_ip(ops, ip, remove, reset) ({ -ENODEV; })
#define ftrace_set_filter_ips(ops, ips, cnt, remove, reset) ({ -ENODEV; })
diff --git a/include/linux/futex.h b/include/linux/futex.h
index b70df27d7e85..09f757f14d01 100644
--- a/include/linux/futex.h
+++ b/include/linux/futex.h
@@ -23,26 +23,30 @@ struct task_struct;
* 01 : Shared futex (PTHREAD_PROCESS_SHARED)
* mapped on a file (reference on the underlying inode)
* 10 : Shared futex (PTHREAD_PROCESS_SHARED)
- * (but private mapping on an mm, and reference taken on it)
+ * (but private mapping on an mm)
*/
-#define FUT_OFF_INODE 1 /* We set bit 0 if key has a reference on inode */
-#define FUT_OFF_MMSHARED 2 /* We set bit 1 if key has a reference on mm */
+#define FUT_OFF_INODE 1 /* We set bit 0 if shared key identifies an inode */
+#define FUT_OFF_MMSHARED 2 /* We set bit 1 if shared key identifies an mm */
union futex_key {
+ /* For FUT_OFF_INODE */
struct {
u64 i_seq;
unsigned long pgoff;
unsigned int offset;
} shared;
+
+ /* For FUT_OFF_MMSHARED or private */
struct {
union {
- struct mm_struct *mm;
- u64 __tmp;
+ struct mm_struct *mm; /* for private */
+ u64 mm_seq; /* for FUT_OFF_MMSHARED */
};
unsigned long address;
unsigned int offset;
} private;
+
struct {
u64 ptr;
unsigned long word;
diff --git a/include/linux/hid.h b/include/linux/hid.h
index 536a4750b905..09f6ba66f1db 100644
--- a/include/linux/hid.h
+++ b/include/linux/hid.h
@@ -80,6 +80,8 @@ struct hid_item {
#define HID_MAIN_ITEM_TAG_FEATURE 11
#define HID_MAIN_ITEM_TAG_BEGIN_COLLECTION 10
#define HID_MAIN_ITEM_TAG_END_COLLECTION 12
+#define HID_MAIN_ITEM_TAG_RESERVED_MIN 13
+#define HID_MAIN_ITEM_TAG_RESERVED_MAX 15
/*
* HID report descriptor main item contents
@@ -837,11 +839,11 @@ struct hid_driver {
void (*feature_mapping)(struct hid_device *hdev,
struct hid_field *field,
struct hid_usage *usage);
-#ifdef CONFIG_PM
+
int (*suspend)(struct hid_device *hdev, pm_message_t message);
int (*resume)(struct hid_device *hdev);
int (*reset_resume)(struct hid_device *hdev);
-#endif
+
/* private: */
struct device_driver driver;
};
@@ -960,6 +962,8 @@ extern void hidinput_disconnect(struct hid_device *);
int hid_set_field(struct hid_field *, unsigned, __s32);
int hid_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data, u32 size,
int interrupt);
+int hid_safe_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data,
+ size_t bufsize, u32 size, int interrupt);
struct hid_field *hidinput_get_led_field(struct hid_device *hid);
unsigned int hidinput_count_leds(struct hid_device *hid);
__s32 hidinput_calc_abs_res(const struct hid_field *field, __u16 code);
@@ -1244,6 +1248,8 @@ do { \
dev_notice(&(hid)->dev, fmt, ##__VA_ARGS__)
#define hid_warn(hid, fmt, ...) \
dev_warn(&(hid)->dev, fmt, ##__VA_ARGS__)
+#define hid_warn_ratelimited(hid, fmt, ...) \
+ dev_warn_ratelimited(&(hid)->dev, fmt, ##__VA_ARGS__)
#define hid_info(hid, fmt, ...) \
dev_info(&(hid)->dev, fmt, ##__VA_ARGS__)
#define hid_dbg(hid, fmt, ...) \
diff --git a/include/linux/hw_random.h b/include/linux/hw_random.h
index 2505d58bd582..e207044a776a 100644
--- a/include/linux/hw_random.h
+++ b/include/linux/hw_random.h
@@ -16,6 +16,7 @@
#include <linux/types.h>
#include <linux/list.h>
#include <linux/kref.h>
+#include <linux/workqueue.h>
/**
* struct hwrng - Hardware Random Number Generator driver
@@ -49,6 +50,7 @@ struct hwrng {
/* internal. */
struct list_head list;
struct kref ref;
+ struct work_struct cleanup_work;
struct completion cleanup_done;
struct completion dying;
};
diff --git a/include/linux/i2c.h b/include/linux/i2c.h
index 87b403a6d7e4..60c5bc674dcc 100644
--- a/include/linux/i2c.h
+++ b/include/linux/i2c.h
@@ -189,6 +189,7 @@ s32 i2c_smbus_read_i2c_block_data_or_emulated(const struct i2c_client *client,
u8 *values);
int i2c_get_device_id(const struct i2c_client *client,
struct i2c_device_identity *id);
+const struct i2c_device_id *i2c_client_get_device_id(const struct i2c_client *client);
#endif /* I2C */
/**
diff --git a/include/linux/i3c/master.h b/include/linux/i3c/master.h
index 2f731c6c16ea..6f8635a13b61 100644
--- a/include/linux/i3c/master.h
+++ b/include/linux/i3c/master.h
@@ -219,6 +219,8 @@ struct i3c_dev_desc {
* every time the I3C device is rediscovered with a different dynamic
* address assigned
* @bus: I3C bus this device is attached to
+ * @node: unregistered device list node, only for use by
+ * i3c_master_register_new_i3c_devs(), it is not protected by a lock
*
* I3C device object exposed to I3C device drivers. The takes care of linking
* this object to the relevant &struct_i3c_dev_desc one.
@@ -229,6 +231,7 @@ struct i3c_device {
struct device dev;
struct i3c_dev_desc *desc;
struct i3c_bus *bus;
+ struct list_head node;
};
/*
diff --git a/include/linux/ieee80211.h b/include/linux/ieee80211.h
index c16e90c5588e..d7ad2e9b0baa 100644
--- a/include/linux/ieee80211.h
+++ b/include/linux/ieee80211.h
@@ -4601,7 +4601,7 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, u8 len)
const struct ieee80211_multi_link_elem *mle = (const void *)data;
u8 fixed = sizeof(*mle);
u8 common = 0;
- bool check_common_len = false;
+ u8 common_len;
u16 control;
if (!data || len < fixed)
@@ -4612,7 +4612,6 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, u8 len)
switch (u16_get_bits(control, IEEE80211_ML_CONTROL_TYPE)) {
case IEEE80211_ML_CONTROL_TYPE_BASIC:
common += sizeof(struct ieee80211_mle_basic_common_info);
- check_common_len = true;
if (control & IEEE80211_MLC_BASIC_PRES_LINK_ID)
common += 1;
if (control & IEEE80211_MLC_BASIC_PRES_BSS_PARAM_CH_CNT)
@@ -4630,15 +4629,14 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, u8 len)
common += sizeof(struct ieee80211_mle_preq_common_info);
if (control & IEEE80211_MLC_PREQ_PRES_MLD_ID)
common += 1;
- check_common_len = true;
break;
case IEEE80211_ML_CONTROL_TYPE_RECONF:
+ common += 1;
if (control & IEEE80211_MLC_RECONF_PRES_MLD_MAC_ADDR)
common += ETH_ALEN;
break;
case IEEE80211_ML_CONTROL_TYPE_TDLS:
common += sizeof(struct ieee80211_mle_tdls_common_info);
- check_common_len = true;
break;
case IEEE80211_ML_CONTROL_TYPE_PRIO_ACCESS:
common = ETH_ALEN + 1;
@@ -4651,11 +4649,9 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, u8 len)
if (len < fixed + common)
return false;
- if (!check_common_len)
- return true;
+ common_len = mle->variable[0];
- /* if present, common length is the first octet there */
- return mle->variable[0] >= common;
+ return common_len >= common && common_len <= len - fixed;
}
enum ieee80211_mle_subelems {
diff --git a/include/linux/if_team.h b/include/linux/if_team.h
index 34bcba5a7067..7a69cd251bfb 100644
--- a/include/linux/if_team.h
+++ b/include/linux/if_team.h
@@ -121,8 +121,7 @@ struct team_mode_ops {
int (*port_enter)(struct team *team, struct team_port *port);
void (*port_leave)(struct team *team, struct team_port *port);
void (*port_change_dev_addr)(struct team *team, struct team_port *port);
- void (*port_enabled)(struct team *team, struct team_port *port);
- void (*port_disabled)(struct team *team, struct team_port *port);
+ void (*port_tx_disabled)(struct team *team, struct team_port *port);
};
extern int team_modeop_port_enter(struct team *team, struct team_port *port);
diff --git a/include/linux/if_vlan.h b/include/linux/if_vlan.h
index f4eb8dd7308a..e63081a9d7db 100644
--- a/include/linux/if_vlan.h
+++ b/include/linux/if_vlan.h
@@ -351,6 +351,9 @@ static inline int __vlan_insert_inner_tag(struct sk_buff *skb,
{
struct vlan_ethhdr *veth;
+ if (unlikely(!pskb_may_pull(skb, mac_len)))
+ return -EINVAL;
+
if (skb_cow_head(skb, VLAN_HLEN) < 0)
return -ENOMEM;
diff --git a/include/linux/iio/common/st_sensors.h b/include/linux/iio/common/st_sensors.h
index c34e648f07e2..ae8b75595535 100644
--- a/include/linux/iio/common/st_sensors.h
+++ b/include/linux/iio/common/st_sensors.h
@@ -224,8 +224,6 @@ struct st_sensor_settings {
* @mount_matrix: The mounting matrix of the sensor.
* @sensor_settings: Pointer to the specific sensor settings in use.
* @current_fullscale: Maximum range of measure by the sensor.
- * @vdd: Pointer to sensor's Vdd power supply
- * @vdd_io: Pointer to sensor's Vdd-IO power supply
* @regmap: Pointer to specific sensor regmap configuration.
* @enabled: Status of the sensor (false->off, true->on).
* @odr: Output data rate of the sensor [Hz].
@@ -244,8 +242,6 @@ struct st_sensor_data {
struct iio_mount_matrix mount_matrix;
struct st_sensor_settings *sensor_settings;
struct st_sensor_fullscale_avl *current_fullscale;
- struct regulator *vdd;
- struct regulator *vdd_io;
struct regmap *regmap;
bool enabled;
diff --git a/include/linux/ima.h b/include/linux/ima.h
index 83ddee788583..21f442c3fa88 100644
--- a/include/linux/ima.h
+++ b/include/linux/ima.h
@@ -145,6 +145,7 @@ static inline int ima_measure_critical_data(const char *event_label,
#ifdef CONFIG_HAVE_IMA_KEXEC
int __init ima_free_kexec_buffer(void);
int __init ima_get_kexec_buffer(void **addr, size_t *size);
+int ima_validate_range(phys_addr_t phys, size_t size);
#endif
#ifdef CONFIG_IMA_SECURE_AND_OR_TRUSTED_BOOT
diff --git a/include/linux/io_uring.h b/include/linux/io_uring.h
index e0dec1bf917a..cb42f1c9dfeb 100644
--- a/include/linux/io_uring.h
+++ b/include/linux/io_uring.h
@@ -19,6 +19,8 @@ enum io_uring_cmd_flags {
/* the request is executed from poll, it should not be freed */
IO_URING_F_MULTISHOT = 32,
+ /* executed by io-wq */
+ IO_URING_F_IOWQ = 64,
};
struct io_uring_cmd {
diff --git a/include/linux/iommu.h b/include/linux/iommu.h
index 2bfa9611be67..780b688e4e04 100644
--- a/include/linux/iommu.h
+++ b/include/linux/iommu.h
@@ -322,12 +322,14 @@ struct iommu_domain_ops {
* @list: Used by the iommu-core to keep a list of registered iommus
* @ops: iommu-ops for talking to this iommu
* @dev: struct device for sysfs handling
+ * @max_pasids: number of supported PASIDs
*/
struct iommu_device {
struct list_head list;
const struct iommu_ops *ops;
struct fwnode_handle *fwnode;
struct device *dev;
+ u32 max_pasids;
};
/**
diff --git a/include/linux/ism.h b/include/linux/ism.h
new file mode 100644
index 000000000000..69bfbf0faaa1
--- /dev/null
+++ b/include/linux/ism.h
@@ -0,0 +1,23 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Internal Shared Memory
+ *
+ * Definitions for the ISM module
+ *
+ * Copyright IBM Corp. 2022
+ */
+#ifndef _ISM_H
+#define _ISM_H
+
+struct ism_dmb {
+ u64 dmb_tok;
+ u64 rgid;
+ u32 dmb_len;
+ u32 sba_idx;
+ u32 vlan_valid;
+ u32 vlan_id;
+ void *cpu_addr;
+ dma_addr_t dma_addr;
+};
+
+#endif /* _ISM_H */
diff --git a/include/linux/kasan.h b/include/linux/kasan.h
index 5614d9da7a9e..b290621c4ee3 100644
--- a/include/linux/kasan.h
+++ b/include/linux/kasan.h
@@ -387,7 +387,7 @@ static inline void kasan_init_hw_tags(void) { }
#if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS)
void kasan_populate_early_vm_area_shadow(void *start, unsigned long size);
-int kasan_populate_vmalloc(unsigned long addr, unsigned long size);
+int kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mask);
void kasan_release_vmalloc(unsigned long start, unsigned long end,
unsigned long free_region_start,
unsigned long free_region_end);
@@ -398,7 +398,7 @@ static inline void kasan_populate_early_vm_area_shadow(void *start,
unsigned long size)
{ }
static inline int kasan_populate_vmalloc(unsigned long start,
- unsigned long size)
+ unsigned long size, gfp_t gfp_mask)
{
return 0;
}
@@ -443,7 +443,7 @@ kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms,
static inline void kasan_populate_early_vm_area_shadow(void *start,
unsigned long size) { }
static inline int kasan_populate_vmalloc(unsigned long start,
- unsigned long size)
+ unsigned long size, gfp_t gfp_mask)
{
return 0;
}
diff --git a/include/linux/key-type.h b/include/linux/key-type.h
index 5caf3ce82373..1336c14138ab 100644
--- a/include/linux/key-type.h
+++ b/include/linux/key-type.h
@@ -158,6 +158,7 @@ struct key_type {
const void *in, const void *in2);
/* internal fields */
+ atomic64_t index_id; /* Opaque keyring index ID */
struct list_head link; /* link in types list */
struct lock_class_key lock_class; /* key->sem lock class */
} __randomize_layout;
diff --git a/include/linux/key.h b/include/linux/key.h
index d27477faf00d..ccd00ac2bbf0 100644
--- a/include/linux/key.h
+++ b/include/linux/key.h
@@ -107,6 +107,7 @@ struct keyring_name;
struct key_tag {
struct rcu_head rcu;
+ atomic64_t index_id; /* Opaque keyring index ID */
refcount_t usage;
bool removed; /* T when subject removed */
};
@@ -129,6 +130,8 @@ struct keyring_index_key {
struct key_type *type;
struct key_tag *domain_tag; /* Domain of operation */
const char *description;
+ u64 type_id; /* Cached index IDs */
+ u64 domain_id;
};
union key_payload {
@@ -250,6 +253,8 @@ struct key {
struct key_type *type; /* type of key */
struct key_tag *domain_tag; /* Domain of operation */
char *description;
+ u64 type_id;
+ u64 domain_id;
};
};
diff --git a/include/linux/libata.h b/include/linux/libata.h
index d7f86de11d18..a1e9a637de59 100644
--- a/include/linux/libata.h
+++ b/include/linux/libata.h
@@ -830,6 +830,9 @@ struct ata_port {
u64 qc_active;
int nr_active_links; /* #links with active qcs */
+ struct work_struct deferred_qc_work;
+ struct ata_queued_cmd *deferred_qc;
+
struct ata_link link; /* host default link */
struct ata_link *slave_link; /* see ata_slave_link_init() */
diff --git a/include/linux/list.h b/include/linux/list.h
index a2f9a62c577a..38e8ed08d598 100644
--- a/include/linux/list.h
+++ b/include/linux/list.h
@@ -662,6 +662,21 @@ static inline void list_splice_tail_init(struct list_head *list,
!list_is_head(pos, (head)); \
pos = n, n = pos->prev)
+/**
+ * list_count_nodes - count nodes in the list
+ * @head: the head for your list.
+ */
+static inline size_t list_count_nodes(struct list_head *head)
+{
+ struct list_head *pos;
+ size_t count = 0;
+
+ list_for_each(pos, head)
+ count++;
+
+ return count;
+}
+
/**
* list_entry_is_head - test if the entry points to the head of the list
* @pos: the type * to cursor
diff --git a/include/linux/lockd/bind.h b/include/linux/lockd/bind.h
index 3bc9f7410e21..2fef35577f01 100644
--- a/include/linux/lockd/bind.h
+++ b/include/linux/lockd/bind.h
@@ -20,11 +20,17 @@
/* Dummy declarations */
struct svc_rqst;
struct rpc_task;
+struct module;
-/*
- * This is the set of functions for lockd->nfsd communication
+/**
+ * struct nlmsvc_binding - lockd -> nfsd callback table
+ * @owner: module that provides this binding.
+ * @fopen: open a file by NFS file handle on behalf of an NLM request.
+ * @fclose: close a file that was previously opened via @fopen.
+ * Implementations MUST be semantically equivalent to fput().
*/
struct nlmsvc_binding {
+ struct module *owner;
__be32 (*fopen)(struct svc_rqst *,
struct nfs_fh *,
struct file **,
@@ -32,7 +38,7 @@ struct nlmsvc_binding {
void (*fclose)(struct file *);
};
-extern const struct nlmsvc_binding *nlmsvc_ops;
+extern const struct nlmsvc_binding __rcu *nlmsvc_ops;
/*
* Similar to nfs_client_initdata, but without the NFS-specific
diff --git a/include/linux/lockd/lockd.h b/include/linux/lockd/lockd.h
index 70ce419e2709..d6034b8bbb7d 100644
--- a/include/linux/lockd/lockd.h
+++ b/include/linux/lockd/lockd.h
@@ -265,7 +265,7 @@ void nsm_release(struct nsm_handle *nsm);
* This is used in garbage collection and resource reclaim
* A return value != 0 means destroy the lock/block/share
*/
-typedef int (*nlm_host_match_fn_t)(void *cur, struct nlm_host *ref);
+typedef int (*nlm_host_match_fn_t)(void *owner, struct nlm_host *ref);
/*
* Server-side lock handling
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 9fcbf5706595..26846484aed7 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -330,7 +330,7 @@ extern void set_zone_contiguous(struct zone *zone);
extern void clear_zone_contiguous(struct zone *zone);
#ifdef CONFIG_MEMORY_HOTPLUG
-extern void __ref free_area_init_core_hotplug(struct pglist_data *pgdat);
+int __ref free_area_init_core_hotplug(struct pglist_data *pgdat);
extern int __add_memory(int nid, u64 start, u64 size, mhp_t mhp_flags);
extern int add_memory(int nid, u64 start, u64 size, mhp_t mhp_flags);
extern int add_memory_resource(int nid, struct resource *resource,
diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h
index cb15308b5cb0..804432fd0f27 100644
--- a/include/linux/mlx5/cq.h
+++ b/include/linux/mlx5/cq.h
@@ -183,6 +183,7 @@ static inline void mlx5_cq_put(struct mlx5_core_cq *cq)
complete(&cq->free);
}
+void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq, struct mlx5_eqe *eqe);
int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
u32 *in, int inlen, u32 *out, int outlen);
int mlx5_core_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq,
diff --git a/include/linux/mlx5/driver.h b/include/linux/mlx5/driver.h
index 0e8705c3a460..6948767d5252 100644
--- a/include/linux/mlx5/driver.h
+++ b/include/linux/mlx5/driver.h
@@ -747,7 +747,6 @@ enum {
enum {
MKEY_CACHE_LAST_STD_ENTRY = 20,
- MLX5_IMR_MTT_CACHE_ENTRY,
MLX5_IMR_KSM_CACHE_ENTRY,
MAX_MKEY_CACHE_ENTRIES
};
diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h
index 9610b325f2b6..6021e8de4e87 100644
--- a/include/linux/mlx5/mlx5_ifc.h
+++ b/include/linux/mlx5/mlx5_ifc.h
@@ -11017,18 +11017,7 @@ struct mlx5_ifc_mcia_reg_bits {
u8 reserved_at_60[0x20];
- u8 dword_0[0x20];
- u8 dword_1[0x20];
- u8 dword_2[0x20];
- u8 dword_3[0x20];
- u8 dword_4[0x20];
- u8 dword_5[0x20];
- u8 dword_6[0x20];
- u8 dword_7[0x20];
- u8 dword_8[0x20];
- u8 dword_9[0x20];
- u8 dword_10[0x20];
- u8 dword_11[0x20];
+ u8 dwords[0x400];
};
struct mlx5_ifc_dcbx_param_bits {
diff --git a/include/linux/mlx5/vport.h b/include/linux/mlx5/vport.h
index aad53cb72f17..5b9c57673c0d 100644
--- a/include/linux/mlx5/vport.h
+++ b/include/linux/mlx5/vport.h
@@ -94,8 +94,8 @@ int mlx5_query_hca_vport_node_guid(struct mlx5_core_dev *dev,
int mlx5_query_nic_vport_mac_list(struct mlx5_core_dev *dev,
u16 vport,
enum mlx5_list_type list_type,
- u8 addr_list[][ETH_ALEN],
- int *list_size);
+ u8 (**mac_list)[ETH_ALEN],
+ int *mac_list_size);
int mlx5_modify_nic_vport_mac_list(struct mlx5_core_dev *dev,
enum mlx5_list_type list_type,
u8 addr_list[][ETH_ALEN],
@@ -132,4 +132,6 @@ int mlx5_nic_vport_affiliate_multiport(struct mlx5_core_dev *master_mdev,
int mlx5_nic_vport_unaffiliate_multiport(struct mlx5_core_dev *port_mdev);
u64 mlx5_query_nic_system_image_guid(struct mlx5_core_dev *mdev);
+int mlx5_vport_get_other_func_cap(struct mlx5_core_dev *dev, u16 function_id, void *out,
+ u16 opmod);
#endif /* __MLX5_VPORT_H__ */
diff --git a/include/linux/mm.h b/include/linux/mm.h
index 4f0aa81257c5..279c121ef6b3 100644
--- a/include/linux/mm.h
+++ b/include/linux/mm.h
@@ -2822,7 +2822,7 @@ extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *);
extern void unlink_file_vma(struct vm_area_struct *);
extern struct vm_area_struct *copy_vma(struct vm_area_struct **,
unsigned long addr, unsigned long len, pgoff_t pgoff,
- bool *need_rmap_locks);
+ bool *need_rmap_locks, bool keep_source);
extern void exit_mmap(struct mm_struct *);
void vma_mas_store(struct vm_area_struct *vma, struct ma_state *mas);
diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h
index d1a370add9bc..690e0ea004a5 100644
--- a/include/linux/mm_types.h
+++ b/include/linux/mm_types.h
@@ -602,6 +602,9 @@ struct mm_struct {
* by mmlist_lock
*/
+#ifdef CONFIG_FUTEX
+ atomic64_t futex_unique_id;
+#endif
unsigned long hiwater_rss; /* High-watermark of RSS usage */
unsigned long hiwater_vm; /* High-water virtual memory usage */
diff --git a/include/linux/mount.h b/include/linux/mount.h
index 55a4abaf6715..76c21958e842 100644
--- a/include/linux/mount.h
+++ b/include/linux/mount.h
@@ -50,7 +50,7 @@ struct path;
#define MNT_INTERNAL_FLAGS (MNT_SHARED | MNT_WRITE_HOLD | MNT_INTERNAL | \
MNT_DOOMED | MNT_SYNC_UMOUNT | MNT_MARKED | \
- MNT_CURSOR)
+ MNT_CURSOR | MNT_UMOUNT)
#define MNT_INTERNAL 0x4000
diff --git a/include/linux/mtd/mtd.h b/include/linux/mtd/mtd.h
index 7c58c44662b8..914a9f974baa 100644
--- a/include/linux/mtd/mtd.h
+++ b/include/linux/mtd/mtd.h
@@ -379,7 +379,7 @@ struct mtd_info {
struct module *owner;
struct device dev;
- int usecount;
+ struct kref refcnt;
struct mtd_debug_info dbg;
struct nvmem_device *nvmem;
struct nvmem_device *otp_user_nvmem;
diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h
index e97967ca2760..683896a9af44 100644
--- a/include/linux/netdevice.h
+++ b/include/linux/netdevice.h
@@ -1938,6 +1938,8 @@ enum netdev_reg_state {
* @qdisc_hash: qdisc hash table
* @watchdog_timeo: Represents the timeout that is used by
* the watchdog (see dev_watchdog())
+ * @watchdog_lock: protect watchdog_ref_held
+ * @watchdog_ref_held: True if the watchdog device ref is taken.
* @watchdog_timer: List of timers
*
* @proto_down_reason: reason a netdev interface is held down
@@ -2267,6 +2269,8 @@ struct net_device {
/* These may be needed for future network-power-down code. */
struct timer_list watchdog_timer;
int watchdog_timeo;
+ spinlock_t watchdog_lock;
+ bool watchdog_ref_held;
u32 proto_down_reason;
diff --git a/include/linux/netfilter.h b/include/linux/netfilter.h
index e5f4b6f8d1c0..031718f764ca 100644
--- a/include/linux/netfilter.h
+++ b/include/linux/netfilter.h
@@ -21,6 +21,16 @@ static inline int NF_DROP_GETERR(int verdict)
return -(verdict >> NF_VERDICT_QBITS);
}
+static __always_inline int
+NF_DROP_REASON(struct sk_buff *skb, enum skb_drop_reason reason, u32 err)
+{
+ BUILD_BUG_ON(err > 0xffff);
+
+ kfree_skb_reason(skb, reason);
+
+ return ((err << 16) | NF_STOLEN);
+}
+
static inline int nf_inet_addr_cmp(const union nf_inet_addr *a1,
const union nf_inet_addr *a2)
{
@@ -83,6 +93,8 @@ enum nf_hook_ops_type {
};
struct nf_hook_ops {
+ struct rcu_head rcu;
+
/* User fills in from here down. */
nf_hookfn *hook;
struct net_device *dev;
diff --git a/include/linux/nfs_xdr.h b/include/linux/nfs_xdr.h
index ec796b85938e..31d77c029bc3 100644
--- a/include/linux/nfs_xdr.h
+++ b/include/linux/nfs_xdr.h
@@ -563,7 +563,6 @@ struct nfs_lock_args {
struct nfs_lowner lock_owner;
unsigned char block : 1;
unsigned char reclaim : 1;
- unsigned char new_lock : 1;
unsigned char new_lock_owner : 1;
};
diff --git a/include/linux/ntb.h b/include/linux/ntb.h
index 191b524e5c0d..5c29e904a9c5 100644
--- a/include/linux/ntb.h
+++ b/include/linux/ntb.h
@@ -256,6 +256,7 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops)
* @msg_clear_mask: See ntb_msg_clear_mask().
* @msg_read: See ntb_msg_read().
* @peer_msg_write: See ntb_peer_msg_write().
+ * @get_dma_dev: See ntb_get_dma_dev().
*/
struct ntb_dev_ops {
int (*port_number)(struct ntb_dev *ntb);
@@ -329,6 +330,7 @@ struct ntb_dev_ops {
int (*msg_clear_mask)(struct ntb_dev *ntb, u64 mask_bits);
u32 (*msg_read)(struct ntb_dev *ntb, int *pidx, int midx);
int (*peer_msg_write)(struct ntb_dev *ntb, int pidx, int midx, u32 msg);
+ struct device *(*get_dma_dev)(struct ntb_dev *ntb);
};
static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
@@ -391,6 +393,8 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops)
/* !ops->msg_clear_mask == !ops->msg_count && */
!ops->msg_read == !ops->msg_count &&
!ops->peer_msg_write == !ops->msg_count &&
+
+ /* ops->get_dma_dev is optional */
1;
}
@@ -1563,6 +1567,26 @@ static inline int ntb_peer_msg_write(struct ntb_dev *ntb, int pidx, int midx,
return ntb->ops->peer_msg_write(ntb, pidx, midx, msg);
}
+/**
+ * ntb_get_dma_dev() - get the device to use for DMA allocations/mappings
+ * @ntb: NTB device context.
+ *
+ * Return a struct device suitable for DMA API allocations and mappings.
+ * This is typically the parent of the NTB device, but may be overridden by a
+ * driver by implementing .get_dma_dev().
+ *
+ * Drivers that implement .get_dma_dev() must return a non-NULL pointer.
+ *
+ * Return: device pointer to use for DMA operations.
+ */
+static inline struct device *ntb_get_dma_dev(struct ntb_dev *ntb)
+{
+ if (!ntb->ops->get_dma_dev)
+ return ntb->dev.parent;
+
+ return ntb->ops->get_dma_dev(ntb);
+}
+
/**
* ntb_peer_resource_idx() - get a resource index for a given peer idx
* @ntb: NTB device context.
diff --git a/include/linux/nvme.h b/include/linux/nvme.h
index b906c244c94d..7f8ef075b0d2 100644
--- a/include/linux/nvme.h
+++ b/include/linux/nvme.h
@@ -1060,8 +1060,8 @@ enum {
NVME_ZRASF_ZONE_STATE_IMP_OPEN = 0x02,
NVME_ZRASF_ZONE_STATE_EXP_OPEN = 0x03,
NVME_ZRASF_ZONE_STATE_CLOSED = 0x04,
- NVME_ZRASF_ZONE_STATE_READONLY = 0x05,
- NVME_ZRASF_ZONE_STATE_FULL = 0x06,
+ NVME_ZRASF_ZONE_STATE_FULL = 0x05,
+ NVME_ZRASF_ZONE_STATE_READONLY = 0x06,
NVME_ZRASF_ZONE_STATE_OFFLINE = 0x07,
NVME_REPORT_ZONE_PARTIAL = 1,
};
diff --git a/include/linux/of.h b/include/linux/of.h
index 2960e609ca05..9ff3ce1719c0 100644
--- a/include/linux/of.h
+++ b/include/linux/of.h
@@ -364,6 +364,7 @@ extern int of_n_addr_cells(struct device_node *np);
extern int of_n_size_cells(struct device_node *np);
extern const struct of_device_id *of_match_node(
const struct of_device_id *matches, const struct device_node *node);
+extern const void *of_device_get_match_data(const struct device *dev);
extern int of_modalias_node(struct device_node *node, char *modalias, int len);
extern void of_print_phandle_args(const char *msg, const struct of_phandle_args *args);
extern int __of_parse_phandle_with_args(const struct device_node *np,
@@ -845,6 +846,11 @@ static inline phys_addr_t of_dma_get_max_cpu_address(struct device_node *np)
return PHYS_ADDR_MAX;
}
+static inline const void *of_device_get_match_data(const struct device *dev)
+{
+ return NULL;
+}
+
#define of_match_ptr(_ptr) NULL
#define of_match_node(_matches, _node) NULL
#endif /* CONFIG_OF */
diff --git a/include/linux/of_device.h b/include/linux/of_device.h
index 1a803e4335d3..24b911d3c89a 100644
--- a/include/linux/of_device.h
+++ b/include/linux/of_device.h
@@ -26,12 +26,6 @@ static inline int of_driver_match_device(struct device *dev,
return of_match_device(drv->of_match_table, dev) != NULL;
}
-extern int of_device_add(struct platform_device *pdev);
-extern int of_device_register(struct platform_device *ofdev);
-extern void of_device_unregister(struct platform_device *ofdev);
-
-extern const void *of_device_get_match_data(const struct device *dev);
-
extern ssize_t of_device_modalias(struct device *dev, char *str, ssize_t len);
extern int of_device_request_module(struct device *dev);
@@ -67,11 +61,6 @@ static inline int of_driver_match_device(struct device *dev,
static inline void of_device_uevent(struct device *dev,
struct kobj_uevent_env *env) { }
-static inline const void *of_device_get_match_data(const struct device *dev)
-{
- return NULL;
-}
-
static inline int of_device_modalias(struct device *dev,
char *str, ssize_t len)
{
diff --git a/include/linux/of_platform.h b/include/linux/of_platform.h
index d15b6cd5e1c3..8ac5cb933dc3 100644
--- a/include/linux/of_platform.h
+++ b/include/linux/of_platform.h
@@ -52,6 +52,11 @@ extern const struct of_device_id of_default_bus_match_table[];
extern struct platform_device *of_device_alloc(struct device_node *np,
const char *bus_id,
struct device *parent);
+
+extern int of_device_add(struct platform_device *pdev);
+extern int of_device_register(struct platform_device *ofdev);
+extern void of_device_unregister(struct platform_device *ofdev);
+
#ifdef CONFIG_OF
extern struct platform_device *of_find_device_by_node(struct device_node *np);
#else
diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h
index dfaa09901867..a40afc0aea3e 100644
--- a/include/linux/pagemap.h
+++ b/include/linux/pagemap.h
@@ -325,6 +325,26 @@ static inline void mapping_set_gfp_mask(struct address_space *m, gfp_t mask)
m->gfp_mask = mask;
}
+/*
+ * There are some parts of the kernel which assume that PMD entries
+ * are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then,
+ * limit the maximum allocation order to PMD size. I'm not aware of any
+ * assumptions about maximum order if THP are disabled, but 8 seems like
+ * a good order (that's 1MB if you're using 4kB pages)
+ */
+#ifdef CONFIG_TRANSPARENT_HUGEPAGE
+#define PREFERRED_MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER
+#else
+#define PREFERRED_MAX_PAGECACHE_ORDER 8
+#endif
+
+/*
+ * xas_split_alloc() does not support arbitrary orders. This implies no
+ * 512MB THP on ARM64 with 64KB base page size.
+ */
+#define MAX_XAS_ORDER (XA_CHUNK_SHIFT * 2 - 1)
+#define MAX_PAGECACHE_ORDER min(MAX_XAS_ORDER, PREFERRED_MAX_PAGECACHE_ORDER)
+
/**
* mapping_set_large_folios() - Indicate the file supports large folios.
* @mapping: The file.
@@ -351,6 +371,14 @@ static inline bool mapping_large_folio_support(struct address_space *mapping)
test_bit(AS_LARGE_FOLIO_SUPPORT, &mapping->flags);
}
+/* Return the maximum folio size for this pagecache mapping, in bytes. */
+static inline size_t mapping_max_folio_size(struct address_space *mapping)
+{
+ if (mapping_large_folio_support(mapping))
+ return PAGE_SIZE << MAX_PAGECACHE_ORDER;
+ return PAGE_SIZE;
+}
+
static inline int filemap_nr_thps(struct address_space *mapping)
{
#ifdef CONFIG_READ_ONLY_THP_FOR_FS
diff --git a/include/linux/pci.h b/include/linux/pci.h
index 282e86b9a35f..39e9aea18f5d 100644
--- a/include/linux/pci.h
+++ b/include/linux/pci.h
@@ -71,12 +71,16 @@
/* return bus from PCI devid = ((u16)bus_number) << 8) | devfn */
#define PCI_BUS_NUM(x) (((x) >> 8) & 0xff)
+/* Used to identify a slot as a placeholder */
+#define PCI_SLOT_PLACEHOLDER 0xffff
+
/* pci_slot represents a physical slot */
struct pci_slot {
struct pci_bus *bus; /* Bus this slot is on */
struct list_head list; /* Node in list of slots */
struct hotplug_slot *hotplug; /* Hotplug info (move here) */
- unsigned char number; /* PCI_SLOT(pci_dev->devfn) */
+ u16 number; /* PCI_SLOT(pci_dev->devfn) or PCI_SLOT_PLACEHOLDER */
+ unsigned int per_func_slot:1; /* Allow per function slot */
struct kobject kobj;
};
@@ -878,6 +882,7 @@ struct module;
* function returns zero when the driver chooses to
* take "ownership" of the device or an error code
* (negative number) otherwise.
+ * The pci_device_id parameter is only valid during probe.
* The probe function always gets called from process
* context, so it can sleep.
* @remove: The remove() function gets called whenever a device
diff --git a/include/linux/perf/riscv_pmu.h b/include/linux/perf/riscv_pmu.h
index e17e86ad6f3a..8d48d7fed11b 100644
--- a/include/linux/perf/riscv_pmu.h
+++ b/include/linux/perf/riscv_pmu.h
@@ -45,7 +45,7 @@ struct riscv_pmu {
irqreturn_t (*handle_irq)(int irq_num, void *dev);
- unsigned long cmask;
+ DECLARE_BITMAP(cmask, RISCV_MAX_COUNTERS);
u64 (*ctr_read)(struct perf_event *event);
int (*ctr_get_idx)(struct perf_event *event);
int (*ctr_get_width)(int idx);
@@ -73,6 +73,9 @@ void riscv_pmu_legacy_skip_init(void);
static inline void riscv_pmu_legacy_skip_init(void) {};
#endif
struct riscv_pmu *riscv_pmu_alloc(void);
+#ifdef CONFIG_RISCV_PMU_SBI
+int riscv_pmu_get_hpm_info(u32 *hw_ctr_width, u32 *num_hw_ctr);
+#endif
#endif /* CONFIG_RISCV_PMU */
diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h
index 41ff70f315a9..1c32eec27b04 100644
--- a/include/linux/perf_event.h
+++ b/include/linux/perf_event.h
@@ -765,7 +765,6 @@ struct perf_event {
struct irq_work pending_irq;
struct callback_head pending_task;
unsigned int pending_work;
- struct rcuwait pending_work_wait;
atomic_t event_limit;
diff --git a/include/linux/power_supply.h b/include/linux/power_supply.h
index a10428884e6a..049ca48a9838 100644
--- a/include/linux/power_supply.h
+++ b/include/linux/power_supply.h
@@ -364,9 +364,9 @@ struct power_supply_vbat_ri_table {
* @charge_voltage_max_uv: maintenance charging voltage that is usually a bit
* lower than the constant_charge_voltage_max_uv. We can apply this settings
* charge_current_max_ua until we get back up to this voltage.
- * @safety_timer_minutes: maintenance charging safety timer, with an expiry
- * time in minutes. We will only use maintenance charging in this setting
- * for a certain amount of time, then we will first move to the next
+ * @charge_safety_timer_minutes: maintenance charging safety timer, with an
+ * expiry time in minutes. We will only use maintenance charging in this
+ * setting for a certain amount of time, then we will first move to the next
* maintenance charge current and voltage pair in respective array and wait
* for the next safety timer timeout, or, if we reached the last maintencance
* charging setting, disable charging until we reach
diff --git a/include/linux/psi.h b/include/linux/psi.h
index e0745873e3f2..7966e3ac03b9 100644
--- a/include/linux/psi.h
+++ b/include/linux/psi.h
@@ -25,7 +25,9 @@ void psi_memstall_leave(unsigned long *flags);
int psi_show(struct seq_file *s, struct psi_group *group, enum psi_res res);
struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
enum psi_res res, struct file *file,
- struct kernfs_open_file *of);
+ struct kernfs_open_file *of,
+ bool *need_rtpoll_worker);
+int psi_trigger_create_rtpoll_worker(struct psi_group *group);
void psi_trigger_destroy(struct psi_trigger *t);
__poll_t psi_trigger_poll(void **trigger_ptr, struct file *file,
diff --git a/include/linux/qcom-geni-se.h b/include/linux/qcom-geni-se.h
index f5672785c0c4..c54580c48076 100644
--- a/include/linux/qcom-geni-se.h
+++ b/include/linux/qcom-geni-se.h
@@ -265,17 +265,12 @@ struct geni_se {
#define QUP_SE_VERSION_2_5 0x20050000
/*
- * Define bandwidth thresholds that cause the underlying Core 2X interconnect
- * clock to run at the named frequency. These baseline values are recommended
- * by the hardware team, and are not dynamically scaled with GENI bandwidth
- * beyond basic on/off.
+ * QUP Core 2X clock votes used by GENI clients through the "qup-core" ICC
+ * path. Values are in Bps and must be converted with Bps_to_icc() before
+ * setting avg_bw.
*/
-#define CORE_2X_19_2_MHZ 960
-#define CORE_2X_50_MHZ 2500
-#define CORE_2X_100_MHZ 5000
-#define CORE_2X_150_MHZ 7500
-#define CORE_2X_200_MHZ 10000
-#define CORE_2X_236_MHZ 16383
+#define CORE_2X_19_2_MHZ 9600000
+#define CORE_2X_50_MHZ 25000000
#define GENI_DEFAULT_BW Bps_to_icc(1000)
diff --git a/include/linux/sched.h b/include/linux/sched.h
index 5b415d3b653e..3cd897f572af 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -1753,7 +1753,7 @@ extern struct pid *cad_pid;
* I am cleaning dirty pages from some other bdi. */
#define PF_KTHREAD 0x00200000 /* I am a kernel thread */
#define PF_RANDOMIZE 0x00400000 /* Randomize virtual address space */
-#define PF__HOLE__00800000 0x00800000
+#define PF_NO_NOTIFY_SIGNAL 0x00800000 /* see no_notify_signal_save() */
#define PF__HOLE__01000000 0x01000000
#define PF__HOLE__02000000 0x02000000
#define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_mask */
diff --git a/include/linux/sched/signal.h b/include/linux/sched/signal.h
index 669e8cff40c7..66e125dc12aa 100644
--- a/include/linux/sched/signal.h
+++ b/include/linux/sched/signal.h
@@ -2,6 +2,7 @@
#ifndef _LINUX_SCHED_SIGNAL_H
#define _LINUX_SCHED_SIGNAL_H
+#include <linux/cleanup.h>
#include <linux/rculist.h>
#include <linux/signal.h>
#include <linux/sched.h>
@@ -389,14 +390,36 @@ static inline int task_sigpending(struct task_struct *p)
return unlikely(test_tsk_thread_flag(p,TIF_SIGPENDING));
}
+/* Prevent TIF_NOTIFY_SIGNAL from interrupting this task. */
+static inline unsigned int no_notify_signal_save(void)
+{
+ unsigned int flags = current->flags;
+
+ current->flags |= PF_NO_NOTIFY_SIGNAL;
+ return flags;
+}
+
+/* Restore the previous PF_NO_NOTIFY_SIGNAL state. */
+static inline void no_notify_signal_restore(unsigned int flags)
+{
+ current_restore_flags(flags, PF_NO_NOTIFY_SIGNAL);
+}
+
+DEFINE_LOCK_GUARD_0(no_notify_signal,
+ _T->flags = no_notify_signal_save(),
+ no_notify_signal_restore(_T->flags),
+ unsigned int flags)
+
static inline int signal_pending(struct task_struct *p)
{
/*
* TIF_NOTIFY_SIGNAL isn't really a signal, but it requires the same
* behavior in terms of ensuring that we break out of wait loops
- * so that notify signal callbacks can be processed.
+ * so that notify signal callbacks can be processed. Not for a task
+ * that asked not to be interrupted by it, see no_notify_signal_save().
*/
- if (unlikely(test_tsk_thread_flag(p, TIF_NOTIFY_SIGNAL)))
+ if (unlikely(test_tsk_thread_flag(p, TIF_NOTIFY_SIGNAL)) &&
+ likely(!(READ_ONCE(p->flags) & PF_NO_NOTIFY_SIGNAL)))
return 1;
return task_sigpending(p);
}
diff --git a/include/linux/sched/user.h b/include/linux/sched/user.h
index 4cc52698e214..ec262b605437 100644
--- a/include/linux/sched/user.h
+++ b/include/linux/sched/user.h
@@ -23,10 +23,13 @@ struct user_struct {
struct hlist_node uidhash_node;
kuid_t uid;
-#if defined(CONFIG_PERF_EVENTS) || defined(CONFIG_BPF_SYSCALL) || \
- defined(CONFIG_NET) || defined(CONFIG_IO_URING) || \
- defined(CONFIG_VFIO_PCI_ZDEV_KVM) || IS_ENABLED(CONFIG_IOMMUFD)
- atomic_long_t locked_vm;
+#if defined(CONFIG_NET) || defined(CONFIG_IO_URING) || \
+ defined(CONFIG_VFIO_PCI_ZDEV_KVM) || IS_ENABLED(CONFIG_IOMMUFD) || \
+ defined(CONFIG_SECRETMEM)
+ atomic_long_t locked_vm; /* pinned pages charged to RLIMIT_MEMLOCK */
+#endif
+#ifdef CONFIG_PERF_EVENTS
+ atomic_long_t perf_mlock; /* perf buffer pages within perf_event_mlock_kb */
#endif
#ifdef CONFIG_WATCH_QUEUE
atomic_t nr_watches; /* The number of watches this user currently has */
diff --git a/include/linux/scmi_protocol.h b/include/linux/scmi_protocol.h
index 4f765bc788ff..0ce5746a4470 100644
--- a/include/linux/scmi_protocol.h
+++ b/include/linux/scmi_protocol.h
@@ -804,11 +804,6 @@ struct scmi_device {
#define to_scmi_dev(d) container_of(d, struct scmi_device, dev)
-struct scmi_device *
-scmi_device_create(struct device_node *np, struct device *parent, int protocol,
- const char *name);
-void scmi_device_destroy(struct scmi_device *scmi_dev);
-
struct scmi_device_id {
u8 protocol_id;
const char *name;
diff --git a/include/linux/seq_buf.h b/include/linux/seq_buf.h
index 5b31c5147969..468d8c5eef4a 100644
--- a/include/linux/seq_buf.h
+++ b/include/linux/seq_buf.h
@@ -14,19 +14,24 @@
* @buffer: pointer to the buffer
* @size: size of the buffer
* @len: the amount of data inside the buffer
- * @readpos: The next position to read in the buffer.
*/
struct seq_buf {
char *buffer;
size_t size;
size_t len;
- loff_t readpos;
};
+#define DECLARE_SEQ_BUF(NAME, SIZE) \
+ struct seq_buf NAME = { \
+ .buffer = (char[SIZE]) { 0 }, \
+ .size = SIZE, \
+ }
+
static inline void seq_buf_clear(struct seq_buf *s)
{
s->len = 0;
- s->readpos = 0;
+ if (s->size)
+ s->buffer[0] = '\0';
}
static inline void
@@ -72,8 +77,8 @@ static inline unsigned int seq_buf_used(struct seq_buf *s)
}
/**
- * seq_buf_terminate - Make sure buffer is nul terminated
- * @s: the seq_buf descriptor to terminate.
+ * seq_buf_str - get %NUL-terminated C string from seq_buf
+ * @s: the seq_buf handle
*
* This makes sure that the buffer in @s is nul terminated and
* safe to read as a string.
@@ -84,16 +89,20 @@ static inline unsigned int seq_buf_used(struct seq_buf *s)
*
* After this function is called, s->buffer is safe to use
* in string operations.
+ *
+ * Returns @s->buf after making sure it is terminated.
*/
-static inline void seq_buf_terminate(struct seq_buf *s)
+static inline const char *seq_buf_str(struct seq_buf *s)
{
if (WARN_ON(s->size == 0))
- return;
+ return "";
if (seq_buf_buffer_left(s))
s->buffer[s->len] = 0;
else
s->buffer[s->size - 1] = 0;
+
+ return s->buffer;
}
/**
@@ -143,7 +152,7 @@ extern __printf(2, 0)
int seq_buf_vprintf(struct seq_buf *s, const char *fmt, va_list args);
extern int seq_buf_print_seq(struct seq_file *m, struct seq_buf *s);
extern int seq_buf_to_user(struct seq_buf *s, char __user *ubuf,
- int cnt);
+ size_t start, int cnt);
extern int seq_buf_puts(struct seq_buf *s, const char *str);
extern int seq_buf_putc(struct seq_buf *s, unsigned char c);
extern int seq_buf_putmem(struct seq_buf *s, const void *mem, unsigned int len);
@@ -159,4 +168,6 @@ extern int
seq_buf_bprintf(struct seq_buf *s, const char *fmt, const u32 *binary);
#endif
+void seq_buf_do_printk(struct seq_buf *s, const char *lvl);
+
#endif /* _LINUX_SEQ_BUF_H */
diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h
index c5a2b0f6516c..1f646364926e 100644
--- a/include/linux/skbuff.h
+++ b/include/linux/skbuff.h
@@ -1487,7 +1487,7 @@ __skb_set_sw_hash(struct sk_buff *skb, __u32 hash, bool is_l4)
__skb_set_hash(skb, hash, true, is_l4);
}
-void __skb_get_hash(struct sk_buff *skb);
+void __skb_get_hash_net(const struct net *net, struct sk_buff *skb);
u32 __skb_get_hash_symmetric(const struct sk_buff *skb);
u32 skb_get_poff(const struct sk_buff *skb);
u32 __skb_get_poff(const struct sk_buff *skb, const void *data,
@@ -1567,10 +1567,18 @@ void skb_flow_dissect_hash(const struct sk_buff *skb,
struct flow_dissector *flow_dissector,
void *target_container);
+static inline __u32 skb_get_hash_net(const struct net *net, struct sk_buff *skb)
+{
+ if (!skb->l4_hash && !skb->sw_hash)
+ __skb_get_hash_net(net, skb);
+
+ return skb->hash;
+}
+
static inline __u32 skb_get_hash(struct sk_buff *skb)
{
if (!skb->l4_hash && !skb->sw_hash)
- __skb_get_hash(skb);
+ __skb_get_hash_net(NULL, skb);
return skb->hash;
}
@@ -3472,6 +3480,16 @@ static inline void skb_frag_ref(struct sk_buff *skb, int f)
__skb_frag_ref(&skb_shinfo(skb)->frags[f]);
}
+static inline void
+skb_page_unref(const struct sk_buff *skb, struct page *page, bool napi_safe)
+{
+#ifdef CONFIG_PAGE_POOL
+ if (skb->pp_recycle && page_pool_return_skb_page(page))
+ return;
+#endif
+ put_page(page);
+}
+
/**
* __skb_frag_unref - release a reference on a paged fragment.
* @frag: the paged fragment
diff --git a/include/linux/stmmac.h b/include/linux/stmmac.h
index 06e879cdbff8..ffa2e307ff40 100644
--- a/include/linux/stmmac.h
+++ b/include/linux/stmmac.h
@@ -115,7 +115,6 @@ struct stmmac_axi {
#define EST_GCL 1024
struct stmmac_est {
- struct mutex lock;
int enable;
u32 btr_reserve[2];
u32 btr_offset[2];
diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h
index fbc4bd423b35..3ac3307ce28d 100644
--- a/include/linux/sunrpc/svc_rdma.h
+++ b/include/linux/sunrpc/svc_rdma.h
@@ -138,6 +138,7 @@ struct svc_rdma_recv_ctxt {
bool rc_temp;
u32 rc_byte_len;
unsigned int rc_page_count;
+ unsigned int rc_maxpages;
u32 rc_inv_rkey;
__be32 rc_msgtype;
diff --git a/include/linux/sunrpc/xdr.h b/include/linux/sunrpc/xdr.h
index f84e2a1358e1..f7e2850c8638 100644
--- a/include/linux/sunrpc/xdr.h
+++ b/include/linux/sunrpc/xdr.h
@@ -188,7 +188,6 @@ xdr_adjust_iovec(struct kvec *iov, __be32 *p)
/*
* XDR buffer helper functions
*/
-extern void xdr_shift_buf(struct xdr_buf *, size_t);
extern void xdr_buf_from_iov(const struct kvec *, struct xdr_buf *);
extern int xdr_buf_subsegment(const struct xdr_buf *, struct xdr_buf *, unsigned int, unsigned int);
extern void xdr_buf_trim(struct xdr_buf *, unsigned int);
diff --git a/include/linux/trace_recursion.h b/include/linux/trace_recursion.h
index c303f7a114e9..2ac1853f056d 100644
--- a/include/linux/trace_recursion.h
+++ b/include/linux/trace_recursion.h
@@ -34,6 +34,13 @@ enum {
TRACE_INTERNAL_SIRQ_BIT,
TRACE_INTERNAL_TRANSITION_BIT,
+ /* Internal event use recursion bits */
+ TRACE_INTERNAL_EVENT_BIT,
+ TRACE_INTERNAL_EVENT_NMI_BIT,
+ TRACE_INTERNAL_EVENT_IRQ_BIT,
+ TRACE_INTERNAL_EVENT_SIRQ_BIT,
+ TRACE_INTERNAL_EVENT_TRANSITION_BIT,
+
TRACE_BRANCH_BIT,
/*
* Abuse of the trace_recursion.
@@ -97,6 +104,8 @@ enum {
#define TRACE_LIST_START TRACE_INTERNAL_BIT
+#define TRACE_EVENT_START TRACE_INTERNAL_EVENT_BIT
+
#define TRACE_CONTEXT_MASK ((1 << (TRACE_LIST_START + TRACE_CONTEXT_BITS)) - 1)
/*
diff --git a/include/linux/trace_seq.h b/include/linux/trace_seq.h
index c230cbd25aee..93dc6ea8d002 100644
--- a/include/linux/trace_seq.h
+++ b/include/linux/trace_seq.h
@@ -14,6 +14,7 @@
struct trace_seq {
char buffer[PAGE_SIZE];
struct seq_buf seq;
+ size_t readpos;
int full;
};
@@ -22,6 +23,7 @@ trace_seq_init(struct trace_seq *s)
{
seq_buf_init(&s->seq, s->buffer, PAGE_SIZE);
s->full = 0;
+ s->readpos = 0;
}
/**
diff --git a/include/linux/tty.h b/include/linux/tty.h
index 730c3301d710..2b90fe8afade 100644
--- a/include/linux/tty.h
+++ b/include/linux/tty.h
@@ -173,6 +173,7 @@ struct tty_operations;
* @write_wait: concurrent writers are waiting in this queue until they are
* allowed to write
* @read_wait: readers wait for data in this queue
+ * @break_wait: wait queue for timed breaks
* @hangup_work: normally a work to perform a hangup (do_tty_hangup()); while
* freeing the tty, (re)used to release_one_tty()
* @disc_data: pointer to @ldisc's private data (e.g. to &struct n_tty_data)
@@ -235,6 +236,7 @@ struct tty_struct {
struct fasync_struct *fasync;
wait_queue_head_t write_wait;
wait_queue_head_t read_wait;
+ wait_queue_head_t break_wait;
struct work_struct hangup_work;
void *disc_data;
void *driver_data;
@@ -387,7 +389,7 @@ extern struct ktermios tty_std_termios;
int vcs_init(void);
-extern struct class *tty_class;
+extern struct class tty_class;
/**
* tty_kref_get - get a tty reference
diff --git a/include/linux/tty_driver.h b/include/linux/tty_driver.h
index ff5791373cf9..1e5241b82385 100644
--- a/include/linux/tty_driver.h
+++ b/include/linux/tty_driver.h
@@ -542,6 +542,11 @@ static inline void tty_set_operations(struct tty_driver *driver,
* Do not create numbered ``/dev`` nodes. For example, create
* ``/dev/ttyprintk`` and not ``/dev/ttyprintk0``. Applicable only when a
* driver for a single tty device is being allocated.
+ *
+ * TTY_DRIVER_RESET_SAVED_TERMIOS
+ * Reset any saved termios settings on device registration when reusing a
+ * minor number. Must only be set by drivers that guarantee that the minor
+ * number is no longer in use.
*/
#define TTY_DRIVER_INSTALLED 0x0001
#define TTY_DRIVER_RESET_TERMIOS 0x0002
@@ -551,6 +556,7 @@ static inline void tty_set_operations(struct tty_driver *driver,
#define TTY_DRIVER_HARDWARE_BREAK 0x0020
#define TTY_DRIVER_DYNAMIC_ALLOC 0x0040
#define TTY_DRIVER_UNNUMBERED_NODE 0x0080
+#define TTY_DRIVER_RESET_SAVED_TERMIOS 0x0200
/* tty driver types */
#define TTY_DRIVER_TYPE_SYSTEM 0x0001
diff --git a/include/linux/tty_port.h b/include/linux/tty_port.h
index 1ec146b8d0f4..660edf9e741a 100644
--- a/include/linux/tty_port.h
+++ b/include/linux/tty_port.h
@@ -234,7 +234,8 @@ int tty_port_carrier_raised(struct tty_port *port);
void tty_port_raise_dtr_rts(struct tty_port *port);
void tty_port_lower_dtr_rts(struct tty_port *port);
void tty_port_hangup(struct tty_port *port);
-void __tty_port_tty_hangup(struct tty_port *port, bool check_clocal, bool async);
+void tty_port_tty_hangup(struct tty_port *port, bool check_clocal);
+void tty_port_tty_vhangup(struct tty_port *port);
void tty_port_tty_wakeup(struct tty_port *port);
int tty_port_block_til_ready(struct tty_port *port, struct tty_struct *tty,
struct file *filp);
@@ -253,23 +254,4 @@ static inline int tty_port_users(struct tty_port *port)
return port->count + port->blocked_open;
}
-/**
- * tty_port_tty_hangup - helper to hang up a tty asynchronously
- * @port: tty port
- * @check_clocal: hang only ttys with %CLOCAL unset?
- */
-static inline void tty_port_tty_hangup(struct tty_port *port, bool check_clocal)
-{
- __tty_port_tty_hangup(port, check_clocal, true);
-}
-
-/**
- * tty_port_tty_vhangup - helper to hang up a tty synchronously
- * @port: tty port
- */
-static inline void tty_port_tty_vhangup(struct tty_port *port)
-{
- __tty_port_tty_hangup(port, false, false);
-}
-
#endif
diff --git a/include/linux/unicode.h b/include/linux/unicode.h
index 4d39e6e11a95..f5ee53dc4dff 100644
--- a/include/linux/unicode.h
+++ b/include/linux/unicode.h
@@ -51,6 +51,7 @@ struct unicode_map {
unsigned int version;
const struct utf8data *ntab[UTF8_NMAX];
const struct utf8data_table *tables;
+ struct rcu_head rcu;
};
int utf8_validate(const struct unicode_map *um, const struct qstr *str);
diff --git a/include/linux/usb/gadget.h b/include/linux/usb/gadget.h
index 14090bafe5ac..ea15f3019857 100644
--- a/include/linux/usb/gadget.h
+++ b/include/linux/usb/gadget.h
@@ -53,6 +53,7 @@ struct usb_ep;
* @short_not_ok: When reading data, makes short packets be
* treated as errors (queue stops advancing till cleanup).
* @dma_mapped: Indicates if request has been mapped to DMA (internal)
+ * @sg_was_mapped: Set if the scatterlist has been mapped before the request
* @complete: Function called when request completes, so this request and
* its buffer may be re-used. The function will always be called with
* interrupts disabled, and it must not sleep.
@@ -113,6 +114,7 @@ struct usb_request {
unsigned zero:1;
unsigned short_not_ok:1;
unsigned dma_mapped:1;
+ unsigned sg_was_mapped:1;
void (*complete)(struct usb_ep *ep,
struct usb_request *req);
diff --git a/include/linux/usb/serial.h b/include/linux/usb/serial.h
index f7bfedb740f5..0950aa316677 100644
--- a/include/linux/usb/serial.h
+++ b/include/linux/usb/serial.h
@@ -278,7 +278,7 @@ struct usb_serial_driver {
int (*set_serial)(struct tty_struct *tty, struct serial_struct *ss);
void (*set_termios)(struct tty_struct *tty, struct usb_serial_port *port,
const struct ktermios *old);
- void (*break_ctl)(struct tty_struct *tty, int break_state);
+ int (*break_ctl)(struct tty_struct *tty, int break_state);
unsigned int (*chars_in_buffer)(struct tty_struct *tty);
void (*wait_until_sent)(struct tty_struct *tty, long timeout);
bool (*tx_empty)(struct usb_serial_port *port);
@@ -377,6 +377,7 @@ void usb_serial_handle_dcd_change(struct usb_serial_port *usb_port,
int usb_serial_bus_register(struct usb_serial_driver *device);
void usb_serial_bus_deregister(struct usb_serial_driver *device);
+void usb_serial_bus_remove_new_id(struct usb_serial_driver *driver);
extern struct bus_type usb_serial_bus_type;
extern struct tty_driver *usb_serial_tty_driver;
diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h
index 9df0b9a762cc..87f3d6e17a67 100644
--- a/include/linux/userfaultfd_k.h
+++ b/include/linux/userfaultfd_k.h
@@ -162,6 +162,8 @@ static inline bool vma_can_userfault(struct vm_area_struct *vma,
vma_is_shmem(vma);
}
+bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma);
+
extern int dup_userfaultfd(struct vm_area_struct *, struct list_head *);
extern void dup_userfaultfd_complete(struct list_head *);
@@ -274,6 +276,11 @@ static inline bool uffd_disable_fault_around(struct vm_area_struct *vma)
return false;
}
+static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma)
+{
+ return false;
+}
+
#endif /* CONFIG_USERFAULTFD */
static inline bool pte_marker_entry_uffd_wp(swp_entry_t entry)
diff --git a/include/linux/virtio.h b/include/linux/virtio.h
index 82a29db54e7d..8982611799e0 100644
--- a/include/linux/virtio.h
+++ b/include/linux/virtio.h
@@ -156,7 +156,7 @@ void virtio_reset_device(struct virtio_device *dev);
size_t virtio_max_dma_size(struct virtio_device *vdev);
#define virtio_device_for_each_vq(vdev, vq) \
- list_for_each_entry(vq, &vdev->vqs, list)
+ list_for_each_entry(vq, &(vdev)->vqs, list)
/**
* struct virtio_driver - operations for a virtio I/O driver
diff --git a/include/media/v4l2-mem2mem.h b/include/media/v4l2-mem2mem.h
index 370c230ad3be..4a2649a4562a 100644
--- a/include/media/v4l2-mem2mem.h
+++ b/include/media/v4l2-mem2mem.h
@@ -537,6 +537,27 @@ v4l2_m2m_register_media_controller(struct v4l2_m2m_dev *m2m_dev,
*/
void v4l2_m2m_release(struct v4l2_m2m_dev *m2m_dev);
+/**
+ * v4l2_m2m_get() - take a reference to the m2m_dev structure
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * This is used to share the M2M device across multiple devices. This
+ * can be used to avoid scheduling two hardware nodes concurrently.
+ */
+void v4l2_m2m_get(struct v4l2_m2m_dev *m2m_dev);
+
+/**
+ * v4l2_m2m_put() - remove a reference to the m2m_dev structure
+ *
+ * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ *
+ * Once the M2M device has no more references, v4l2_m2m_release() will be
+ * called automatically. Users of this method should never call
+ * v4l2_m2m_release() directly. See v4l2_m2m_get() for more details.
+ */
+void v4l2_m2m_put(struct v4l2_m2m_dev *m2m_dev);
+
/**
* v4l2_m2m_ctx_init() - allocate and initialize a m2m context
*
diff --git a/include/net/af_vsock.h b/include/net/af_vsock.h
index 568a87c5e0d0..a1aebfa14a3e 100644
--- a/include/net/af_vsock.h
+++ b/include/net/af_vsock.h
@@ -207,6 +207,9 @@ void vsock_remove_connected(struct vsock_sock *vsk);
struct sock *vsock_find_bound_socket(struct sockaddr_vm *addr);
struct sock *vsock_find_connected_socket(struct sockaddr_vm *src,
struct sockaddr_vm *dst);
+bool vsock_check_source(const struct vsock_sock *vsk,
+ const struct vsock_transport *transport,
+ const struct sockaddr_vm *src);
void vsock_remove_sock(struct vsock_sock *vsk);
void vsock_for_each_connected_socket(struct vsock_transport *transport,
void (*fn)(struct sock *sk));
diff --git a/include/net/amt.h b/include/net/amt.h
index c881bc8b673b..ad844d65a64d 100644
--- a/include/net/amt.h
+++ b/include/net/amt.h
@@ -231,10 +231,6 @@ struct amt_relay_headers {
};
} __packed;
-struct amt_skb_cb {
- struct amt_tunnel_list *tunnel;
-};
-
struct amt_tunnel_list {
struct list_head list;
/* Protect All resources under an amt_tunne_list */
diff --git a/include/net/ax25.h b/include/net/ax25.h
index e9465aa07a4e..6b29f3a30cfa 100644
--- a/include/net/ax25.h
+++ b/include/net/ax25.h
@@ -419,7 +419,6 @@ void ax25_rt_device_down(struct net_device *);
int ax25_rt_ioctl(unsigned int, void __user *);
extern const struct seq_operations ax25_rt_seqops;
ax25_route *ax25_get_route(ax25_address *addr, struct net_device *dev);
-int ax25_rt_autobind(ax25_cb *, ax25_address *);
struct sk_buff *ax25_rt_build_path(struct sk_buff *, ax25_address *,
ax25_address *, ax25_digi *);
void ax25_rt_free(void);
diff --git a/include/net/bluetooth/hci.h b/include/net/bluetooth/hci.h
index fcab978894b1..473d13f9e5dd 100644
--- a/include/net/bluetooth/hci.h
+++ b/include/net/bluetooth/hci.h
@@ -1532,6 +1532,15 @@ struct hci_cp_le_set_event_mask {
__u8 mask[8];
} __packed;
+/* BLUETOOTH CORE SPECIFICATION Version 5.4 | Vol 4, Part E
+ * 7.8.2 LE Read Buffer Size command
+ * MAX_LE_MTU is 0xffff.
+ * 0 is also valid. It means that no dedicated LE Buffer exists.
+ * It should use the HCI_Read_Buffer_Size command and mtu is shared
+ * between BR/EDR and LE.
+ */
+#define HCI_MIN_LE_MTU 0x001b
+
#define HCI_OP_LE_READ_BUFFER_SIZE 0x2002
struct hci_rp_le_read_buffer_size {
__u8 status;
diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h
index 95e3fc06ee20..1502affe5e5e 100644
--- a/include/net/bluetooth/hci_core.h
+++ b/include/net/bluetooth/hci_core.h
@@ -63,7 +63,7 @@ struct inquiry_entry {
NAME_PENDING,
NAME_KNOWN,
} name_state;
- __u32 timestamp;
+ unsigned long timestamp;
struct inquiry_data data;
};
@@ -79,7 +79,7 @@ struct discovery_state {
struct list_head all; /* All devices found during inquiry */
struct list_head unknown; /* Name state not known */
struct list_head resolve; /* Name needs to be resolved */
- __u32 timestamp;
+ unsigned long timestamp;
bdaddr_t last_adv_addr;
u8 last_adv_addr_type;
s8 last_adv_rssi;
@@ -675,6 +675,7 @@ struct hci_conn {
__u16 handle;
__u16 sync_handle;
__u16 state;
+ __u16 mtu;
__u8 mode;
__u8 type;
__u8 role;
diff --git a/include/net/bluetooth/l2cap.h b/include/net/bluetooth/l2cap.h
index ce799df6cbb3..b317c89935e9 100644
--- a/include/net/bluetooth/l2cap.h
+++ b/include/net/bluetooth/l2cap.h
@@ -980,6 +980,7 @@ void __l2cap_physical_cfm(struct l2cap_chan *chan, int result);
struct l2cap_conn *l2cap_conn_get(struct l2cap_conn *conn);
void l2cap_conn_put(struct l2cap_conn *conn);
+struct l2cap_conn *l2cap_conn_hold_unless_zero(struct l2cap_conn *conn);
int l2cap_register_user(struct l2cap_conn *conn, struct l2cap_user *user);
void l2cap_unregister_user(struct l2cap_conn *conn, struct l2cap_user *user);
diff --git a/include/net/bluetooth/rfcomm.h b/include/net/bluetooth/rfcomm.h
index ba8d3702853d..d483830188f3 100644
--- a/include/net/bluetooth/rfcomm.h
+++ b/include/net/bluetooth/rfcomm.h
@@ -210,6 +210,7 @@ struct rfcomm_dlc {
#define RFCOMM_AUTH_REJECT 7
#define RFCOMM_DEFER_SETUP 8
#define RFCOMM_ENC_DROP 9
+#define RFCOMM_CLOSED 10
/* Scheduling flags and events */
#define RFCOMM_SCHED_WAKEUP 31
diff --git a/include/net/cfg80211.h b/include/net/cfg80211.h
index 297a88a4e14e..691d328e0507 100644
--- a/include/net/cfg80211.h
+++ b/include/net/cfg80211.h
@@ -1858,6 +1858,99 @@ struct cfg80211_tid_stats {
#define IEEE80211_MAX_CHAINS 4
+/**
+ * struct link_station_info - link station information
+ *
+ * Link station information filled by driver for get_station() and
+ * dump_station().
+ * @filled: bit flag of flags using the bits of &enum nl80211_sta_info to
+ * indicate the relevant values in this struct for them
+ * @connected_time: time(in secs) since a link of station is last connected
+ * @inactive_time: time since last activity for link station(tx/rx)
+ * in milliseconds
+ * @assoc_at: bootime (ns) of the last association of link of station
+ * @rx_bytes: bytes (size of MPDUs) received from this link of station
+ * @tx_bytes: bytes (size of MPDUs) transmitted to this link of station
+ * @signal: The signal strength, type depends on the wiphy's signal_type.
+ * For CFG80211_SIGNAL_TYPE_MBM, value is expressed in _dBm_.
+ * @signal_avg: Average signal strength, type depends on the wiphy's
+ * signal_type. For CFG80211_SIGNAL_TYPE_MBM, value is expressed in _dBm_
+ * @chains: bitmask for filled values in @chain_signal, @chain_signal_avg
+ * @chain_signal: per-chain signal strength of last received packet in dBm
+ * @chain_signal_avg: per-chain signal strength average in dBm
+ * @txrate: current unicast bitrate from this link of station
+ * @rxrate: current unicast bitrate to this link of station
+ * @rx_packets: packets (MSDUs & MMPDUs) received from this link of station
+ * @tx_packets: packets (MSDUs & MMPDUs) transmitted to this link of station
+ * @tx_retries: cumulative retry counts (MPDUs) for this link of station
+ * @tx_failed: number of failed transmissions (MPDUs) (retries exceeded, no ACK)
+ * @rx_dropped_misc: Dropped for un-specified reason.
+ * @bss_param: current BSS parameters
+ * @beacon_loss_count: Number of times beacon loss event has triggered.
+ * @expected_throughput: expected throughput in kbps (including 802.11 headers)
+ * towards this station.
+ * @rx_beacon: number of beacons received from this peer
+ * @rx_beacon_signal_avg: signal strength average (in dBm) for beacons received
+ * from this peer
+ * @rx_duration: aggregate PPDU duration(usecs) for all the frames from a peer
+ * @tx_duration: aggregate PPDU duration(usecs) for all the frames to a peer
+ * @airtime_weight: current airtime scheduling weight
+ * @pertid: per-TID statistics, see &struct cfg80211_tid_stats, using the last
+ * (IEEE80211_NUM_TIDS) index for MSDUs not encapsulated in QoS-MPDUs.
+ * Note that this doesn't use the @filled bit, but is used if non-NULL.
+ * @ack_signal: signal strength (in dBm) of the last ACK frame.
+ * @avg_ack_signal: average rssi value of ack packet for the no of msdu's has
+ * been sent.
+ * @rx_mpdu_count: number of MPDUs received from this station
+ * @fcs_err_count: number of packets (MPDUs) received from this station with
+ * an FCS error. This counter should be incremented only when TA of the
+ * received packet with an FCS error matches the peer MAC address.
+ * @addr: For MLO STA connection, filled with address of the link of station.
+ */
+struct link_station_info {
+ u64 filled;
+ u32 connected_time;
+ u32 inactive_time;
+ u64 assoc_at;
+ u64 rx_bytes;
+ u64 tx_bytes;
+ s8 signal;
+ s8 signal_avg;
+
+ u8 chains;
+ s8 chain_signal[IEEE80211_MAX_CHAINS];
+ s8 chain_signal_avg[IEEE80211_MAX_CHAINS];
+
+ struct rate_info txrate;
+ struct rate_info rxrate;
+ u32 rx_packets;
+ u32 tx_packets;
+ u32 tx_retries;
+ u32 tx_failed;
+ u32 rx_dropped_misc;
+ struct sta_bss_parameters bss_param;
+
+ u32 beacon_loss_count;
+
+ u32 expected_throughput;
+
+ u64 tx_duration;
+ u64 rx_duration;
+ u64 rx_beacon;
+ u8 rx_beacon_signal_avg;
+
+ u16 airtime_weight;
+
+ s8 ack_signal;
+ s8 avg_ack_signal;
+ struct cfg80211_tid_stats *pertid;
+
+ u32 rx_mpdu_count;
+ u32 fcs_err_count;
+
+ u8 addr[ETH_ALEN] __aligned(2);
+};
+
/**
* struct station_info - station information
*
@@ -1924,6 +2017,11 @@ struct cfg80211_tid_stats {
* received packet with an FCS error matches the peer MAC address.
* @airtime_link_metric: mesh airtime link metric.
* @connected_to_as: true if mesh STA has a path to authentication server
+ * @valid_links: bitmap of valid links, or 0 for non-MLO. Drivers fill this
+ * information in cfg80211_new_sta(), cfg80211_del_sta_sinfo(),
+ * get_station() and dump_station() callbacks.
+ * @links: reference to Link sta entries for MLO STA, all link specific
+ * information is accessed through links[link_id].
*/
struct station_info {
u64 filled;
@@ -1983,6 +2081,9 @@ struct station_info {
u32 airtime_link_metric;
u8 connected_to_as;
+
+ u16 valid_links;
+ struct link_station_info *links[IEEE80211_MLD_MAX_NUM_LINKS];
};
/**
@@ -5917,7 +6018,7 @@ struct wireless_dev {
enum ieee80211_bss_type conn_bss_type;
u32 conn_owner_nlportid;
- struct work_struct disconnect_wk;
+ struct wiphy_work disconnect_wk;
u8 disconnect_bssid[ETH_ALEN];
struct list_head event_list;
@@ -5959,7 +6060,7 @@ struct wireless_dev {
struct list_head pmsr_list;
spinlock_t pmsr_lock;
- struct work_struct pmsr_free_wk;
+ struct wiphy_work pmsr_free_wk;
unsigned long unprot_beacon_reported;
diff --git a/include/net/codel_impl.h b/include/net/codel_impl.h
index 78a27ac73070..b2c359c6dd1b 100644
--- a/include/net/codel_impl.h
+++ b/include/net/codel_impl.h
@@ -158,6 +158,7 @@ static struct sk_buff *codel_dequeue(void *ctx,
bool drop;
if (!skb) {
+ vars->first_above_time = 0;
vars->dropping = false;
return skb;
}
diff --git a/include/net/dropreason.h b/include/net/dropreason.h
index c1cbcdbaf149..df701ddb8324 100644
--- a/include/net/dropreason.h
+++ b/include/net/dropreason.h
@@ -68,6 +68,7 @@
FN(IP_INADDRERRORS) \
FN(IP_INNOROUTES) \
FN(PKT_TOO_BIG) \
+ FN(TC_CHAIN_NOTFOUND) \
FNe(MAX)
/**
@@ -298,6 +299,8 @@ enum skb_drop_reason {
* MTU)
*/
SKB_DROP_REASON_PKT_TOO_BIG,
+ /** @SKB_DROP_REASON_TC_CHAIN_NOTFOUND: tc chain lookup failed. */
+ SKB_DROP_REASON_TC_CHAIN_NOTFOUND,
/**
* @SKB_DROP_REASON_MAX: the maximum of drop reason, which shouldn't be
* used as a real 'reason'
diff --git a/include/net/dsa.h b/include/net/dsa.h
index e26bace77479..9e5f04dac420 100644
--- a/include/net/dsa.h
+++ b/include/net/dsa.h
@@ -310,6 +310,7 @@ struct dsa_port {
struct devlink_port devlink_port;
struct phylink *pl;
struct phylink_config pl_config;
+ netdevice_tracker conduit_tracker;
struct dsa_lag *lag;
struct net_device *hsr_dev;
diff --git a/include/net/inet_connection_sock.h b/include/net/inet_connection_sock.h
index 9760ba0516b6..51be99667a23 100644
--- a/include/net/inet_connection_sock.h
+++ b/include/net/inet_connection_sock.h
@@ -63,6 +63,9 @@ struct inet_connection_sock_af_ops {
* @icsk_bind2_hash: Bind node in the bhash2 table
* @icsk_timeout: Timeout
* @icsk_retransmit_timer: Resend (no ack)
+ * @icsk_delack_timer: Delayed ACK timer
+ * @icsk_keepalive_timer: Keepalive timer
+ * @mptcp_tout_timer: MPTCP timeout timer
* @icsk_rto: Retransmit timeout
* @icsk_pmtu_cookie Last pmtu seen by socket
* @icsk_ca_ops Pluggable congestion control hook
@@ -89,8 +92,12 @@ struct inet_connection_sock {
struct inet_bind_bucket *icsk_bind_hash;
struct inet_bind2_bucket *icsk_bind2_hash;
unsigned long icsk_timeout;
- struct timer_list icsk_retransmit_timer;
- struct timer_list icsk_delack_timer;
+ struct timer_list icsk_retransmit_timer;
+ struct timer_list icsk_delack_timer;
+ union {
+ struct timer_list icsk_keepalive_timer;
+ struct timer_list mptcp_tout_timer;
+ };
__u32 icsk_rto;
__u32 icsk_rto_min;
__u32 icsk_delack_max;
diff --git a/include/net/ip.h b/include/net/ip.h
index 49e86b8bf9c9..855998087701 100644
--- a/include/net/ip.h
+++ b/include/net/ip.h
@@ -459,9 +459,14 @@ static inline unsigned int ip_dst_mtu_maybe_forward(const struct dst_entry *dst,
bool forwarding)
{
const struct rtable *rt = container_of(dst, struct rtable, dst);
- struct net *net = dev_net(dst_dev(dst));
- unsigned int mtu;
+ const struct net_device *dev;
+ unsigned int mtu, res;
+ struct net *net;
+
+ rcu_read_lock();
+ dev = dst_dev_rcu(dst);
+ net = dev_net_rcu(dev);
if (READ_ONCE(net->ipv4.sysctl_ip_fwd_use_pmtu) ||
ip_mtu_locked(dst) ||
!forwarding) {
@@ -475,7 +480,7 @@ static inline unsigned int ip_dst_mtu_maybe_forward(const struct dst_entry *dst,
if (mtu)
goto out;
- mtu = READ_ONCE(dst_dev(dst)->mtu);
+ mtu = READ_ONCE(dev->mtu);
if (unlikely(ip_mtu_locked(dst))) {
if (rt->rt_uses_gateway && mtu > 576)
@@ -485,7 +490,11 @@ static inline unsigned int ip_dst_mtu_maybe_forward(const struct dst_entry *dst,
out:
mtu = min_t(unsigned int, mtu, IP_MAX_MTU);
- return mtu - lwtunnel_headroom(dst->lwtstate, mtu);
+ res = mtu - lwtunnel_headroom(dst->lwtstate, mtu);
+
+ rcu_read_unlock();
+
+ return res;
}
static inline unsigned int ip_skb_dst_mtu(struct sock *sk,
diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h
index 9df55c2ee058..6b7dae7d47db 100644
--- a/include/net/ip6_route.h
+++ b/include/net/ip6_route.h
@@ -337,7 +337,7 @@ static inline unsigned int ip6_dst_mtu_maybe_forward(const struct dst_entry *dst
mtu = IPV6_MIN_MTU;
rcu_read_lock();
- idev = __in6_dev_get(dst->dev);
+ idev = __in6_dev_get(dst_dev_rcu(dst));
if (idev)
mtu = idev->cnf.mtu6;
rcu_read_unlock();
diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h
index 9aeb44f03f17..9ac0ffb54beb 100644
--- a/include/net/ip_vs.h
+++ b/include/net/ip_vs.h
@@ -1583,7 +1583,7 @@ static inline char ip_vs_fwd_tag(struct ip_vs_conn *cp)
return fwd;
}
-void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
+bool ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct ip_vs_conn *cp, int dir, unsigned int toff,
bool has_ports, struct ip_vs_iphdr *ciph);
diff --git a/include/net/netfilter/nf_conntrack_expect.h b/include/net/netfilter/nf_conntrack_expect.h
index 80f50fd0f7ad..be4a120d549e 100644
--- a/include/net/netfilter/nf_conntrack_expect.h
+++ b/include/net/netfilter/nf_conntrack_expect.h
@@ -54,8 +54,8 @@ struct nf_conntrack_expect {
/* The conntrack of the master connection */
struct nf_conn *master;
- /* Timer function; deletes the expectation. */
- struct timer_list timeout;
+ /* jiffies32 when this expectation expires */
+ u32 timeout;
#if IS_ENABLED(CONFIG_NF_NAT)
union nf_inet_addr saved_addr;
@@ -69,6 +69,14 @@ struct nf_conntrack_expect {
struct rcu_head rcu;
};
+static inline bool nf_ct_exp_is_expired(const struct nf_conntrack_expect *exp)
+{
+ if (READ_ONCE(exp->flags) & NF_CT_EXPECT_DEAD)
+ return true;
+
+ return (__s32)(READ_ONCE(exp->timeout) - nfct_time_stamp) <= 0;
+}
+
static inline struct net *nf_ct_exp_net(struct nf_conntrack_expect *exp)
{
return read_pnet(&exp->net);
@@ -130,7 +138,6 @@ static inline void nf_ct_unlink_expect(struct nf_conntrack_expect *exp)
void nf_ct_remove_expectations(struct nf_conn *ct);
void nf_ct_unexpect_related(struct nf_conntrack_expect *exp);
-bool nf_ct_remove_expect(struct nf_conntrack_expect *exp);
void nf_ct_expect_iterate_destroy(bool (*iter)(struct nf_conntrack_expect *e, void *data), void *data);
void nf_ct_expect_iterate_net(struct net *net,
@@ -153,5 +160,8 @@ static inline int nf_ct_expect_related(struct nf_conntrack_expect *expect,
return nf_ct_expect_related_report(expect, 0, 0, flags);
}
+struct nf_conn_help;
+void nf_ct_expectation_gc(struct nf_conn_help *master_help);
+
#endif /*_NF_CONNTRACK_EXPECT_H*/
diff --git a/include/net/netfilter/nf_conntrack_timeout.h b/include/net/netfilter/nf_conntrack_timeout.h
index 3a66d4abb6d6..d60aa86be019 100644
--- a/include/net/netfilter/nf_conntrack_timeout.h
+++ b/include/net/netfilter/nf_conntrack_timeout.h
@@ -12,6 +12,7 @@
#define CTNL_TIMEOUT_NAME_MAX 32
struct nf_ct_timeout {
+ refcount_t refcnt;
__u16 l3num;
const struct nf_conntrack_l4proto *l4proto;
struct rcu_head rcu;
@@ -22,6 +23,22 @@ struct nf_conn_timeout {
struct nf_ct_timeout __rcu *timeout;
};
+static inline void nf_ct_timeout_put(const struct nf_conn *ct)
+{
+#ifdef CONFIG_NF_CONNTRACK_TIMEOUT
+ struct nf_conn_timeout *timeout_ext;
+ struct nf_ct_timeout *timeout;
+
+ timeout_ext = nf_ct_ext_find(ct, NF_CT_EXT_TIMEOUT);
+ if (!timeout_ext)
+ return;
+
+ timeout = rcu_dereference(timeout_ext->timeout);
+ if (timeout && refcount_dec_and_test(&timeout->refcnt))
+ kfree_rcu(timeout, rcu);
+#endif
+}
+
static inline unsigned int *
nf_ct_timeout_data(const struct nf_conn_timeout *t)
{
@@ -56,8 +73,14 @@ struct nf_conn_timeout *nf_ct_timeout_ext_add(struct nf_conn *ct,
#ifdef CONFIG_NF_CONNTRACK_TIMEOUT
struct nf_conn_timeout *timeout_ext;
+ if (!timeout)
+ return NULL;
+
timeout_ext = nf_ct_ext_add(ct, NF_CT_EXT_TIMEOUT, gfp);
- if (timeout_ext == NULL)
+ if (!timeout_ext || timeout_ext->timeout)
+ return NULL;
+
+ if (!refcount_inc_not_zero(&timeout->refcnt))
return NULL;
rcu_assign_pointer(timeout_ext->timeout, timeout);
@@ -75,7 +98,7 @@ static inline unsigned int *nf_ct_timeout_lookup(const struct nf_conn *ct)
struct nf_conn_timeout *timeout_ext;
timeout_ext = nf_ct_timeout_find(ct);
- if (timeout_ext)
+ if (timeout_ext && rcu_access_pointer(timeout_ext->timeout))
timeouts = nf_ct_timeout_data(timeout_ext);
#endif
return timeouts;
diff --git a/include/net/netfilter/nf_queue.h b/include/net/netfilter/nf_queue.h
index c576479b9aa5..4cf158b5ea69 100644
--- a/include/net/netfilter/nf_queue.h
+++ b/include/net/netfilter/nf_queue.h
@@ -21,6 +21,7 @@ struct nf_queue_entry {
struct net_device *physout;
#endif
struct nf_hook_state state;
+ bool nf_ct_is_unconfirmed;
u16 size; /* sizeof(entry) + saved route keys */
/* extra space to store route keys */
diff --git a/include/net/netfilter/nf_tables.h b/include/net/netfilter/nf_tables.h
index 7e63281f2d5a..cdb2e1a34436 100644
--- a/include/net/netfilter/nf_tables.h
+++ b/include/net/netfilter/nf_tables.h
@@ -187,6 +187,13 @@ static inline u64 nft_reg_load64(const u32 *sreg)
return get_unaligned((u64 *)sreg);
}
+static inline bool nft_reg_overlap(u8 src, u8 dst, u32 len)
+{
+ unsigned int n = DIV_ROUND_UP(len, sizeof(u32));
+
+ return src != dst && src < dst + n && dst < src + n;
+}
+
static inline void nft_data_copy(u32 *dst, const struct nft_data *src,
unsigned int len)
{
@@ -302,11 +309,13 @@ struct nft_set_elem {
*
* @NFT_ITER_READ: read-only iteration over set elements
* @NFT_ITER_UPDATE: iteration under mutex to update set element state
+ * @NFT_ITER_UPDATE_CLONE: clone set before iteration under mutex to update element
*/
enum nft_iter_type {
NFT_ITER_UNSPEC,
NFT_ITER_READ,
NFT_ITER_UPDATE,
+ NFT_ITER_UPDATE_CLONE,
};
struct nft_set;
@@ -454,7 +463,7 @@ struct nft_set_ops {
int (*insert)(const struct net *net,
const struct nft_set *set,
const struct nft_set_elem *elem,
- struct nft_set_ext **ext);
+ void **priv);
void (*activate)(const struct net *net,
const struct nft_set *set,
const struct nft_set_elem *elem);
@@ -843,8 +852,6 @@ void *nft_set_elem_init(const struct nft_set *set,
const struct nft_set_ext_tmpl *tmpl,
const u32 *key, const u32 *key_end, const u32 *data,
u64 timeout, u64 expiration, gfp_t gfp);
-int nft_set_elem_expr_clone(const struct nft_ctx *ctx, struct nft_set *set,
- struct nft_expr *expr_array[]);
void nft_set_elem_expr_destroy(const struct nft_ctx *ctx,
struct nft_set_elem_expr *elem_expr);
void nft_set_elem_destroy(const struct nft_set *set, void *elem,
@@ -930,7 +937,8 @@ struct nft_expr_ops {
void (*destroy_clone)(const struct nft_ctx *ctx,
const struct nft_expr *expr);
int (*dump)(struct sk_buff *skb,
- const struct nft_expr *expr);
+ const struct nft_expr *expr,
+ bool reset);
int (*validate)(const struct nft_ctx *ctx,
const struct nft_expr *expr);
bool (*reduce)(struct nft_regs_track *track,
diff --git a/include/net/netfilter/nft_fib.h b/include/net/netfilter/nft_fib.h
index fc6ee031b35c..38cae7113de4 100644
--- a/include/net/netfilter/nft_fib.h
+++ b/include/net/netfilter/nft_fib.h
@@ -18,7 +18,7 @@ nft_fib_is_loopback(const struct sk_buff *skb, const struct net_device *in)
return skb->pkt_type == PACKET_LOOPBACK || in->flags & IFF_LOOPBACK;
}
-int nft_fib_dump(struct sk_buff *skb, const struct nft_expr *expr);
+int nft_fib_dump(struct sk_buff *skb, const struct nft_expr *expr, bool reset);
int nft_fib_init(const struct nft_ctx *ctx, const struct nft_expr *expr,
const struct nlattr * const tb[]);
int nft_fib_validate(const struct nft_ctx *ctx, const struct nft_expr *expr);
diff --git a/include/net/netfilter/nft_meta.h b/include/net/netfilter/nft_meta.h
index 22e98ac336f7..b31a9e907417 100644
--- a/include/net/netfilter/nft_meta.h
+++ b/include/net/netfilter/nft_meta.h
@@ -24,10 +24,10 @@ int nft_meta_set_init(const struct nft_ctx *ctx,
const struct nlattr * const tb[]);
int nft_meta_get_dump(struct sk_buff *skb,
- const struct nft_expr *expr);
+ const struct nft_expr *expr, bool reset);
int nft_meta_set_dump(struct sk_buff *skb,
- const struct nft_expr *expr);
+ const struct nft_expr *expr, bool reset);
void nft_meta_get_eval(const struct nft_expr *expr,
struct nft_regs *regs,
diff --git a/include/net/netfilter/nft_reject.h b/include/net/netfilter/nft_reject.h
index bce67b3a42aa..19060212988a 100644
--- a/include/net/netfilter/nft_reject.h
+++ b/include/net/netfilter/nft_reject.h
@@ -21,7 +21,8 @@ int nft_reject_init(const struct nft_ctx *ctx,
const struct nft_expr *expr,
const struct nlattr * const tb[]);
-int nft_reject_dump(struct sk_buff *skb, const struct nft_expr *expr);
+int nft_reject_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset);
int nft_reject_icmp_code(u8 code);
int nft_reject_icmpv6_code(u8 code);
diff --git a/include/net/netns/mpls.h b/include/net/netns/mpls.h
index 19ad2574b267..f364286adf28 100644
--- a/include/net/netns/mpls.h
+++ b/include/net/netns/mpls.h
@@ -16,6 +16,7 @@ struct netns_mpls {
int default_ttl;
size_t platform_labels;
struct mpls_route __rcu * __rcu *platform_label;
+ seqcount_t platform_label_seq;
struct ctl_table_header *ctl;
};
diff --git a/include/net/netns/xfrm.h b/include/net/netns/xfrm.h
index 7922ec72eaa6..eb9edddd7309 100644
--- a/include/net/netns/xfrm.h
+++ b/include/net/netns/xfrm.h
@@ -23,6 +23,7 @@ struct xfrm_policy_hash {
struct xfrm_policy_hthresh {
struct work_struct work;
seqlock_t lock;
+ bool work_disabled;
u8 lbits4;
u8 rbits4;
u8 lbits6;
diff --git a/include/net/nexthop.h b/include/net/nexthop.h
index 2b12725de9c0..f37e96f93f20 100644
--- a/include/net/nexthop.h
+++ b/include/net/nexthop.h
@@ -97,7 +97,7 @@ struct nh_res_table {
struct nh_grp_entry {
struct nexthop *nh;
- u8 weight;
+ u16 weight;
union {
struct {
@@ -173,14 +173,14 @@ struct nh_notifier_single_info {
__be32 ipv4;
struct in6_addr ipv6;
};
+ u32 id;
u8 is_reject:1,
is_fdb:1,
has_encap:1;
};
struct nh_notifier_grp_entry_info {
- u8 weight;
- u32 id;
+ u16 weight;
struct nh_notifier_single_info nh;
};
diff --git a/include/net/pkt_cls.h b/include/net/pkt_cls.h
index 857d9ea60d46..7ab833b41d05 100644
--- a/include/net/pkt_cls.h
+++ b/include/net/pkt_cls.h
@@ -70,6 +70,10 @@ static inline bool tcf_block_non_null_shared(struct tcf_block *block)
return block && block->index;
}
+#ifdef CONFIG_NET_CLS_ACT
+DECLARE_STATIC_KEY_FALSE(tcf_sw_enabled_key);
+#endif
+
static inline struct Qdisc *tcf_block_q(struct tcf_block *block)
{
WARN_ON(tcf_block_shared(block));
@@ -156,8 +160,20 @@ static inline int tcf_classify(struct sk_buff *skb,
{
return TC_ACT_UNSPEC;
}
-
#endif
+static inline int tcf_classify_qdisc(struct sk_buff *skb,
+ const struct tcf_proto *tp,
+ struct tcf_result *res, bool compat_mode)
+{
+ int ret = tcf_classify(skb, NULL, tp, res, compat_mode);
+
+ /* TC_ACT_REDIRECT from qdisc filter chains is not supported.
+ * Use BPF via tcx or mirred redirect instead.
+ */
+ if (unlikely(ret == TC_ACT_REDIRECT))
+ ret = TC_ACT_SHOT;
+ return ret;
+}
static inline unsigned long
__cls_set_class(unsigned long *clp, unsigned long cl)
@@ -165,6 +181,22 @@ __cls_set_class(unsigned long *clp, unsigned long cl)
return xchg(clp, cl);
}
+struct tc_skb_cb;
+
+static inline struct tc_skb_cb *tc_skb_cb(const struct sk_buff *skb);
+
+static inline enum skb_drop_reason
+tcf_get_drop_reason(const struct sk_buff *skb)
+{
+ return tc_skb_cb(skb)->drop_reason;
+}
+
+static inline void tcf_set_drop_reason(const struct sk_buff *skb,
+ enum skb_drop_reason reason)
+{
+ tc_skb_cb(skb)->drop_reason = reason;
+}
+
static inline void
__tcf_bind_filter(struct Qdisc *q, struct tcf_result *r, unsigned long base)
{
@@ -248,7 +280,7 @@ static inline int tcf_exts_init(struct tcf_exts *exts, struct net *net,
*/
exts->net = net;
exts->actions = kcalloc(TCA_ACT_MAX_PRIO, sizeof(struct tc_action *),
- GFP_KERNEL);
+ GFP_KERNEL_ACCOUNT);
if (!exts->actions)
return -ENOMEM;
#endif
@@ -750,6 +782,15 @@ tc_cls_common_offload_init(struct flow_cls_common_offload *cls_common,
cls_common->extack = extack;
}
+static inline void tcf_proto_update_usesw(struct tcf_proto *tp, u32 flags)
+{
+ if (tp->usesw)
+ return;
+ if (tc_skip_sw(flags) && tc_in_hw(flags))
+ return;
+ tp->usesw = true;
+}
+
#if IS_ENABLED(CONFIG_NET_TC_SKB_EXT)
static inline struct tc_skb_ext *tc_skb_ext_alloc(struct sk_buff *skb)
{
diff --git a/include/net/pkt_sched.h b/include/net/pkt_sched.h
index 7795b2f68e36..2ff91bfeb80d 100644
--- a/include/net/pkt_sched.h
+++ b/include/net/pkt_sched.h
@@ -218,12 +218,13 @@ static inline void skb_txtime_consumed(struct sk_buff *skb)
struct tc_skb_cb {
struct qdisc_skb_cb qdisc_cb;
+ u32 drop_reason;
+ u16 zone; /* Only valid if post_ct = true */
u16 mru;
u8 post_ct:1;
u8 post_ct_snat:1;
u8 post_ct_dnat:1;
- u16 zone; /* Only valid if post_ct = true */
};
static inline struct tc_skb_cb *tc_skb_cb(const struct sk_buff *skb)
diff --git a/include/net/route.h b/include/net/route.h
index ce608f883040..4e2f8c3e1f98 100644
--- a/include/net/route.h
+++ b/include/net/route.h
@@ -362,7 +362,7 @@ static inline int ip4_dst_hoplimit(const struct dst_entry *dst)
const struct net *net;
rcu_read_lock();
- net = dev_net_rcu(dst_dev(dst));
+ net = dst_dev_net_rcu(dst);
hoplimit = READ_ONCE(net->ipv4.sysctl_ip_default_ttl);
rcu_read_unlock();
}
diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h
index 7ec1f5aa6ada..2bc9b63cf9f2 100644
--- a/include/net/sch_generic.h
+++ b/include/net/sch_generic.h
@@ -117,6 +117,7 @@ struct Qdisc {
struct qdisc_skb_head q;
struct gnet_stats_basic_sync bstats;
struct gnet_stats_queue qstats;
+ int owner;
unsigned long state;
unsigned long state2; /* must be written under qdisc spinlock */
struct Qdisc *next_sched;
@@ -326,7 +327,6 @@ struct Qdisc_ops {
struct module *owner;
};
-
struct tcf_result {
union {
struct {
@@ -334,7 +334,6 @@ struct tcf_result {
u32 classid;
};
const struct tcf_proto *goto_tp;
-
};
};
@@ -378,6 +377,10 @@ struct tcf_proto_ops {
struct nlattr **tca,
struct netlink_ext_ack *extack);
void (*tmplt_destroy)(void *tmplt_priv);
+ void (*tmplt_reoffload)(struct tcf_chain *chain,
+ bool add,
+ flow_setup_cb_t *cb,
+ void *cb_priv);
/* rtnetlink specific */
int (*dump)(struct net*, struct tcf_proto*, void *,
@@ -424,6 +427,8 @@ struct tcf_proto {
*/
spinlock_t lock;
bool deleting;
+ bool counted;
+ bool usesw;
refcount_t refcnt;
struct rcu_head rcu;
struct hlist_node destroy_ht_node;
@@ -472,6 +477,7 @@ struct tcf_block {
struct flow_block flow_block;
struct list_head owner_list;
bool keep_dst;
+ atomic_t useswcnt;
atomic_t offloadcnt; /* Number of oddloaded filters */
unsigned int nooffloaddevcnt; /* Number of devs unable to do offload */
unsigned int lockeddevcnt; /* Number of devs that require rtnl lock. */
@@ -1043,12 +1049,17 @@ static inline struct sk_buff *qdisc_dequeue_internal(struct Qdisc *sch, bool dir
skb = __skb_dequeue(&sch->gso_skb);
if (skb) {
sch->q.qlen--;
+ qdisc_qstats_backlog_dec(sch, skb);
return skb;
}
- if (direct)
- return __qdisc_dequeue_head(&sch->q);
- else
+ if (direct) {
+ skb = __qdisc_dequeue_head(&sch->q);
+ if (skb)
+ qdisc_qstats_backlog_dec(sch, skb);
+ return skb;
+ } else {
return sch->dequeue(sch);
+ }
}
static inline struct sk_buff *qdisc_dequeue_head(struct Qdisc *sch)
diff --git a/include/net/secure_seq.h b/include/net/secure_seq.h
index 21e7fa2a1813..899612c4bec3 100644
--- a/include/net/secure_seq.h
+++ b/include/net/secure_seq.h
@@ -5,17 +5,50 @@
#include <linux/types.h>
struct net;
+extern struct net init_net;
+
+union tcp_seq_and_ts_off {
+ struct {
+ u32 seq;
+ u32 ts_off;
+ };
+ u64 hash64;
+};
u64 secure_ipv4_port_ephemeral(__be32 saddr, __be32 daddr, __be16 dport);
u64 secure_ipv6_port_ephemeral(const __be32 *saddr, const __be32 *daddr,
__be16 dport);
-u32 secure_tcp_seq(__be32 saddr, __be32 daddr,
- __be16 sport, __be16 dport);
-u32 secure_tcp_ts_off(const struct net *net, __be32 saddr, __be32 daddr);
-u32 secure_tcpv6_seq(const __be32 *saddr, const __be32 *daddr,
- __be16 sport, __be16 dport);
-u32 secure_tcpv6_ts_off(const struct net *net,
- const __be32 *saddr, const __be32 *daddr);
+union tcp_seq_and_ts_off
+secure_tcp_seq_and_ts_off(const struct net *net, __be32 saddr, __be32 daddr,
+ __be16 sport, __be16 dport);
+
+static inline u32 secure_tcp_seq(__be32 saddr, __be32 daddr,
+ __be16 sport, __be16 dport)
+{
+ union tcp_seq_and_ts_off ts;
+
+ ts = secure_tcp_seq_and_ts_off(&init_net, saddr, daddr,
+ sport, dport);
+
+ return ts.seq;
+}
+
+union tcp_seq_and_ts_off
+secure_tcpv6_seq_and_ts_off(const struct net *net, const __be32 *saddr,
+ const __be32 *daddr,
+ __be16 sport, __be16 dport);
+
+static inline u32 secure_tcpv6_seq(const __be32 *saddr, const __be32 *daddr,
+ __be16 sport, __be16 dport)
+{
+ union tcp_seq_and_ts_off ts;
+
+ ts = secure_tcpv6_seq_and_ts_off(&init_net, saddr, daddr,
+ sport, dport);
+
+ return ts.seq;
+}
+
u64 secure_dccp_sequence_number(__be32 saddr, __be32 daddr,
__be16 sport, __be16 dport);
u64 secure_dccpv6_sequence_number(__be32 *saddr, __be32 *daddr,
diff --git a/include/net/sock.h b/include/net/sock.h
index 9fe621f98574..13c112bd2d31 100644
--- a/include/net/sock.h
+++ b/include/net/sock.h
@@ -315,9 +315,13 @@ struct sk_filter;
* @sk_txrehash: enable TX hash rethink
* @sk_filter: socket filtering instructions
* @sk_timer: sock cleanup timer
+ * @mptcp_retransmit_timer: MPTCP retransmit timer
* @sk_stamp: time stamp of last packet received
* @sk_stamp_seq: lock for accessing sk_stamp on 32 bit architectures only
* @sk_tsflags: SO_TIMESTAMPING flags
+ * @sk_use_task_frag: allow sk_page_frag() to use current->task_frag.
+ * Sockets that can be used under memory reclaim should
+ * set this to false.
* @sk_bind_phc: SO_TIMESTAMPING bind PHC index of PTP virtual clock
* for timestamping
* @sk_tskey: counter to disambiguate concurrent tstamp requests
@@ -459,7 +463,10 @@ struct sock {
__u32 sk_dst_pending_confirm;
u32 sk_pacing_status; /* see enum sk_pacing */
long sk_sndtimeo;
- struct timer_list sk_timer;
+ union {
+ struct timer_list sk_timer;
+ struct timer_list mptcp_retransmit_timer;
+ };
__u32 sk_priority;
__u32 sk_mark;
unsigned long sk_pacing_rate; /* bytes per second */
@@ -516,6 +523,7 @@ struct sock {
u8 sk_txtime_deadline_mode : 1,
sk_txtime_report_errors : 1,
sk_txtime_unused : 6;
+ bool sk_use_task_frag;
struct socket *sk_socket;
void *sk_user_data;
@@ -1977,6 +1985,8 @@ static inline struct sk_buff *sock_alloc_send_skb(struct sock *sk,
}
void *sock_kmalloc(struct sock *sk, int size, gfp_t priority);
+void *sock_kmemdup(struct sock *sk, const void *src,
+ int size, gfp_t priority);
void sock_kfree_s(struct sock *sk, void *mem, int size);
void sock_kzfree_s(struct sock *sk, void *mem, int size);
void sk_send_sigurg(struct sock *sk);
@@ -2679,14 +2689,17 @@ static inline void sk_stream_moderate_sndbuf(struct sock *sk)
* socket operations and end up recursing into sk_page_frag()
* while it's already in use: explicitly avoid task page_frag
* usage if the caller is potentially doing any of them.
- * This assumes that page fault handlers use the GFP_NOFS flags.
+ * This assumes that page fault handlers use the GFP_NOFS flags or
+ * explicitly disable sk_use_task_frag.
*
* Return: a per task page_frag if context allows that,
* otherwise a per socket one.
*/
static inline struct page_frag *sk_page_frag(struct sock *sk)
{
- if ((sk->sk_allocation & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC | __GFP_FS)) ==
+ if (sk->sk_use_task_frag &&
+ (sk->sk_allocation & (__GFP_DIRECT_RECLAIM | __GFP_MEMALLOC |
+ __GFP_FS)) ==
(__GFP_DIRECT_RECLAIM | __GFP_FS))
return ¤t->task_frag;
diff --git a/include/net/strparser.h b/include/net/strparser.h
index 0a83010b3a64..6db67c2526e3 100644
--- a/include/net/strparser.h
+++ b/include/net/strparser.h
@@ -91,11 +91,11 @@ static inline struct strp_msg *strp_msg(struct sk_buff *skb)
struct strparser {
struct sock *sk;
- u32 stopped : 1;
- u32 paused : 1;
- u32 aborted : 1;
- u32 interrupted : 1;
- u32 unrecov_intr : 1;
+ u8 stopped;
+ u8 paused;
+ u8 aborted;
+ u8 interrupted;
+ u8 unrecov_intr;
struct sk_buff **skb_nextp;
struct sk_buff *skb_head;
diff --git a/include/net/tc_act/tc_pedit.h b/include/net/tc_act/tc_pedit.h
index a26d4cd3b8d6..fa4cd4fc3674 100644
--- a/include/net/tc_act/tc_pedit.h
+++ b/include/net/tc_act/tc_pedit.h
@@ -36,17 +36,15 @@ static inline bool is_tcf_pedit(const struct tc_action *a)
return false;
}
-static inline int tcf_pedit_nkeys(const struct tc_action *a)
+/* Must be called with act->tcfa_lock held to ensure consistency of parallel
+ * reads of the same action's pedit keys (e.g. flow_offload count vs fill).
+ * Note, this is only used for pedit offload.
+ */
+static inline int tcf_pedit_nkeys_locked(const struct tc_action *a)
{
- struct tcf_pedit_parms *parms;
- int nkeys;
-
- rcu_read_lock();
- parms = to_pedit_parms(a);
- nkeys = parms->tcfp_nkeys;
- rcu_read_unlock();
-
- return nkeys;
+ lockdep_assert_held(&a->tcfa_lock);
+ return rcu_dereference_protected(to_pedit(a)->parms,
+ lockdep_is_held(&a->tcfa_lock))->tcfp_nkeys;
}
static inline u32 tcf_pedit_htype(const struct tc_action *a, int index)
diff --git a/include/net/tcp.h b/include/net/tcp.h
index c25cd0df7a90..ff8025ff25a4 100644
--- a/include/net/tcp.h
+++ b/include/net/tcp.h
@@ -41,6 +41,7 @@
#include <net/dst.h>
#include <net/mptcp.h>
#include <net/xfrm.h>
+#include <net/secure_seq.h>
#include <linux/seq_file.h>
#include <linux/memcontrol.h>
@@ -1019,6 +1020,7 @@ static inline bool tcp_skb_can_collapse(const struct sk_buff *to,
{
return likely(tcp_skb_can_collapse_to(to) &&
mptcp_skb_can_collapse(to, from) &&
+ !skb_cmp_decrypted(to, from) &&
skb_pure_zcopy_same(to, from));
}
@@ -1481,6 +1483,11 @@ static inline int tcp_full_space(const struct sock *sk)
return tcp_win_from_space(sk, READ_ONCE(sk->sk_rcvbuf));
}
+static inline u32 tcp_dst_advmss(const struct dst_entry *dst)
+{
+ return max_t(u32, dst_metric_advmss(dst), TCP_MIN_MSS);
+}
+
static inline void __tcp_adjust_rcv_ssthresh(struct sock *sk, u32 new_ssthresh)
{
int unused_mem = sk_unused_reserved_mem(sk);
@@ -1639,6 +1646,8 @@ static inline bool tcp_paws_reject(const struct tcp_options_received *rx_opt,
bool tcp_oow_rate_limited(struct net *net, const struct sk_buff *skb,
int mib_idx, u32 *last_oow_ack_time);
+void tcp_reqsk_send_challenge_ack(struct sock *sk, struct sk_buff *skb,
+ struct request_sock *req);
static inline void tcp_mib_init(struct net *net)
{
@@ -2142,8 +2151,9 @@ struct tcp_request_sock_ops {
struct sk_buff *skb,
struct flowi *fl,
struct request_sock *req);
- u32 (*init_seq)(const struct sk_buff *skb);
- u32 (*init_ts_off)(const struct net *net, const struct sk_buff *skb);
+ union tcp_seq_and_ts_off (*init_seq_and_ts_off)(
+ const struct net *net,
+ const struct sk_buff *skb);
int (*send_synack)(const struct sock *sk, struct dst_entry *dst,
struct flowi *fl, struct request_sock *req,
struct tcp_fastopen_cookie *foc,
diff --git a/include/net/tls.h b/include/net/tls.h
index b5856a28070d..cd0dfa25c591 100644
--- a/include/net/tls.h
+++ b/include/net/tls.h
@@ -405,8 +405,12 @@ tls_offload_ctx_tx(const struct tls_context *tls_ctx)
static inline bool tls_sw_has_ctx_tx(const struct sock *sk)
{
- struct tls_context *ctx = tls_get_ctx(sk);
+ struct tls_context *ctx;
+
+ if (!sk_is_inet(sk) || !inet_sk(sk)->is_icsk)
+ return false;
+ ctx = tls_get_ctx(sk);
if (!ctx)
return false;
return !!tls_sw_ctx_tx(ctx);
@@ -414,8 +418,12 @@ static inline bool tls_sw_has_ctx_tx(const struct sock *sk)
static inline bool tls_sw_has_ctx_rx(const struct sock *sk)
{
- struct tls_context *ctx = tls_get_ctx(sk);
+ struct tls_context *ctx;
+
+ if (!sk_is_inet(sk) || !inet_sk(sk)->is_icsk)
+ return false;
+ ctx = tls_get_ctx(sk);
if (!ctx)
return false;
return !!tls_sw_ctx_rx(ctx);
diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h
index 149b63e3534a..4be73c771ed0 100644
--- a/include/scsi/scsi_host.h
+++ b/include/scsi/scsi_host.h
@@ -623,6 +623,9 @@ struct Scsi_Host {
*/
unsigned nr_hw_queues;
unsigned nr_maps;
+ /* Don't resume host in EH */
+ bool eh_noresume;
+
unsigned active_mode:2;
/*
@@ -644,9 +647,6 @@ struct Scsi_Host {
/* Asynchronous scan in progress */
unsigned async_scan:1;
- /* Don't resume host in EH */
- unsigned eh_noresume:1;
-
/* The controller does not support WRITE SAME */
unsigned no_write_same:1;
@@ -704,6 +704,9 @@ struct Scsi_Host {
*/
struct device *dma_dev;
+ /* Used for an rcu-synchronizing eh wakeup */
+ struct work_struct eh_work;
+
/*
* We should ensure that this is aligned, both for better performance
* and also because some compilers (m68k) don't automatically force
diff --git a/include/trace/events/btrfs.h b/include/trace/events/btrfs.h
index b2b41b98ec6d..dcb861e813f2 100644
--- a/include/trace/events/btrfs.h
+++ b/include/trace/events/btrfs.h
@@ -225,8 +225,8 @@ DECLARE_EVENT_CLASS(btrfs__inode,
__entry->generation = BTRFS_I(inode)->generation;
__entry->last_trans = BTRFS_I(inode)->last_trans;
__entry->logged_trans = BTRFS_I(inode)->logged_trans;
- __entry->root_objectid =
- BTRFS_I(inode)->root->root_key.objectid;
+ __entry->root_objectid = BTRFS_I(inode)->root ?
+ BTRFS_I(inode)->root->root_key.objectid : 0;
),
TP_printk_btrfs("root=%llu(%s) gen=%llu ino=%llu blocks=%llu "
diff --git a/include/trace/events/rv.h b/include/trace/events/rv.h
index 56592da9301c..da213b91da33 100644
--- a/include/trace/events/rv.h
+++ b/include/trace/events/rv.h
@@ -16,23 +16,23 @@ DECLARE_EVENT_CLASS(event_da_monitor,
TP_ARGS(state, event, next_state, final_state),
TP_STRUCT__entry(
- __array( char, state, MAX_DA_NAME_LEN )
- __array( char, event, MAX_DA_NAME_LEN )
- __array( char, next_state, MAX_DA_NAME_LEN )
- __field( bool, final_state )
+ __string( state, state )
+ __string( event, event )
+ __string( next_state, next_state )
+ __field( bool, final_state )
),
TP_fast_assign(
- memcpy(__entry->state, state, MAX_DA_NAME_LEN);
- memcpy(__entry->event, event, MAX_DA_NAME_LEN);
- memcpy(__entry->next_state, next_state, MAX_DA_NAME_LEN);
- __entry->final_state = final_state;
+ __assign_str(state, state);
+ __assign_str(event, event);
+ __assign_str(next_state, next_state);
+ __entry->final_state = final_state;
),
TP_printk("%s x %s -> %s %s",
- __entry->state,
- __entry->event,
- __entry->next_state,
+ __get_str(state),
+ __get_str(event),
+ __get_str(next_state),
__entry->final_state ? "(final)" : "")
);
@@ -43,18 +43,18 @@ DECLARE_EVENT_CLASS(error_da_monitor,
TP_ARGS(state, event),
TP_STRUCT__entry(
- __array( char, state, MAX_DA_NAME_LEN )
- __array( char, event, MAX_DA_NAME_LEN )
+ __string( state, state )
+ __string( event, event )
),
TP_fast_assign(
- memcpy(__entry->state, state, MAX_DA_NAME_LEN);
- memcpy(__entry->event, event, MAX_DA_NAME_LEN);
+ __assign_str(state, state);
+ __assign_str(event, event);
),
TP_printk("event %s not expected in the state %s",
- __entry->event,
- __entry->state)
+ __get_str(event),
+ __get_str(state))
);
#ifdef CONFIG_RV_MON_WIP
@@ -76,26 +76,26 @@ DECLARE_EVENT_CLASS(event_da_monitor_id,
TP_ARGS(id, state, event, next_state, final_state),
TP_STRUCT__entry(
- __field( int, id )
- __array( char, state, MAX_DA_NAME_LEN )
- __array( char, event, MAX_DA_NAME_LEN )
- __array( char, next_state, MAX_DA_NAME_LEN )
- __field( bool, final_state )
+ __field( int, id )
+ __string( state, state )
+ __string( event, event )
+ __string( next_state, next_state )
+ __field( bool, final_state )
),
TP_fast_assign(
- memcpy(__entry->state, state, MAX_DA_NAME_LEN);
- memcpy(__entry->event, event, MAX_DA_NAME_LEN);
- memcpy(__entry->next_state, next_state, MAX_DA_NAME_LEN);
- __entry->id = id;
- __entry->final_state = final_state;
+ __assign_str(state, state);
+ __assign_str(event, event);
+ __assign_str(next_state, next_state);
+ __entry->id = id;
+ __entry->final_state = final_state;
),
TP_printk("%d: %s x %s -> %s %s",
__entry->id,
- __entry->state,
- __entry->event,
- __entry->next_state,
+ __get_str(state),
+ __get_str(event),
+ __get_str(next_state),
__entry->final_state ? "(final)" : "")
);
@@ -106,21 +106,21 @@ DECLARE_EVENT_CLASS(error_da_monitor_id,
TP_ARGS(id, state, event),
TP_STRUCT__entry(
- __field( int, id )
- __array( char, state, MAX_DA_NAME_LEN )
- __array( char, event, MAX_DA_NAME_LEN )
+ __field( int, id )
+ __string( state, state )
+ __string( event, event )
),
TP_fast_assign(
- memcpy(__entry->state, state, MAX_DA_NAME_LEN);
- memcpy(__entry->event, event, MAX_DA_NAME_LEN);
- __entry->id = id;
+ __assign_str(state, state);
+ __assign_str(event, event);
+ __entry->id = id;
),
TP_printk("%d: event %s not expected in the state %s",
__entry->id,
- __entry->event,
- __entry->state)
+ __get_str(event),
+ __get_str(state))
);
#ifdef CONFIG_RV_MON_WWNR
diff --git a/include/uapi/linux/ioprio.h b/include/uapi/linux/ioprio.h
index f70f2596a6bf..99440b2e8c35 100644
--- a/include/uapi/linux/ioprio.h
+++ b/include/uapi/linux/ioprio.h
@@ -2,22 +2,23 @@
#ifndef _UAPI_LINUX_IOPRIO_H
#define _UAPI_LINUX_IOPRIO_H
+#include <linux/stddef.h>
+#include <linux/types.h>
+
/*
* Gives us 8 prio classes with 13-bits of data for each class
*/
#define IOPRIO_CLASS_SHIFT 13
-#define IOPRIO_CLASS_MASK 0x07
+#define IOPRIO_NR_CLASSES 8
+#define IOPRIO_CLASS_MASK (IOPRIO_NR_CLASSES - 1)
#define IOPRIO_PRIO_MASK ((1UL << IOPRIO_CLASS_SHIFT) - 1)
#define IOPRIO_PRIO_CLASS(ioprio) \
(((ioprio) >> IOPRIO_CLASS_SHIFT) & IOPRIO_CLASS_MASK)
#define IOPRIO_PRIO_DATA(ioprio) ((ioprio) & IOPRIO_PRIO_MASK)
-#define IOPRIO_PRIO_VALUE(class, data) \
- ((((class) & IOPRIO_CLASS_MASK) << IOPRIO_CLASS_SHIFT) | \
- ((data) & IOPRIO_PRIO_MASK))
/*
- * These are the io priority groups as implemented by the BFQ and mq-deadline
+ * These are the io priority classes as implemented by the BFQ and mq-deadline
* schedulers. RT is the realtime class, it always gets premium service. For
* ATA disks supporting NCQ IO priority, RT class IOs will be processed using
* high priority NCQ commands. BE is the best-effort scheduling class, the
@@ -25,18 +26,30 @@
* served when no one else is using the disk.
*/
enum {
- IOPRIO_CLASS_NONE,
- IOPRIO_CLASS_RT,
- IOPRIO_CLASS_BE,
- IOPRIO_CLASS_IDLE,
+ IOPRIO_CLASS_NONE = 0,
+ IOPRIO_CLASS_RT = 1,
+ IOPRIO_CLASS_BE = 2,
+ IOPRIO_CLASS_IDLE = 3,
+
+ /* Special class to indicate an invalid ioprio value */
+ IOPRIO_CLASS_INVALID = 7,
};
/*
- * The RT and BE priority classes both support up to 8 priority levels.
+ * The RT and BE priority classes both support up to 8 priority levels that
+ * can be specified using the lower 3-bits of the priority data.
*/
-#define IOPRIO_NR_LEVELS 8
-#define IOPRIO_BE_NR IOPRIO_NR_LEVELS
+#define IOPRIO_LEVEL_NR_BITS 3
+#define IOPRIO_NR_LEVELS (1 << IOPRIO_LEVEL_NR_BITS)
+#define IOPRIO_LEVEL_MASK (IOPRIO_NR_LEVELS - 1)
+#define IOPRIO_PRIO_LEVEL(ioprio) ((ioprio) & IOPRIO_LEVEL_MASK)
+
+#define IOPRIO_BE_NR IOPRIO_NR_LEVELS
+/*
+ * Possible values for the "which" argument of the ioprio_get() and
+ * ioprio_set() system calls (see "man ioprio_set").
+ */
enum {
IOPRIO_WHO_PROCESS = 1,
IOPRIO_WHO_PGRP,
@@ -44,9 +57,70 @@ enum {
};
/*
- * Fallback BE priority level.
+ * Fallback BE class priority level.
*/
#define IOPRIO_NORM 4
#define IOPRIO_BE_NORM IOPRIO_NORM
+/*
+ * The 10 bits between the priority class and the priority level are used to
+ * optionally define I/O hints for any combination of I/O priority class and
+ * level. Depending on the kernel configuration, I/O scheduler being used and
+ * the target I/O device being used, hints can influence how I/Os are processed
+ * without affecting the I/O scheduling ordering defined by the I/O priority
+ * class and level.
+ */
+#define IOPRIO_HINT_SHIFT IOPRIO_LEVEL_NR_BITS
+#define IOPRIO_HINT_NR_BITS 10
+#define IOPRIO_NR_HINTS (1 << IOPRIO_HINT_NR_BITS)
+#define IOPRIO_HINT_MASK (IOPRIO_NR_HINTS - 1)
+#define IOPRIO_PRIO_HINT(ioprio) \
+ (((ioprio) >> IOPRIO_HINT_SHIFT) & IOPRIO_HINT_MASK)
+
+/*
+ * I/O hints.
+ */
+enum {
+ /* No hint */
+ IOPRIO_HINT_NONE = 0,
+
+ /*
+ * Device command duration limits: indicate to the device a desired
+ * duration limit for the commands that will be used to process an I/O.
+ * These will currently only be effective for SCSI and ATA devices that
+ * support the command duration limits feature. If this feature is
+ * enabled, then the commands issued to the device to process an I/O with
+ * one of these hints set will have the duration limit index (dld field)
+ * set to the value of the hint.
+ */
+ IOPRIO_HINT_DEV_DURATION_LIMIT_1 = 1,
+ IOPRIO_HINT_DEV_DURATION_LIMIT_2 = 2,
+ IOPRIO_HINT_DEV_DURATION_LIMIT_3 = 3,
+ IOPRIO_HINT_DEV_DURATION_LIMIT_4 = 4,
+ IOPRIO_HINT_DEV_DURATION_LIMIT_5 = 5,
+ IOPRIO_HINT_DEV_DURATION_LIMIT_6 = 6,
+ IOPRIO_HINT_DEV_DURATION_LIMIT_7 = 7,
+};
+
+#define IOPRIO_BAD_VALUE(val, max) ((val) < 0 || (val) >= (max))
+
+/*
+ * Return an I/O priority value based on a class, a level and a hint.
+ */
+static __always_inline __u16 ioprio_value(int class, int level, int hint)
+{
+ if (IOPRIO_BAD_VALUE(class, IOPRIO_NR_CLASSES) ||
+ IOPRIO_BAD_VALUE(level, IOPRIO_NR_LEVELS) ||
+ IOPRIO_BAD_VALUE(hint, IOPRIO_NR_HINTS))
+ return IOPRIO_CLASS_INVALID << IOPRIO_CLASS_SHIFT;
+
+ return (class << IOPRIO_CLASS_SHIFT) |
+ (hint << IOPRIO_HINT_SHIFT) | level;
+}
+
+#define IOPRIO_PRIO_VALUE(class, level) \
+ ioprio_value(class, level, IOPRIO_HINT_NONE)
+#define IOPRIO_PRIO_VALUE_HINT(class, level, hint) \
+ ioprio_value(class, level, hint)
+
#endif /* _UAPI_LINUX_IOPRIO_H */
diff --git a/include/uapi/linux/netfilter/nf_conntrack_common.h b/include/uapi/linux/netfilter/nf_conntrack_common.h
index 56b6b60a814f..ee51045ae1d6 100644
--- a/include/uapi/linux/netfilter/nf_conntrack_common.h
+++ b/include/uapi/linux/netfilter/nf_conntrack_common.h
@@ -160,6 +160,7 @@ enum ip_conntrack_expect_events {
#define NF_CT_EXPECT_USERSPACE 0x4
#ifdef __KERNEL__
+#define NF_CT_EXPECT_DEAD 0x8
#define NF_CT_EXPECT_MASK (NF_CT_EXPECT_PERMANENT | NF_CT_EXPECT_INACTIVE | \
NF_CT_EXPECT_USERSPACE)
#endif
diff --git a/include/uapi/linux/nexthop.h b/include/uapi/linux/nexthop.h
index d8ffa8c9ca78..6b0de563beec 100644
--- a/include/uapi/linux/nexthop.h
+++ b/include/uapi/linux/nexthop.h
@@ -16,10 +16,15 @@ struct nhmsg {
struct nexthop_grp {
__u32 id; /* nexthop id - must exist */
__u8 weight; /* weight of this nexthop */
- __u8 resvd1;
+ __u8 weight_high; /* high order bits of weight */
__u16 resvd2;
};
+static inline __u16 nexthop_grp_weight(const struct nexthop_grp *entry)
+{
+ return ((entry->weight_high << 8) | entry->weight) + 1;
+}
+
enum {
NEXTHOP_GRP_TYPE_MPATH, /* hash-threshold nexthop group
* default type if not specified
diff --git a/include/uapi/linux/stddef.h b/include/uapi/linux/stddef.h
index ef892cb1cbb7..f039ec8e5455 100644
--- a/include/uapi/linux/stddef.h
+++ b/include/uapi/linux/stddef.h
@@ -51,4 +51,9 @@
struct { } __empty_ ## NAME; \
TYPE NAME[]; \
}
+
+#ifndef __counted_by
+#define __counted_by(m)
#endif
+
+#endif /* _UAPI_LINUX_STDDEF_H */
diff --git a/include/uapi/linux/virtio_crypto.h b/include/uapi/linux/virtio_crypto.h
index 71a54a6849ca..2fccb64c9d6b 100644
--- a/include/uapi/linux/virtio_crypto.h
+++ b/include/uapi/linux/virtio_crypto.h
@@ -329,6 +329,7 @@ struct virtio_crypto_op_header {
VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AKCIPHER, 0x00)
#define VIRTIO_CRYPTO_AKCIPHER_DECRYPT \
VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AKCIPHER, 0x01)
+ /* akcipher sign/verify opcodes are deprecated */
#define VIRTIO_CRYPTO_AKCIPHER_SIGN \
VIRTIO_CRYPTO_OPCODE(VIRTIO_CRYPTO_SERVICE_AKCIPHER, 0x02)
#define VIRTIO_CRYPTO_AKCIPHER_VERIFY \
diff --git a/include/xen/events.h b/include/xen/events.h
index b303bd24e2a6..07184beaa0a5 100644
--- a/include/xen/events.h
+++ b/include/xen/events.h
@@ -102,8 +102,8 @@ void xen_poll_irq_timeout(int irq, u64 timeout);
/* Determine the IRQ which is bound to an event channel */
unsigned int irq_from_evtchn(evtchn_port_t evtchn);
-int irq_from_virq(unsigned int cpu, unsigned int virq);
-evtchn_port_t evtchn_from_irq(unsigned irq);
+int irq_evtchn_from_virq(unsigned int cpu, unsigned int virq,
+ evtchn_port_t *evtchn);
int xen_set_callback_via(uint64_t via);
int xen_evtchn_do_upcall(void);
diff --git a/io_uring/cancel.c b/io_uring/cancel.c
index bfaf8f9f1def..91ed58712b43 100644
--- a/io_uring/cancel.c
+++ b/io_uring/cancel.c
@@ -69,7 +69,16 @@ static bool io_cancel_cb(struct io_wq_work *work, void *data)
struct io_kiocb *req = container_of(work, struct io_kiocb, work);
struct io_cancel_data *cd = data;
- return io_cancel_req_match(req, cd);
+ if (!io_cancel_req_match(req, cd))
+ return false;
+ /*
+ * Pending work is gone once it has matched. Work that is running may
+ * park the request where the same cancellation looks next. Don't make
+ * it skip the request as one that it has dealt with.
+ */
+ if (cd->flags & (IORING_ASYNC_CANCEL_ALL | IORING_ASYNC_CANCEL_ANY))
+ req->work.cancel_seq = cd->seq - 1;
+ return true;
}
static int io_async_cancel_one(struct io_uring_task *tctx,
@@ -114,6 +123,8 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd,
*/
if (!ret)
return 0;
+ if (ret == -EALREADY)
+ cd->running = true;
ret = io_poll_cancel(ctx, cd, issue_flags);
if (ret != -ENOENT)
@@ -162,6 +173,7 @@ static int __io_async_cancel(struct io_cancel_data *cd,
struct io_tctx_node *node;
int ret, nr = 0;
+ cd->running = false;
do {
ret = io_try_cancel(tctx, cd, issue_flags);
if (ret == -ENOENT)
@@ -175,9 +187,10 @@ static int __io_async_cancel(struct io_cancel_data *cd,
io_ring_submit_lock(ctx, issue_flags);
ret = -ENOENT;
list_for_each_entry(node, &ctx->tctx_list, ctx_node) {
- struct io_uring_task *tctx = node->task->io_uring;
-
- ret = io_async_cancel_one(tctx, cd);
+ /* has been done above, don't count what is running there */
+ if (all && node->task->io_uring == tctx)
+ continue;
+ ret = io_async_cancel_one(node->task->io_uring, cd);
if (ret != -ENOENT) {
if (!all)
break;
@@ -185,7 +198,15 @@ static int __io_async_cancel(struct io_cancel_data *cd,
}
}
io_ring_submit_unlock(ctx, issue_flags);
- return all ? nr : ret;
+ if (all)
+ return nr;
+ /*
+ * The worker may be done with the work that was found running, and
+ * flagged, above. That isn't the same as not having found it.
+ */
+ if (ret == -ENOENT && cd->running)
+ ret = -EALREADY;
+ return ret;
}
int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags)
@@ -295,7 +316,7 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg)
if (ret != -EALREADY)
goto out;
- if (sc.timeout.tv_sec != -1UL || sc.timeout.tv_nsec != -1UL) {
+ if (sc.timeout.tv_sec != -1 || sc.timeout.tv_nsec != -1) {
struct timespec64 ts = {
.tv_sec = sc.timeout.tv_sec,
.tv_nsec = sc.timeout.tv_nsec
diff --git a/io_uring/cancel.h b/io_uring/cancel.h
index 7e6d0fca7db2..d6433c5cba4d 100644
--- a/io_uring/cancel.h
+++ b/io_uring/cancel.h
@@ -9,6 +9,8 @@ struct io_cancel_data {
u8 opcode;
u32 flags;
int seq;
+ /* io_try_cancel() has found it being run by io-wq, and flagged it */
+ bool running;
};
int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe);
diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c
index f1bc212309c8..b4a102bcbe30 100644
--- a/io_uring/io-wq.c
+++ b/io_uring/io-wq.c
@@ -53,6 +53,8 @@ struct io_worker {
struct io_wq_work *cur_work;
struct io_wq_work *next_work;
+ /* rest of the hashed chain ->next_work or ->cur_work came with */
+ struct io_wq_work *chain_work;
raw_spinlock_t lock;
struct completion ref_done;
@@ -118,6 +120,7 @@ struct io_wqe {
struct io_wq {
unsigned long state;
+ put_work_fn *put_work;
free_work_fn *free_work;
io_wq_work_fn *do_work;
@@ -547,6 +550,9 @@ static void io_assign_current_work(struct io_worker *worker,
raw_spin_lock(&worker->lock);
worker->cur_work = work;
worker->next_work = NULL;
+ /* stays valid while in between two items of a chain */
+ if (work)
+ worker->chain_work = wq_next_work(work);
raw_spin_unlock(&worker->lock);
}
@@ -583,6 +589,7 @@ static void io_worker_handle_work(struct io_worker *worker)
*/
raw_spin_lock(&worker->lock);
worker->next_work = work;
+ worker->chain_work = wq_next_work(work);
raw_spin_unlock(&worker->lock);
} else {
break;
@@ -593,17 +600,25 @@ static void io_worker_handle_work(struct io_worker *worker)
/* handle a whole dependent link */
do {
bool do_kill = test_bit(IO_WQ_BIT_EXIT, &wq->state);
- struct io_wq_work *next_hashed, *linked;
+ struct io_wq_work *next_hashed, *linked, *done = work;
unsigned int hash = io_get_work_hash(work);
+ bool last;
next_hashed = wq_next_work(work);
+ /*
+ * io_wq_exit_workers() signals us once, so that we
+ * don't sleep for ever. If that was before this work
+ * became the current one, the signal is gone.
+ */
if (unlikely(do_kill) && (work->flags & IO_WQ_WORK_UNBOUND))
work->flags |= IO_WQ_WORK_CANCEL;
+ else if (unlikely(do_kill))
+ set_notify_signal(current);
wq->do_work(work);
io_assign_current_work(worker, NULL);
- linked = wq->free_work(work);
+ last = wq->put_work(work, &linked);
work = next_hashed;
if (!work && linked && !io_wq_is_hashed(linked)) {
work = linked;
@@ -612,6 +627,13 @@ static void io_worker_handle_work(struct io_worker *worker)
io_assign_current_work(worker, work);
if (linked)
io_wqe_enqueue(wqe, linked);
+ /*
+ * Not before the link is current or pending work again.
+ * This may be the last time that a task is woken up
+ * which exits and looks for requests to cancel.
+ */
+ if (last)
+ wq->free_work(done);
if (hash != -1U && !next_hashed) {
/* serialize hash clear with wake_up() */
@@ -626,6 +648,36 @@ static void io_worker_handle_work(struct io_worker *worker)
} while (1);
}
+/*
+ * The worker has decided to exit, and has given up its ->nr_workers slot and
+ * its place on the free list. io_wqe_enqueue() may have picked it from that
+ * list, or seen it running, after it last looked at the work list. Leave
+ * ->nr_running as well and check again. Returns false if the worker has to
+ * stay.
+ */
+static bool io_worker_retire(struct io_wqe *wqe, struct io_worker *worker)
+{
+ struct io_wqe_acct *acct = io_wqe_get_acct(worker);
+
+ /* turns io_wqe_dec_running() and the scheduler hooks into no-ops */
+ worker->flags &= ~IO_WORKER_F_UP;
+ atomic_dec(&acct->nr_running);
+ if (!io_acct_run_queue(acct))
+ return true;
+
+ raw_spin_lock(&wqe->lock);
+ if (acct->nr_workers >= acct->max_workers) {
+ /* a worker was created in our place, the work is its business */
+ raw_spin_unlock(&wqe->lock);
+ return true;
+ }
+ acct->nr_workers++;
+ raw_spin_unlock(&wqe->lock);
+ atomic_inc(&acct->nr_running);
+ worker->flags |= IO_WORKER_F_UP;
+ return false;
+}
+
static int io_wqe_worker(void *data)
{
struct io_worker *worker = data;
@@ -651,12 +703,23 @@ static int io_wqe_worker(void *data)
/*
* Last sleep timed out. Exit if we're not the last worker,
* or if someone modified our affinity.
+ * Not as long as a request of ours for another worker is
+ * pending, though. It is accounted as a running worker, which
+ * others rely on, and io_worker_exit() would cancel it.
*/
- if (last_timeout && (exit_mask || acct->nr_workers > 1)) {
+ if (last_timeout && (exit_mask || acct->nr_workers > 1) &&
+ !test_bit(0, &worker->create_state)) {
acct->nr_workers--;
+ if (worker->flags & IO_WORKER_F_FREE) {
+ worker->flags &= ~IO_WORKER_F_FREE;
+ hlist_nulls_del_init_rcu(&worker->nulls_node);
+ }
raw_spin_unlock(&wqe->lock);
__set_current_state(TASK_RUNNING);
- break;
+ if (io_worker_retire(wqe, worker))
+ break;
+ last_timeout = false;
+ continue;
}
last_timeout = false;
__io_worker_idle(wqe, worker);
@@ -892,9 +955,13 @@ static void io_run_cancel(struct io_wq_work *work, struct io_wqe *wqe)
struct io_wq *wq = wqe->wq;
do {
+ struct io_wq_work *linked;
+
work->flags |= IO_WQ_WORK_CANCEL;
wq->do_work(work);
- work = wq->free_work(work);
+ if (wq->put_work(work, &linked))
+ wq->free_work(work);
+ work = linked;
} while (work);
}
@@ -1012,6 +1079,7 @@ static bool __io_wq_worker_cancel(struct io_worker *worker,
static bool io_wq_worker_cancel(struct io_worker *worker, void *data)
{
struct io_cb_cancel_data *match = data;
+ struct io_wq_work *work;
/*
* Hold the lock to avoid ->cur_work going out of scope, caller
@@ -1021,6 +1089,19 @@ static bool io_wq_worker_cancel(struct io_worker *worker, void *data)
if (__io_wq_worker_cancel(worker, match, worker->cur_work) ||
__io_wq_worker_cancel(worker, match, worker->next_work))
match->nr_running++;
+ /*
+ * Hashed work is taken off the work list a chain at a time, what the
+ * worker hasn't started yet can only be found here. It doesn't move on
+ * without the lock. Just mark the work, it fails once it is its turn.
+ */
+ for (work = worker->chain_work; work; work = wq_next_work(work)) {
+ if (match->nr_running && !match->cancel_all)
+ break;
+ if (match->fn(work, match->data)) {
+ work->flags |= IO_WQ_WORK_CANCEL;
+ match->nr_running++;
+ }
+ }
raw_spin_unlock(&worker->lock);
return match->nr_running && !match->cancel_all;
@@ -1163,7 +1244,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
struct io_wq *wq;
cpumask_var_t allowed_mask;
- if (WARN_ON_ONCE(!data->free_work || !data->do_work))
+ if (WARN_ON_ONCE(!data->put_work || !data->free_work || !data->do_work))
return ERR_PTR(-EINVAL);
if (WARN_ON_ONCE(!bounded))
return ERR_PTR(-EINVAL);
@@ -1177,6 +1258,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data)
refcount_inc(&data->hash->refs);
wq->hash = data->hash;
+ wq->put_work = data->put_work;
wq->free_work = data->free_work;
wq->do_work = data->do_work;
@@ -1251,6 +1333,8 @@ static bool io_task_work_match(struct callback_head *cb, void *data)
void io_wq_exit_start(struct io_wq *wq)
{
set_bit(IO_WQ_BIT_EXIT, &wq->state);
+ /* Pairs with task_work_add() in io_queue_worker_create(). */
+ smp_mb__after_atomic();
}
static void io_wq_cancel_tw_create(struct io_wq *wq)
diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h
index 2b2a6406dd8e..8bd6645e7463 100644
--- a/io_uring/io-wq.h
+++ b/io_uring/io-wq.h
@@ -21,7 +21,8 @@ enum io_wq_cancel {
IO_WQ_CANCEL_NOTFOUND, /* work not found */
};
-typedef struct io_wq_work *(free_work_fn)(struct io_wq_work *);
+typedef bool (put_work_fn)(struct io_wq_work *, struct io_wq_work **);
+typedef void (free_work_fn)(struct io_wq_work *);
typedef void (io_wq_work_fn)(struct io_wq_work *);
struct io_wq_hash {
@@ -40,6 +41,7 @@ struct io_wq_data {
struct io_wq_hash *hash;
struct task_struct *task;
io_wq_work_fn *do_work;
+ put_work_fn *put_work;
free_work_fn *free_work;
};
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index fdb8afdb0135..655674be3859 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -417,17 +417,45 @@ static void io_prep_async_link(struct io_kiocb *req)
}
}
+/* Called for every request that is issued: keep it inline */
+static __always_inline bool io_assign_file(struct io_kiocb *req,
+ unsigned int issue_flags)
+{
+ if (req->file || !io_op_defs[req->opcode].needs_file)
+ return true;
+
+ if (req->flags & REQ_F_FIXED_FILE)
+ req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags);
+ else
+ req->file = io_file_get_normal(req, req->cqe.fd);
+
+ return !!req->file;
+}
+
static void io_queue_iowq(struct io_kiocb *req)
{
struct io_uring_task *tctx = req->task->io_uring;
BUG_ON(!tctx);
+ lockdep_assert_held(&req->ctx->uring_lock);
if ((current->flags & PF_KTHREAD) || !tctx->io_wq) {
io_req_task_queue_fail(req, -ECANCELED);
return;
}
+ /*
+ * The file is looked up as late as possible, as it may be the result
+ * of an earlier request of the link. This one is up now, and what kind
+ * of work it is depends on the file. If that fails, leave the error
+ * handling to the worker.
+ *
+ * Not for a splice: whatever it writes to, it may wait for its input
+ * for ever. It has to remain unbound work that isn't hashed.
+ */
+ if (req->opcode != IORING_OP_SPLICE)
+ io_assign_file(req, 0);
+
/* init ->work of the whole link before punting */
io_prep_async_link(req);
@@ -1348,22 +1376,6 @@ static void __io_submit_flush_completions(struct io_ring_ctx *ctx)
INIT_WQ_LIST(&state->compl_reqs);
}
-/*
- * Drop reference to request, return next in chain (if there is one) if this
- * was the last reference to this request.
- */
-static inline struct io_kiocb *io_put_req_find_next(struct io_kiocb *req)
-{
- struct io_kiocb *nxt = NULL;
-
- if (req_ref_put_and_test(req)) {
- if (unlikely(req->flags & IO_REQ_LINK_FLAGS))
- nxt = io_req_find_next(req);
- io_free_req(req);
- }
- return nxt;
-}
-
static unsigned io_cqring_events(struct io_ring_ctx *ctx)
{
/* See comment at the top of this file */
@@ -1709,19 +1721,6 @@ static void io_clean_op(struct io_kiocb *req)
req->flags &= ~IO_REQ_CLEAN_FLAGS;
}
-static bool io_assign_file(struct io_kiocb *req, unsigned int issue_flags)
-{
- if (req->file || !io_op_defs[req->opcode].needs_file)
- return true;
-
- if (req->flags & REQ_F_FIXED_FILE)
- req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags);
- else
- req->file = io_file_get_normal(req, req->cqe.fd);
-
- return !!req->file;
-}
-
#define REQ_ISSUE_SLOW_FLAGS (REQ_F_CREDS | REQ_F_ARM_LTIMEOUT)
static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
@@ -1757,12 +1756,18 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags)
}
if (ret == IOU_OK) {
+ /* not all handlers take care of the internal restart codes */
+ if (unlikely(req->cqe.res <= -ERESTARTSYS))
+ req->cqe.res = io_fixup_restart_res(req->cqe.res);
if (issue_flags & IO_URING_F_COMPLETE_DEFER)
io_req_complete_defer(req);
else
io_req_complete_post(req);
- } else if (ret != IOU_ISSUE_SKIP_COMPLETE)
+ } else if (ret != IOU_ISSUE_SKIP_COMPLETE) {
+ if (unlikely(ret <= -ERESTARTSYS))
+ ret = io_fixup_restart_res(ret);
return ret;
+ }
/* If the op doesn't have a file, we're not polling for it */
if ((req->ctx->flags & IORING_SETUP_IOPOLL) && def->iopoll_queue)
@@ -1781,23 +1786,41 @@ int io_poll_issue(struct io_kiocb *req, bool *locked)
return io_issue_sqe(req, IO_URING_F_NONBLOCK|IO_URING_F_MULTISHOT);
}
-struct io_wq_work *io_wq_free_work(struct io_wq_work *work)
+/*
+ * io-wq is done with @work. Returns true if it held the last reference, and
+ * hands out the next request of the link then. The caller has to make sure
+ * that cancellations can find the latter before it calls io_wq_free_work().
+ */
+bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked)
{
struct io_kiocb *req = container_of(work, struct io_kiocb, work);
+ struct io_kiocb *nxt = NULL;
- req = io_put_req_find_next(req);
- return req ? &req->work : NULL;
+ *linked = NULL;
+ if (!req_ref_put_and_test(req))
+ return false;
+
+ if (unlikely(req->flags & IO_REQ_LINK_FLAGS))
+ nxt = io_req_find_next(req);
+ if (nxt)
+ *linked = &nxt->work;
+ return true;
+}
+
+void io_wq_free_work(struct io_wq_work *work)
+{
+ io_free_req(container_of(work, struct io_kiocb, work));
}
void io_wq_submit_work(struct io_wq_work *work)
{
struct io_kiocb *req = container_of(work, struct io_kiocb, work);
const struct io_op_def *def = &io_op_defs[req->opcode];
- unsigned int issue_flags = IO_URING_F_UNLOCKED;
+ unsigned int issue_flags = IO_URING_F_UNLOCKED | IO_URING_F_IOWQ;
bool needs_poll = false;
int ret = 0, err = -ECANCELED;
- /* one will be dropped by ->io_free_work() after returning to io-wq */
+ /* one will be dropped by ->put_work() after returning to io-wq */
if (!(req->flags & REQ_F_REFCOUNT))
__io_req_set_refcount(req, 2);
else
@@ -2366,7 +2389,8 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode,
*/
if (io_should_wake(iowq) || io_has_work(ctx))
return autoremove_wake_function(curr, mode, wake_flags, key);
- return -1;
+ /* keep going, a waiter further down may be content with less */
+ return 0;
}
int io_run_task_work_sig(struct io_ring_ctx *ctx)
@@ -2827,6 +2851,7 @@ struct io_tctx_exit {
struct callback_head task_work;
struct completion completion;
struct io_ring_ctx *ctx;
+ bool in_cancel;
};
static __cold void io_tctx_exit_cb(struct callback_head *cb)
@@ -2841,7 +2866,8 @@ static __cold void io_tctx_exit_cb(struct callback_head *cb)
* tctx can be NULL if the queueing of this task_work raced with
* work cancelation off the exec path.
*/
- if (tctx && !atomic_read(&tctx->in_idle))
+ work->in_cancel = tctx && atomic_read(&tctx->in_idle);
+ if (tctx && !work->in_cancel)
io_uring_del_tctx_node((unsigned long)work->ctx);
complete(&work->completion);
}
@@ -2929,6 +2955,14 @@ static __cold void io_ring_exit_work(struct work_struct *work)
* wait is marked as interruptible.
*/
wait_for_completion_interruptible(&exit.completion);
+ /*
+ * The task removes the node when it is done with its
+ * cancellation. Give it the time for that: it doesn't get back
+ * to it as long as it finds new task_work each time it has run
+ * the old one, and we may be one of many who send it.
+ */
+ if (exit.in_cancel)
+ schedule_timeout_idle(HZ / 20);
mutex_lock(&ctx->uring_lock);
}
mutex_unlock(&ctx->uring_lock);
diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h
index 194e3230f853..640015da944c 100644
--- a/io_uring/io_uring.h
+++ b/io_uring/io_uring.h
@@ -70,7 +70,8 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin);
void io_free_batch_list(struct io_ring_ctx *ctx, struct io_wq_work_node *node);
int io_req_prep_async(struct io_kiocb *req);
-struct io_wq_work *io_wq_free_work(struct io_wq_work *work);
+bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked);
+void io_wq_free_work(struct io_wq_work *work);
void io_wq_submit_work(struct io_wq_work *work);
void io_free_req(struct io_kiocb *req);
@@ -154,6 +155,24 @@ static inline bool __io_fill_cqe_req(struct io_ring_ctx *ctx,
return true;
}
+static inline ssize_t io_fixup_restart_res(ssize_t ret)
+{
+ switch (ret) {
+ case -ERESTARTSYS:
+ case -ERESTARTNOINTR:
+ case -ERESTARTNOHAND:
+ case -ERESTART_RESTARTBLOCK:
+ /*
+ * We can't just restart the syscall, since previously
+ * submitted sqes may already be in progress. Just fail
+ * this IO with EINTR.
+ */
+ return -EINTR;
+ default:
+ return ret;
+ }
+}
+
static inline void req_set_fail(struct io_kiocb *req)
{
req->flags |= REQ_F_FAIL;
@@ -202,6 +221,29 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx,
lockdep_assert_held(&ctx->uring_lock);
}
+/*
+ * For a handler that is run by io-wq and parks the request on a list where
+ * cancellations find it, rather than completing it. Whoever tried to cancel
+ * the request before that found it running. It has flagged the work item,
+ * has searched the list in vain, and won't come back. To be called with
+ * the lock the search takes held: if the search came first, the flag is
+ * visible. For most lists that is ->uring_lock, timeouts use ->timeout_lock.
+ */
+static inline bool __io_wq_cancel_seen(struct io_kiocb *req,
+ unsigned int issue_flags)
+{
+ return (issue_flags & IO_URING_F_IOWQ) &&
+ (req->work.flags & IO_WQ_WORK_CANCEL);
+}
+
+static inline bool io_wq_cancel_seen(struct io_kiocb *req,
+ unsigned int issue_flags)
+{
+ lockdep_assert_held(&req->ctx->uring_lock);
+
+ return __io_wq_cancel_seen(req, issue_flags);
+}
+
static inline void io_commit_cqring(struct io_ring_ctx *ctx)
{
/* order cqe stores with ring update */
diff --git a/io_uring/poll.c b/io_uring/poll.c
index cdf1f2f57b9a..904a5a0bae2e 100644
--- a/io_uring/poll.c
+++ b/io_uring/poll.c
@@ -239,12 +239,14 @@ enum {
static int io_poll_check_events(struct io_kiocb *req, bool *locked)
{
struct io_ring_ctx *ctx = req->ctx;
+ bool again;
int v;
if (unlikely(io_should_terminate_tw(ctx)))
return -ECANCELED;
do {
+ again = false;
v = atomic_read(&req->poll_refs);
/* tw handler should be the owner, and so have some references */
@@ -305,10 +307,17 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked)
} else {
int ret;
- /* multiple refs and HUP, ensure we loop once more */
+ /*
+ * Multiple refs and HUP, ensure we loop once more. Drop
+ * down to one ref now, so the next pass ends unless a
+ * wake up came in.
+ */
if ((req->cqe.res & (POLLHUP | POLLRDHUP)) &&
- (v & IO_POLL_REF_MASK) != 1)
- v--;
+ (v & IO_POLL_REF_MASK) != 1) {
+ atomic_sub((v & IO_POLL_REF_MASK) - 1,
+ &req->poll_refs);
+ again = true;
+ }
ret = io_poll_issue(req, locked);
io_kbuf_recycle(req, 0);
@@ -325,8 +334,8 @@ static int io_poll_check_events(struct io_kiocb *req, bool *locked)
* Release all references, retry if someone tried to restart
* task_work while we were executing it.
*/
- } while (atomic_sub_return(v & IO_POLL_REF_MASK, &req->poll_refs) &
- IO_POLL_REF_MASK);
+ v &= IO_POLL_REF_MASK;
+ } while (again || atomic_cmpxchg(&req->poll_refs, v, 0) != v);
return IOU_POLL_NO_ACTION;
}
@@ -652,6 +661,17 @@ static int __io_arm_poll_handler(struct io_kiocb *req,
io_poll_add_hash(req);
+ /*
+ * Whoever tried to cancel the request while io-wq was running it has
+ * flagged the work item. If it searched the hash table before the
+ * request got there it found nothing, and it won't come back. Both
+ * the search and the insertion are done with the hash bucket lock
+ * held, and the flag is set before the search. If the search came
+ * first, the flag is visible by now.
+ */
+ if (ipt->owning && (req->work.flags & IO_WQ_WORK_CANCEL))
+ io_poll_mark_cancelled(req);
+
if (mask && (poll->events & EPOLLET) &&
io_poll_can_finish_inline(req, ipt)) {
__io_poll_execute(req, mask);
@@ -988,6 +1008,18 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags)
if (req->ctx->flags & (IORING_SETUP_SQPOLL|IORING_SETUP_SINGLE_ISSUER))
req->flags |= REQ_F_HASH_LOCKED;
+ /*
+ * A poll update or a spurious wake up arms the request again. Forget
+ * the entries of the previous round like io_arm_apoll() does, a stale
+ * second entry would be taken for a third wait queue.
+ */
+ if (req_has_async_data(req)) {
+ kfree(req->async_data);
+ req->async_data = NULL;
+ req->flags &= ~REQ_F_ASYNC_DATA;
+ }
+ req->flags &= ~(REQ_F_SINGLE_POLL | REQ_F_DOUBLE_POLL);
+
ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags);
if (ret > 0) {
io_req_set_res(req, ipt.result_mask, 0);
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index ac658cfa89c6..1c5abe2dcd5e 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -983,7 +983,12 @@ struct page **io_pin_pages(unsigned long ubuf, unsigned long len, int *npages)
struct page **pages = NULL;
int i, pret, ret = -ENOMEM;
- end = (ubuf + len + PAGE_SIZE - 1) >> PAGE_SHIFT;
+ if (check_add_overflow(ubuf, len, &end))
+ return ERR_PTR(-EOVERFLOW);
+ if (check_add_overflow(end, PAGE_SIZE - 1, &end))
+ return ERR_PTR(-EOVERFLOW);
+
+ end = end >> PAGE_SHIFT;
start = ubuf >> PAGE_SHIFT;
nr_pages = end - start;
diff --git a/io_uring/rw.c b/io_uring/rw.c
index cbd3ee9a9373..8be0d4fd33ea 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -129,24 +129,6 @@ void io_readv_writev_cleanup(struct io_kiocb *req)
kfree(io->free_iovec);
}
-static inline ssize_t io_fixup_restart_res(ssize_t ret)
-{
- switch (ret) {
- case -ERESTARTSYS:
- case -ERESTARTNOINTR:
- case -ERESTARTNOHAND:
- case -ERESTART_RESTARTBLOCK:
- /*
- * We can't just restart the syscall, since previously
- * submitted sqes may already be in progress. Just fail
- * this IO with EINTR.
- */
- return -EINTR;
- default:
- return ret;
- }
-}
-
static inline void io_rw_done(struct kiocb *kiocb, ssize_t ret)
{
/* IO was queued async, completion will happen later */
diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c
index 0d66bef60285..adc01f65464b 100644
--- a/io_uring/sqpoll.c
+++ b/io_uring/sqpoll.c
@@ -293,6 +293,13 @@ static int io_sq_thread(void *data)
*/
smp_mb__after_atomic();
+ /*
+ * Nothing gets submitted before the ring is
+ * enabled, which comes with a wake up.
+ */
+ if (READ_ONCE(ctx->flags) & IORING_SETUP_R_DISABLED)
+ continue;
+
if (io_sqring_entries(ctx)) {
needs_sched = false;
break;
diff --git a/io_uring/tctx.c b/io_uring/tctx.c
index 51b05fda8e45..e4ec8c0411ab 100644
--- a/io_uring/tctx.c
+++ b/io_uring/tctx.c
@@ -35,6 +35,7 @@ static struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx,
data.hash = hash;
data.task = task;
+ data.put_work = io_wq_put_work;
data.free_work = io_wq_free_work;
data.do_work = io_wq_submit_work;
diff --git a/io_uring/timeout.c b/io_uring/timeout.c
index 49cf800eff37..6238187e7966 100644
--- a/io_uring/timeout.c
+++ b/io_uring/timeout.c
@@ -78,6 +78,14 @@ __cold void io_flush_timeouts(struct io_ring_ctx *ctx)
spin_lock_irq(&ctx->timeout_lock);
seq = ctx->cached_cq_tail - atomic_read(&ctx->cq_timeouts);
+ /*
+ * A fired timeout bumps ->cq_timeouts before its CQE is posted, which
+ * leaves the sequence behind the last flush until it is. No event has
+ * happened then, see io_timeout().
+ */
+ if ((s32)(seq - ctx->cq_last_tm_flush) < 0)
+ seq = ctx->cq_last_tm_flush;
+
list_for_each_entry_safe(timeout, tmp, &ctx->timeout_list, list) {
struct io_kiocb *req = cmd_to_io_kiocb(timeout);
u32 events_needed, events_got;
@@ -534,11 +542,19 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout);
struct io_ring_ctx *ctx = req->ctx;
struct io_timeout_data *data = req->async_data;
+ struct io_timeout *first;
struct list_head *entry;
- u32 tail, off = timeout->off;
+ u32 tail, base, off = timeout->off;
spin_lock_irq(&ctx->timeout_lock);
+ /* the list is searched under ->timeout_lock, see io_wq_cancel_seen() */
+ if (__io_wq_cancel_seen(req, issue_flags)) {
+ spin_unlock_irq(&ctx->timeout_lock);
+ io_req_task_queue_fail(req, -ECANCELED);
+ return IOU_ISSUE_SKIP_COMPLETE;
+ }
+
/*
* sqe->off holds how many events that need to occur for this
* timeout event to be satisfied. If it isn't set, then this is
@@ -550,13 +566,27 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
}
tail = data_race(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts);
- timeout->target_seq = tail + off;
- /* Update the last seq here in case io_flush_timeouts() hasn't.
- * This is safe because ->completion_lock is held, and submissions
- * and completions are never mixed in the same ->completion_lock section.
+ /*
+ * Update the last seq here in case io_flush_timeouts() hasn't, it only
+ * gets called once sequences are in use. Leave it alone if there are
+ * timeouts waiting for one. CQEs may have been posted since the last
+ * flush, and a target that has been reached must not fall behind it.
+ */
+ first = list_first_entry_or_null(&ctx->timeout_list, struct io_timeout,
+ list);
+ if (!first || io_is_timeout_noseq(cmd_to_io_kiocb(first)))
+ ctx->cq_last_tm_flush = tail;
+ base = ctx->cq_last_tm_flush;
+
+ /*
+ * A fired timeout bumps ->cq_timeouts before its CQE is posted, which
+ * leaves the sequence behind the last flush until it is. Nothing else
+ * separates the two then, so count from the flush.
*/
- ctx->cq_last_tm_flush = tail;
+ if ((s32)(tail - base) < 0)
+ tail = base;
+ timeout->target_seq = tail + off;
/*
* Insertion sort, ensuring the first entry in the list is always
@@ -568,8 +598,8 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags)
if (io_is_timeout_noseq(nxt))
continue;
- /* nxt.seq is behind @tail, otherwise would've been completed */
- if (off >= nextt->target_seq - tail)
+ /* no target is behind the last flush, see io_flush_timeouts() */
+ if (timeout->target_seq - base >= nextt->target_seq - base)
break;
}
add:
diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c
index 17125ac3a573..e87c11345b7d 100644
--- a/kernel/audit_tree.c
+++ b/kernel/audit_tree.c
@@ -543,22 +543,38 @@ static void kill_rules(struct audit_context *context, struct audit_tree *tree)
{
struct audit_krule *rule, *next;
struct audit_entry *entry;
+ bool need_sync = false;
list_for_each_entry_safe(rule, next, &tree->rules, rlist) {
entry = container_of(rule, struct audit_entry, rule);
- list_del_init(&rule->rlist);
if (rule->tree) {
/* not a half-baked one */
audit_tree_log_remove_rule(context, rule);
- if (entry->rule.exe)
- audit_remove_mark(entry->rule.exe);
rule->tree = NULL;
list_del_rcu(&entry->list);
list_del(&entry->rule.list);
- call_rcu(&entry->rcu, audit_free_rule_rcu);
+ if (entry->rule.exe)
+ need_sync = true;
+ } else {
+ list_del_init(&rule->rlist);
}
}
+
+ if (list_empty(&tree->rules))
+ return;
+
+ if (need_sync)
+ synchronize_rcu();
+
+ list_for_each_entry_safe(rule, next, &tree->rules, rlist) {
+ entry = container_of(rule, struct audit_entry, rule);
+
+ list_del_init(&rule->rlist);
+ if (entry->rule.exe)
+ audit_remove_mark(entry->rule.exe);
+ call_rcu(&entry->rcu, audit_free_rule_rcu);
+ }
}
/*
diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c
index 50b9bf57a4e3..6fc07a983dd0 100644
--- a/kernel/bpf/arraymap.c
+++ b/kernel/bpf/arraymap.c
@@ -889,22 +889,44 @@ static void *prog_fd_array_get_ptr(struct bpf_map *map,
struct file *map_file, int fd)
{
struct bpf_prog *prog = bpf_prog_get(fd);
+ bool is_extended;
if (IS_ERR(prog))
return prog;
- if (!bpf_prog_map_compatible(map, prog)) {
+ if (prog->type == BPF_PROG_TYPE_EXT ||
+ !bpf_prog_map_compatible(map, prog)) {
bpf_prog_put(prog);
return ERR_PTR(-EINVAL);
}
+ mutex_lock(&prog->aux->ext_mutex);
+ is_extended = prog->aux->is_extended;
+ if (!is_extended)
+ prog->aux->prog_array_member_cnt++;
+ mutex_unlock(&prog->aux->ext_mutex);
+ if (is_extended) {
+ /* Extended prog can not be tail callee. It's to prevent a
+ * potential infinite loop like:
+ * tail callee prog entry -> tail callee prog subprog ->
+ * freplace prog entry --tailcall-> tail callee prog entry.
+ */
+ bpf_prog_put(prog);
+ return ERR_PTR(-EBUSY);
+ }
+
return prog;
}
static void prog_fd_array_put_ptr(struct bpf_map *map, void *ptr, bool need_defer)
{
+ struct bpf_prog *prog = ptr;
+
+ mutex_lock(&prog->aux->ext_mutex);
+ prog->aux->prog_array_member_cnt--;
+ mutex_unlock(&prog->aux->ext_mutex);
/* bpf_prog is freed after one RCU or tasks trace grace period */
- bpf_prog_put(ptr);
+ bpf_prog_put(prog);
}
static u32 prog_fd_array_sys_lookup_elem(void *ptr)
@@ -918,8 +940,10 @@ static void bpf_fd_array_map_clear(struct bpf_map *map)
struct bpf_array *array = container_of(map, struct bpf_array, map);
int i;
- for (i = 0; i < array->map.max_entries; i++)
+ for (i = 0; i < array->map.max_entries; i++) {
fd_array_map_delete_elem(map, &i);
+ cond_resched();
+ }
}
static void prog_array_map_seq_show_elem(struct bpf_map *map, void *key,
diff --git a/kernel/bpf/bpf_struct_ops.c b/kernel/bpf/bpf_struct_ops.c
index 84b2d9dba79a..4daedbb6bf13 100644
--- a/kernel/bpf/bpf_struct_ops.c
+++ b/kernel/bpf/bpf_struct_ops.c
@@ -11,6 +11,7 @@
#include <linux/refcount.h>
#include <linux/mutex.h>
#include <linux/btf_ids.h>
+#include <linux/rcupdate_wait.h>
enum bpf_struct_ops_state {
BPF_STRUCT_OPS_STATE_INIT,
@@ -249,6 +250,7 @@ int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,
struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
struct bpf_struct_ops_value *uvalue, *kvalue;
enum bpf_struct_ops_state state;
+ s64 refcnt;
if (unlikely(*(u32 *)key != 0))
return -ENOENT;
@@ -267,7 +269,14 @@ int bpf_struct_ops_map_sys_lookup_elem(struct bpf_map *map, void *key,
uvalue = value;
memcpy(uvalue, st_map->uvalue, map->value_size);
uvalue->state = state;
- refcount_set(&uvalue->refcnt, refcount_read(&kvalue->refcnt));
+
+ /* This value offers the user space a general estimate of how
+ * many sockets are still utilizing this struct_ops for TCP
+ * congestion control. The number might not be exact, but it
+ * should sufficiently meet our present goals.
+ */
+ refcnt = atomic64_read(&map->refcnt) - atomic64_read(&map->usercnt);
+ refcount_set(&uvalue->refcnt, max_t(s64, refcnt, 0));
return 0;
}
@@ -491,7 +500,6 @@ static int bpf_struct_ops_map_update_elem(struct bpf_map *map, void *key,
*(unsigned long *)(udata + moff) = prog->aux->id;
}
- refcount_set(&kvalue->refcnt, 1);
bpf_map_inc(map);
set_memory_ro((long)st_map->image, 1);
@@ -537,8 +545,7 @@ static int bpf_struct_ops_map_delete_elem(struct bpf_map *map, void *key)
switch (prev_state) {
case BPF_STRUCT_OPS_STATE_INUSE:
st_map->st_ops->unreg(&st_map->kvalue.data);
- if (refcount_dec_and_test(&st_map->kvalue.refcnt))
- bpf_map_put(map);
+ bpf_map_put(map);
return 0;
case BPF_STRUCT_OPS_STATE_TOBEFREE:
return -EINPROGRESS;
@@ -571,7 +578,7 @@ static void bpf_struct_ops_map_seq_show_elem(struct bpf_map *map, void *key,
kfree(value);
}
-static void bpf_struct_ops_map_free(struct bpf_map *map)
+static void __bpf_struct_ops_map_free(struct bpf_map *map)
{
struct bpf_struct_ops_map *st_map = (struct bpf_struct_ops_map *)map;
@@ -583,6 +590,28 @@ static void bpf_struct_ops_map_free(struct bpf_map *map)
bpf_map_area_free(st_map);
}
+static void bpf_struct_ops_map_free(struct bpf_map *map)
+{
+ /* The struct_ops's function may switch to another struct_ops.
+ *
+ * For example, bpf_tcp_cc_x->init() may switch to
+ * another tcp_cc_y by calling
+ * setsockopt(TCP_CONGESTION, "tcp_cc_y").
+ * During the switch, bpf_struct_ops_put(tcp_cc_x) is called
+ * and its refcount may reach 0 which then free its
+ * trampoline image while tcp_cc_x is still running.
+ *
+ * A vanilla rcu gp is to wait for all bpf-tcp-cc prog
+ * to finish. bpf-tcp-cc prog is non sleepable.
+ * A rcu_tasks gp is to wait for the last few insn
+ * in the tramopline image to finish before releasing
+ * the trampoline image.
+ */
+ synchronize_rcu_mult(call_rcu, call_rcu_tasks);
+
+ __bpf_struct_ops_map_free(map);
+}
+
static int bpf_struct_ops_map_alloc_check(union bpf_attr *attr)
{
if (attr->key_size != sizeof(unsigned int) || attr->max_entries != 1 ||
@@ -631,7 +660,7 @@ static struct bpf_map *bpf_struct_ops_map_alloc(union bpf_attr *attr)
NUMA_NO_NODE);
st_map->image = bpf_jit_alloc_exec(PAGE_SIZE);
if (!st_map->uvalue || !st_map->links || !st_map->image) {
- bpf_struct_ops_map_free(map);
+ __bpf_struct_ops_map_free(map);
return ERR_PTR(-ENOMEM);
}
@@ -661,41 +690,23 @@ const struct bpf_map_ops bpf_struct_ops_map_ops = {
bool bpf_struct_ops_get(const void *kdata)
{
struct bpf_struct_ops_value *kvalue;
+ struct bpf_struct_ops_map *st_map;
+ struct bpf_map *map;
kvalue = container_of(kdata, struct bpf_struct_ops_value, data);
+ st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue);
- return refcount_inc_not_zero(&kvalue->refcnt);
-}
-
-static void bpf_struct_ops_put_rcu(struct rcu_head *head)
-{
- struct bpf_struct_ops_map *st_map;
-
- st_map = container_of(head, struct bpf_struct_ops_map, rcu);
- bpf_map_put(&st_map->map);
+ map = __bpf_map_inc_not_zero(&st_map->map, false);
+ return !IS_ERR(map);
}
void bpf_struct_ops_put(const void *kdata)
{
struct bpf_struct_ops_value *kvalue;
+ struct bpf_struct_ops_map *st_map;
kvalue = container_of(kdata, struct bpf_struct_ops_value, data);
- if (refcount_dec_and_test(&kvalue->refcnt)) {
- struct bpf_struct_ops_map *st_map;
-
- st_map = container_of(kvalue, struct bpf_struct_ops_map,
- kvalue);
- /* The struct_ops's function may switch to another struct_ops.
- *
- * For example, bpf_tcp_cc_x->init() may switch to
- * another tcp_cc_y by calling
- * setsockopt(TCP_CONGESTION, "tcp_cc_y").
- * During the switch, bpf_struct_ops_put(tcp_cc_x) is called
- * and its map->refcnt may reach 0 which then free its
- * trampoline image while tcp_cc_x is still running.
- *
- * Thus, a rcu grace period is needed here.
- */
- call_rcu(&st_map->rcu, bpf_struct_ops_put_rcu);
- }
+ st_map = container_of(kvalue, struct bpf_struct_ops_map, kvalue);
+
+ bpf_map_put(&st_map->map);
}
diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c
index e20b4d024a82..850ea69e05fd 100644
--- a/kernel/bpf/btf.c
+++ b/kernel/bpf/btf.c
@@ -6345,6 +6345,11 @@ static int btf_check_func_arg_match(struct bpf_verifier_env *env,
}
kfunc_meta->r0_size = reg->var_off.value;
+ if (kfunc_meta->r0_size > U32_MAX) {
+ bpf_log(log, "R%d rdonly/rdwr_buf_size exceeds u32 max\n",
+ regno);
+ return -EINVAL;
+ }
ret = mark_chain_precision(env, regno);
if (ret)
return ret;
diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c
index 6158b021f617..3140fb095077 100644
--- a/kernel/bpf/cgroup.c
+++ b/kernel/bpf/cgroup.c
@@ -630,8 +630,10 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
struct bpf_prog *old_prog = NULL;
struct bpf_cgroup_storage *storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {};
struct bpf_cgroup_storage *new_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {};
+ struct bpf_cgroup_storage *old_storage[MAX_BPF_CGROUP_STORAGE_TYPE] = {};
struct bpf_prog *new_prog = prog ? : link->link.prog;
enum cgroup_bpf_attach_type atype;
+ u32 old_flags;
struct bpf_prog_list *pl;
struct hlist_head *progs;
int err;
@@ -677,6 +679,7 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
if (pl) {
old_prog = pl->prog;
+ bpf_cgroup_storages_assign(old_storage, pl->storage);
} else {
struct hlist_node *last = NULL;
@@ -699,6 +702,7 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
pl->prog = prog;
pl->link = link;
bpf_cgroup_storages_assign(pl->storage, storage);
+ old_flags = cgrp->bpf.flags[atype];
cgrp->bpf.flags[atype] = saved_flags;
if (type == BPF_LSM_CGROUP) {
@@ -729,12 +733,14 @@ static int __cgroup_bpf_attach(struct cgroup *cgrp,
if (old_prog) {
pl->prog = old_prog;
pl->link = NULL;
+ bpf_cgroup_storages_assign(pl->storage, old_storage);
}
bpf_cgroup_storages_free(new_storage);
if (!old_prog) {
hlist_del(&pl->node);
kfree(pl);
}
+ cgrp->bpf.flags[atype] = old_flags;
return err;
}
@@ -797,6 +803,20 @@ static void replace_effective_prog(struct cgroup *cgrp,
}
}
+static bool cgroup_bpf_storages_compatible(struct bpf_prog *old_prog,
+ struct bpf_prog *new_prog)
+{
+ enum bpf_cgroup_storage_type stype;
+
+ for_each_cgroup_storage_type(stype) {
+ if (old_prog->aux->cgroup_storage[stype] !=
+ new_prog->aux->cgroup_storage[stype])
+ return false;
+ }
+
+ return true;
+}
+
/**
* __cgroup_bpf_replace() - Replace link's program and propagate the change
* to descendants
@@ -834,6 +854,9 @@ static int __cgroup_bpf_replace(struct cgroup *cgrp,
if (!found)
return -ENOENT;
+ if (!cgroup_bpf_storages_compatible(link->link.prog, new_prog))
+ return -EINVAL;
+
old_prog = xchg(&link->link.prog, new_prog);
replace_effective_prog(cgrp, atype, link);
bpf_prog_put(old_prog);
diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c
index 4ea21434bf78..a9e7b17af480 100644
--- a/kernel/bpf/core.c
+++ b/kernel/bpf/core.c
@@ -117,6 +117,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag
INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode);
mutex_init(&fp->aux->used_maps_mutex);
+ mutex_init(&fp->aux->ext_mutex);
mutex_init(&fp->aux->dst_mutex);
return fp;
diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c
index 291c2befa120..efe73a684c48 100644
--- a/kernel/bpf/hashtab.c
+++ b/kernel/bpf/hashtab.c
@@ -197,12 +197,12 @@ static bool htab_is_percpu(const struct bpf_htab *htab)
static inline void htab_elem_set_ptr(struct htab_elem *l, u32 key_size,
void __percpu *pptr)
{
- *(void __percpu **)(l->key + key_size) = pptr;
+ *(void __percpu **)(l->key + roundup(key_size, 8)) = pptr;
}
static inline void __percpu *htab_elem_get_ptr(struct htab_elem *l, u32 key_size)
{
- return *(void __percpu **)(l->key + key_size);
+ return *(void __percpu **)(l->key + roundup(key_size, 8));
}
static void *fd_htab_map_get_ptr(const struct bpf_map *map, struct htab_elem *l)
@@ -873,10 +873,12 @@ static int htab_map_get_next_key(struct bpf_map *map, void *key, void *next_key)
static void htab_elem_free(struct bpf_htab *htab, struct htab_elem *l)
{
+ migrate_disable();
if (htab->map.map_type == BPF_MAP_TYPE_PERCPU_HASH)
bpf_mem_cache_free(&htab->pcpu_ma, l->ptr_to_pptr);
check_and_free_fields(htab, l);
bpf_mem_cache_free(&htab->ma, l);
+ migrate_enable();
}
static void htab_put_fd_value(struct bpf_htab *htab, struct htab_elem *l)
@@ -921,7 +923,7 @@ static void free_htab_elem(struct bpf_htab *htab, struct htab_elem *l)
if (htab_is_prealloc(htab)) {
check_and_free_fields(htab, l);
- __pcpu_freelist_push(&htab->freelist, &l->fnode);
+ pcpu_freelist_push(&htab->freelist, &l->fnode);
} else {
dec_elem_count(htab);
htab_elem_free(htab, l);
@@ -961,8 +963,8 @@ static void pcpu_init_value(struct bpf_htab *htab, void __percpu *pptr,
for_each_possible_cpu(cpu) {
if (cpu == current_cpu)
- bpf_long_memcpy(per_cpu_ptr(pptr, cpu), value,
- size);
+ memcpy(per_cpu_ptr(pptr, cpu), value,
+ htab->map.value_size);
else
memset(per_cpu_ptr(pptr, cpu), 0, size);
}
@@ -994,7 +996,6 @@ static struct htab_elem *alloc_htab_elem(struct bpf_htab *htab, void *key,
*/
pl_new = this_cpu_ptr(htab->extra_elems);
l_new = *pl_new;
- htab_put_fd_value(htab, old_elem);
*pl_new = old_elem;
} else {
struct pcpu_freelist_node *l;
@@ -1079,6 +1080,7 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value,
struct htab_elem *l_new = NULL, *l_old;
struct hlist_nulls_head *head;
unsigned long flags;
+ void *old_map_ptr;
struct bucket *b;
u32 key_size, hash;
int ret;
@@ -1156,12 +1158,27 @@ static int htab_map_update_elem(struct bpf_map *map, void *key, void *value,
hlist_nulls_add_head_rcu(&l_new->hash_node, head);
if (l_old) {
hlist_nulls_del_rcu(&l_old->hash_node);
+
+ /* l_old has already been stashed in htab->extra_elems, free
+ * its special fields before it is available for reuse. Also
+ * save the old map pointer in htab of maps before unlock
+ * and release it after unlock.
+ */
+ old_map_ptr = NULL;
+ if (htab_is_prealloc(htab)) {
+ if (map->ops->map_fd_put_ptr)
+ old_map_ptr = fd_htab_map_get_ptr(map, l_old);
+ check_and_free_fields(htab, l_old);
+ }
+ }
+ htab_unlock_bucket(htab, b, hash, flags);
+ if (l_old) {
+ if (old_map_ptr)
+ map->ops->map_fd_put_ptr(map, old_map_ptr, true);
if (!htab_is_prealloc(htab))
free_htab_elem(htab, l_old);
- else
- check_and_free_fields(htab, l_old);
}
- ret = 0;
+ return 0;
err:
htab_unlock_bucket(htab, b, hash, flags);
return ret;
@@ -1398,15 +1415,15 @@ static int htab_map_delete_elem(struct bpf_map *map, void *key)
return ret;
l = lookup_elem_raw(head, hash, key, key_size);
-
- if (l) {
+ if (l)
hlist_nulls_del_rcu(&l->hash_node);
- free_htab_elem(htab, l);
- } else {
+ else
ret = -ENOENT;
- }
htab_unlock_bucket(htab, b, hash, flags);
+
+ if (l)
+ free_htab_elem(htab, l);
return ret;
}
@@ -1824,13 +1841,14 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
* may cause deadlock. See comments in function
* prealloc_lru_pop(). Let us do bpf_lru_push_free()
* after releasing the bucket lock.
+ *
+ * For htab of maps, htab_put_fd_value() in
+ * free_htab_elem() may acquire a spinlock with bucket
+ * lock being held and it violates the lock rule, so
+ * invoke free_htab_elem() after unlock as well.
*/
- if (is_lru_map) {
- l->batch_flink = node_to_free;
- node_to_free = l;
- } else {
- free_htab_elem(htab, l);
- }
+ l->batch_flink = node_to_free;
+ node_to_free = l;
}
dst_key += key_size;
dst_val += value_size;
@@ -1842,7 +1860,10 @@ __htab_map_lookup_and_delete_batch(struct bpf_map *map,
while (node_to_free) {
l = node_to_free;
node_to_free = node_to_free->batch_flink;
- htab_lru_push_free(htab, l);
+ if (is_lru_map)
+ htab_lru_push_free(htab, l);
+ else
+ free_htab_elem(htab, l);
}
next_batch:
diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c
index 4cd785493294..a89a88b5fb98 100644
--- a/kernel/bpf/helpers.c
+++ b/kernel/bpf/helpers.c
@@ -1110,11 +1110,15 @@ const struct bpf_func_proto bpf_snprintf_proto = {
*/
struct bpf_hrtimer {
struct hrtimer timer;
+ atomic_t cancelling;
struct bpf_map *map;
struct bpf_prog *prog;
void __rcu *callback_fn;
void *value;
- struct rcu_head rcu;
+ union {
+ struct rcu_head rcu;
+ struct work_struct delete_work;
+ };
};
/* the actual struct hidden inside uapi struct bpf_timer */
@@ -1168,6 +1172,22 @@ static enum hrtimer_restart bpf_timer_cb(struct hrtimer *hrtimer)
return HRTIMER_NORESTART;
}
+static void bpf_timer_delete_work(struct work_struct *work)
+{
+ struct bpf_hrtimer *t = container_of(work, struct bpf_hrtimer,
+ delete_work);
+
+ /* Cancel the timer and wait for callback to complete if it was running.
+ * If hrtimer_cancel() can be safely called it's safe to call
+ * kfree_rcu(t) right after for both preallocated and non-preallocated
+ * maps. The timer->timer = NULL was already done and no code path can see
+ * address 't' anymore. A timer armed before bpf_timer_cancel_and_free()
+ * will have been cancelled.
+ */
+ hrtimer_cancel(&t->timer);
+ kfree_rcu(t, rcu);
+}
+
BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern *, timer, struct bpf_map *, map,
u64, flags)
{
@@ -1194,8 +1214,11 @@ BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern *, timer, struct bpf_map *, map
ret = -EBUSY;
goto out;
}
- /* allocate hrtimer via map_kmalloc to use memcg accounting */
- t = bpf_map_kmalloc_node(map, sizeof(*t), GFP_ATOMIC, map->numa_node);
+ /* Allocate via bpf_map_kmalloc_node() for memcg accounting. Until
+ * kmalloc_nolock() is available, avoid locking issues by using
+ * __GFP_HIGH (GFP_ATOMIC & ~__GFP_RECLAIM).
+ */
+ t = bpf_map_kmalloc_node(map, sizeof(*t), __GFP_HIGH, map->numa_node);
if (!t) {
ret = -ENOMEM;
goto out;
@@ -1204,6 +1227,8 @@ BPF_CALL_3(bpf_timer_init, struct bpf_timer_kern *, timer, struct bpf_map *, map
t->map = map;
t->prog = NULL;
rcu_assign_pointer(t->callback_fn, NULL);
+ atomic_set(&t->cancelling, 0);
+ INIT_WORK(&t->delete_work, bpf_timer_delete_work);
hrtimer_init(&t->timer, clockid, HRTIMER_MODE_REL_SOFT);
t->timer.function = bpf_timer_cb;
WRITE_ONCE(timer->timer, t);
@@ -1331,7 +1356,8 @@ static void drop_prog_refcnt(struct bpf_hrtimer *t)
BPF_CALL_1(bpf_timer_cancel, struct bpf_timer_kern *, timer)
{
- struct bpf_hrtimer *t;
+ struct bpf_hrtimer *t, *cur_t;
+ bool inc = false;
int ret = 0;
if (in_nmi())
@@ -1343,14 +1369,40 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_timer_kern *, timer)
ret = -EINVAL;
goto out;
}
- if (this_cpu_read(hrtimer_running) == t) {
+ cur_t = this_cpu_read(hrtimer_running);
+ if (cur_t == t) {
/* If bpf callback_fn is trying to bpf_timer_cancel()
* its own timer the hrtimer_cancel() will deadlock
- * since it waits for callback_fn to finish
+ * since it waits for callback_fn to finish.
*/
ret = -EDEADLK;
goto out;
}
+
+ /* Only account in-flight cancellations when invoked from a timer
+ * callback, since we want to avoid waiting only if other _callbacks_
+ * are waiting on us, to avoid introducing lockups. Non-callback paths
+ * are ok, since nobody would synchronously wait for their completion.
+ */
+ if (!cur_t)
+ goto drop;
+ atomic_inc(&t->cancelling);
+ /* Need full barrier after relaxed atomic_inc */
+ smp_mb__after_atomic();
+ inc = true;
+ if (atomic_read(&cur_t->cancelling)) {
+ /* We're cancelling timer t, while some other timer callback is
+ * attempting to cancel us. In such a case, it might be possible
+ * that timer t belongs to the other callback, or some other
+ * callback waiting upon it (creating transitive dependencies
+ * upon us), and we will enter a deadlock if we continue
+ * cancelling and waiting for it synchronously, since it might
+ * do the same. Bail!
+ */
+ ret = -EDEADLK;
+ goto out;
+ }
+drop:
drop_prog_refcnt(t);
out:
__bpf_spin_unlock_irqrestore(&timer->lock);
@@ -1358,6 +1410,8 @@ BPF_CALL_1(bpf_timer_cancel, struct bpf_timer_kern *, timer)
* if it was running.
*/
ret = ret ?: hrtimer_cancel(&t->timer);
+ if (inc)
+ atomic_dec(&t->cancelling);
rcu_read_unlock();
return ret;
}
@@ -1395,14 +1449,8 @@ void bpf_timer_cancel_and_free(void *val)
__bpf_spin_unlock_irqrestore(&timer->lock);
if (!t)
return;
- /* Cancel the timer and wait for callback to complete if it was running.
- * If hrtimer_cancel() can be safely called it's safe to call kfree(t)
- * right after for both preallocated and non-preallocated maps.
- * The timer->timer = NULL was already done and no code path can
- * see address 't' anymore.
- *
- * Check that bpf_map_delete/update_elem() wasn't called from timer
- * callback_fn. In such case don't call hrtimer_cancel() (since it will
+ /* We check that bpf_map_delete/update_elem() was called from timer
+ * callback_fn. In such case we don't call hrtimer_cancel() (since it will
* deadlock) and don't call hrtimer_try_to_cancel() (since it will just
* return -1). Though callback_fn is still running on this cpu it's
* safe to do kfree(t) because bpf_timer_cb() read everything it needed
@@ -1410,10 +1458,30 @@ void bpf_timer_cancel_and_free(void *val)
* since timer->timer = NULL was already done. The timer will be
* effectively cancelled because bpf_timer_cb() will return
* HRTIMER_NORESTART.
+ *
+ * However, it is possible the timer callback_fn calling us armed the
+ * timer _before_ calling us, such that failing to cancel it here will
+ * cause it to possibly use struct hrtimer after freeing bpf_hrtimer.
+ * Therefore, we _need_ to cancel any outstanding timers before we do
+ * kfree_rcu, even though no more timers can be armed.
+ *
+ * Moreover, we need to schedule work even if timer does not belong to
+ * the calling callback_fn, as on two different CPUs, we can end up in a
+ * situation where both sides run in parallel, try to cancel one
+ * another, and we end up waiting on both sides in hrtimer_cancel
+ * without making forward progress, since timer1 depends on timer2
+ * callback to finish, and vice versa.
+ *
+ * CPU 1 (timer1_cb) CPU 2 (timer2_cb)
+ * bpf_timer_cancel_and_free(timer2) bpf_timer_cancel_and_free(timer1)
+ *
+ * To avoid these issues, punt to workqueue context when we are in a
+ * timer callback.
*/
- if (this_cpu_read(hrtimer_running) != t)
- hrtimer_cancel(&t->timer);
- kfree_rcu(t, rcu);
+ if (this_cpu_read(hrtimer_running))
+ queue_work(system_unbound_wq, &t->delete_work);
+ else
+ bpf_timer_delete_work(&t->delete_work);
}
BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr)
diff --git a/kernel/bpf/mmap_unlock_work.h b/kernel/bpf/mmap_unlock_work.h
index 5d18d7d85bef..1834db20b861 100644
--- a/kernel/bpf/mmap_unlock_work.h
+++ b/kernel/bpf/mmap_unlock_work.h
@@ -4,12 +4,15 @@
#ifndef __MMAP_UNLOCK_WORK_H__
#define __MMAP_UNLOCK_WORK_H__
+#include <linux/atomic.h>
+#include <linux/err.h>
#include <linux/irq_work.h>
/* irq_work to run mmap_read_unlock() in irq_work */
struct mmap_unlock_irq_work {
struct irq_work irq_work;
struct mm_struct *mm;
+ atomic_t active;
};
DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
@@ -18,32 +21,36 @@ DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work);
* We cannot do mmap_read_unlock() when the irq is disabled, because of
* risk to deadlock with rq_lock. To look up vma when the irqs are
* disabled, we need to run mmap_read_unlock() in irq_work. We use a
- * percpu variable to do the irq_work. If the irq_work is already used
- * by another lookup, we fall over.
+ * percpu variable to do the irq_work. The active flag reserves the slot
+ * before mmap_read_trylock() and until the irq_work callback consumes mm.
*/
-static inline bool bpf_mmap_unlock_get_irq_work(struct mmap_unlock_irq_work **work_ptr)
+static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void)
{
- struct mmap_unlock_irq_work *work = NULL;
- bool irq_work_busy = false;
+ struct mmap_unlock_irq_work *work;
- if (irqs_disabled()) {
- if (!IS_ENABLED(CONFIG_PREEMPT_RT)) {
- work = this_cpu_ptr(&mmap_unlock_work);
- if (irq_work_is_busy(&work->irq_work)) {
- /* cannot queue more up_read, fallback */
- irq_work_busy = true;
- }
- } else {
- /*
- * PREEMPT_RT does not allow to trylock mmap sem in
- * interrupt disabled context. Force the fallback code.
- */
- irq_work_busy = true;
- }
- }
+ if (!irqs_disabled())
+ return NULL;
+
+ /*
+ * PREEMPT_RT does not allow to trylock mmap sem in interrupt
+ * disabled context. Force the fallback code.
+ */
+ if (IS_ENABLED(CONFIG_PREEMPT_RT))
+ return ERR_PTR(-EBUSY);
+
+ work = this_cpu_ptr(&mmap_unlock_work);
+ if (irq_work_is_busy(&work->irq_work) ||
+ atomic_cmpxchg_acquire(&work->active, 0, 1))
+ return ERR_PTR(-EBUSY);
- *work_ptr = work;
- return irq_work_busy;
+ return work;
+}
+
+static inline void
+bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work)
+{
+ if (work)
+ atomic_set_release(&work->active, 0);
}
static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm)
diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c
index a8fe640318c6..a088132163b1 100644
--- a/kernel/bpf/queue_stack_maps.c
+++ b/kernel/bpf/queue_stack_maps.c
@@ -103,8 +103,10 @@ static int __queue_map_get(struct bpf_map *map, void *value, bool delete)
void *ptr;
if (in_nmi()) {
- if (!raw_spin_trylock_irqsave(&qs->lock, flags))
+ if (!raw_spin_trylock_irqsave(&qs->lock, flags)) {
+ memset(value, 0, qs->map.value_size);
return -EBUSY;
+ }
} else {
raw_spin_lock_irqsave(&qs->lock, flags);
}
@@ -138,8 +140,10 @@ static int __stack_map_get(struct bpf_map *map, void *value, bool delete)
u32 index;
if (in_nmi()) {
- if (!raw_spin_trylock_irqsave(&qs->lock, flags))
+ if (!raw_spin_trylock_irqsave(&qs->lock, flags)) {
+ memset(value, 0, qs->map.value_size);
return -EBUSY;
+ }
} else {
raw_spin_lock_irqsave(&qs->lock, flags);
}
diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c
index f0fd936cef31..3de70a1578ca 100644
--- a/kernel/bpf/stackmap.c
+++ b/kernel/bpf/stackmap.c
@@ -131,24 +131,20 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
u64 *ips, u32 trace_nr, bool user)
{
int i;
- struct mmap_unlock_irq_work *work = NULL;
- bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
+ struct mmap_unlock_irq_work *work;
struct vm_area_struct *vma, *prev_vma = NULL;
const char *prev_build_id;
- /* If the irq_work is in use, fall back to report ips. Same
- * fallback is used for kernel stack (!user) on a stackmap with
- * build_id.
- */
- if (!user || !current || !current->mm || irq_work_busy ||
- !mmap_read_trylock(current->mm)) {
- /* cannot access current->mm, fall back to ips */
- for (i = 0; i < trace_nr; i++) {
- id_offs[i].status = BPF_STACK_BUILD_ID_IP;
- id_offs[i].ip = ips[i];
- memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
- }
- return;
+ if (!user || !current || !current->mm)
+ goto fallback;
+
+ work = bpf_mmap_unlock_guard_get();
+ if (IS_ERR(work))
+ goto fallback;
+
+ if (!mmap_read_trylock(current->mm)) {
+ bpf_mmap_unlock_guard_put(work);
+ goto fallback;
}
for (i = 0; i < trace_nr; i++) {
@@ -174,6 +170,15 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs,
prev_build_id = id_offs[i].build_id;
}
bpf_mmap_unlock_mm(work, current->mm);
+ return;
+
+fallback:
+ /* cannot access current->mm, fall back to ips */
+ for (i = 0; i < trace_nr; i++) {
+ id_offs[i].status = BPF_STACK_BUILD_ID_IP;
+ id_offs[i].ip = ips[i];
+ memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX);
+ }
}
static struct perf_callchain_entry *
@@ -219,7 +224,7 @@ static long __bpf_get_stackid(struct bpf_map *map,
struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map);
struct stack_map_bucket *bucket, *new_bucket, *old_bucket;
u32 skip = flags & BPF_F_SKIP_FIELD_MASK;
- u32 hash, id, trace_nr, trace_len;
+ u32 hash, id, trace_nr, trace_len, max_depth;
bool user = flags & BPF_F_USER_STACK;
u64 *ips;
bool hash_matches;
@@ -228,7 +233,10 @@ static long __bpf_get_stackid(struct bpf_map *map,
/* skipping more than usable stack trace */
return -EFAULT;
- trace_nr = trace->nr - skip;
+ max_depth = map->value_size / stack_map_data_size(map) + skip;
+ if (max_depth > sysctl_perf_event_max_stack)
+ max_depth = sysctl_perf_event_max_stack;
+ trace_nr = min_t(u32, trace->nr - skip, max_depth - skip);
trace_len = trace_nr * sizeof(u64);
ips = trace->ip + skip;
hash = jhash2((u32 *)ips, trace_len / sizeof(u32), 0);
@@ -291,6 +299,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,
bool user = flags & BPF_F_USER_STACK;
struct perf_callchain_entry *trace;
bool kernel = !user;
+ long ret;
if (unlikely(flags & ~(BPF_F_SKIP_FIELD_MASK | BPF_F_USER_STACK |
BPF_F_FAST_STACK_CMP | BPF_F_REUSE_STACKID)))
@@ -300,14 +309,17 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map,
if (max_depth > sysctl_perf_event_max_stack)
max_depth = sysctl_perf_event_max_stack;
- trace = get_perf_callchain(regs, 0, kernel, user, max_depth,
- false, false);
+ scoped_guard(preempt) {
+ trace = get_perf_callchain(regs, 0, kernel, user, max_depth,
+ false, false);
+ if (unlikely(!trace))
+ /* couldn't fetch the stack trace */
+ return -EFAULT;
- if (unlikely(!trace))
- /* couldn't fetch the stack trace */
- return -EFAULT;
+ ret = __bpf_get_stackid(map, trace, flags);
+ }
- return __bpf_get_stackid(map, trace, flags);
+ return ret;
}
const struct bpf_func_proto bpf_get_stackid_proto = {
@@ -357,15 +369,11 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx,
return -EFAULT;
nr_kernel = count_kernel_ip(trace);
+ __u64 nr = trace->nr; /* save original */
if (kernel) {
- __u64 nr = trace->nr;
-
trace->nr = nr_kernel;
ret = __bpf_get_stackid(map, trace, flags);
-
- /* restore nr */
- trace->nr = nr;
} else { /* user */
u64 skip = flags & BPF_F_SKIP_FIELD_MASK;
@@ -376,6 +384,10 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx,
flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip;
ret = __bpf_get_stackid(map, trace, flags);
}
+
+ /* restore nr */
+ trace->nr = nr;
+
return ret;
}
@@ -430,6 +442,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
if (sysctl_perf_event_max_stack < max_depth)
max_depth = sysctl_perf_event_max_stack;
+ preempt_disable();
if (trace_in)
trace = trace_in;
else if (kernel && task)
@@ -437,11 +450,15 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
else
trace = get_perf_callchain(regs, 0, kernel, user, max_depth,
crosstask, false);
- if (unlikely(!trace))
+ if (unlikely(!trace)) {
+ preempt_enable();
goto err_fault;
+ }
- if (trace->nr < skip)
+ if (trace->nr < skip) {
+ preempt_enable();
goto err_fault;
+ }
trace_nr = trace->nr - skip;
trace_nr = (trace_nr <= num_elem) ? trace_nr : num_elem;
@@ -452,6 +469,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task,
stack_map_get_build_id_offset(buf, ips, trace_nr, user);
else
memcpy(buf, ips, copy_len);
+ preempt_enable();
if (size > copy_len)
memset(buf + copy_len, 0, size - copy_len);
@@ -486,14 +504,17 @@ BPF_CALL_4(bpf_get_task_stack, struct task_struct *, task, void *, buf,
struct pt_regs *regs;
long res = -EINVAL;
- if (!try_get_task_stack(task))
+ if (!try_get_task_stack(task)) {
+ memset(buf, 0, size);
return -EFAULT;
+ }
regs = task_pt_regs(task);
if (regs)
res = __bpf_get_stack(regs, task, NULL, buf, size, flags);
+ else
+ memset(buf, 0, size);
put_task_stack(task);
-
return res;
}
diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c
index d630d1665c6b..97dbfb8832ae 100644
--- a/kernel/bpf/syscall.c
+++ b/kernel/bpf/syscall.c
@@ -1298,8 +1298,10 @@ struct bpf_map *bpf_map_get_with_uref(u32 ufd)
return map;
}
-/* map_idr_lock should have been held */
-static struct bpf_map *__bpf_map_inc_not_zero(struct bpf_map *map, bool uref)
+/* map_idr_lock should have been held or the map should have been
+ * protected by rcu read lock.
+ */
+struct bpf_map *__bpf_map_inc_not_zero(struct bpf_map *map, bool uref)
{
int refold;
@@ -2723,6 +2725,7 @@ static int bpf_obj_get(const union bpf_attr *attr)
void bpf_link_init(struct bpf_link *link, enum bpf_link_type type,
const struct bpf_link_ops *ops, struct bpf_prog *prog)
{
+ WARN_ON(ops->dealloc && ops->dealloc_deferred);
atomic64_set(&link->refcnt, 1);
link->type = type;
link->id = 0;
@@ -2760,17 +2763,46 @@ void bpf_link_inc(struct bpf_link *link)
atomic64_inc(&link->refcnt);
}
+static void bpf_link_defer_dealloc_rcu_gp(struct rcu_head *rcu)
+{
+ struct bpf_link *link = container_of(rcu, struct bpf_link, rcu);
+
+ /* free bpf_link and its containing memory */
+ link->ops->dealloc_deferred(link);
+}
+
+static void bpf_link_defer_dealloc_mult_rcu_gp(struct rcu_head *rcu)
+{
+ if (rcu_trace_implies_rcu_gp())
+ bpf_link_defer_dealloc_rcu_gp(rcu);
+ else
+ call_rcu(rcu, bpf_link_defer_dealloc_rcu_gp);
+}
+
/* bpf_link_free is guaranteed to be called from process context */
static void bpf_link_free(struct bpf_link *link)
{
+ const struct bpf_link_ops *ops = link->ops;
+ bool sleepable = false;
+
bpf_link_free_id(link->id);
if (link->prog) {
+ sleepable = link->prog->aux->sleepable;
/* detach BPF program, clean up used resources */
- link->ops->release(link);
+ ops->release(link);
bpf_prog_put(link->prog);
}
- /* free bpf_link and its containing memory */
- link->ops->dealloc(link);
+ if (ops->dealloc_deferred) {
+ /* schedule BPF link deallocation; if underlying BPF program
+ * is sleepable, we need to first wait for RCU tasks trace
+ * sync, then go through "classic" RCU grace period
+ */
+ if (sleepable)
+ call_rcu_tasks_trace(&link->rcu, bpf_link_defer_dealloc_mult_rcu_gp);
+ else
+ call_rcu(&link->rcu, bpf_link_defer_dealloc_rcu_gp);
+ } else if (ops->dealloc)
+ ops->dealloc(link);
}
static void bpf_link_put_deferred(struct work_struct *work)
@@ -2820,9 +2852,10 @@ static const char *bpf_link_type_strs[] = {
static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
{
const struct bpf_link *link = filp->private_data;
- const struct bpf_prog *prog = link->prog;
+ const struct bpf_prog *prog;
enum bpf_link_type type = link->type;
char prog_tag[sizeof(prog->tag) * 2 + 1] = { };
+ u32 prog_id = 0;
if (type < ARRAY_SIZE(bpf_link_type_strs) && bpf_link_type_strs[type]) {
seq_printf(m, "link_type:\t%s\n", bpf_link_type_strs[type]);
@@ -2832,12 +2865,21 @@ static void bpf_link_show_fdinfo(struct seq_file *m, struct file *filp)
}
seq_printf(m, "link_id:\t%u\n", link->id);
- bin2hex(prog_tag, prog->tag, sizeof(prog->tag));
- seq_printf(m,
- "prog_tag:\t%s\n"
- "prog_id:\t%u\n",
- prog_tag,
- prog->aux->id);
+ rcu_read_lock();
+ prog = READ_ONCE(link->prog);
+ if (prog) {
+ bin2hex(prog_tag, prog->tag, sizeof(prog->tag));
+ prog_id = prog->aux->id;
+ }
+ rcu_read_unlock();
+
+ if (prog) {
+ seq_printf(m,
+ "prog_tag:\t%s\n"
+ "prog_id:\t%u\n",
+ prog_tag,
+ prog_id);
+ }
if (link->ops->show_fdinfo)
link->ops->show_fdinfo(link, m);
}
@@ -2951,7 +2993,8 @@ static void bpf_tracing_link_release(struct bpf_link *link)
container_of(link, struct bpf_tracing_link, link.link);
WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link,
- tr_link->trampoline));
+ tr_link->trampoline,
+ tr_link->tgt_prog));
bpf_trampoline_put(tr_link->trampoline);
@@ -3079,7 +3122,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
* in prog->aux
*
* - if prog->aux->dst_trampoline is NULL, the program has already been
- * attached to a target and its initial target was cleared (below)
+ * attached to a target and its initial target was cleared (below)
*
* - if tgt_prog != NULL, the caller specified tgt_prog_fd +
* target_btf_id using the link_create API.
@@ -3149,7 +3192,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog,
if (err)
goto out_unlock;
- err = bpf_trampoline_link_prog(&link->link, tr);
+ err = bpf_trampoline_link_prog(&link->link, tr, tgt_prog);
if (err) {
bpf_link_cleanup(&link_primer);
link = NULL;
@@ -3256,7 +3299,7 @@ static int bpf_raw_tp_link_fill_link_info(const struct bpf_link *link,
static const struct bpf_link_ops bpf_raw_tp_link_lops = {
.release = bpf_raw_tp_link_release,
- .dealloc = bpf_raw_tp_link_dealloc,
+ .dealloc_deferred = bpf_raw_tp_link_dealloc,
.show_fdinfo = bpf_raw_tp_link_show_fdinfo,
.fill_link_info = bpf_raw_tp_link_fill_link_info,
};
@@ -3427,9 +3470,14 @@ static int bpf_raw_tracepoint_open(const union bpf_attr *attr)
return fd;
}
+static enum bpf_prog_type
+attach_type_to_prog_type(enum bpf_attach_type attach_type);
+
static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
enum bpf_attach_type attach_type)
{
+ enum bpf_prog_type ptype;
+
switch (prog->type) {
case BPF_PROG_TYPE_CGROUP_SOCK:
case BPF_PROG_TYPE_CGROUP_SOCK_ADDR:
@@ -3442,6 +3490,11 @@ static int bpf_prog_attach_check_attach_type(const struct bpf_prog *prog,
* check permissions at attach time.
*/
return -EPERM;
+
+ ptype = attach_type_to_prog_type(attach_type);
+ if (prog->type != ptype)
+ return -EINVAL;
+
return prog->enforce_expected_attach_type &&
prog->expected_attach_type != attach_type ?
-EINVAL : 0;
@@ -4313,6 +4366,7 @@ static int bpf_link_get_info_by_fd(struct file *file,
{
struct bpf_link_info __user *uinfo = u64_to_user_ptr(attr->info.info);
struct bpf_link_info info;
+ const struct bpf_prog *prog;
u32 info_len = attr->info.info_len;
int err;
@@ -4327,7 +4381,12 @@ static int bpf_link_get_info_by_fd(struct file *file,
info.type = link->type;
info.id = link->id;
- info.prog_id = link->prog->aux->id;
+
+ rcu_read_lock();
+ prog = READ_ONCE(link->prog);
+ if (prog)
+ info.prog_id = prog->aux->id;
+ rcu_read_unlock();
if (link->ops->fill_link_info) {
err = link->ops->fill_link_info(link, &info);
diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c
index 30dd2b31a38d..481e2b43f0ed 100644
--- a/kernel/bpf/task_iter.c
+++ b/kernel/bpf/task_iter.c
@@ -780,9 +780,8 @@ static struct bpf_iter_reg task_vma_reg_info = {
BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start,
bpf_callback_t, callback_fn, void *, callback_ctx, u64, flags)
{
- struct mmap_unlock_irq_work *work = NULL;
+ struct mmap_unlock_irq_work *work;
struct vm_area_struct *vma;
- bool irq_work_busy = false;
bool __maybe_unused mmput_needed = false;
struct mm_struct *mm;
int ret = -ENOENT;
@@ -819,9 +818,14 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start,
if (!mm)
return -ENOENT;
- irq_work_busy = bpf_mmap_unlock_get_irq_work(&work);
+ work = bpf_mmap_unlock_guard_get();
+ if (IS_ERR(work)) {
+ ret = PTR_ERR(work);
+ goto out;
+ }
- if (irq_work_busy || !mmap_read_trylock(mm)) {
+ if (!mmap_read_trylock(mm)) {
+ bpf_mmap_unlock_guard_put(work);
ret = -EBUSY;
goto out;
}
@@ -864,6 +868,8 @@ static void do_mmap_read_unlock(struct irq_work *entry)
work = container_of(entry, struct mmap_unlock_irq_work, irq_work);
mmap_read_unlock_non_owner(work->mm);
+ work->mm = NULL;
+ bpf_mmap_unlock_guard_put(work);
}
static int __init task_iter_init(void)
diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c
index 17763af54179..3473e86a112a 100644
--- a/kernel/bpf/trampoline.c
+++ b/kernel/bpf/trampoline.c
@@ -418,6 +418,13 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key)
return ERR_PTR(err);
}
+void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags)
+{
+ mutex_lock(&tr->mutex);
+ tr->flags |= flags;
+ mutex_unlock(&tr->mutex);
+}
+
static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex)
{
struct bpf_tramp_image *im;
@@ -539,7 +546,27 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog)
}
}
-static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr)
+static int bpf_freplace_check_tgt_prog(struct bpf_prog *tgt_prog)
+{
+ struct bpf_prog_aux *aux = tgt_prog->aux;
+
+ guard(mutex)(&aux->ext_mutex);
+ if (aux->prog_array_member_cnt)
+ /* Program extensions can not extend target prog when the target
+ * prog has been updated to any prog_array map as tail callee.
+ * It's to prevent a potential infinite loop like:
+ * tgt prog entry -> tgt prog subprog -> freplace prog entry
+ * --tailcall-> tgt prog entry.
+ */
+ return -EBUSY;
+
+ aux->is_extended = true;
+ return 0;
+}
+
+static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link,
+ struct bpf_trampoline *tr,
+ struct bpf_prog *tgt_prog)
{
enum bpf_tramp_prog_type kind;
struct bpf_tramp_link *link_exiting;
@@ -560,6 +587,9 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_tr
/* Cannot attach extension if fentry/fexit are in use. */
if (cnt)
return -EBUSY;
+ err = bpf_freplace_check_tgt_prog(tgt_prog);
+ if (err)
+ return err;
tr->extension_prog = link->link.prog;
return bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, NULL,
link->link.prog->bpf_func);
@@ -586,17 +616,21 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_tr
return err;
}
-int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr)
+int bpf_trampoline_link_prog(struct bpf_tramp_link *link,
+ struct bpf_trampoline *tr,
+ struct bpf_prog *tgt_prog)
{
int err;
mutex_lock(&tr->mutex);
- err = __bpf_trampoline_link_prog(link, tr);
+ err = __bpf_trampoline_link_prog(link, tr, tgt_prog);
mutex_unlock(&tr->mutex);
return err;
}
-static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr)
+static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
+ struct bpf_trampoline *tr,
+ struct bpf_prog *tgt_prog)
{
enum bpf_tramp_prog_type kind;
int err;
@@ -607,6 +641,8 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_
err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP,
tr->extension_prog->bpf_func, NULL);
tr->extension_prog = NULL;
+ guard(mutex)(&tgt_prog->aux->ext_mutex);
+ tgt_prog->aux->is_extended = false;
return err;
}
hlist_del_init(&link->tramp_hlist);
@@ -615,12 +651,14 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_
}
/* bpf_trampoline_unlink_prog() should never fail. */
-int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr)
+int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link,
+ struct bpf_trampoline *tr,
+ struct bpf_prog *tgt_prog)
{
int err;
mutex_lock(&tr->mutex);
- err = __bpf_trampoline_unlink_prog(link, tr);
+ err = __bpf_trampoline_unlink_prog(link, tr, tgt_prog);
mutex_unlock(&tr->mutex);
return err;
}
@@ -635,7 +673,7 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link)
if (!shim_link->trampoline)
return;
- WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline));
+ WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline, NULL));
bpf_trampoline_put(shim_link->trampoline);
}
@@ -747,7 +785,7 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog,
goto err;
}
- err = __bpf_trampoline_link_prog(&shim_link->link, tr);
+ err = __bpf_trampoline_link_prog(&shim_link->link, tr, NULL);
if (err)
goto err;
diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c
index e2ce9f664a40..b79581a941cd 100644
--- a/kernel/bpf/verifier.c
+++ b/kernel/bpf/verifier.c
@@ -473,6 +473,13 @@ static bool is_cmpxchg_insn(const struct bpf_insn *insn)
insn->imm == BPF_CMPXCHG;
}
+static bool is_atomic_fetch_insn(const struct bpf_insn *insn)
+{
+ return BPF_CLASS(insn->code) == BPF_STX &&
+ BPF_MODE(insn->code) == BPF_ATOMIC &&
+ (insn->imm & BPF_FETCH);
+}
+
/* string representation of 'enum bpf_reg_type'
*
* Note that reg_type_str() can not appear more than once in a single verbose()
@@ -1026,7 +1033,10 @@ static int resize_reference_state(struct bpf_func_state *state, size_t n)
return 0;
}
-static int grow_stack_state(struct bpf_func_state *state, int size)
+/* Possibly update state->allocated_stack to be at least size bytes. Also
+ * possibly update the function's high-water mark in its bpf_subprog_info.
+ */
+static int grow_stack_state(struct bpf_verifier_env *env, struct bpf_func_state *state, int size)
{
size_t old_n = state->allocated_stack / BPF_REG_SIZE, n = size / BPF_REG_SIZE;
@@ -1038,6 +1048,11 @@ static int grow_stack_state(struct bpf_func_state *state, int size)
return -ENOMEM;
state->allocated_stack = size;
+
+ /* update known max for given subprogram */
+ if (env->subprog_info[state->subprogno].stack_depth < size)
+ env->subprog_info[state->subprogno].stack_depth = size;
+
return 0;
}
@@ -2600,10 +2615,25 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
* dreg still needs precision before this insn
*/
}
- } else if (class == BPF_LDX) {
- if (!(*reg_mask & dreg))
+ } else if (class == BPF_LDX || is_atomic_fetch_insn(insn)) {
+ u32 load_reg = dreg;
+ u8 base_reg = insn->src_reg;
+
+ /*
+ * Atomic fetch operation writes the old value into
+ * a register (sreg or r0) and if it was tracked for
+ * precision, propagate to the stack slot like we do
+ * in regular ldx.
+ */
+ if (is_atomic_fetch_insn(insn)) {
+ load_reg = insn->imm == BPF_CMPXCHG ?
+ 1u << BPF_REG_0 : sreg;
+ base_reg = insn->dst_reg;
+ }
+
+ if (!(*reg_mask & load_reg))
return 0;
- *reg_mask &= ~dreg;
+ *reg_mask &= ~load_reg;
/* scalars can only be spilled into stack w/o losing precision.
* Load from any other memory can be zero extended.
@@ -2611,7 +2641,7 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx,
* by 'precise' mark in corresponding register of this state.
* No further tracking necessary.
*/
- if (insn->src_reg != BPF_REG_FP)
+ if (base_reg != BPF_REG_FP)
return 0;
/* dreg = *(u64 *)[fp - off] was a fill from the stack.
@@ -3214,9 +3244,6 @@ static int check_stack_write_fixed_off(struct bpf_verifier_env *env,
struct bpf_reg_state *reg = NULL;
u32 dst_reg = insn->dst_reg;
- err = grow_stack_state(state, round_up(slot + 1, BPF_REG_SIZE));
- if (err)
- return err;
/* caller checked that off % size == 0 and -MAX_BPF_STACK <= off < 0,
* so it's aligned access and [off, off + size) are within stack limits
*/
@@ -3367,11 +3394,6 @@ static int check_stack_write_var_off(struct bpf_verifier_env *env,
if (value_reg && register_is_null(value_reg))
writing_zero = true;
- err = grow_stack_state(state, round_up(-min_off, BPF_REG_SIZE));
- if (err)
- return err;
-
-
/* Variable offset writes destroy any spilled pointers in range. */
for (i = min_off; i < max_off; i++) {
u8 new_type, *stype;
@@ -4406,20 +4428,6 @@ static int check_ptr_alignment(struct bpf_verifier_env *env,
strict);
}
-static int update_stack_depth(struct bpf_verifier_env *env,
- const struct bpf_func_state *func,
- int off)
-{
- u16 stack = env->subprog_info[func->subprogno].stack_depth;
-
- if (stack >= -off)
- return 0;
-
- /* update known max for given subprogram */
- env->subprog_info[func->subprogno].stack_depth = -off;
- return 0;
-}
-
/* starting from main bpf function walk all instructions of the function
* and recursively walk all callees that given function can call.
* Ignore jump and exit insns.
@@ -4849,13 +4857,14 @@ static int check_ptr_to_map_access(struct bpf_verifier_env *env,
* The minimum valid offset is -MAX_BPF_STACK for writes, and
* -state->allocated_stack for reads.
*/
-static int check_stack_slot_within_bounds(s64 off,
- struct bpf_func_state *state,
- enum bpf_access_type t)
+static int check_stack_slot_within_bounds(struct bpf_verifier_env *env,
+ s64 off,
+ struct bpf_func_state *state,
+ enum bpf_access_type t)
{
int min_valid_off;
- if (t == BPF_WRITE)
+ if (t == BPF_WRITE || env->allow_uninit_stack)
min_valid_off = -MAX_BPF_STACK;
else
min_valid_off = -state->allocated_stack;
@@ -4904,7 +4913,7 @@ static int check_stack_access_within_bounds(
max_off = reg->smax_value + off + access_size;
}
- err = check_stack_slot_within_bounds(min_off, state, type);
+ err = check_stack_slot_within_bounds(env, min_off, state, type);
if (!err && max_off > 0)
err = -EINVAL; /* out of stack access into non-negative offsets */
if (!err && access_size < 0)
@@ -4924,8 +4933,10 @@ static int check_stack_access_within_bounds(
verbose(env, "invalid variable-offset%s stack R%d var_off=%s size=%d\n",
err_extra, regno, tn_buf, access_size);
}
+ return err;
}
- return err;
+
+ return grow_stack_state(env, state, round_up(-min_off, BPF_REG_SIZE));
}
/* check whether memory at (regno + off) is accessible for t = (read | write)
@@ -4940,7 +4951,6 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
{
struct bpf_reg_state *regs = cur_regs(env);
struct bpf_reg_state *reg = regs + regno;
- struct bpf_func_state *state;
int size, err = 0;
size = bpf_size_to_bytes(bpf_size);
@@ -5084,11 +5094,6 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn
if (err)
return err;
- state = func(env, reg);
- err = update_stack_depth(env, state, off);
- if (err)
- return err;
-
if (t == BPF_READ)
err = check_stack_read(env, regno, off, size,
value_regno);
@@ -5281,7 +5286,8 @@ static int check_atomic(struct bpf_verifier_env *env, int insn_idx, struct bpf_i
/* When register 'regno' is used to read the stack (either directly or through
* a helper function) make sure that it's within stack boundary and, depending
- * on the access type, that all elements of the stack are initialized.
+ * on the access type and privileges, that all elements of the stack are
+ * initialized.
*
* 'off' includes 'regno->off', but not its dynamic part (if any).
*
@@ -5364,8 +5370,11 @@ static int check_stack_range_initialized(
slot = -i - 1;
spi = slot / BPF_REG_SIZE;
- if (state->allocated_stack <= slot)
- goto err;
+ if (state->allocated_stack <= slot) {
+ verbose(env, "verifier bug: allocated_stack too small");
+ return -EFAULT;
+ }
+
stype = &state->stack[spi].slot_type[slot % BPF_REG_SIZE];
if (*stype == STACK_MISC)
goto mark;
@@ -5389,7 +5398,6 @@ static int check_stack_range_initialized(
goto mark;
}
-err:
if (tnum_is_const(reg->var_off)) {
verbose(env, "invalid%s read from stack R%d off %d+%d size %d\n",
err_extra, regno, min_off, i - min_off, access_size);
@@ -5414,7 +5422,7 @@ static int check_stack_range_initialized(
* helper may write to the entire memory range.
*/
}
- return update_stack_depth(env, state, min_off);
+ return 0;
}
static int check_helper_mem_access(struct bpf_verifier_env *env, int regno,
@@ -15425,6 +15433,12 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -EINVAL;
}
+ if (prog->aux->sleepable && prog->type == BPF_PROG_TYPE_LSM &&
+ prog->expected_attach_type == BPF_LSM_CGROUP) {
+ verbose(env, "BPF_LSM_CGROUP programs cannot be sleepable\n");
+ return -EINVAL;
+ }
+
if (prog->type == BPF_PROG_TYPE_STRUCT_OPS)
return check_struct_ops_btf_id(env);
@@ -15479,7 +15493,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env)
return -ENOMEM;
if (tgt_prog && tgt_prog->aux->tail_call_reachable)
- tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX;
+ bpf_trampoline_set_flags(tr, BPF_TRAMP_F_TAIL_CALL_CTX);
prog->aux->dst_trampoline = tr;
return 0;
diff --git a/kernel/cgroup/Makefile b/kernel/cgroup/Makefile
index 12f8457ad1f9..005ac4c675cb 100644
--- a/kernel/cgroup/Makefile
+++ b/kernel/cgroup/Makefile
@@ -4,6 +4,6 @@ obj-y := cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o
obj-$(CONFIG_CGROUP_FREEZER) += legacy_freezer.o
obj-$(CONFIG_CGROUP_PIDS) += pids.o
obj-$(CONFIG_CGROUP_RDMA) += rdma.o
-obj-$(CONFIG_CPUSETS) += cpuset.o
+obj-$(CONFIG_CPUSETS) += cpuset.o cpuset-v1.o
obj-$(CONFIG_CGROUP_MISC) += misc.o
obj-$(CONFIG_CGROUP_DEBUG) += debug.o
diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c
index 5ff761930145..92283899cd95 100644
--- a/kernel/cgroup/cgroup.c
+++ b/kernel/cgroup/cgroup.c
@@ -3786,33 +3786,62 @@ static int cgroup_cpu_pressure_show(struct seq_file *seq, void *v)
static ssize_t pressure_write(struct kernfs_open_file *of, char *buf,
size_t nbytes, enum psi_res res)
{
- struct cgroup_file_ctx *ctx = of->priv;
+ struct cgroup_file_ctx *ctx;
struct psi_trigger *new;
struct cgroup *cgrp;
struct psi_group *psi;
+ bool need_rtpoll_worker;
+ ssize_t ret = 0;
cgrp = cgroup_kn_lock_live(of->kn, false);
if (!cgrp)
return -ENODEV;
- cgroup_get(cgrp);
- cgroup_kn_unlock(of->kn);
+ ctx = of->priv;
+ if (!ctx) {
+ ret = -ENODEV;
+ goto out_unlock;
+ }
/* Allow only one trigger per file descriptor */
if (ctx->psi.trigger) {
- cgroup_put(cgrp);
- return -EBUSY;
+ ret = -EBUSY;
+ goto out_unlock;
}
psi = cgroup_psi(cgrp);
- new = psi_trigger_create(psi, buf, res, of->file, of);
+ new = psi_trigger_create(psi, buf, res, of->file, of,
+ &need_rtpoll_worker);
if (IS_ERR(new)) {
- cgroup_put(cgrp);
- return PTR_ERR(new);
+ ret = PTR_ERR(new);
+ goto out_unlock;
+ }
+
+ /*
+ * The worker fork must run with neither cgroup_mutex nor the file's
+ * kernfs active reference held. The latter is broken since
+ * cgroup_kn_lock_live(). @of->priv may be released while unlocked, so
+ * recheck before publishing @new.
+ */
+ if (need_rtpoll_worker) {
+ cgroup_unlock();
+ ret = psi_trigger_create_rtpoll_worker(psi);
+ cgroup_lock();
+
+ if (!ret && !of->priv)
+ ret = -ENODEV;
+ if (ret) {
+ psi_trigger_destroy(new);
+ goto out_unlock;
+ }
}
smp_store_release(&ctx->psi.trigger, new);
- cgroup_put(cgrp);
+
+out_unlock:
+ cgroup_kn_unlock(of->kn);
+ if (ret)
+ return ret;
return nbytes;
}
@@ -6435,10 +6464,7 @@ static int cgroup_css_set_fork(struct kernel_clone_args *kargs)
spin_lock_irq(&css_set_lock);
cset = task_css_set(current);
get_css_set(cset);
- if (kargs->cgrp)
- kargs->kill_seq = kargs->cgrp->kill_seq;
- else
- kargs->kill_seq = cset->dfl_cgrp->kill_seq;
+ kargs->kill_seq = cset->dfl_cgrp->kill_seq;
spin_unlock_irq(&css_set_lock);
if (!(kargs->flags & CLONE_INTO_CGROUP)) {
@@ -6503,6 +6529,7 @@ static int cgroup_css_set_fork(struct kernel_clone_args *kargs)
put_css_set(cset);
fput(f);
kargs->cgrp = dst_cgrp;
+ kargs->kill_seq = dst_cgrp->kill_seq;
return ret;
err:
diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h
new file mode 100644
index 000000000000..8e5c275d730d
--- /dev/null
+++ b/kernel/cgroup/cpuset-internal.h
@@ -0,0 +1,222 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+
+#ifndef __CPUSET_INTERNAL_H
+#define __CPUSET_INTERNAL_H
+
+#include <linux/cpumask.h>
+#include <linux/spinlock.h>
+#include <linux/cpuset.h>
+#include <linux/cgroup.h>
+
+/* See "Frequency meter" comments, below. */
+
+struct fmeter {
+ int cnt; /* unprocessed events count */
+ int val; /* most recent output value */
+ time64_t time; /* clock (secs) when val computed */
+ spinlock_t lock; /* guards read or write of above */
+};
+
+/*
+ * Invalid partition error code
+ */
+enum prs_errcode {
+ PERR_NONE = 0,
+ PERR_INVCPUS,
+ PERR_INVPARENT,
+ PERR_NOTPART,
+ PERR_NOTEXCL,
+ PERR_NOCPUS,
+ PERR_HOTPLUG,
+ PERR_CPUSEMPTY,
+};
+
+/* bits in struct cpuset flags field */
+typedef enum {
+ CS_ONLINE,
+ CS_CPU_EXCLUSIVE,
+ CS_MEM_EXCLUSIVE,
+ CS_MEM_HARDWALL,
+ CS_MEMORY_MIGRATE,
+ CS_SCHED_LOAD_BALANCE,
+ CS_SPREAD_PAGE,
+ CS_SPREAD_SLAB,
+} cpuset_flagbits_t;
+
+/* The various types of files and directories in a cpuset file system */
+
+typedef enum {
+ FILE_MEMORY_MIGRATE,
+ FILE_CPULIST,
+ FILE_MEMLIST,
+ FILE_EFFECTIVE_CPULIST,
+ FILE_EFFECTIVE_MEMLIST,
+ FILE_SUBPARTS_CPULIST,
+ FILE_CPU_EXCLUSIVE,
+ FILE_MEM_EXCLUSIVE,
+ FILE_MEM_HARDWALL,
+ FILE_SCHED_LOAD_BALANCE,
+ FILE_PARTITION_ROOT,
+ FILE_SCHED_RELAX_DOMAIN_LEVEL,
+ FILE_MEMORY_PRESSURE_ENABLED,
+ FILE_MEMORY_PRESSURE,
+ FILE_SPREAD_PAGE,
+ FILE_SPREAD_SLAB,
+} cpuset_filetype_t;
+
+struct cpuset {
+ struct cgroup_subsys_state css;
+
+ unsigned long flags; /* "unsigned long" so bitops work */
+
+ /*
+ * On default hierarchy:
+ *
+ * The user-configured masks can only be changed by writing to
+ * cpuset.cpus and cpuset.mems, and won't be limited by the
+ * parent masks.
+ *
+ * The effective masks is the real masks that apply to the tasks
+ * in the cpuset. They may be changed if the configured masks are
+ * changed or hotplug happens.
+ *
+ * effective_mask == configured_mask & parent's effective_mask,
+ * and if it ends up empty, it will inherit the parent's mask.
+ *
+ *
+ * On legacy hierarchy:
+ *
+ * The user-configured masks are always the same with effective masks.
+ */
+
+ /* user-configured CPUs and Memory Nodes allow to tasks */
+ cpumask_var_t cpus_allowed;
+ nodemask_t mems_allowed;
+
+ /* effective CPUs and Memory Nodes allow to tasks */
+ cpumask_var_t effective_cpus;
+ nodemask_t effective_mems;
+
+ /*
+ * CPUs allocated to child sub-partitions (default hierarchy only)
+ * - CPUs granted by the parent = effective_cpus U subparts_cpus
+ * - effective_cpus and subparts_cpus are mutually exclusive.
+ *
+ * effective_cpus contains only onlined CPUs, but subparts_cpus
+ * may have offlined ones.
+ */
+ cpumask_var_t subparts_cpus;
+
+ /*
+ * This is old Memory Nodes tasks took on.
+ *
+ * - top_cpuset.old_mems_allowed is initialized to mems_allowed.
+ * - A new cpuset's old_mems_allowed is initialized when some
+ * task is moved into it.
+ * - old_mems_allowed is used in cpuset_migrate_mm() when we change
+ * cpuset.mems_allowed and have tasks' nodemask updated, and
+ * then old_mems_allowed is updated to mems_allowed.
+ */
+ nodemask_t old_mems_allowed;
+
+ struct fmeter fmeter; /* memory_pressure filter */
+
+ /*
+ * Tasks are being attached to this cpuset. Used to prevent
+ * zeroing cpus/mems_allowed between ->can_attach() and ->attach().
+ */
+ int attach_in_progress;
+
+ /* partition number for rebuild_sched_domains() */
+ int pn;
+
+ /* for custom sched domain */
+ int relax_domain_level;
+
+ /* number of CPUs in subparts_cpus */
+ int nr_subparts_cpus;
+
+ /* partition root state */
+ int partition_root_state;
+
+ /*
+ * Default hierarchy only:
+ * use_parent_ecpus - set if using parent's effective_cpus
+ * child_ecpus_count - # of children with use_parent_ecpus set
+ */
+ int use_parent_ecpus;
+ int child_ecpus_count;
+
+ /*
+ * number of SCHED_DEADLINE tasks attached to this cpuset, so that we
+ * know when to rebuild associated root domain bandwidth information.
+ */
+ atomic_t nr_deadline_tasks;
+ int nr_migrate_dl_tasks;
+ u64 sum_migrate_dl_bw;
+
+ /* Invalid partition error code, not lock protected */
+ enum prs_errcode prs_err;
+
+ /* Handle for cpuset.cpus.partition */
+ struct cgroup_file partition_file;
+};
+
+static inline struct cpuset *css_cs(struct cgroup_subsys_state *css)
+{
+ return css ? container_of(css, struct cpuset, css) : NULL;
+}
+
+/* Retrieve the cpuset for a task */
+static inline struct cpuset *task_cs(struct task_struct *task)
+{
+ return css_cs(task_css(task, cpuset_cgrp_id));
+}
+
+static inline struct cpuset *parent_cs(struct cpuset *cs)
+{
+ return css_cs(cs->css.parent);
+}
+
+/* convenient tests for these bits */
+static inline bool is_cpuset_online(struct cpuset *cs)
+{
+ return test_bit(CS_ONLINE, &cs->flags) && !css_is_dying(&cs->css);
+}
+
+static inline int is_cpu_exclusive(const struct cpuset *cs)
+{
+ return test_bit(CS_CPU_EXCLUSIVE, &cs->flags);
+}
+
+static inline int is_mem_exclusive(const struct cpuset *cs)
+{
+ return test_bit(CS_MEM_EXCLUSIVE, &cs->flags);
+}
+
+static inline int is_mem_hardwall(const struct cpuset *cs)
+{
+ return test_bit(CS_MEM_HARDWALL, &cs->flags);
+}
+
+static inline int is_sched_load_balance(const struct cpuset *cs)
+{
+ return test_bit(CS_SCHED_LOAD_BALANCE, &cs->flags);
+}
+
+static inline int is_memory_migrate(const struct cpuset *cs)
+{
+ return test_bit(CS_MEMORY_MIGRATE, &cs->flags);
+}
+
+static inline int is_spread_page(const struct cpuset *cs)
+{
+ return test_bit(CS_SPREAD_PAGE, &cs->flags);
+}
+
+static inline int is_spread_slab(const struct cpuset *cs)
+{
+ return test_bit(CS_SPREAD_SLAB, &cs->flags);
+}
+
+#endif /* __CPUSET_INTERNAL_H */
diff --git a/kernel/cgroup/cpuset-v1.c b/kernel/cgroup/cpuset-v1.c
new file mode 100644
index 000000000000..bdec4b196986
--- /dev/null
+++ b/kernel/cgroup/cpuset-v1.c
@@ -0,0 +1,3 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include "cpuset-internal.h"
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index 7123bb020a24..61736a4e105b 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -22,6 +22,7 @@
* distribution for more details.
*/
#include "cgroup-internal.h"
+#include "cpuset-internal.h"
#include <linux/cpu.h>
#include <linux/cpumask.h>
@@ -78,29 +79,6 @@ DEFINE_STATIC_KEY_FALSE(cpusets_enabled_key);
*/
DEFINE_STATIC_KEY_FALSE(cpusets_insane_config_key);
-/* See "Frequency meter" comments, below. */
-
-struct fmeter {
- int cnt; /* unprocessed events count */
- int val; /* most recent output value */
- time64_t time; /* clock (secs) when val computed */
- spinlock_t lock; /* guards read or write of above */
-};
-
-/*
- * Invalid partition error code
- */
-enum prs_errcode {
- PERR_NONE = 0,
- PERR_INVCPUS,
- PERR_INVPARENT,
- PERR_NOTPART,
- PERR_NOTEXCL,
- PERR_NOCPUS,
- PERR_HOTPLUG,
- PERR_CPUSEMPTY,
-};
-
static const char * const perr_strings[] = {
[PERR_INVCPUS] = "Invalid cpu list in cpuset.cpus",
[PERR_INVPARENT] = "Parent is an invalid partition root",
@@ -111,104 +89,6 @@ static const char * const perr_strings[] = {
[PERR_CPUSEMPTY] = "cpuset.cpus is empty",
};
-struct cpuset {
- struct cgroup_subsys_state css;
-
- unsigned long flags; /* "unsigned long" so bitops work */
-
- /*
- * On default hierarchy:
- *
- * The user-configured masks can only be changed by writing to
- * cpuset.cpus and cpuset.mems, and won't be limited by the
- * parent masks.
- *
- * The effective masks is the real masks that apply to the tasks
- * in the cpuset. They may be changed if the configured masks are
- * changed or hotplug happens.
- *
- * effective_mask == configured_mask & parent's effective_mask,
- * and if it ends up empty, it will inherit the parent's mask.
- *
- *
- * On legacy hierarchy:
- *
- * The user-configured masks are always the same with effective masks.
- */
-
- /* user-configured CPUs and Memory Nodes allow to tasks */
- cpumask_var_t cpus_allowed;
- nodemask_t mems_allowed;
-
- /* effective CPUs and Memory Nodes allow to tasks */
- cpumask_var_t effective_cpus;
- nodemask_t effective_mems;
-
- /*
- * CPUs allocated to child sub-partitions (default hierarchy only)
- * - CPUs granted by the parent = effective_cpus U subparts_cpus
- * - effective_cpus and subparts_cpus are mutually exclusive.
- *
- * effective_cpus contains only onlined CPUs, but subparts_cpus
- * may have offlined ones.
- */
- cpumask_var_t subparts_cpus;
-
- /*
- * This is old Memory Nodes tasks took on.
- *
- * - top_cpuset.old_mems_allowed is initialized to mems_allowed.
- * - A new cpuset's old_mems_allowed is initialized when some
- * task is moved into it.
- * - old_mems_allowed is used in cpuset_migrate_mm() when we change
- * cpuset.mems_allowed and have tasks' nodemask updated, and
- * then old_mems_allowed is updated to mems_allowed.
- */
- nodemask_t old_mems_allowed;
-
- struct fmeter fmeter; /* memory_pressure filter */
-
- /*
- * Tasks are being attached to this cpuset. Used to prevent
- * zeroing cpus/mems_allowed between ->can_attach() and ->attach().
- */
- int attach_in_progress;
-
- /* partition number for rebuild_sched_domains() */
- int pn;
-
- /* for custom sched domain */
- int relax_domain_level;
-
- /* number of CPUs in subparts_cpus */
- int nr_subparts_cpus;
-
- /* partition root state */
- int partition_root_state;
-
- /*
- * Default hierarchy only:
- * use_parent_ecpus - set if using parent's effective_cpus
- * child_ecpus_count - # of children with use_parent_ecpus set
- */
- int use_parent_ecpus;
- int child_ecpus_count;
-
- /*
- * number of SCHED_DEADLINE tasks attached to this cpuset, so that we
- * know when to rebuild associated root domain bandwidth information.
- */
- int nr_deadline_tasks;
- int nr_migrate_dl_tasks;
- u64 sum_migrate_dl_bw;
-
- /* Invalid partition error code, not lock protected */
- enum prs_errcode prs_err;
-
- /* Handle for cpuset.cpus.partition */
- struct cgroup_file partition_file;
-};
-
/*
* Partition root states:
*
@@ -238,87 +118,18 @@ struct tmpmasks {
cpumask_var_t new_cpus; /* For update_cpumasks_hier() */
};
-static inline struct cpuset *css_cs(struct cgroup_subsys_state *css)
-{
- return css ? container_of(css, struct cpuset, css) : NULL;
-}
-
-/* Retrieve the cpuset for a task */
-static inline struct cpuset *task_cs(struct task_struct *task)
-{
- return css_cs(task_css(task, cpuset_cgrp_id));
-}
-
-static inline struct cpuset *parent_cs(struct cpuset *cs)
-{
- return css_cs(cs->css.parent);
-}
-
void inc_dl_tasks_cs(struct task_struct *p)
{
struct cpuset *cs = task_cs(p);
- cs->nr_deadline_tasks++;
+ atomic_inc(&cs->nr_deadline_tasks);
}
void dec_dl_tasks_cs(struct task_struct *p)
{
struct cpuset *cs = task_cs(p);
- cs->nr_deadline_tasks--;
-}
-
-/* bits in struct cpuset flags field */
-typedef enum {
- CS_ONLINE,
- CS_CPU_EXCLUSIVE,
- CS_MEM_EXCLUSIVE,
- CS_MEM_HARDWALL,
- CS_MEMORY_MIGRATE,
- CS_SCHED_LOAD_BALANCE,
- CS_SPREAD_PAGE,
- CS_SPREAD_SLAB,
-} cpuset_flagbits_t;
-
-/* convenient tests for these bits */
-static inline bool is_cpuset_online(struct cpuset *cs)
-{
- return test_bit(CS_ONLINE, &cs->flags) && !css_is_dying(&cs->css);
-}
-
-static inline int is_cpu_exclusive(const struct cpuset *cs)
-{
- return test_bit(CS_CPU_EXCLUSIVE, &cs->flags);
-}
-
-static inline int is_mem_exclusive(const struct cpuset *cs)
-{
- return test_bit(CS_MEM_EXCLUSIVE, &cs->flags);
-}
-
-static inline int is_mem_hardwall(const struct cpuset *cs)
-{
- return test_bit(CS_MEM_HARDWALL, &cs->flags);
-}
-
-static inline int is_sched_load_balance(const struct cpuset *cs)
-{
- return test_bit(CS_SCHED_LOAD_BALANCE, &cs->flags);
-}
-
-static inline int is_memory_migrate(const struct cpuset *cs)
-{
- return test_bit(CS_MEMORY_MIGRATE, &cs->flags);
-}
-
-static inline int is_spread_page(const struct cpuset *cs)
-{
- return test_bit(CS_SPREAD_PAGE, &cs->flags);
-}
-
-static inline int is_spread_slab(const struct cpuset *cs)
-{
- return test_bit(CS_SPREAD_SLAB, &cs->flags);
+ atomic_dec(&cs->nr_deadline_tasks);
}
static inline int is_partition_valid(const struct cpuset *cs)
@@ -1114,7 +925,7 @@ static void dl_update_tasks_root_domain(struct cpuset *cs)
struct css_task_iter it;
struct task_struct *task;
- if (cs->nr_deadline_tasks == 0)
+ if (atomic_read(&cs->nr_deadline_tasks) == 0)
return;
css_task_iter_start(&cs->css, 0, &it);
@@ -2704,8 +2515,8 @@ static void cpuset_attach(struct cgroup_taskset *tset)
cs->old_mems_allowed = cpuset_attach_nodemask_to;
if (cs->nr_migrate_dl_tasks) {
- cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks;
- oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks;
+ atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks);
+ atomic_sub(cs->nr_migrate_dl_tasks, &oldcs->nr_deadline_tasks);
reset_migrate_dl_data(cs);
}
@@ -2716,27 +2527,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
mutex_unlock(&cpuset_mutex);
}
-/* The various types of files and directories in a cpuset file system */
-
-typedef enum {
- FILE_MEMORY_MIGRATE,
- FILE_CPULIST,
- FILE_MEMLIST,
- FILE_EFFECTIVE_CPULIST,
- FILE_EFFECTIVE_MEMLIST,
- FILE_SUBPARTS_CPULIST,
- FILE_CPU_EXCLUSIVE,
- FILE_MEM_EXCLUSIVE,
- FILE_MEM_HARDWALL,
- FILE_SCHED_LOAD_BALANCE,
- FILE_PARTITION_ROOT,
- FILE_SCHED_RELAX_DOMAIN_LEVEL,
- FILE_MEMORY_PRESSURE_ENABLED,
- FILE_MEMORY_PRESSURE,
- FILE_SPREAD_PAGE,
- FILE_SPREAD_SLAB,
-} cpuset_filetype_t;
-
static int cpuset_write_u64(struct cgroup_subsys_state *css, struct cftype *cft,
u64 val)
{
diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c
index 0ae90c15cad8..0660e860afa2 100644
--- a/kernel/cgroup/rstat.c
+++ b/kernel/cgroup/rstat.c
@@ -1,6 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-only
#include "cgroup-internal.h"
+#include <linux/cpumask.h>
#include <linux/sched/cputime.h>
#include <linux/bpf.h>
@@ -18,7 +19,7 @@ static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu)
}
/**
- * cgroup_rstat_updated - keep track of updated rstat_cpu
+ * __cgroup_rstat_updated - keep track of updated rstat_cpu
* @cgrp: target cgroup
* @cpu: cpu on which rstat_cpu was updated
*
@@ -26,7 +27,7 @@ static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu)
* rstat_cpu->updated_children list. See the comment on top of
* cgroup_rstat_cpu definition for details.
*/
-void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
+void __cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
{
raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu);
unsigned long flags;
@@ -73,6 +74,18 @@ void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
raw_spin_unlock_irqrestore(cpu_lock, flags);
}
+/*
+ * BPF-facing wrapper for __cgroup_rstat_updated(). Validate the caller-provided
+ * CPU before passing it to the internal rstat updater.
+ */
+void cgroup_rstat_updated(struct cgroup *cgrp, int cpu)
+{
+ if (unlikely(cpu < 0 || cpu >= nr_cpu_ids || !cpu_possible(cpu)))
+ return;
+
+ __cgroup_rstat_updated(cgrp, cpu);
+}
+
/**
* cgroup_rstat_cpu_pop_updated - iterate and dismantle rstat_cpu updated tree
* @pos: current position
@@ -402,7 +415,7 @@ static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp,
unsigned long flags)
{
u64_stats_update_end_irqrestore(&rstatc->bsync, flags);
- cgroup_rstat_updated(cgrp, smp_processor_id());
+ __cgroup_rstat_updated(cgrp, smp_processor_id());
put_cpu_ptr(rstatc);
}
diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c
index d4215739efc7..88ec1b5cb506 100644
--- a/kernel/dma/direct.c
+++ b/kernel/dma/direct.c
@@ -44,10 +44,11 @@ u64 dma_direct_get_required_mask(struct device *dev)
return (1ULL << (fls64(max_dma) - 1)) * 2 - 1;
}
-static gfp_t dma_direct_optimal_gfp_mask(struct device *dev, u64 dma_mask,
- u64 *phys_limit)
+static gfp_t dma_direct_optimal_gfp_mask(struct device *dev, u64 *phys_limit)
{
- u64 dma_limit = min_not_zero(dma_mask, dev->bus_dma_limit);
+ u64 dma_limit = min_not_zero(
+ dev->coherent_dma_mask,
+ dev->bus_dma_limit);
/*
* Optimistically try the zone that the physical address mask falls
@@ -126,8 +127,7 @@ static struct page *__dma_direct_alloc_pages(struct device *dev, size_t size,
if (is_swiotlb_for_alloc(dev))
return dma_direct_alloc_swiotlb(dev, size);
- gfp |= dma_direct_optimal_gfp_mask(dev, dev->coherent_dma_mask,
- &phys_limit);
+ gfp |= dma_direct_optimal_gfp_mask(dev, &phys_limit);
page = dma_alloc_contiguous(dev, size, gfp);
if (page) {
if (!dma_coherent_ok(dev, page_to_phys(page), size) ||
@@ -168,23 +168,21 @@ static bool dma_direct_use_pool(struct device *dev, gfp_t gfp)
return !gfpflags_allow_blocking(gfp) && !is_swiotlb_for_alloc(dev);
}
-static void *dma_direct_alloc_from_pool(struct device *dev, size_t size,
- dma_addr_t *dma_handle, gfp_t gfp)
+static struct page *dma_direct_alloc_from_pool(struct device *dev, size_t size,
+ dma_addr_t *dma_handle, void **cpu_addr, gfp_t gfp)
{
struct page *page;
- u64 phys_mask;
- void *ret;
+ u64 phys_limit;
if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_DMA_COHERENT_POOL)))
return NULL;
- gfp |= dma_direct_optimal_gfp_mask(dev, dev->coherent_dma_mask,
- &phys_mask);
- page = dma_alloc_from_pool(dev, size, &ret, gfp, dma_coherent_ok);
+ gfp |= dma_direct_optimal_gfp_mask(dev, &phys_limit);
+ page = dma_alloc_from_pool(dev, size, cpu_addr, gfp, dma_coherent_ok);
if (!page)
return NULL;
*dma_handle = phys_to_dma_direct(dev, page_to_phys(page));
- return ret;
+ return page;
}
static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size,
@@ -241,28 +239,26 @@ void *dma_direct_alloc(struct device *dev, size_t size,
dma_handle);
/*
- * Otherwise remap if the architecture is asking for it. But
- * given that remapping memory is a blocking operation we'll
- * instead have to dip into the atomic pools.
+ * Otherwise we require the architecture to either be able to
+ * mark arbitrary parts of the kernel direct mapping uncached,
+ * or remapped it uncached.
*/
+ set_uncached = IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED);
remap = IS_ENABLED(CONFIG_DMA_DIRECT_REMAP);
- if (remap) {
- if (dma_direct_use_pool(dev, gfp))
- return dma_direct_alloc_from_pool(dev, size,
- dma_handle, gfp);
- } else {
- if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED))
- return NULL;
- set_uncached = true;
- }
+ if (!set_uncached && !remap)
+ return NULL;
}
/*
- * Decrypting memory may block, so allocate the memory from the atomic
- * pools if we can't block.
+ * Remapping or decrypting memory may block, allocate the memory from
+ * the atomic pools instead if we aren't allowed block.
*/
- if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp))
- return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp);
+ if ((remap || force_dma_unencrypted(dev)) &&
+ dma_direct_use_pool(dev, gfp)) {
+ page = dma_direct_alloc_from_pool(dev, size, dma_handle,
+ &ret, gfp);
+ return page ? ret : NULL;
+ }
/* we always manually zero the memory once we are done */
page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true);
@@ -373,7 +369,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size,
void *ret;
if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp))
- return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp);
+ return dma_direct_alloc_from_pool(dev, size, dma_handle, &ret, gfp);
page = __dma_direct_alloc_pages(dev, size, gfp, false);
if (!page)
diff --git a/kernel/events/core.c b/kernel/events/core.c
index a0bb33854c4a..e256ad0bfa4c 100644
--- a/kernel/events/core.c
+++ b/kernel/events/core.c
@@ -2317,6 +2317,7 @@ event_sched_out(struct perf_event *event,
!event->pending_work &&
!task_work_add(current, &event->pending_task, TWA_RESUME)) {
event->pending_work = 1;
+ WARN_ON_ONCE(!atomic_long_inc_not_zero(&event->refcount));
} else {
local_dec(&event->ctx->nr_pending);
}
@@ -4344,7 +4345,8 @@ static void perf_event_enable_on_exec(int ctxn)
static void perf_remove_from_owner(struct perf_event *event);
static void perf_event_exit_event(struct perf_event *event,
- struct perf_event_context *ctx);
+ struct perf_event_context *ctx,
+ unsigned long detach_flags);
/*
* Removes all events from the current task that have been marked
@@ -4375,7 +4377,7 @@ static void perf_event_remove_on_exec(int ctxn)
modified = true;
- perf_event_exit_event(event, ctx);
+ perf_event_exit_event(event, ctx, DETACH_GROUP);
}
raw_spin_lock_irqsave(&ctx->lock, flags);
@@ -4800,6 +4802,7 @@ static void free_event_rcu(struct rcu_head *head)
if (event->ns)
put_pid_ns(event->ns);
perf_event_free_filter(event);
+ kfree(event->addr_filter_ranges);
kmem_cache_free(perf_event_cache, event);
}
@@ -5016,35 +5019,9 @@ static bool exclusive_event_installable(struct perf_event *event,
static void perf_addr_filters_splice(struct perf_event *event,
struct list_head *head);
-static void perf_pending_task_sync(struct perf_event *event)
-{
- struct callback_head *head = &event->pending_task;
-
- if (!event->pending_work)
- return;
- /*
- * If the task is queued to the current task's queue, we
- * obviously can't wait for it to complete. Simply cancel it.
- */
- if (task_work_cancel(current, head)) {
- event->pending_work = 0;
- local_dec(&event->ctx->nr_pending);
- return;
- }
-
- /*
- * All accesses related to the event are within the same
- * non-preemptible section in perf_pending_task(). The RCU
- * grace period before the event is freed will make sure all
- * those accesses are complete by then.
- */
- rcuwait_wait_event(&event->pending_work_wait, !event->pending_work, TASK_UNINTERRUPTIBLE);
-}
-
static void _free_event(struct perf_event *event)
{
irq_work_sync(&event->pending_irq);
- perf_pending_task_sync(event);
unaccount_event(event);
@@ -5072,7 +5049,6 @@ static void _free_event(struct perf_event *event)
perf_event_free_bpf_prog(event);
perf_addr_filters_splice(event, NULL);
- kfree(event->addr_filter_ranges);
if (event->destroy)
event->destroy(event);
@@ -5166,10 +5142,17 @@ static void perf_remove_from_owner(struct perf_event *event)
static void put_event(struct perf_event *event)
{
+ struct perf_event *parent;
+
if (!atomic_long_dec_and_test(&event->refcount))
return;
+ parent = event->parent;
_free_event(event);
+
+ /* Matches the refcount bump in inherit_event() */
+ if (parent)
+ put_event(parent);
}
/*
@@ -5253,11 +5236,6 @@ int perf_event_release_kernel(struct perf_event *event)
if (tmp == child) {
perf_remove_from_context(child, DETACH_GROUP);
list_move(&child->child_list, &free_list);
- /*
- * This matches the refcount bump in inherit_event();
- * this can't be the last reference.
- */
- put_event(event);
} else {
var = &ctx->refcount;
}
@@ -5283,7 +5261,8 @@ int perf_event_release_kernel(struct perf_event *event)
void *var = &child->ctx->refcount;
list_del(&child->child_list);
- free_event(child);
+ /* Last reference unless ->pending_task work is pending */
+ put_event(child);
/*
* Wake any perf_event_free_task() waiting for this event to be
@@ -5294,7 +5273,11 @@ int perf_event_release_kernel(struct perf_event *event)
}
no_ctx:
- put_event(event); /* Must be the 'last' reference */
+ /*
+ * Last reference unless ->pending_task work is pending on this event
+ * or any of its children.
+ */
+ put_event(event);
return 0;
}
EXPORT_SYMBOL_GPL(perf_event_release_kernel);
@@ -6212,7 +6195,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)
perf_pmu_output_stop(event);
/* now it's safe to free the pages */
- atomic_long_sub(rb->aux_nr_pages - rb->aux_mmap_locked, &mmap_user->locked_vm);
+ atomic_long_sub(rb->aux_nr_pages - rb->aux_mmap_locked, &mmap_user->perf_mlock);
atomic64_sub(rb->aux_mmap_locked, &vma->vm_mm->pinned_vm);
/* this has to be the last one */
@@ -6287,7 +6270,7 @@ static void perf_mmap_close(struct vm_area_struct *vma)
*/
atomic_long_sub((size >> PAGE_SHIFT) + 1 - mmap_locked,
- &mmap_user->locked_vm);
+ &mmap_user->perf_mlock);
atomic64_sub(mmap_locked, &vma->vm_mm->pinned_vm);
free_uid(mmap_user);
@@ -6447,11 +6430,11 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)
*/
user_lock_limit *= num_online_cpus();
- user_locked = atomic_long_read(&user->locked_vm);
+ user_locked = atomic_long_read(&user->perf_mlock);
/*
* sysctl_perf_event_mlock may have changed, so that
- * user->locked_vm > user_lock_limit
+ * user->perf_mlock > user_lock_limit
*/
if (user_locked > user_lock_limit)
user_locked = user_lock_limit;
@@ -6459,7 +6442,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)
if (user_locked > user_lock_limit) {
/*
- * charge locked_vm until it hits user_lock_limit;
+ * charge perf_mlock until it hits user_lock_limit;
* charge the rest from pinned_vm
*/
extra = user_locked - user_lock_limit;
@@ -6511,7 +6494,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma)
unlock:
if (!ret) {
- atomic_long_add(user_extra, &user->locked_vm);
+ atomic_long_add(user_extra, &user->perf_mlock);
atomic64_add(extra, &vma->vm_mm->pinned_vm);
atomic_inc(&event->mmap_count);
@@ -6595,18 +6578,20 @@ void perf_event_wakeup(struct perf_event *event)
static void perf_sigtrap(struct perf_event *event)
{
/*
- * We'd expect this to only occur if the irq_work is delayed and either
- * ctx->task or current has changed in the meantime. This can be the
- * case on architectures that do not implement arch_irq_work_raise().
+ * Both perf_pending_task() and perf_pending_irq() can race with the
+ * task exiting or exec-ing. We can determine if such a race has
+ * occurred by checking if perf_event_exit_task(), which will set
+ * ctx->task to TASK_TOMBSTONE, has already been called.
*/
- if (WARN_ON_ONCE(event->ctx->task != current))
+ if (event->ctx->task == TASK_TOMBSTONE)
return;
/*
- * Both perf_pending_task() and perf_pending_irq() can race with the
- * task exiting.
+ * We'd expect this to only occur if the irq_work is delayed and either
+ * ctx->task or current has changed in the meantime. This can be the
+ * case on architectures that do not implement arch_irq_work_raise().
*/
- if (current->flags & PF_EXITING)
+ if (WARN_ON_ONCE(event->ctx->task != current))
return;
send_sig_perf((void __user *)event->pending_addr,
@@ -6697,11 +6682,6 @@ static void perf_pending_task(struct callback_head *head)
struct perf_event *event = container_of(head, struct perf_event, pending_task);
int rctx;
- /*
- * All accesses to the event must belong to the same implicit RCU read-side
- * critical section as the ->pending_work reset. See comment in
- * perf_pending_task_sync().
- */
preempt_disable_notrace();
/*
* If we 'fail' here, that's OK, it means recursion is already disabled
@@ -6713,12 +6693,13 @@ static void perf_pending_task(struct callback_head *head)
event->pending_work = 0;
perf_sigtrap(event);
local_dec(&event->ctx->nr_pending);
- rcuwait_wake_up(&event->pending_work_wait);
}
if (rctx >= 0)
perf_swevent_put_recursion_context(rctx);
preempt_enable_notrace();
+
+ put_event(event);
}
#ifdef CONFIG_GUEST_PERF_EVENTS
@@ -11913,7 +11894,6 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
init_waitqueue_head(&event->waitq);
init_irq_work(&event->pending_irq, perf_pending_irq);
init_task_work(&event->pending_task, perf_pending_task);
- rcuwait_init(&event->pending_work_wait);
mutex_init(&event->mmap_mutex);
raw_spin_lock_init(&event->addr_filters.lock);
@@ -12078,8 +12058,6 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu,
put_callchain_buffers();
}
err_addr_filters:
- kfree(event->addr_filter_ranges);
-
err_per_task:
exclusive_event_destroy(event);
@@ -12456,7 +12434,7 @@ SYSCALL_DEFINE5(perf_event_open,
if (err)
return err;
- if (!attr.exclude_kernel) {
+ if (!attr.exclude_kernel || attr.text_poke) {
err = perf_allow_kernel(&attr);
if (err)
return err;
@@ -13065,10 +13043,10 @@ static void sync_child_event(struct perf_event *child_event)
}
static void
-perf_event_exit_event(struct perf_event *event, struct perf_event_context *ctx)
+perf_event_exit_event(struct perf_event *event, struct perf_event_context *ctx,
+ unsigned long detach_flags)
{
struct perf_event *parent_event = event->parent;
- unsigned long detach_flags = 0;
if (parent_event) {
/*
@@ -13083,7 +13061,7 @@ perf_event_exit_event(struct perf_event *event, struct perf_event_context *ctx)
* Do destroy all inherited groups, we don't care about those
* and being thorough is better.
*/
- detach_flags = DETACH_GROUP | DETACH_CHILD;
+ detach_flags |= DETACH_GROUP | DETACH_CHILD;
mutex_lock(&parent_event->child_mutex);
}
@@ -13098,8 +13076,7 @@ perf_event_exit_event(struct perf_event *event, struct perf_event_context *ctx)
* Kick perf_poll() for is_event_hup();
*/
perf_event_wakeup(parent_event);
- free_event(event);
- put_event(parent_event);
+ put_event(event);
return;
}
@@ -13163,7 +13140,7 @@ static void perf_event_exit_task_context(struct task_struct *child, int ctxn)
perf_event_task(child, child_ctx, 0);
list_for_each_entry_safe(child_event, next, &child_ctx->event_list, event_entry)
- perf_event_exit_event(child_event, child_ctx);
+ perf_event_exit_event(child_event, child_ctx, 0);
mutex_unlock(&child_ctx->mutex);
@@ -13219,13 +13196,11 @@ static void perf_free_event(struct perf_event *event,
list_del_init(&event->child_list);
mutex_unlock(&parent->child_mutex);
- put_event(parent);
-
raw_spin_lock_irq(&ctx->lock);
perf_group_detach(event);
list_del_event(event, ctx);
raw_spin_unlock_irq(&ctx->lock);
- free_event(event);
+ put_event(event);
}
/*
diff --git a/kernel/exit.c b/kernel/exit.c
index 413540df3e18..b3cfeba72dc0 100644
--- a/kernel/exit.c
+++ b/kernel/exit.c
@@ -1588,16 +1588,10 @@ static int do_wait_pid(struct wait_opts *wo)
return 0;
}
-static long do_wait(struct wait_opts *wo)
+static long __do_wait(struct wait_opts *wo)
{
- int retval;
-
- trace_sched_process_wait(wo->wo_pid);
+ long retval;
- init_waitqueue_func_entry(&wo->child_wait, child_wait_callback);
- wo->child_wait.private = current;
- add_wait_queue(¤t->signal->wait_chldexit, &wo->child_wait);
-repeat:
/*
* If there is nothing that can match our criteria, just get out.
* We will clear ->notask_error to zero if we see any child that
@@ -1609,24 +1603,23 @@ static long do_wait(struct wait_opts *wo)
(!wo->wo_pid || !pid_has_task(wo->wo_pid, wo->wo_type)))
goto notask;
- set_current_state(TASK_INTERRUPTIBLE);
read_lock(&tasklist_lock);
if (wo->wo_type == PIDTYPE_PID) {
retval = do_wait_pid(wo);
if (retval)
- goto end;
+ return retval;
} else {
struct task_struct *tsk = current;
do {
retval = do_wait_thread(wo, tsk);
if (retval)
- goto end;
+ return retval;
retval = ptrace_do_wait(wo, tsk);
if (retval)
- goto end;
+ return retval;
if (wo->wo_flags & __WNOTHREAD)
break;
@@ -1636,14 +1629,32 @@ static long do_wait(struct wait_opts *wo)
notask:
retval = wo->notask_error;
- if (!retval && !(wo->wo_flags & WNOHANG)) {
- retval = -ERESTARTSYS;
- if (!signal_pending(current)) {
- schedule();
- goto repeat;
- }
- }
-end:
+ if (!retval && !(wo->wo_flags & WNOHANG))
+ return -ERESTARTSYS;
+
+ return retval;
+}
+
+static long do_wait(struct wait_opts *wo)
+{
+ int retval;
+
+ trace_sched_process_wait(wo->wo_pid);
+
+ init_waitqueue_func_entry(&wo->child_wait, child_wait_callback);
+ wo->child_wait.private = current;
+ add_wait_queue(¤t->signal->wait_chldexit, &wo->child_wait);
+
+ do {
+ set_current_state(TASK_INTERRUPTIBLE | TASK_FREEZABLE);
+ retval = __do_wait(wo);
+ if (retval != -ERESTARTSYS)
+ break;
+ if (signal_pending(current))
+ break;
+ schedule();
+ } while (1);
+
__set_current_state(TASK_RUNNING);
remove_wait_queue(¤t->signal->wait_chldexit, &wo->child_wait);
return retval;
diff --git a/kernel/fork.c b/kernel/fork.c
index 39b8483e0700..37c5d6123cf6 100644
--- a/kernel/fork.c
+++ b/kernel/fork.c
@@ -669,6 +669,24 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
} else if (anon_vma_fork(tmp, mpnt))
goto fail_nomem_anon_vma_fork;
tmp->vm_flags &= ~(VM_LOCKED | VM_LOCKONFAULT);
+ /*
+ * Copy/update hugetlb private vma information.
+ */
+ if (is_vm_hugetlb_page(tmp))
+ hugetlb_dup_vma_private(tmp);
+
+ /* Link the vma into the MT */
+ mas.index = tmp->vm_start;
+ mas.last = tmp->vm_end - 1;
+ mas_store(&mas, tmp);
+ if (mas_is_err(&mas))
+ goto fail_nomem_mas_store;
+
+ mm->map_count++;
+
+ if (tmp->vm_ops && tmp->vm_ops->open)
+ tmp->vm_ops->open(tmp);
+
file = tmp->vm_file;
if (file) {
struct address_space *mapping = file->f_mapping;
@@ -685,26 +703,9 @@ static __latent_entropy int dup_mmap(struct mm_struct *mm,
i_mmap_unlock_write(mapping);
}
- /*
- * Copy/update hugetlb private vma information.
- */
- if (is_vm_hugetlb_page(tmp))
- hugetlb_dup_vma_private(tmp);
-
- /* Link the vma into the MT */
- mas.index = tmp->vm_start;
- mas.last = tmp->vm_end - 1;
- mas_store(&mas, tmp);
- if (mas_is_err(&mas))
- goto fail_nomem_mas_store;
-
- mm->map_count++;
if (!(tmp->vm_flags & VM_WIPEONFORK))
retval = copy_page_range(tmp, mpnt);
- if (tmp->vm_ops && tmp->vm_ops->open)
- tmp->vm_ops->open(tmp);
-
if (retval)
goto loop_out;
}
@@ -1158,6 +1159,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p,
#endif
mm_init_uprobes_state(mm);
hugetlb_count_init(mm);
+#ifdef CONFIG_FUTEX
+ atomic64_set(&mm->futex_unique_id, 0);
+#endif
if (current->mm) {
mm->flags = current->mm->flags & MMF_INIT_MASK;
diff --git a/kernel/futex/core.c b/kernel/futex/core.c
index b370fdea7c21..5944e5728506 100644
--- a/kernel/futex/core.c
+++ b/kernel/futex/core.c
@@ -151,39 +151,27 @@ futex_setup_timer(ktime_t *time, struct hrtimer_sleeper *timeout,
}
/*
- * Generate a machine wide unique identifier for this inode.
+ * Generate a machine wide unique identifier for this object (inode or mm).
*
* This relies on u64 not wrapping in the life-time of the machine; which with
* 1ns resolution means almost 585 years.
- *
- * This further relies on the fact that a well formed program will not unmap
- * the file while it has a (shared) futex waiting on it. This mapping will have
- * a file reference which pins the mount and inode.
- *
- * If for some reason an inode gets evicted and read back in again, it will get
- * a new sequence number and will _NOT_ match, even though it is the exact same
- * file.
- *
- * It is important that futex_match() will never have a false-positive, esp.
- * for PI futexes that can mess up the state. The above argues that false-negatives
- * are only possible for malformed programs.
*/
-static u64 get_inode_sequence_number(struct inode *inode)
+static u64 get_object_id(atomic64_t *object_seq)
{
- static atomic64_t i_seq;
+ static atomic64_t last_assigned_seq;
u64 old;
- /* Does the inode already have a sequence number? */
- old = atomic64_read(&inode->i_sequence);
+ /* Does the object already have a sequence number? */
+ old = atomic64_read(object_seq);
if (likely(old))
return old;
for (;;) {
- u64 new = atomic64_add_return(1, &i_seq);
+ u64 new = atomic64_add_return(1, &last_assigned_seq);
if (WARN_ON_ONCE(!new))
continue;
- old = atomic64_cmpxchg_relaxed(&inode->i_sequence, 0, new);
+ old = atomic64_cmpxchg_relaxed(object_seq, 0, new);
if (old)
return old;
return new;
@@ -206,7 +194,7 @@ static u64 get_inode_sequence_number(struct inode *inode)
*
* ( inode->i_sequence, page->index, offset_within_page )
*
- * [ also see get_inode_sequence_number() ]
+ * [ also see get_object_id() ]
*
* For private mappings (or when !@fshared), the key is:
*
@@ -358,8 +346,8 @@ int get_futex_key(u32 __user *uaddr, bool fshared, union futex_key *key,
goto out;
}
- key->both.offset |= FUT_OFF_MMSHARED; /* ref taken on mm */
- key->private.mm = mm;
+ key->both.offset |= FUT_OFF_MMSHARED;
+ key->private.mm_seq = get_object_id(&mm->futex_unique_id);
key->private.address = address;
} else {
@@ -394,7 +382,22 @@ int get_futex_key(u32 __user *uaddr, bool fshared, union futex_key *key,
}
key->both.offset |= FUT_OFF_INODE; /* inode-based key */
- key->shared.i_seq = get_inode_sequence_number(inode);
+ /*
+ * This relies on the fact that a well formed program will not
+ * unmap the file while it has a (shared) futex waiting on it.
+ * This mapping will have a file reference which pins the mount
+ * and inode.
+ *
+ * If for some reason an inode gets evicted and read back in
+ * again, it will get a new sequence number and will _NOT_ match,
+ * even though it is the exact same file.
+ *
+ * It is important that futex_match() will never have a
+ * false-positive, esp. for PI futexes that can mess up the state.
+ * The above argues that false-negatives are only possible for
+ * malformed programs.
+ */
+ key->shared.i_seq = get_object_id(&inode->i_sequence);
key->shared.pgoff = page_to_pgoff(tail);
rcu_read_unlock();
}
diff --git a/kernel/kprobes.c b/kernel/kprobes.c
index bc0ebb9cf375..58c04d6d1b37 100644
--- a/kernel/kprobes.c
+++ b/kernel/kprobes.c
@@ -491,14 +491,16 @@ static bool kprobe_queued(struct kprobe *p)
static struct kprobe *get_optimized_kprobe(kprobe_opcode_t *addr)
{
int i;
- struct kprobe *p = NULL;
+ struct kprobe *p;
struct optimized_kprobe *op;
/* Don't check i == 0, since that is a breakpoint case. */
- for (i = 1; !p && i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++)
+ for (i = 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++) {
p = get_kprobe(addr - i);
+ /* A disabled probe can have prepared, but inactive, optinsns. */
+ if (!p || !kprobe_optready(p) || kprobe_disarmed(p))
+ continue;
- if (p && kprobe_optready(p)) {
op = container_of(p, struct optimized_kprobe, kp);
if (arch_within_optimized_kprobe(op, addr))
return p;
diff --git a/kernel/params.c b/kernel/params.c
index 82109c0cf991..1b699c763259 100644
--- a/kernel/params.c
+++ b/kernel/params.c
@@ -1,18 +1,20 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* Helpers for initial module or kernel cmdline parsing
- Copyright (C) 2001 Rusty Russell.
-
-*/
-#include <linux/kernel.h>
-#include <linux/string.h>
+/*
+ * Helpers for initial module or kernel cmdline parsing
+ * Copyright (C) 2001 Rusty Russell.
+ */
+#include <linux/ctype.h>
+#include <linux/device.h>
+#include <linux/err.h>
#include <linux/errno.h>
+#include <linux/kernel.h>
+#include <linux/kstrtox.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
-#include <linux/device.h>
-#include <linux/err.h>
-#include <linux/slab.h>
-#include <linux/ctype.h>
+#include <linux/overflow.h>
#include <linux/security.h>
+#include <linux/slab.h>
+#include <linux/string.h>
#ifdef CONFIG_SYSFS
/* Protects all built-in parameters, modules use their own param_lock */
@@ -47,7 +49,7 @@ static void *kmalloc_parameter(unsigned int size)
{
struct kmalloced_param *p;
- p = kmalloc(sizeof(*p) + size, GFP_KERNEL);
+ p = kmalloc(size_add(sizeof(*p), size), GFP_KERNEL);
if (!p)
return NULL;
@@ -263,22 +265,29 @@ EXPORT_SYMBOL_GPL(param_set_uint_minmax);
int param_set_charp(const char *val, const struct kernel_param *kp)
{
- if (strlen(val) > 1024) {
+ char *tmp;
+ size_t len, maxlen = 1024;
+
+ len = strnlen(val, maxlen + 1);
+ if (len == maxlen + 1) {
pr_err("%s: string parameter too long\n", kp->name);
return -ENOSPC;
}
- maybe_kfree_parameter(*(char **)kp->arg);
-
- /* This is a hack. We can't kmalloc in early boot, and we
- * don't need to; this mangled commandline is preserved. */
+ /*
+ * This is a hack. We can't kmalloc() in early boot, and we
+ * don't need to; this mangled commandline is preserved.
+ */
if (slab_is_available()) {
- *(char **)kp->arg = kmalloc_parameter(strlen(val)+1);
- if (!*(char **)kp->arg)
+ tmp = kmalloc_parameter(len + 1);
+ if (!tmp)
return -ENOMEM;
- strcpy(*(char **)kp->arg, val);
+ memcpy(tmp, val, len + 1);
} else
- *(const char **)kp->arg = val;
+ tmp = (char *)val;
+
+ maybe_kfree_parameter(*(char **)kp->arg);
+ *(char **)kp->arg = tmp;
return 0;
}
@@ -310,7 +319,7 @@ int param_set_bool(const char *val, const struct kernel_param *kp)
if (!val) val = "1";
/* One of =[yYnN01] */
- return strtobool(val, kp->arg);
+ return kstrtobool(val, kp->arg);
}
EXPORT_SYMBOL(param_set_bool);
@@ -511,7 +520,7 @@ int param_set_copystring(const char *val, const struct kernel_param *kp)
{
const struct kparam_string *kps = kp->str;
- if (strlen(val)+1 > kps->maxlen) {
+ if (strnlen(val, kps->maxlen) == kps->maxlen) {
pr_err("%s: string doesn't fit in %u chars.\n",
kp->name, kps->maxlen-1);
return -ENOSPC;
@@ -736,8 +745,10 @@ void module_param_sysfs_remove(struct module *mod)
{
if (mod->mkobj.mp) {
sysfs_remove_group(&mod->mkobj.kobj, &mod->mkobj.mp->grp);
- /* We are positive that no one is using any param
- * attrs at this point. Deallocate immediately. */
+ /*
+ * We are positive that no one is using any param
+ * attrs at this point. Deallocate immediately.
+ */
free_module_param_attrs(&mod->mkobj);
}
}
diff --git a/kernel/pid_namespace.c b/kernel/pid_namespace.c
index a575fabf697e..084f43e4f715 100644
--- a/kernel/pid_namespace.c
+++ b/kernel/pid_namespace.c
@@ -212,9 +212,10 @@ void zap_pid_ns_processes(struct pid_namespace *pid_ns)
* kernel_wait4() will also block until our children traced from the
* parent namespace are detached and become EXIT_DEAD.
*/
+ /* Task work must not busy-loop the reaper, see signal_pending(). */
+ guard(no_notify_signal)();
do {
clear_thread_flag(TIF_SIGPENDING);
- clear_thread_flag(TIF_NOTIFY_SIGNAL);
rc = kernel_wait4(-1, NULL, __WALL, NULL);
} while (rc != -ECHILD);
@@ -242,7 +243,7 @@ void zap_pid_ns_processes(struct pid_namespace *pid_ns)
* free_pid() will awaken this task.
*/
for (;;) {
- set_current_state(TASK_INTERRUPTIBLE);
+ set_current_state(TASK_IDLE | TASK_FREEZABLE);
if (pid_ns->pid_allocated == init_pids)
break;
/*
diff --git a/kernel/rcu/tasks.h b/kernel/rcu/tasks.h
index bb6b037ef30f..a7c86baebce8 100644
--- a/kernel/rcu/tasks.h
+++ b/kernel/rcu/tasks.h
@@ -31,6 +31,7 @@ typedef void (*postgp_func_t)(struct rcu_tasks *rtp);
* @barrier_q_head: RCU callback for barrier operation.
* @rtp_blkd_tasks: List of tasks blocked as readers.
* @cpu: CPU number corresponding to this entry.
+ * @index: Index of this CPU in rtpcp_array of the rcu_tasks structure.
* @rtpp: Pointer to the rcu_tasks structure.
*/
struct rcu_tasks_percpu {
@@ -43,6 +44,7 @@ struct rcu_tasks_percpu {
struct rcu_head barrier_q_head;
struct list_head rtp_blkd_tasks;
int cpu;
+ int index;
struct rcu_tasks *rtpp;
};
@@ -68,6 +70,7 @@ struct rcu_tasks_percpu {
* @postgp_func: This flavor's post-grace-period function (optional).
* @call_func: This flavor's call_rcu()-equivalent function.
* @rtpcpu: This flavor's rcu_tasks_percpu structure.
+ * @rtpcp_array: Array of pointers to rcu_tasks_percpu structure of CPUs in cpu_possible_mask.
* @percpu_enqueue_shift: Shift down CPU ID this much when enqueuing callbacks.
* @percpu_enqueue_lim: Number of per-CPU callback queues in use for enqueuing.
* @percpu_dequeue_lim: Number of per-CPU callback queues in use for dequeuing.
@@ -100,6 +103,7 @@ struct rcu_tasks {
postgp_func_t postgp_func;
call_rcu_func_t call_func;
struct rcu_tasks_percpu __percpu *rtpcpu;
+ struct rcu_tasks_percpu **rtpcp_array;
int percpu_enqueue_shift;
int percpu_enqueue_lim;
int percpu_dequeue_lim;
@@ -164,6 +168,8 @@ module_param(rcu_task_contend_lim, int, 0444);
static int rcu_task_collapse_lim __read_mostly = 10;
module_param(rcu_task_collapse_lim, int, 0444);
+static int rcu_task_cpu_ids;
+
/* RCU tasks grace-period state for debugging. */
#define RTGS_INIT 0
#define RTGS_WAIT_WAIT_CBS 1
@@ -228,6 +234,13 @@ static void cblist_init_generic(struct rcu_tasks *rtp)
unsigned long flags;
int lim;
int shift;
+ int maxcpu;
+ int index = 0;
+
+ if (!rtp->rtpcp_array) {
+ rtp->rtpcp_array = kcalloc(num_possible_cpus(), sizeof(struct rcu_tasks_percpu *), GFP_ATOMIC);
+ BUG_ON(!rtp->rtpcp_array);
+ }
raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
if (rcu_task_enqueue_lim < 0) {
@@ -238,14 +251,6 @@ static void cblist_init_generic(struct rcu_tasks *rtp)
}
lim = rcu_task_enqueue_lim;
- if (lim > nr_cpu_ids)
- lim = nr_cpu_ids;
- shift = ilog2(nr_cpu_ids / lim);
- if (((nr_cpu_ids - 1) >> shift) >= lim)
- shift++;
- WRITE_ONCE(rtp->percpu_enqueue_shift, shift);
- WRITE_ONCE(rtp->percpu_dequeue_lim, lim);
- smp_store_release(&rtp->percpu_enqueue_lim, lim);
for_each_possible_cpu(cpu) {
struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rtp->rtpcpu, cpu);
@@ -258,16 +263,31 @@ static void cblist_init_generic(struct rcu_tasks *rtp)
INIT_WORK(&rtpcp->rtp_work, rcu_tasks_invoke_cbs_wq);
rtpcp->cpu = cpu;
rtpcp->rtpp = rtp;
+ rtpcp->index = index;
+ rtp->rtpcp_array[index] = rtpcp;
+ index++;
if (!rtpcp->rtp_blkd_tasks.next)
INIT_LIST_HEAD(&rtpcp->rtp_blkd_tasks);
+ maxcpu = cpu;
raw_spin_unlock_rcu_node(rtpcp); // irqs remain disabled.
}
+
+ rcu_task_cpu_ids = maxcpu + 1;
+ if (lim > rcu_task_cpu_ids)
+ lim = rcu_task_cpu_ids;
+ shift = ilog2(rcu_task_cpu_ids / lim);
+ if (((rcu_task_cpu_ids - 1) >> shift) >= lim)
+ shift++;
+ WRITE_ONCE(rtp->percpu_enqueue_shift, shift);
+ WRITE_ONCE(rtp->percpu_dequeue_lim, lim);
+ smp_store_release(&rtp->percpu_enqueue_lim, lim);
raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
if (rcu_task_cb_adjust)
pr_info("%s: Setting adjustable number of callback queues.\n", __func__);
- pr_info("%s: Setting shift to %d and lim to %d.\n", __func__, data_race(rtp->percpu_enqueue_shift), data_race(rtp->percpu_enqueue_lim));
+ pr_info("%s: Setting shift to %d and lim to %d rcu_task_cpu_ids=%d.\n", __func__,
+ data_race(rtp->percpu_enqueue_shift), data_race(rtp->percpu_enqueue_lim), rcu_task_cpu_ids);
}
// IRQ-work handler that does deferred wakeup for call_rcu_tasks_generic().
@@ -307,7 +327,7 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
rtpcp->rtp_n_lock_retries = 0;
}
if (rcu_task_cb_adjust && ++rtpcp->rtp_n_lock_retries > rcu_task_contend_lim &&
- READ_ONCE(rtp->percpu_enqueue_lim) != nr_cpu_ids)
+ READ_ONCE(rtp->percpu_enqueue_lim) != rcu_task_cpu_ids)
needadjust = true; // Defer adjustment to avoid deadlock.
}
if (!rcu_segcblist_is_enabled(&rtpcp->cblist)) {
@@ -320,10 +340,10 @@ static void call_rcu_tasks_generic(struct rcu_head *rhp, rcu_callback_t func,
raw_spin_unlock_irqrestore_rcu_node(rtpcp, flags);
if (unlikely(needadjust)) {
raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
- if (rtp->percpu_enqueue_lim != nr_cpu_ids) {
+ if (rtp->percpu_enqueue_lim != rcu_task_cpu_ids) {
WRITE_ONCE(rtp->percpu_enqueue_shift, 0);
- WRITE_ONCE(rtp->percpu_dequeue_lim, nr_cpu_ids);
- smp_store_release(&rtp->percpu_enqueue_lim, nr_cpu_ids);
+ WRITE_ONCE(rtp->percpu_dequeue_lim, rcu_task_cpu_ids);
+ smp_store_release(&rtp->percpu_enqueue_lim, rcu_task_cpu_ids);
pr_info("Switching %s to per-CPU callback queuing.\n", rtp->name);
}
raw_spin_unlock_irqrestore(&rtp->cbs_gbl_lock, flags);
@@ -396,6 +416,8 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)
for (cpu = 0; cpu < smp_load_acquire(&rtp->percpu_dequeue_lim); cpu++) {
struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rtp->rtpcpu, cpu);
+ if (!cpu_possible(cpu))
+ continue;
/* Advance and accelerate any new callbacks. */
if (!rcu_segcblist_n_cbs(&rtpcp->cblist))
continue;
@@ -426,7 +448,7 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)
if (rcu_task_cb_adjust && ncbs <= rcu_task_collapse_lim) {
raw_spin_lock_irqsave(&rtp->cbs_gbl_lock, flags);
if (rtp->percpu_enqueue_lim > 1) {
- WRITE_ONCE(rtp->percpu_enqueue_shift, order_base_2(nr_cpu_ids));
+ WRITE_ONCE(rtp->percpu_enqueue_shift, order_base_2(rcu_task_cpu_ids));
smp_store_release(&rtp->percpu_enqueue_lim, 1);
rtp->percpu_dequeue_gpseq = get_state_synchronize_rcu();
gpdone = false;
@@ -441,7 +463,9 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)
pr_info("Completing switch %s to CPU-0 callback queuing.\n", rtp->name);
}
if (rtp->percpu_dequeue_lim == 1) {
- for (cpu = rtp->percpu_dequeue_lim; cpu < nr_cpu_ids; cpu++) {
+ for (cpu = rtp->percpu_dequeue_lim; cpu < rcu_task_cpu_ids; cpu++) {
+ if (!cpu_possible(cpu))
+ continue;
struct rcu_tasks_percpu *rtpcp = per_cpu_ptr(rtp->rtpcpu, cpu);
WARN_ON_ONCE(rcu_segcblist_n_cbs(&rtpcp->cblist));
@@ -456,30 +480,32 @@ static int rcu_tasks_need_gpcb(struct rcu_tasks *rtp)
// Advance callbacks and invoke any that are ready.
static void rcu_tasks_invoke_cbs(struct rcu_tasks *rtp, struct rcu_tasks_percpu *rtpcp)
{
- int cpu;
- int cpunext;
int cpuwq;
unsigned long flags;
int len;
+ int index;
struct rcu_head *rhp;
struct rcu_cblist rcl = RCU_CBLIST_INITIALIZER(rcl);
struct rcu_tasks_percpu *rtpcp_next;
- cpu = rtpcp->cpu;
- cpunext = cpu * 2 + 1;
- if (cpunext < smp_load_acquire(&rtp->percpu_dequeue_lim)) {
- rtpcp_next = per_cpu_ptr(rtp->rtpcpu, cpunext);
- cpuwq = rcu_cpu_beenfullyonline(cpunext) ? cpunext : WORK_CPU_UNBOUND;
- queue_work_on(cpuwq, system_wq, &rtpcp_next->rtp_work);
- cpunext++;
- if (cpunext < smp_load_acquire(&rtp->percpu_dequeue_lim)) {
- rtpcp_next = per_cpu_ptr(rtp->rtpcpu, cpunext);
- cpuwq = rcu_cpu_beenfullyonline(cpunext) ? cpunext : WORK_CPU_UNBOUND;
+ index = rtpcp->index * 2 + 1;
+ if (index < num_possible_cpus()) {
+ rtpcp_next = rtp->rtpcp_array[index];
+ if (rtpcp_next->cpu < smp_load_acquire(&rtp->percpu_dequeue_lim)) {
+ cpuwq = rcu_cpu_beenfullyonline(rtpcp_next->cpu) ? rtpcp_next->cpu : WORK_CPU_UNBOUND;
queue_work_on(cpuwq, system_wq, &rtpcp_next->rtp_work);
+ index++;
+ if (index < num_possible_cpus()) {
+ rtpcp_next = rtp->rtpcp_array[index];
+ if (rtpcp_next->cpu < smp_load_acquire(&rtp->percpu_dequeue_lim)) {
+ cpuwq = rcu_cpu_beenfullyonline(rtpcp_next->cpu) ? rtpcp_next->cpu : WORK_CPU_UNBOUND;
+ queue_work_on(cpuwq, system_wq, &rtpcp_next->rtp_work);
+ }
+ }
}
}
- if (rcu_segcblist_empty(&rtpcp->cblist) || !cpu_possible(cpu))
+ if (rcu_segcblist_empty(&rtpcp->cblist))
return;
raw_spin_lock_irqsave_rcu_node(rtpcp, flags);
rcu_segcblist_advance(&rtpcp->cblist, rcu_seq_current(&rtp->tasks_gp_seq));
diff --git a/kernel/rcu/tree.h b/kernel/rcu/tree.h
index aa16d3cd62ba..e5f74554e3e6 100644
--- a/kernel/rcu/tree.h
+++ b/kernel/rcu/tree.h
@@ -158,6 +158,17 @@ union rcu_noqs {
u16 s; /* Set of bits, aggregate OR here. */
};
+/*
+ * An IRQ work (deferred_qs_iw) is used by RCU to get the scheduler's attention.
+ * to report quiescent states at the soonest possible time.
+ * The request can be in one of the following states:
+ * - DEFER_QS_IDLE: An IRQ work is yet to be scheduled.
+ * - DEFER_QS_PENDING: An IRQ work was scheduled but either not yet run, or it
+ * ran and we still haven't reported a quiescent state.
+ */
+#define DEFER_QS_IDLE 0
+#define DEFER_QS_PENDING 1
+
/* Per-CPU data for read-copy update. */
struct rcu_data {
/* 1) quiescent-state and grace-period handling : */
@@ -175,7 +186,7 @@ struct rcu_data {
/* during and after the last grace */
/* period it is aware of. */
struct irq_work defer_qs_iw; /* Obtain later scheduler attention. */
- bool defer_qs_iw_pending; /* Scheduler attention pending? */
+ int defer_qs_pending; /* irqwork or softirq pending? */
struct work_struct strict_work; /* Schedule readers for strict GPs. */
/* 2) batch handling */
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 4cb5c901a204..9958f73d7ee1 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -474,13 +474,16 @@ rcu_preempt_deferred_qs_irqrestore(struct task_struct *t, unsigned long flags)
struct rcu_node *rnp;
union rcu_special special;
+ rdp = this_cpu_ptr(&rcu_data);
+ if (rdp->defer_qs_pending == DEFER_QS_PENDING)
+ rdp->defer_qs_pending = DEFER_QS_IDLE;
+
/*
* If RCU core is waiting for this CPU to exit its critical section,
* report the fact that it has exited. Because irqs are disabled,
* t->rcu_read_unlock_special cannot change.
*/
special = t->rcu_read_unlock_special;
- rdp = this_cpu_ptr(&rcu_data);
if (!special.s && !rdp->cpu_no_qs.b.exp) {
local_irq_restore(flags);
return;
@@ -617,7 +620,23 @@ static void rcu_preempt_deferred_qs_handler(struct irq_work *iwp)
rdp = container_of(iwp, struct rcu_data, defer_qs_iw);
local_irq_save(flags);
- rdp->defer_qs_iw_pending = false;
+
+ /*
+ * If the IRQ work handler happens to run in the middle of RCU read-side
+ * critical section, it could be ineffective in getting the scheduler's
+ * attention to report a deferred quiescent state (the whole point of the
+ * IRQ work). For this reason, requeue the IRQ work.
+ *
+ * Basically, we want to avoid following situation:
+ * 1. rcu_read_unlock() queues IRQ work (state -> DEFER_QS_PENDING)
+ * 2. CPU enters new rcu_read_lock()
+ * 3. IRQ work runs but cannot report QS due to rcu_preempt_depth() > 0
+ * 4. rcu_read_unlock() does not re-queue work (state still PENDING)
+ * 5. Deferred QS reporting does not happen.
+ */
+ if (rcu_preempt_depth() > 0)
+ WRITE_ONCE(rdp->defer_qs_pending, DEFER_QS_IDLE);
+
local_irq_restore(flags);
}
@@ -655,7 +674,10 @@ static void rcu_read_unlock_special(struct task_struct *t)
// Using softirq, safe to awaken, and either the
// wakeup is free or there is either an expedited
// GP in flight or a potential need to deboost.
- raise_softirq_irqoff(RCU_SOFTIRQ);
+ if (rdp->defer_qs_pending != DEFER_QS_PENDING) {
+ rdp->defer_qs_pending = DEFER_QS_PENDING;
+ raise_softirq_irqoff(RCU_SOFTIRQ);
+ }
} else {
// Enabling BH or preempt does reschedule, so...
// Also if no expediting and no possible deboosting,
@@ -664,17 +686,13 @@ static void rcu_read_unlock_special(struct task_struct *t)
set_tsk_need_resched(current);
set_preempt_need_resched();
if (IS_ENABLED(CONFIG_IRQ_WORK) && irqs_were_disabled &&
- expboost && !rdp->defer_qs_iw_pending && cpu_online(rdp->cpu)) {
+ expboost && rdp->defer_qs_pending != DEFER_QS_PENDING &&
+ cpu_online(rdp->cpu)) {
// Get scheduler to re-evaluate and call hooks.
// If !IRQ_WORK, FQS scan will eventually IPI.
- if (IS_ENABLED(CONFIG_RCU_STRICT_GRACE_PERIOD) &&
- IS_ENABLED(CONFIG_PREEMPT_RT))
- rdp->defer_qs_iw = IRQ_WORK_INIT_HARD(
- rcu_preempt_deferred_qs_handler);
- else
- init_irq_work(&rdp->defer_qs_iw,
- rcu_preempt_deferred_qs_handler);
- rdp->defer_qs_iw_pending = true;
+ rdp->defer_qs_iw =
+ IRQ_WORK_INIT_HARD(rcu_preempt_deferred_qs_handler);
+ rdp->defer_qs_pending = DEFER_QS_PENDING;
irq_work_queue_on(&rdp->defer_qs_iw, rdp->cpu);
}
}
diff --git a/kernel/sched/core.c b/kernel/sched/core.c
index fa9008e5d8bd..cf470ac63af7 100644
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -345,6 +345,17 @@ static void __sched_core_flip(bool enabled)
sched_core_lock(cpu, &flags);
+ /*
+ * A core-wide selection may have the shared rq lock temporarily
+ * released by a lock-dropping ->pick_task(). Flipping would
+ * rebind rq_lockp() under it. Wait it out.
+ */
+ while (cpu_rq(cpu)->core->core_pick_in_flight) {
+ sched_core_unlock(cpu, &flags);
+ cpu_relax();
+ sched_core_lock(cpu, &flags);
+ }
+
for_each_cpu(t, smt_mask)
cpu_rq(t)->core_enabled = enabled;
@@ -5959,6 +5970,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
return __pick_next_task(rq, prev, rf);
}
+ rq->core->core_pick_in_flight++;
+
/*
* If there were no {en,de}queues since we picked (IOW, the task
* pointers are all still valid), and we haven't scheduled the last
@@ -6152,6 +6165,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
out_set_next:
set_next_task(rq, next);
out:
+ rq->core->core_pick_in_flight--;
if (rq->core->core_forceidle_count && next == rq->idle)
queue_core_balance(rq);
@@ -6344,6 +6358,13 @@ static void sched_core_cpu_deactivate(unsigned int cpu)
core_rq->core_forceidle_seq = rq->core_forceidle_seq;
core_rq->core_forceidle_occupation = rq->core_forceidle_occupation;
+ /*
+ * A stale leftover would bias the count forever if this CPU later
+ * returns as its own leader. Move, don't copy.
+ */
+ core_rq->core_pick_in_flight = rq->core_pick_in_flight;
+ rq->core_pick_in_flight = 0;
+
/*
* Accounting edge for forced idle is handled in pick_next_task().
* Don't need another one here, since the hotplug thread shouldn't
@@ -7368,6 +7389,33 @@ static struct task_struct *find_process_by_pid(pid_t pid)
*/
#define SETPARAM_POLICY -1
+#ifdef CONFIG_RT_MUTEXES
+static inline void __setscheduler_dl_pi(int newprio, int policy,
+ struct task_struct *p, int *queue_flags)
+{
+ /*
+ * In case a DEADLINE task (either proper or boosted) gets
+ * setscheduled to a lower priority class, check if it neeeds to
+ * inherit parameters from a potential pi_task. In that case make
+ * sure replenishment happens with the next enqueue.
+ */
+
+ if (dl_prio(newprio) && !dl_policy(policy)) {
+ struct task_struct *pi_task = rt_mutex_get_top_task(p);
+
+ if (pi_task) {
+ p->dl.pi_se = pi_task->dl.pi_se;
+ *queue_flags |= ENQUEUE_REPLENISH;
+ }
+ }
+}
+#else /* !CONFIG_RT_MUTEXES */
+static inline void __setscheduler_dl_pi(int newprio, int policy,
+ struct task_struct *p, int *queue_flags)
+{
+}
+#endif /* !CONFIG_RT_MUTEXES */
+
static void __setscheduler_params(struct task_struct *p,
const struct sched_attr *attr)
{
@@ -7669,6 +7717,7 @@ static int __sched_setscheduler(struct task_struct *p,
if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)) {
__setscheduler_params(p, attr);
__setscheduler_prio(p, newprio);
+ __setscheduler_dl_pi(newprio, policy, p, &queue_flags);
}
__setscheduler_uclamp(p, attr);
@@ -9834,6 +9883,7 @@ void __init sched_init(void)
rq->core_forceidle_count = 0;
rq->core_forceidle_occupation = 0;
rq->core_forceidle_start = 0;
+ rq->core_pick_in_flight = 0;
rq->core_cookie = 0UL;
#endif
diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c
index 62a7c4ed8675..972a5e728c10 100644
--- a/kernel/sched/deadline.c
+++ b/kernel/sched/deadline.c
@@ -1656,6 +1656,7 @@ enqueue_dl_entity(struct sched_dl_entity *dl_se, int flags)
} else if (flags & ENQUEUE_REPLENISH) {
replenish_dl_entity(dl_se);
} else if ((flags & ENQUEUE_RESTORE) &&
+ !is_dl_boosted(dl_se) &&
dl_time_before(dl_se->deadline,
rq_clock(rq_of_dl_rq(dl_rq_of_se(dl_se))))) {
setup_new_dl_entity(dl_se);
@@ -1689,8 +1690,12 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags)
* The replenish timer needs to be canceled. No
* problem if it fires concurrently: boosted threads
* are ignored in dl_task_timer().
+ *
+ * If the timer callback was running (hrtimer_try_to_cancel == -1),
+ * it will eventually call put_task_struct().
*/
- hrtimer_try_to_cancel(&p->dl.dl_timer);
+ if (hrtimer_try_to_cancel(&p->dl.dl_timer) == 1)
+ put_task_struct(p);
p->dl.dl_throttled = 0;
}
} else if (!dl_prio(p->normal_prio)) {
diff --git a/kernel/sched/psi.c b/kernel/sched/psi.c
index 5294ba4664c6..2d61e95bb398 100644
--- a/kernel/sched/psi.c
+++ b/kernel/sched/psi.c
@@ -1291,9 +1291,44 @@ int psi_show(struct seq_file *m, struct psi_group *group, enum psi_res res)
return 0;
}
+/*
+ * Create @group's rtpoll worker after psi_trigger_create() reported the need
+ * for one. kthread creation depends on the whole fork path and we don't want
+ * all of that nested inside cgroup_mutex, so the caller must drop it and any
+ * other lock that forks can wait behind. If two callers race, the loser stops
+ * its never-woken kthread.
+ */
+int psi_trigger_create_rtpoll_worker(struct psi_group *group)
+{
+ struct task_struct *task;
+
+ task = kthread_create(psi_rtpoll_worker, group, "psimon");
+ if (IS_ERR(task))
+ return PTR_ERR(task);
+
+ scoped_guard(mutex, &group->rtpoll_trigger_lock) {
+ if (!rcu_access_pointer(group->rtpoll_task)) {
+ atomic_set(&group->rtpoll_wakeup, 0);
+ wake_up_process(task);
+ rcu_assign_pointer(group->rtpoll_task, task);
+
+ /*
+ * Poll once to catch up on scheduling attempts dropped
+ * while there was no rtpoll worker.
+ */
+ psi_schedule_rtpoll_work(group, 1, true);
+ return 0;
+ }
+ }
+
+ kthread_stop(task);
+ return 0;
+}
+
struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
enum psi_res res, struct file *file,
- struct kernfs_open_file *of)
+ struct kernfs_open_file *of,
+ bool *need_rtpoll_worker)
{
struct psi_trigger *t;
enum psi_states state;
@@ -1301,6 +1336,8 @@ struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
bool privileged;
u32 window_us;
+ *need_rtpoll_worker = false;
+
if (static_branch_likely(&psi_disabled))
return ERR_PTR(-EOPNOTSUPP);
@@ -1362,26 +1399,14 @@ struct psi_trigger *psi_trigger_create(struct psi_group *group, char *buf,
if (privileged) {
mutex_lock(&group->rtpoll_trigger_lock);
- if (!rcu_access_pointer(group->rtpoll_task)) {
- struct task_struct *task;
-
- task = kthread_create(psi_rtpoll_worker, group, "psimon");
- if (IS_ERR(task)) {
- kfree(t);
- mutex_unlock(&group->rtpoll_trigger_lock);
- return ERR_CAST(task);
- }
- atomic_set(&group->rtpoll_wakeup, 0);
- wake_up_process(task);
- rcu_assign_pointer(group->rtpoll_task, task);
- }
-
list_add(&t->node, &group->rtpoll_triggers);
group->rtpoll_min_period = min(group->rtpoll_min_period,
div_u64(t->win.size, UPDATES_PER_WINDOW));
group->rtpoll_nr_triggers[t->state]++;
group->rtpoll_states |= (1 << t->state);
+ *need_rtpoll_worker = !rcu_access_pointer(group->rtpoll_task);
+
mutex_unlock(&group->rtpoll_trigger_lock);
} else {
mutex_lock(&group->avgs_lock);
@@ -1536,6 +1561,8 @@ static ssize_t psi_write(struct file *file, const char __user *user_buf,
size_t buf_size;
struct seq_file *seq;
struct psi_trigger *new;
+ bool need_rtpoll_worker;
+ int ret;
if (static_branch_likely(&psi_disabled))
return -EOPNOTSUPP;
@@ -1560,12 +1587,22 @@ static ssize_t psi_write(struct file *file, const char __user *user_buf,
return -EBUSY;
}
- new = psi_trigger_create(&psi_system, buf, res, file, NULL);
+ new = psi_trigger_create(&psi_system, buf, res, file, NULL,
+ &need_rtpoll_worker);
if (IS_ERR(new)) {
mutex_unlock(&seq->lock);
return PTR_ERR(new);
}
+ if (need_rtpoll_worker) {
+ ret = psi_trigger_create_rtpoll_worker(&psi_system);
+ if (ret) {
+ psi_trigger_destroy(new);
+ mutex_unlock(&seq->lock);
+ return ret;
+ }
+ }
+
smp_store_release(&seq->private, new);
mutex_unlock(&seq->lock);
diff --git a/kernel/sched/rt.c b/kernel/sched/rt.c
index 49e19ed896f8..2eddf5653f6a 100644
--- a/kernel/sched/rt.c
+++ b/kernel/sched/rt.c
@@ -1972,6 +1972,26 @@ static int find_lowest_rq(struct task_struct *task)
return -1;
}
+static struct task_struct *pick_next_pushable_task(struct rq *rq)
+{
+ struct task_struct *p;
+
+ if (!has_pushable_tasks(rq))
+ return NULL;
+
+ p = plist_first_entry(&rq->rt.pushable_tasks,
+ struct task_struct, pushable_tasks);
+
+ BUG_ON(rq->cpu != task_cpu(p));
+ BUG_ON(task_current(rq, p));
+ BUG_ON(p->nr_cpus_allowed <= 1);
+
+ BUG_ON(!task_on_rq_queued(p));
+ BUG_ON(!rt_task(p));
+
+ return p;
+}
+
/* Will lock the rq it finds */
static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
{
@@ -2002,18 +2022,16 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
/*
* We had to unlock the run queue. In
* the mean time, task could have
- * migrated already or had its affinity changed.
- * Also make sure that it wasn't scheduled on its rq.
+ * migrated already or had its affinity changed,
+ * therefore check if the task is still at the
+ * head of the pushable tasks list.
* It is possible the task was scheduled, set
* "migrate_disabled" and then got preempted, so we must
* check the task migration disable flag here too.
*/
- if (unlikely(task_rq(task) != rq ||
+ if (unlikely(is_migration_disabled(task) ||
!cpumask_test_cpu(lowest_rq->cpu, &task->cpus_mask) ||
- task_on_cpu(rq, task) ||
- !rt_task(task) ||
- is_migration_disabled(task) ||
- !task_on_rq_queued(task))) {
+ task != pick_next_pushable_task(rq))) {
double_unlock_balance(rq, lowest_rq);
lowest_rq = NULL;
@@ -2033,26 +2051,6 @@ static struct rq *find_lock_lowest_rq(struct task_struct *task, struct rq *rq)
return lowest_rq;
}
-static struct task_struct *pick_next_pushable_task(struct rq *rq)
-{
- struct task_struct *p;
-
- if (!has_pushable_tasks(rq))
- return NULL;
-
- p = plist_first_entry(&rq->rt.pushable_tasks,
- struct task_struct, pushable_tasks);
-
- BUG_ON(rq->cpu != task_cpu(p));
- BUG_ON(task_current(rq, p));
- BUG_ON(p->nr_cpus_allowed <= 1);
-
- BUG_ON(!task_on_rq_queued(p));
- BUG_ON(!rt_task(p));
-
- return p;
-}
-
/*
* If the current CPU has more than one RT task, see if the non
* running task can migrate over to a CPU that is running a task
diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h
index a6c2b8f3045e..accff24e091e 100644
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -1151,6 +1151,7 @@ struct rq {
unsigned int core_forceidle_seq;
unsigned int core_forceidle_occupation;
u64 core_forceidle_start;
+ unsigned int core_pick_in_flight;
#endif
};
diff --git a/kernel/signal.c b/kernel/signal.c
index efdd01ffdccd..1c169fa29486 100644
--- a/kernel/signal.c
+++ b/kernel/signal.c
@@ -2942,6 +2942,10 @@ static void retarget_shared_pending(struct task_struct *tsk, sigset_t *which)
sigset_t retarget;
struct task_struct *t;
+ /* Nobody dequeues them in a dying group, see get_signal(). */
+ if (tsk->signal->flags & SIGNAL_GROUP_EXIT)
+ return;
+
sigandsets(&retarget, &tsk->signal->shared_pending.signal, which);
if (sigisemptyset(&retarget))
return;
diff --git a/kernel/time/namespace.c b/kernel/time/namespace.c
index aec832801c26..439a5e6f5ae4 100644
--- a/kernel/time/namespace.c
+++ b/kernel/time/namespace.c
@@ -398,6 +398,11 @@ int proc_timens_set_offset(struct file *file, struct task_struct *p,
off->val.tv_sec < -KTIME_SEC_MAX)
goto out;
+ if (off->val.tv_nsec < 0 || off->val.tv_nsec >= NSEC_PER_SEC) {
+ err = -EINVAL;
+ goto out;
+ }
+
tp = timespec64_add(tp, off->val);
/*
* KTIME_SEC_MAX is divided by 2 to be sure that KTIME_MAX is
diff --git a/kernel/trace/blktrace.c b/kernel/trace/blktrace.c
index a5b35bcfb060..5f76d59489fb 100644
--- a/kernel/trace/blktrace.c
+++ b/kernel/trace/blktrace.c
@@ -251,8 +251,6 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
cpu = raw_smp_processor_id();
if (blk_tracer) {
- tracing_record_cmdline(current);
-
buffer = blk_tr->array_buffer.buffer;
trace_ctx = tracing_gen_ctx_flags(0);
event = trace_buffer_lock_reserve(buffer, TRACE_BLK,
@@ -260,6 +258,7 @@ static void __blk_add_trace(struct blk_trace *bt, sector_t sector, int bytes,
trace_ctx);
if (!event)
return;
+ tracing_record_cmdline(current);
t = ring_buffer_event_data(event);
goto record_it;
}
diff --git a/kernel/trace/bpf_trace.c b/kernel/trace/bpf_trace.c
index fcddc223c3f0..db5484a1dbd3 100644
--- a/kernel/trace/bpf_trace.c
+++ b/kernel/trace/bpf_trace.c
@@ -2565,7 +2565,7 @@ static void bpf_kprobe_multi_link_dealloc(struct bpf_link *link)
static const struct bpf_link_ops bpf_kprobe_multi_link_lops = {
.release = bpf_kprobe_multi_link_release,
- .dealloc = bpf_kprobe_multi_link_dealloc,
+ .dealloc_deferred = bpf_kprobe_multi_link_dealloc,
};
static void bpf_kprobe_multi_cookie_swap(void *a, void *b, int size, const void *priv)
@@ -2730,6 +2730,10 @@ int bpf_kprobe_multi_link_attach(const union bpf_attr *attr, struct bpf_prog *pr
if (prog->expected_attach_type != BPF_TRACE_KPROBE_MULTI)
return -EINVAL;
+ /* kprobe_multi is not allowed to be sleepable. */
+ if (prog->aux->sleepable)
+ return -EINVAL;
+
flags = attr->link_create.kprobe_multi.flags;
if (flags & ~BPF_F_KPROBE_MULTI_RETURN)
return -EINVAL;
diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c
index 218cd95bf8e4..383c8c7f1131 100644
--- a/kernel/trace/fgraph.c
+++ b/kernel/trace/fgraph.c
@@ -623,6 +623,7 @@ int register_ftrace_graph(struct fgraph_ops *gops)
ret = start_graph_tracing();
if (ret) {
ftrace_graph_active--;
+ unregister_pm_notifier(&ftrace_suspend_notifier);
goto out;
}
diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c
index ad112022dad7..2b276e948216 100644
--- a/kernel/trace/ftrace.c
+++ b/kernel/trace/ftrace.c
@@ -71,6 +71,8 @@
#define INIT_OPS_HASH(opsname) \
.func_hash = &opsname.local_hash, \
.local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock),
+/* Used only to synchronize the initialization of ftrace_ops */
+static DEFINE_MUTEX(ops_mutex);
#else
#define INIT_OPS_HASH(opsname)
#endif
@@ -128,10 +130,17 @@ void ftrace_ops_list_func(unsigned long ip, unsigned long parent_ip,
static inline void ftrace_ops_init(struct ftrace_ops *ops)
{
#ifdef CONFIG_DYNAMIC_FTRACE
- if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) {
+ unsigned long flags = smp_load_acquire(&ops->flags);
+
+ if (!(flags & FTRACE_OPS_FL_INITIALIZED)) {
+ guard(mutex)(&ops_mutex);
+ /* Could have been initialized before lock taken */
+ if (unlikely(ops->flags & FTRACE_OPS_FL_INITIALIZED))
+ return;
mutex_init(&ops->local_hash.regex_lock);
ops->func_hash = &ops->local_hash;
- ops->flags |= FTRACE_OPS_FL_INITIALIZED;
+ flags = ops->flags | FTRACE_OPS_FL_INITIALIZED;
+ smp_store_release(&ops->flags, flags);
}
#endif
}
@@ -3733,9 +3742,9 @@ subsys_initcall(ftrace_check_for_weak_functions);
static int print_rec(struct seq_file *m, unsigned long ip)
{
- unsigned long offset;
+ unsigned long offset = 0;
char str[KSYM_SYMBOL_LEN];
- char *modname;
+ char *modname = NULL;
const char *ret;
ret = kallsyms_lookup(ip, NULL, &offset, &modname, str);
@@ -3889,7 +3898,8 @@ ftrace_enabled_open(struct inode *inode, struct file *file)
/**
* ftrace_regex_open - initialize function tracer filter files
- * @ops: The ftrace_ops that hold the hash filters
+ * @tr: The trace_array that holds the ftrace_ops [optional]
+ * @ops: The ftrace_ops that hold the hash filters [optional]
* @flag: The type of filter to process
* @inode: The inode, usually passed in to your open routine
* @file: The file, usually passed in to your open routine
@@ -3902,25 +3912,46 @@ ftrace_enabled_open(struct inode *inode, struct file *file)
* ftrace_notrace_write() if @flag has FTRACE_ITER_NOTRACE set.
* tracing_lseek() should be used as the lseek routine, and
* release must call ftrace_regex_release().
+ *
+ * Note, If @tr is not NULL, its reference has to be taken before
+ * @ops may be referenced.
+ * If @ops is NULL and @tr is not, then @tr->ops is used.
+ * If @tr is NULL and @ops is not then @ops->private is uesd for @tr.
+ * If both @tr and @ops are NULL, then the &global_ops is
+ * to be used, and @tr will be the global_ops.private pointer.
+ *
+ * Returns: 0 on success or a negative errno value on failure
*/
int
-ftrace_regex_open(struct ftrace_ops *ops, int flag,
+ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag,
struct inode *inode, struct file *file)
{
- struct ftrace_iterator *iter;
+ struct ftrace_iterator *iter = NULL;
struct ftrace_hash *hash;
struct list_head *mod_head;
- struct trace_array *tr = ops->private;
- int ret = -ENOMEM;
-
- ftrace_ops_init(ops);
+ int ret = -ENODEV;
if (unlikely(ftrace_disabled))
return -ENODEV;
+ if (!tr) {
+ if (!ops)
+ ops = &global_ops;
+ tr = ops->private;
+ }
+
if (tracing_check_open_get_tr(tr))
return -ENODEV;
+ if (!ops)
+ ops = tr->ops;
+
+ if (WARN_ON_ONCE(!ops))
+ goto out;
+
+ ftrace_ops_init(ops);
+
+ ret = -ENOMEM;
iter = kzalloc(sizeof(*iter), GFP_KERNEL);
if (!iter)
goto out;
@@ -3998,21 +4029,19 @@ ftrace_regex_open(struct ftrace_ops *ops, int flag,
static int
ftrace_filter_open(struct inode *inode, struct file *file)
{
- struct ftrace_ops *ops = inode->i_private;
+ struct trace_array *tr = inode->i_private;
- /* Checks for tracefs lockdown */
- return ftrace_regex_open(ops,
- FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES,
- inode, file);
+ return ftrace_regex_open(tr, NULL,
+ FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES,
+ inode, file);
}
static int
ftrace_notrace_open(struct inode *inode, struct file *file)
{
- struct ftrace_ops *ops = inode->i_private;
+ struct trace_array *tr = inode->i_private;
- /* Checks for tracefs lockdown */
- return ftrace_regex_open(ops, FTRACE_ITER_NOTRACE,
+ return ftrace_regex_open(tr, NULL, FTRACE_ITER_NOTRACE,
inode, file);
}
@@ -6621,15 +6650,15 @@ static const struct file_operations ftrace_graph_notrace_fops = {
};
#endif /* CONFIG_FUNCTION_GRAPH_TRACER */
-void ftrace_create_filter_files(struct ftrace_ops *ops,
+void ftrace_create_filter_files(struct trace_array *tr,
struct dentry *parent)
{
trace_create_file("set_ftrace_filter", TRACE_MODE_WRITE, parent,
- ops, &ftrace_filter_fops);
+ tr, &ftrace_filter_fops);
trace_create_file("set_ftrace_notrace", TRACE_MODE_WRITE, parent,
- ops, &ftrace_notrace_fops);
+ tr, &ftrace_notrace_fops);
}
/*
@@ -6654,14 +6683,13 @@ void ftrace_destroy_filter_files(struct ftrace_ops *ops)
static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer)
{
-
trace_create_file("available_filter_functions", TRACE_MODE_READ,
d_tracer, NULL, &ftrace_avail_fops);
trace_create_file("enabled_functions", TRACE_MODE_READ,
d_tracer, NULL, &ftrace_enabled_fops);
- ftrace_create_filter_files(&global_ops, d_tracer);
+ ftrace_create_filter_files(NULL, d_tracer);
#ifdef CONFIG_FUNCTION_GRAPH_TRACER
trace_create_file("set_graph_function", TRACE_MODE_WRITE, d_tracer,
diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c
index 33770d2b3489..bf8f3ca9df07 100644
--- a/kernel/trace/trace.c
+++ b/kernel/trace/trace.c
@@ -1692,15 +1692,15 @@ static ssize_t trace_seq_to_buffer(struct trace_seq *s, void *buf, size_t cnt)
{
int len;
- if (trace_seq_used(s) <= s->seq.readpos)
+ if (trace_seq_used(s) <= s->readpos)
return -EBUSY;
- len = trace_seq_used(s) - s->seq.readpos;
+ len = trace_seq_used(s) - s->readpos;
if (cnt > len)
cnt = len;
- memcpy(buf, s->buffer + s->seq.readpos, cnt);
+ memcpy(buf, s->buffer + s->readpos, cnt);
- s->seq.readpos += cnt;
+ s->readpos += cnt;
return cnt;
}
@@ -3043,6 +3043,11 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
struct ftrace_stack *fstack;
struct stack_entry *entry;
int stackidx;
+ int bit;
+
+ bit = trace_test_and_set_recursion(_THIS_IP_, _RET_IP_, TRACE_EVENT_START);
+ if (bit < 0)
+ return;
/*
* Add one, for this function and the call to save_stack_trace()
@@ -3099,7 +3104,7 @@ static void __ftrace_trace_stack(struct trace_buffer *buffer,
barrier();
__this_cpu_dec(ftrace_stack_reserve);
preempt_enable_notrace();
-
+ trace_clear_recursion(bit);
}
static inline void ftrace_trace_stack(struct trace_array *tr,
@@ -3740,15 +3745,6 @@ static bool trace_safe_str(struct trace_iterator *iter, const char *str,
return false;
}
-static const char *show_buffer(struct trace_seq *s)
-{
- struct seq_buf *seq = &s->seq;
-
- seq_buf_terminate(seq);
-
- return seq->buffer;
-}
-
static DEFINE_STATIC_KEY_FALSE(trace_no_verify);
static int test_can_verify_check(const char *fmt, ...)
@@ -3888,7 +3884,7 @@ void trace_check_vprintf(struct trace_iterator *iter, const char *fmt,
*/
if (WARN_ONCE(!trace_safe_str(iter, str, star, len),
"fmt: '%s' current_buffer: '%s'",
- fmt, show_buffer(&iter->seq))) {
+ fmt, seq_buf_str(&iter->seq.seq))) {
int ret;
/* Try to safely read the string */
@@ -6931,7 +6927,7 @@ tracing_read_pipe(struct file *filp, char __user *ubuf,
/* Now copy what we have to the user */
sret = trace_seq_to_user(&iter->seq, ubuf, cnt);
- if (iter->seq.seq.readpos >= trace_seq_used(&iter->seq))
+ if (iter->seq.readpos >= trace_seq_used(&iter->seq))
trace_seq_init(&iter->seq);
/*
@@ -9037,11 +9033,72 @@ trace_options_core_write(struct file *filp, const char __user *ubuf, size_t cnt,
return cnt;
}
+/*
+ * The tr_index is the address of a trace_array->trace_flags_index[]
+ * element that holds the index of the trace flag. But since the
+ * trace_array reference has not been taken yet, it cannot be referenced
+ * as it could have been freed by a rmdir of the instance the trace_array
+ * represents.
+ *
+ * Search the list of trace_arrays and compare the tr_index to the
+ * address of the entire trace_array trace_flags_index array for each
+ * trace_array in the list. If one is matched, then take the reference
+ * and return it. If not, the trace_array no longer exits.
+ */
+static int trace_array_options_get(void *tr_index)
+{
+ struct trace_array *tr;
+ int ret;
+
+ ret = security_locked_down(LOCKDOWN_TRACEFS);
+ if (ret)
+ return ret;
+
+ if (tracing_disabled)
+ return -ENODEV;
+
+ guard(mutex)(&trace_types_lock);
+ list_for_each_entry(tr, &ftrace_trace_arrays, list) {
+ if (tr_index >= (void *)&tr->trace_flags_index[0] &&
+ tr_index < (void *)&tr->trace_flags_index[TRACE_FLAGS_MAX_SIZE]) {
+ tr->ref++;
+ return 0;
+ }
+ }
+ return -ENODEV;
+}
+
+static int trace_options_open(struct inode *inode, struct file *filp)
+{
+ void *tr_index = inode->i_private;
+
+ if (trace_array_options_get(tr_index) < 0)
+ return -ENODEV;
+
+ filp->private_data = tr_index;
+
+ return 0;
+}
+
+static int trace_options_release(struct inode *inode, struct file *filp)
+{
+ void *tr_index = filp->private_data;
+ struct trace_array *tr;
+ unsigned int index;
+
+ get_tr_index(tr_index, &tr, &index);
+
+ trace_array_put(tr);
+
+ return 0;
+}
+
static const struct file_operations trace_options_core_fops = {
- .open = tracing_open_generic,
- .read = trace_options_core_read,
- .write = trace_options_core_write,
- .llseek = generic_file_llseek,
+ .open = trace_options_open,
+ .read = trace_options_core_read,
+ .write = trace_options_core_write,
+ .llseek = generic_file_llseek,
+ .release = trace_options_release,
};
struct dentry *trace_create_file(const char *name,
@@ -9948,6 +10005,12 @@ static struct notifier_block trace_module_nb = {
static __init void tracer_init_tracefs_work_func(struct work_struct *work)
{
+ /*
+ * Do not attempt to initialize tracefs if tracing was disabled or
+ * failed to allocate
+ */
+ if (tracing_disabled)
+ return;
event_trace_init();
diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h
index 8cdb8e1a33c1..f86b6f263ddf 100644
--- a/kernel/trace/trace.h
+++ b/kernel/trace/trace.h
@@ -1092,7 +1092,7 @@ extern void clear_ftrace_function_probes(struct trace_array *tr);
int register_ftrace_command(struct ftrace_func_command *cmd);
int unregister_ftrace_command(struct ftrace_func_command *cmd);
-void ftrace_create_filter_files(struct ftrace_ops *ops,
+void ftrace_create_filter_files(struct trace_array *tr,
struct dentry *parent);
void ftrace_destroy_filter_files(struct ftrace_ops *ops);
@@ -1115,11 +1115,11 @@ static inline void clear_ftrace_function_probes(struct trace_array *tr)
{
}
+#define ftrace_create_filter_files(tr, parent) do { } while (0)
/*
* The ops parameter passed in is usually undefined.
* This must be a macro.
*/
-#define ftrace_create_filter_files(ops, parent) do { } while (0)
#define ftrace_destroy_filter_files(ops) do { } while (0)
#endif /* CONFIG_FUNCTION_TRACER && CONFIG_DYNAMIC_FTRACE */
diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c
index 7e3cb495406c..1c4e81b3fff1 100644
--- a/kernel/trace/trace_events_filter.c
+++ b/kernel/trace/trace_events_filter.c
@@ -573,6 +573,12 @@ predicate_parse(const char *str, int nr_parens, int nr_preds,
top--; /* #12 */
}
}
+ /* Only the inner loop can complete a non-empty expression. */
+ if (N) {
+ ret = -EINVAL;
+ parse_error(pe, FILT_ERR_INVALID_FILTER, ptr - str);
+ goto out_free;
+ }
out:
if (top != op_stack) {
/* Too many '(' */
diff --git a/kernel/trace/trace_events_hist.c b/kernel/trace/trace_events_hist.c
index 8e5efdfa0fc7..e97369d462c0 100644
--- a/kernel/trace/trace_events_hist.c
+++ b/kernel/trace/trace_events_hist.c
@@ -8,6 +8,7 @@
#include <linux/module.h>
#include <linux/kallsyms.h>
#include <linux/security.h>
+#include <linux/seq_buf.h>
#include <linux/mutex.h>
#include <linux/slab.h>
#include <linux/stacktrace.h>
@@ -280,7 +281,7 @@ static u64 hist_field_log2(struct hist_field *hist_field,
u64 val = hist_fn_call(operand, elt, buffer, rbe, event);
- return (u64) ilog2(roundup_pow_of_two(val));
+ return order_base_2(val);
}
static u64 hist_field_bucket(struct hist_field *hist_field,
@@ -457,7 +458,7 @@ static u64 hist_field_unary_minus(struct hist_field *hist_field,
{ \
type *addr = (type *)(event + hist_field->field->offset); \
\
- return (u64)(unsigned long)*addr; \
+ return (u64)*addr; \
}
DEFINE_HIST_FIELD_FN(s64);
@@ -538,7 +539,7 @@ struct field_var {
};
struct field_var_hist {
- struct hist_trigger_data *hist_data;
+ struct trace_event_file *file;
char *cmd;
};
@@ -862,7 +863,7 @@ static u64 hist_field_timestamp(struct hist_field *hist_field,
struct hist_trigger_data *hist_data = hist_field->hist_data;
struct trace_array *tr = hist_data->event_file->tr;
- u64 ts = ring_buffer_event_time_stamp(buffer, rbe);
+ u64 ts = tracing_event_time_stamp(buffer, rbe);
if (hist_data->attrs->ts_in_usecs && trace_clock_in_ns(tr))
ts = ns2usecs(ts);
@@ -2066,7 +2067,8 @@ static int init_var_ref(struct hist_field *ref_field,
ref_field->size = var_field->size;
ref_field->is_signed = var_field->is_signed;
ref_field->flags |= var_field->flags &
- (HIST_FIELD_FL_TIMESTAMP | HIST_FIELD_FL_TIMESTAMP_USECS);
+ (HIST_FIELD_FL_TIMESTAMP | HIST_FIELD_FL_TIMESTAMP_USECS |
+ HIST_FIELD_FL_STACKTRACE);
if (system) {
ref_field->system = kstrdup(system, GFP_KERNEL);
@@ -2882,13 +2884,22 @@ find_synthetic_field_var(struct hist_trigger_data *target_hist_data,
{
struct hist_field *event_var;
char *synthetic_name;
+ struct seq_buf s;
synthetic_name = kzalloc(MAX_FILTER_STR_VAL, GFP_KERNEL);
if (!synthetic_name)
return ERR_PTR(-ENOMEM);
- strcpy(synthetic_name, "synthetic_");
- strcat(synthetic_name, field_name);
+ seq_buf_init(&s, synthetic_name, MAX_FILTER_STR_VAL);
+ seq_buf_printf(&s, "synthetic_%s", field_name);
+
+ /* Terminate synthetic_name with a NUL. */
+ seq_buf_str(&s);
+
+ if (seq_buf_has_overflowed(&s)) {
+ kfree(synthetic_name);
+ return ERR_PTR(-E2BIG);
+ }
event_var = find_event_var(target_hist_data, system, event_name, synthetic_name);
@@ -2934,6 +2945,7 @@ create_field_var_hist(struct hist_trigger_data *target_hist_data,
struct hist_field *key_field;
struct hist_field *event_var;
char *saved_filter;
+ struct seq_buf s;
char *cmd;
int ret;
@@ -2978,28 +2990,34 @@ create_field_var_hist(struct hist_trigger_data *target_hist_data,
return ERR_PTR(-ENOMEM);
}
+ seq_buf_init(&s, cmd, MAX_FILTER_STR_VAL);
+
/* Use the same keys as the compatible histogram */
- strcat(cmd, "keys=");
+ seq_buf_puts(&s, "keys=");
for_each_hist_key_field(i, hist_data) {
key_field = hist_data->fields[i];
if (!first)
- strcat(cmd, ",");
- strcat(cmd, key_field->field->name);
+ seq_buf_putc(&s, ',');
+ seq_buf_puts(&s, key_field->field->name);
first = false;
}
/* Create the synthetic field variable specification */
- strcat(cmd, ":synthetic_");
- strcat(cmd, field_name);
- strcat(cmd, "=");
- strcat(cmd, field_name);
+ seq_buf_printf(&s, ":synthetic_%s=%s", field_name, field_name);
/* Use the same filter as the compatible histogram */
saved_filter = find_trigger_filter(hist_data, file);
- if (saved_filter) {
- strcat(cmd, " if ");
- strcat(cmd, saved_filter);
+ if (saved_filter)
+ seq_buf_printf(&s, " if %s", saved_filter);
+
+ /* Terminate cmd with a NUL. */
+ seq_buf_str(&s);
+
+ if (seq_buf_has_overflowed(&s)) {
+ kfree(cmd);
+ kfree(var_hist);
+ return ERR_PTR(-E2BIG);
}
var_hist->cmd = kstrdup(cmd, GFP_KERNEL);
@@ -3009,8 +3027,8 @@ create_field_var_hist(struct hist_trigger_data *target_hist_data,
return ERR_PTR(-ENOMEM);
}
- /* Save the compatible histogram information */
- var_hist->hist_data = hist_data;
+ /* Needed to remove the histogram when the target goes away */
+ var_hist->file = file;
/* Create the new histogram with our variable */
ret = event_hist_trigger_parse(&trigger_hist_cmd, file,
@@ -3025,20 +3043,18 @@ create_field_var_hist(struct hist_trigger_data *target_hist_data,
kfree(cmd);
+ n = target_hist_data->n_field_var_hists;
+ target_hist_data->field_var_hists[n] = var_hist;
+ target_hist_data->n_field_var_hists++;
+
/* If we can't find the variable, something went wrong */
event_var = find_synthetic_field_var(target_hist_data, subsys_name,
event_name, field_name);
if (IS_ERR_OR_NULL(event_var)) {
- kfree(var_hist->cmd);
- kfree(var_hist);
hist_err(tr, HIST_ERR_SYNTH_VAR_NOT_FOUND, errpos(field_name));
return ERR_PTR(-EINVAL);
}
- n = target_hist_data->n_field_var_hists;
- target_hist_data->field_var_hists[n] = var_hist;
- target_hist_data->n_field_var_hists++;
-
return event_var;
}
@@ -3203,6 +3219,7 @@ static struct field_var *create_field_var(struct hist_trigger_data *hist_data,
ret = -ENOMEM;
goto err;
}
+ var->flags |= val->flags & HIST_FIELD_FL_STACKTRACE;
field_var->var = var;
field_var->val = val;
@@ -3820,7 +3837,10 @@ static int check_synth_field(struct synth_event *event,
&& field->is_dynamic)
return 0;
- if (strstr(hist_field->type, "long[") && field->is_stack)
+ if (!!(hist_field->flags & HIST_FIELD_FL_STACKTRACE) != field->is_stack)
+ return -EINVAL;
+
+ if (field->is_stack)
return 0;
if (strcmp(field->type, hist_field->type) != 0) {
@@ -5971,7 +5991,7 @@ static void unregister_field_var_hists(struct hist_trigger_data *hist_data)
int ret;
for (i = 0; i < hist_data->n_field_var_hists; i++) {
- file = hist_data->field_var_hists[i]->hist_data->event_file;
+ file = hist_data->field_var_hists[i]->file;
cmd = hist_data->field_var_hists[i]->cmd;
ret = event_hist_trigger_parse(&trigger_hist_cmd, file,
"!hist", "hist", cmd);
@@ -6244,12 +6264,6 @@ static int hist_register_trigger(char *glob,
data->ops = &event_hist_trigger_named_ops;
}
- if (data->ops->init) {
- ret = data->ops->init(data);
- if (ret < 0)
- goto out;
- }
-
if (hist_data->enable_timestamps) {
char *clock = hist_data->attrs->clock;
@@ -6262,6 +6276,15 @@ static int hist_register_trigger(char *glob,
tracing_set_filter_buffering(file->tr, true);
}
+ if (data->ops->init) {
+ ret = data->ops->init(data);
+ if (ret < 0) {
+ if (hist_data->enable_timestamps)
+ tracing_set_filter_buffering(file->tr, false);
+ goto out;
+ }
+ }
+
if (named_data) {
remove_hist_vars(hist_data);
destroy_hist_data(hist_data);
@@ -6393,7 +6416,6 @@ static void hist_unreg_all(struct trace_event_file *file)
struct event_trigger_data *test, *n;
struct hist_trigger_data *hist_data;
struct synth_event *se;
- const char *se_name;
lockdep_assert_held(&event_mutex);
@@ -6406,8 +6428,7 @@ static void hist_unreg_all(struct trace_event_file *file)
list_del_rcu(&test->list);
trace_event_trigger_enable_disable(file, 0);
- se_name = trace_event_name(file->event_call);
- se = find_synth_event(se_name);
+ se = find_synth_event_call(file->event_call);
if (se)
se->ref--;
@@ -6431,7 +6452,6 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops,
struct hist_trigger_data *hist_data;
char *param, *filter, *p, *start;
struct synth_event *se;
- const char *se_name;
bool remove;
int ret = 0;
@@ -6525,8 +6545,7 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops,
}
event_trigger_unregister(cmd_ops, file, glob+1, trigger_data);
- se_name = trace_event_name(file->event_call);
- se = find_synth_event(se_name);
+ se = find_synth_event_call(file->event_call);
if (se)
se->ref--;
ret = 0;
@@ -6561,8 +6580,7 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops,
if (ret)
goto out_unreg;
- se_name = trace_event_name(file->event_call);
- se = find_synth_event(se_name);
+ se = find_synth_event_call(file->event_call);
if (se)
se->ref++;
out:
@@ -6579,6 +6597,8 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops,
trigger_data_free(trigger_data);
+ unregister_field_var_hists(hist_data);
+
destroy_hist_data(hist_data);
goto out;
}
diff --git a/kernel/trace/trace_events_synth.c b/kernel/trace/trace_events_synth.c
index c2817d0c05b6..b3b236389cd5 100644
--- a/kernel/trace/trace_events_synth.c
+++ b/kernel/trace/trace_events_synth.c
@@ -867,6 +867,16 @@ struct synth_event *find_synth_event(const char *name)
return NULL;
}
+struct synth_event *find_synth_event_call(struct trace_event_call *call)
+{
+ struct synth_event *se;
+ const char *name;
+
+ name = trace_event_name(call);
+ se = find_synth_event(name);
+ return se && &se->call == call ? se : NULL;
+}
+
static struct trace_event_fields synth_event_fields_array[] = {
{ .type = TRACE_FUNCTION_TYPE,
.define_fields = synth_event_define_fields },
diff --git a/kernel/trace/trace_events_user.c b/kernel/trace/trace_events_user.c
index bff699fe9e71..6186474c027f 100644
--- a/kernel/trace/trace_events_user.c
+++ b/kernel/trace/trace_events_user.c
@@ -430,10 +430,9 @@ static void user_event_destroy_validators(struct user_event *user)
}
}
-static void user_event_destroy_fields(struct user_event *user)
+static void user_event_destroy_fields(struct list_head *head)
{
struct ftrace_event_field *field, *next;
- struct list_head *head = &user->fields;
list_for_each_entry_safe(field, next, head, link) {
list_del(&field->link);
@@ -821,15 +820,30 @@ static int user_event_set_call_visible(struct user_event *user, bool visible)
static int destroy_user_event(struct user_event *user)
{
+ LIST_HEAD(fields);
int ret = 0;
- /* Must destroy fields before call removal */
- user_event_destroy_fields(user);
+ /*
+ * Detach the fields before removing the call. Removing the event
+ * frees the field list memory (trace_destroy_fields() is run on
+ * successful removal and kmem_cache_free()s the fields), but the
+ * fields here are allocated and owned by user_events. Destroy
+ * them separately once removal has succeeded.
+ */
+ list_splice_init(&user->fields, &fields);
ret = user_event_set_call_visible(user, false);
- if (ret)
+ if (ret) {
+ /*
+ * Removal failed and the event stays registered, recover
+ * the fields so it is left in a consistent state.
+ */
+ list_splice(&fields, &user->fields);
return ret;
+ }
+
+ user_event_destroy_fields(&fields);
dyn_event_remove(&user->devent);
@@ -1360,7 +1374,7 @@ static int user_event_parse(struct user_event_group *group, char *name,
put_user_lock:
mutex_unlock(&event_mutex);
put_user:
- user_event_destroy_fields(user);
+ user_event_destroy_fields(&user->fields);
user_event_destroy_validators(user);
kfree(user->call.print_fmt);
kfree(user);
diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c
index 44ae51d1dc45..eecac19a2bcb 100644
--- a/kernel/trace/trace_functions.c
+++ b/kernel/trace/trace_functions.c
@@ -90,7 +90,7 @@ int ftrace_create_function_files(struct trace_array *tr,
if (!tr->ops)
return -EINVAL;
- ftrace_create_filter_files(tr->ops, parent);
+ ftrace_create_filter_files(tr, parent);
return 0;
}
diff --git a/kernel/trace/trace_seq.c b/kernel/trace/trace_seq.c
index 9c90b3a7dce2..6e6faeb542af 100644
--- a/kernel/trace/trace_seq.c
+++ b/kernel/trace/trace_seq.c
@@ -369,8 +369,12 @@ EXPORT_SYMBOL_GPL(trace_seq_path);
*/
int trace_seq_to_user(struct trace_seq *s, char __user *ubuf, int cnt)
{
+ int ret;
__trace_seq_init(s);
- return seq_buf_to_user(&s->seq, ubuf, cnt);
+ ret = seq_buf_to_user(&s->seq, ubuf, s->readpos, cnt);
+ if (ret > 0)
+ s->readpos += ret;
+ return ret;
}
EXPORT_SYMBOL_GPL(trace_seq_to_user);
diff --git a/kernel/trace/trace_stack.c b/kernel/trace/trace_stack.c
index 5a48dba912ea..c6b2f622ffa1 100644
--- a/kernel/trace/trace_stack.c
+++ b/kernel/trace/trace_stack.c
@@ -499,7 +499,7 @@ stack_trace_filter_open(struct inode *inode, struct file *file)
struct ftrace_ops *ops = inode->i_private;
/* Checks for tracefs lockdown */
- return ftrace_regex_open(ops, FTRACE_ITER_FILTER,
+ return ftrace_regex_open(NULL, ops, FTRACE_ITER_FILTER,
inode, file);
}
diff --git a/kernel/trace/trace_synth.h b/kernel/trace/trace_synth.h
index 43f6fb6078db..d4ebf01b7049 100644
--- a/kernel/trace/trace_synth.h
+++ b/kernel/trace/trace_synth.h
@@ -37,5 +37,6 @@ struct synth_event {
};
extern struct synth_event *find_synth_event(const char *name);
+extern struct synth_event *find_synth_event_call(struct trace_event_call *call);
#endif /* __TRACE_SYNTH_H */
diff --git a/kernel/watch_queue.c b/kernel/watch_queue.c
index a72ea1c2f59f..506776c11379 100644
--- a/kernel/watch_queue.c
+++ b/kernel/watch_queue.c
@@ -102,7 +102,7 @@ static bool post_one_notification(struct watch_queue *wqueue,
struct pipe_inode_info *pipe = wqueue->pipe;
struct pipe_buffer *buf;
struct page *page;
- unsigned int head, tail, mask, note, offset, len;
+ unsigned int head, tail, mask, note, offset, len, nr_notes;
bool done = false;
spin_lock_irq(&pipe->rd_wait.lock);
@@ -113,8 +113,10 @@ static bool post_one_notification(struct watch_queue *wqueue,
if (pipe_full(head, tail, pipe->ring_size))
goto lost;
- note = find_first_bit(wqueue->notes_bitmap, wqueue->nr_notes);
- if (note >= wqueue->nr_notes)
+ /* Pairs with the smp_store_release() in watch_queue_set_size(). */
+ nr_notes = smp_load_acquire(&wqueue->nr_notes);
+ note = find_first_bit(wqueue->notes_bitmap, nr_notes);
+ if (note >= nr_notes)
goto lost;
page = wqueue->notes[note / WATCH_QUEUE_NOTES_PER_PAGE];
@@ -299,7 +301,8 @@ long watch_queue_set_size(struct pipe_inode_info *pipe, unsigned int nr_notes)
wqueue->notes = pages;
wqueue->notes_bitmap = bitmap;
wqueue->nr_pages = nr_pages;
- wqueue->nr_notes = nr_notes;
+ /* Pairs with the smp_load_acquire() in post_one_notification(). */
+ smp_store_release(&wqueue->nr_notes, nr_notes);
return 0;
error_p:
diff --git a/kernel/workqueue.c b/kernel/workqueue.c
index 485cd05b60ae..70d6e7779c2e 100644
--- a/kernel/workqueue.c
+++ b/kernel/workqueue.c
@@ -1382,7 +1382,7 @@ static bool is_chained_work(struct workqueue_struct *wq)
* Return %true iff I'm a worker executing a work item on @wq. If
* I'm @worker, it's safe to dereference it without locking.
*/
- return worker && worker->current_pwq->wq == wq;
+ return worker && worker->current_pwq && worker->current_pwq->wq == wq;
}
/*
@@ -2607,23 +2607,27 @@ static int rescuer_thread(void *__rescuer)
* check_flush_dependency - check for flush dependency sanity
* @target_wq: workqueue being flushed
* @target_work: work item being flushed (NULL for workqueue flushes)
+ * @from_cancel: are we called from the work cancel path
*
* %current is trying to flush the whole @target_wq or @target_work on it.
- * If @target_wq doesn't have %WQ_MEM_RECLAIM, verify that %current is not
- * reclaiming memory or running on a workqueue which doesn't have
- * %WQ_MEM_RECLAIM as that can break forward-progress guarantee leading to
- * a deadlock.
+ * If this is not the cancel path (which implies work being flushed is either
+ * already running, or will not be at all), check if @target_wq doesn't have
+ * %WQ_MEM_RECLAIM and verify that %current is not reclaiming memory or running
+ * on a workqueue which doesn't have %WQ_MEM_RECLAIM as that can break forward-
+ * progress guarantee leading to a deadlock.
*/
static void check_flush_dependency(struct workqueue_struct *target_wq,
- struct work_struct *target_work)
+ struct work_struct *target_work,
+ bool from_cancel)
{
- work_func_t target_func = target_work ? target_work->func : NULL;
+ work_func_t target_func;
struct worker *worker;
- if (target_wq->flags & WQ_MEM_RECLAIM)
+ if (from_cancel || target_wq->flags & WQ_MEM_RECLAIM)
return;
worker = current_wq_worker();
+ target_func = target_work ? target_work->func : NULL;
WARN_ONCE(current->flags & PF_MEMALLOC,
"workqueue: PF_MEMALLOC task %d(%s) is flushing !WQ_MEM_RECLAIM %s:%ps",
@@ -2857,7 +2861,7 @@ void __flush_workqueue(struct workqueue_struct *wq)
list_add_tail(&this_flusher.list, &wq->flusher_overflow);
}
- check_flush_dependency(wq, NULL);
+ check_flush_dependency(wq, NULL, false);
mutex_unlock(&wq->mutex);
@@ -3032,7 +3036,7 @@ static bool start_flush_work(struct work_struct *work, struct wq_barrier *barr,
pwq = worker->current_pwq;
}
- check_flush_dependency(pwq->wq, work);
+ check_flush_dependency(pwq->wq, work, from_cancel);
insert_wq_barrier(pwq, barr, work, worker);
raw_spin_unlock_irq(&pool->lock);
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index d00c695bf492..ee70b79a4308 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -737,6 +737,8 @@ config DEBUG_KMEMLEAK
select STACKTRACE if STACKTRACE_SUPPORT
select KALLSYMS
select CRC32
+ select STACKDEPOT
+ select STACKDEPOT_ALWAYS_INIT if !DEBUG_KMEMLEAK_DEFAULT_OFF
help
Say Y here if you want to enable the memory leak
detector. The memory allocation/freeing is traced in a way
diff --git a/lib/seq_buf.c b/lib/seq_buf.c
index 0a68f7aa85d6..23518f77ea9c 100644
--- a/lib/seq_buf.c
+++ b/lib/seq_buf.c
@@ -93,6 +93,36 @@ int seq_buf_printf(struct seq_buf *s, const char *fmt, ...)
}
EXPORT_SYMBOL_GPL(seq_buf_printf);
+/**
+ * seq_buf_do_printk - printk seq_buf line by line
+ * @s: seq_buf descriptor
+ * @lvl: printk level
+ *
+ * printk()-s a multi-line sequential buffer line by line. The function
+ * makes sure that the buffer in @s is nul terminated and safe to read
+ * as a string.
+ */
+void seq_buf_do_printk(struct seq_buf *s, const char *lvl)
+{
+ const char *start, *lf;
+
+ if (s->size == 0 || s->len == 0)
+ return;
+
+ start = seq_buf_str(s);
+ while ((lf = strchr(start, '\n'))) {
+ int len = lf - start + 1;
+
+ printk("%s%.*s", lvl, len, start);
+ start = ++lf;
+ }
+
+ /* No trailing LF */
+ if (start < s->buffer + s->len)
+ printk("%s%s\n", lvl, start);
+}
+EXPORT_SYMBOL_GPL(seq_buf_do_printk);
+
#ifdef CONFIG_BINARY_PRINTF
/**
* seq_buf_bprintf - Write the printf string from binary arguments
@@ -292,23 +322,24 @@ int seq_buf_path(struct seq_buf *s, const struct path *path, const char *esc)
* seq_buf_to_user - copy the sequence buffer to user space
* @s: seq_buf descriptor
* @ubuf: The userspace memory location to copy to
+ * @start: The first byte in the buffer to copy
* @cnt: The amount to copy
*
* Copies the sequence buffer into the userspace memory pointed to
- * by @ubuf. It starts from the last read position (@s->readpos)
- * and writes up to @cnt characters or till it reaches the end of
- * the content in the buffer (@s->len), which ever comes first.
+ * by @ubuf. It starts from @start and writes up to @cnt characters
+ * or until it reaches the end of the content in the buffer (@s->len),
+ * whichever comes first.
*
* On success, it returns a positive number of the number of bytes
* it copied.
*
* On failure it returns -EBUSY if all of the content in the
* sequence has been already read, which includes nothing in the
- * sequence (@s->len == @s->readpos).
+ * sequence (@s->len == @start).
*
* Returns -EFAULT if the copy to userspace fails.
*/
-int seq_buf_to_user(struct seq_buf *s, char __user *ubuf, int cnt)
+int seq_buf_to_user(struct seq_buf *s, char __user *ubuf, size_t start, int cnt)
{
int len;
int ret;
@@ -318,20 +349,17 @@ int seq_buf_to_user(struct seq_buf *s, char __user *ubuf, int cnt)
len = seq_buf_used(s);
- if (len <= s->readpos)
+ if (len <= start)
return -EBUSY;
- len -= s->readpos;
+ len -= start;
if (cnt > len)
cnt = len;
- ret = copy_to_user(ubuf, s->buffer + s->readpos, cnt);
+ ret = copy_to_user(ubuf, s->buffer + start, cnt);
if (ret == cnt)
return -EFAULT;
- cnt -= ret;
-
- s->readpos += cnt;
- return cnt;
+ return cnt - ret;
}
/**
diff --git a/lib/test_kprobes.c b/lib/test_kprobes.c
index eeb1d728d974..7f4d2a77b10f 100644
--- a/lib/test_kprobes.c
+++ b/lib/test_kprobes.c
@@ -12,6 +12,12 @@
#define div_factor 3
+#define KP_CLEAR(_kp) \
+do { \
+ (_kp).addr = NULL; \
+ (_kp).flags = 0; \
+} while (0)
+
static u32 rand1, preh_val, posth_val;
static u32 (*target)(u32 value);
static u32 (*target2)(u32 value);
@@ -115,10 +121,6 @@ static void test_kprobes(struct kunit *test)
current_test = test;
- /* addr and flags should be cleard for reusing kprobe. */
- kp.addr = NULL;
- kp.flags = 0;
-
KUNIT_EXPECT_EQ(test, 0, register_kprobes(kps, 2));
preh_val = 0;
posth_val = 0;
@@ -193,9 +195,6 @@ static void test_kretprobes(struct kunit *test)
struct kretprobe *rps[2] = {&rp, &rp2};
current_test = test;
- /* addr and flags should be cleard for reusing kprobe. */
- rp.kp.addr = NULL;
- rp.kp.flags = 0;
KUNIT_EXPECT_EQ(test, 0, register_kretprobes(rps, 2));
krph_val = 0;
@@ -257,8 +256,6 @@ static void test_stacktrace_on_kretprobe(struct kunit *test)
unsigned long myretaddr = (unsigned long)__builtin_return_address(0);
current_test = test;
- rp3.kp.addr = NULL;
- rp3.kp.flags = 0;
/*
* Run the stacktrace_driver() to record correct return address in
@@ -319,8 +316,6 @@ static void test_stacktrace_on_nested_kretprobe(struct kunit *test)
struct kretprobe *rps[2] = {&rp3, &rp4};
current_test = test;
- rp3.kp.addr = NULL;
- rp3.kp.flags = 0;
//KUNIT_ASSERT_NE(test, myretaddr, stacktrace_driver());
@@ -334,6 +329,17 @@ static void test_stacktrace_on_nested_kretprobe(struct kunit *test)
static int kprobes_test_init(struct kunit *test)
{
+ KP_CLEAR(kp);
+ KP_CLEAR(kp2);
+#ifdef CONFIG_KRETPROBES
+ KP_CLEAR(rp.kp);
+ KP_CLEAR(rp2.kp);
+#ifdef CONFIG_ARCH_CORRECT_STACKTRACE_ON_KRETPROBE
+ KP_CLEAR(rp3.kp);
+ KP_CLEAR(rp4.kp);
+#endif
+#endif
+
target = kprobe_target;
target2 = kprobe_target2;
stacktrace_target = kprobe_stacktrace_target;
diff --git a/mm/damon/core.c b/mm/damon/core.c
index 7afbc20833ea..f0da5cb2b88a 100644
--- a/mm/damon/core.c
+++ b/mm/damon/core.c
@@ -756,6 +756,7 @@ static void damon_do_apply_schemes(struct damon_ctx *c,
unsigned long sz = damon_sz_region(r);
struct timespec64 begin, end;
unsigned long sz_applied = 0;
+ bool skip = false;
if (!s->wmarks.activated)
continue;
@@ -768,14 +769,11 @@ static void damon_do_apply_schemes(struct damon_ctx *c,
if (quota->charge_target_from) {
if (t != quota->charge_target_from)
continue;
- if (r == damon_last_region(t)) {
- quota->charge_target_from = NULL;
- quota->charge_addr_from = 0;
- continue;
- }
if (quota->charge_addr_from &&
- r->ar.end <= quota->charge_addr_from)
- continue;
+ r->ar.end <= quota->charge_addr_from) {
+ skip = true;
+ goto clean_charge_from_skip;
+ }
if (quota->charge_addr_from && r->ar.start <
quota->charge_addr_from) {
@@ -783,17 +781,23 @@ static void damon_do_apply_schemes(struct damon_ctx *c,
r->ar.start, DAMON_MIN_REGION);
if (!sz) {
if (damon_sz_region(r) <=
- DAMON_MIN_REGION)
- continue;
+ DAMON_MIN_REGION) {
+ skip = true;
+ goto clean_charge_from_skip;
+ }
sz = DAMON_MIN_REGION;
}
damon_split_region_at(t, r, sz);
- r = damon_next_region(r);
- sz = damon_sz_region(r);
+ skip = true;
}
+ }
+clean_charge_from_skip:
+ if (r == damon_last_region(t)) {
quota->charge_target_from = NULL;
quota->charge_addr_from = 0;
}
+ if (skip)
+ continue;
if (!damos_valid_target(c, t, r, s))
continue;
@@ -858,7 +862,7 @@ static void damos_set_effective_quota(struct damos_quota *quota)
static void kdamond_apply_schemes(struct damon_ctx *c)
{
struct damon_target *t;
- struct damon_region *r, *next_r;
+ struct damon_region *r;
struct damos *s;
damon_for_each_scheme(s, c) {
@@ -869,7 +873,7 @@ static void kdamond_apply_schemes(struct damon_ctx *c)
if (!s->wmarks.activated)
continue;
- if (!quota->ms && !quota->sz)
+ if (!quota->esz && !quota->ms && !quota->sz)
continue;
/* New charge window starts */
@@ -912,7 +916,7 @@ static void kdamond_apply_schemes(struct damon_ctx *c)
}
damon_for_each_target(t, c) {
- damon_for_each_region_safe(r, next_r, t)
+ damon_for_each_region(r, t)
damon_do_apply_schemes(c, t, r);
}
}
diff --git a/mm/filemap.c b/mm/filemap.c
index 40c186c7210b..62bad5bb5776 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -3112,7 +3112,7 @@ static struct file *do_sync_mmap_readahead(struct vm_fault *vmf)
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
/* Use the readahead code, even if readahead is disabled */
- if (vm_flags & VM_HUGEPAGE) {
+ if ((vm_flags & VM_HUGEPAGE) && HPAGE_PMD_ORDER <= MAX_PAGECACHE_ORDER) {
fpin = maybe_unlock_mmap_for_io(vmf, fpin);
ractl._index &= ~((unsigned long)HPAGE_PMD_NR - 1);
ra->size = HPAGE_PMD_NR;
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index 483cdc373a20..5807f276fe22 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -1701,6 +1701,9 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma,
if (arch_needs_pgtable_deposit())
zap_deposited_table(tlb->mm, pmd);
add_mm_counter(tlb->mm, mm_counter_file(page), -HPAGE_PMD_NR);
+
+ if (pmd_present(orig_pmd) && pmd_dirty(orig_pmd))
+ folio_mark_dirty(page_folio(page));
}
spin_unlock(ptl);
@@ -1736,6 +1739,16 @@ static pmd_t move_soft_dirty_pmd(pmd_t pmd)
return pmd;
}
+static pmd_t clear_uffd_wp_pmd(pmd_t pmd)
+{
+ if (pmd_present(pmd))
+ pmd = pmd_clear_uffd_wp(pmd);
+ else if (is_swap_pmd(pmd))
+ pmd = pmd_swp_clear_uffd_wp(pmd);
+
+ return pmd;
+}
+
bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd)
{
@@ -1773,6 +1786,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr,
pgtable_trans_huge_deposit(mm, new_pmd, pgtable);
}
pmd = move_soft_dirty_pmd(pmd);
+ if (vma_has_uffd_without_event_remap(vma))
+ pmd = clear_uffd_wp_pmd(pmd);
set_pmd_at(mm, new_addr, new_pmd, pmd);
if (force_flush)
flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE);
@@ -2738,6 +2753,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list)
goto out;
}
+ mapping_set_update(&xas, mapping);
+
xas_split_alloc(&xas, folio, folio_order(folio), gfp);
if (xas_error(&xas)) {
ret = xas_error(&xas);
diff --git a/mm/hugetlb.c b/mm/hugetlb.c
index bfcb936084e8..f549adc0b670 100644
--- a/mm/hugetlb.c
+++ b/mm/hugetlb.c
@@ -1397,8 +1397,7 @@ static unsigned long available_huge_pages(struct hstate *h)
static struct page *dequeue_huge_page_vma(struct hstate *h,
struct vm_area_struct *vma,
- unsigned long address, int avoid_reserve,
- long chg)
+ unsigned long address, long chg)
{
struct page *page = NULL;
struct mempolicy *mpol;
@@ -1414,10 +1413,6 @@ static struct page *dequeue_huge_page_vma(struct hstate *h,
if (!vma_has_reserves(vma, chg) && !available_huge_pages(h))
goto err;
- /* If reserves cannot be used, ensure enough pages are in the pool */
- if (avoid_reserve && !available_huge_pages(h))
- goto err;
-
gfp_mask = htlb_alloc_mask(h);
nid = huge_node(vma, address, gfp_mask, &mpol, &nodemask);
@@ -1431,7 +1426,7 @@ static struct page *dequeue_huge_page_vma(struct hstate *h,
if (!page)
page = dequeue_huge_page_nodemask(h, gfp_mask, nid, nodemask);
- if (page && !avoid_reserve && vma_has_reserves(vma, chg)) {
+ if (page && vma_has_reserves(vma, chg)) {
SetHPageRestoreReserve(page);
h->resv_huge_pages--;
}
@@ -3088,7 +3083,8 @@ struct page *alloc_huge_page(struct vm_area_struct *vma,
long map_chg, map_commit;
long gbl_chg;
int ret, idx;
- struct hugetlb_cgroup *h_cg;
+ struct hugetlb_cgroup *h_cg = NULL;
+ struct hugetlb_cgroup *h_cg_rsvd = NULL;
bool deferred_reserve;
idx = hstate_index(h);
@@ -3114,17 +3110,6 @@ struct page *alloc_huge_page(struct vm_area_struct *vma,
vma_end_reservation(h, vma, addr);
return ERR_PTR(-ENOSPC);
}
-
- /*
- * Even though there was no reservation in the region/reserve
- * map, there could be reservations associated with the
- * subpool that can be used. This would be indicated if the
- * return value of hugepage_subpool_get_pages() is zero.
- * However, if avoid_reserve is specified we still avoid even
- * the subpool reservations.
- */
- if (avoid_reserve)
- gbl_chg = 1;
}
/* If this allocation is not consuming a reservation, charge it now.
@@ -3132,7 +3117,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma,
deferred_reserve = map_chg || avoid_reserve;
if (deferred_reserve) {
ret = hugetlb_cgroup_charge_cgroup_rsvd(
- idx, pages_per_huge_page(h), &h_cg);
+ idx, pages_per_huge_page(h), &h_cg_rsvd);
if (ret)
goto out_subpool_put;
}
@@ -3147,7 +3132,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma,
* from the global free pool (global change). gbl_chg == 0 indicates
* a reservation exists for the allocation.
*/
- page = dequeue_huge_page_vma(h, vma, addr, avoid_reserve, gbl_chg);
+ page = dequeue_huge_page_vma(h, vma, addr, gbl_chg);
if (!page) {
spin_unlock_irq(&hugetlb_lock);
@@ -3170,7 +3155,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma,
*/
if (deferred_reserve) {
hugetlb_cgroup_commit_charge_rsvd(idx, pages_per_huge_page(h),
- h_cg, page);
+ h_cg_rsvd, page);
}
spin_unlock_irq(&hugetlb_lock);
@@ -3206,7 +3191,7 @@ struct page *alloc_huge_page(struct vm_area_struct *vma,
out_uncharge_cgroup_reservation:
if (deferred_reserve)
hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h),
- h_cg);
+ h_cg_rsvd);
out_subpool_put:
if (map_chg || avoid_reserve)
hugepage_subpool_put_pages(spool, 1);
@@ -5224,6 +5209,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src,
static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
unsigned long new_addr, pte_t *src_pte, pte_t *dst_pte)
{
+ bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma);
struct hstate *h = hstate_vma(vma);
struct mm_struct *mm = vma->vm_mm;
spinlock_t *src_ptl, *dst_ptl;
@@ -5240,7 +5226,18 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr,
spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING);
pte = huge_ptep_get_and_clear(mm, old_addr, src_pte);
- set_huge_pte_at(mm, new_addr, dst_pte, pte);
+
+ if (need_clear_uffd_wp && pte_marker_uffd_wp(pte))
+ huge_pte_clear(mm, new_addr, dst_pte, huge_page_size(h));
+ else {
+ if (need_clear_uffd_wp) {
+ if (pte_present(pte))
+ pte = huge_pte_clear_uffd_wp(pte);
+ else if (is_swap_pte(pte))
+ pte = pte_swp_clear_uffd_wp(pte);
+ }
+ set_huge_pte_at(mm, new_addr, dst_pte, pte);
+ }
if (src_ptl != dst_ptl)
spin_unlock(src_ptl);
@@ -5278,18 +5275,21 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma,
hugetlb_vma_lock_write(vma);
i_mmap_lock_write(mapping);
for (; old_addr < old_end; old_addr += sz, new_addr += sz) {
+ const unsigned long offset_to_last_entry =
+ (old_addr | last_addr_mask) - old_addr;
+
src_pte = huge_pte_offset(mm, old_addr, sz);
if (!src_pte) {
- old_addr |= last_addr_mask;
- new_addr |= last_addr_mask;
+ old_addr += offset_to_last_entry;
+ new_addr += offset_to_last_entry;
continue;
}
if (huge_pte_none(huge_ptep_get(src_pte)))
continue;
if (huge_pmd_unshare(&tlb, vma, old_addr, src_pte)) {
- old_addr |= last_addr_mask;
- new_addr |= last_addr_mask;
+ old_addr += offset_to_last_entry;
+ new_addr += offset_to_last_entry;
continue;
}
diff --git a/mm/kasan/quarantine.c b/mm/kasan/quarantine.c
index 75585077eb6d..11bd18e851a0 100644
--- a/mm/kasan/quarantine.c
+++ b/mm/kasan/quarantine.c
@@ -99,7 +99,6 @@ static unsigned long quarantine_size;
static DEFINE_RAW_SPINLOCK(quarantine_lock);
DEFINE_STATIC_SRCU(remove_cache_srcu);
-#ifdef CONFIG_PREEMPT_RT
struct cpu_shrink_qlist {
raw_spinlock_t lock;
struct qlist_head qlist;
@@ -108,7 +107,6 @@ struct cpu_shrink_qlist {
static DEFINE_PER_CPU(struct cpu_shrink_qlist, shrink_qlist) = {
.lock = __RAW_SPIN_LOCK_UNLOCKED(shrink_qlist.lock),
};
-#endif
/* Maximum size of the global queue. */
static unsigned long quarantine_max_size;
@@ -319,16 +317,6 @@ static void qlist_move_cache(struct qlist_head *from,
}
}
-#ifndef CONFIG_PREEMPT_RT
-static void __per_cpu_remove_cache(struct qlist_head *q, void *arg)
-{
- struct kmem_cache *cache = arg;
- struct qlist_head to_free = QLIST_INIT;
-
- qlist_move_cache(q, &to_free, cache);
- qlist_free_all(&to_free, cache);
-}
-#else
static void __per_cpu_remove_cache(struct qlist_head *q, void *arg)
{
struct kmem_cache *cache = arg;
@@ -340,7 +328,6 @@ static void __per_cpu_remove_cache(struct qlist_head *q, void *arg)
qlist_move_cache(q, &sq->qlist, cache);
raw_spin_unlock_irqrestore(&sq->lock, flags);
}
-#endif
static void per_cpu_remove_cache(void *arg)
{
@@ -362,6 +349,8 @@ void kasan_quarantine_remove_cache(struct kmem_cache *cache)
{
unsigned long flags, i;
struct qlist_head to_free = QLIST_INIT;
+ int cpu;
+ struct cpu_shrink_qlist *sq;
/*
* Must be careful to not miss any objects that are being moved from
@@ -372,20 +361,18 @@ void kasan_quarantine_remove_cache(struct kmem_cache *cache)
*/
on_each_cpu(per_cpu_remove_cache, cache, 1);
-#ifdef CONFIG_PREEMPT_RT
- {
- int cpu;
- struct cpu_shrink_qlist *sq;
-
- for_each_online_cpu(cpu) {
- sq = per_cpu_ptr(&shrink_qlist, cpu);
- raw_spin_lock_irqsave(&sq->lock, flags);
- qlist_move_cache(&sq->qlist, &to_free, cache);
- raw_spin_unlock_irqrestore(&sq->lock, flags);
- }
- qlist_free_all(&to_free, cache);
+ /*
+ * A CPU can go offline after on_each_cpu() returns, leaving cache
+ * objects on that CPU's shrink list. Scan all possible CPUs to
+ * drain those lists.
+ */
+ for_each_possible_cpu(cpu) {
+ sq = per_cpu_ptr(&shrink_qlist, cpu);
+ raw_spin_lock_irqsave(&sq->lock, flags);
+ qlist_move_cache(&sq->qlist, &to_free, cache);
+ raw_spin_unlock_irqrestore(&sq->lock, flags);
}
-#endif
+ qlist_free_all(&to_free, cache);
raw_spin_lock_irqsave(&quarantine_lock, flags);
for (i = 0; i < QUARANTINE_BATCHES; i++) {
diff --git a/mm/kasan/report.c b/mm/kasan/report.c
index 240560ec8d99..a69d0812a922 100644
--- a/mm/kasan/report.c
+++ b/mm/kasan/report.c
@@ -477,7 +477,7 @@ void kasan_report_invalid_free(void *ptr, unsigned long ip, enum kasan_report_ty
start_report(&flags, true);
- memset(&info, 0, sizeof(info));
+ __memset(&info, 0, sizeof(info));
info.type = type;
info.access_addr = ptr;
info.access_size = 0;
@@ -512,7 +512,7 @@ bool kasan_report(unsigned long addr, size_t size, bool is_write,
start_report(&irq_flags, true);
- memset(&info, 0, sizeof(info));
+ __memset(&info, 0, sizeof(info));
info.type = KASAN_REPORT_ACCESS;
info.access_addr = ptr;
info.access_size = size;
diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c
index ecb7acb3897c..c651f3266e06 100644
--- a/mm/kasan/shadow.c
+++ b/mm/kasan/shadow.c
@@ -259,34 +259,117 @@ void __init __weak kasan_populate_early_vm_area_shadow(void *start,
{
}
+struct vmalloc_populate_data {
+ unsigned long start;
+ struct page **pages;
+};
+
static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr,
- void *unused)
+ void *_data)
{
- unsigned long page;
+ struct vmalloc_populate_data *data = _data;
+ struct page *page;
pte_t pte;
+ int index;
if (likely(!pte_none(*ptep)))
return 0;
- page = __get_free_page(GFP_KERNEL);
- if (!page)
- return -ENOMEM;
-
- memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE);
- pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL);
+ index = PFN_DOWN(addr - data->start);
+ page = data->pages[index];
+ __memset(page_to_virt(page), KASAN_VMALLOC_INVALID, PAGE_SIZE);
+ pte = pfn_pte(page_to_pfn(page), PAGE_KERNEL);
spin_lock(&init_mm.page_table_lock);
if (likely(pte_none(*ptep))) {
set_pte_at(&init_mm, addr, ptep, pte);
- page = 0;
+ data->pages[index] = NULL;
}
spin_unlock(&init_mm.page_table_lock);
- if (page)
- free_page(page);
+
+ return 0;
+}
+
+static void ___free_pages_bulk(struct page **pages, int nr_pages)
+{
+ int i;
+
+ for (i = 0; i < nr_pages; i++) {
+ if (pages[i]) {
+ __free_pages(pages[i], 0);
+ pages[i] = NULL;
+ }
+ }
+}
+
+static int ___alloc_pages_bulk(struct page **pages, int nr_pages, gfp_t gfp_mask)
+{
+ unsigned long nr_populated, nr_total = nr_pages;
+ struct page **page_array = pages;
+
+ while (nr_pages) {
+ nr_populated = alloc_pages_bulk_array(gfp_mask, nr_pages, pages);
+ if (!nr_populated) {
+ ___free_pages_bulk(page_array, nr_total - nr_pages);
+ return -ENOMEM;
+ }
+ pages += nr_populated;
+ nr_pages -= nr_populated;
+ }
+
return 0;
}
-int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
+static int __kasan_populate_vmalloc(unsigned long start, unsigned long end, gfp_t gfp_mask)
+{
+ unsigned long nr_pages, nr_total = PFN_UP(end - start);
+ struct vmalloc_populate_data data;
+ unsigned int flags;
+ int ret = 0;
+
+ data.pages = (struct page **)__get_free_page(gfp_mask | __GFP_ZERO);
+ if (!data.pages)
+ return -ENOMEM;
+
+ while (nr_total) {
+ nr_pages = min(nr_total, PAGE_SIZE / sizeof(data.pages[0]));
+ ret = ___alloc_pages_bulk(data.pages, nr_pages, gfp_mask);
+ if (ret)
+ break;
+
+ data.start = start;
+
+ /*
+ * page tables allocations ignore external gfp mask, enforce it
+ * by the scope API
+ */
+ if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)
+ flags = memalloc_nofs_save();
+ else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == 0)
+ flags = memalloc_noio_save();
+
+ ret = apply_to_page_range(&init_mm, start, nr_pages * PAGE_SIZE,
+ kasan_populate_vmalloc_pte, &data);
+
+ if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO)
+ memalloc_nofs_restore(flags);
+ else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == 0)
+ memalloc_noio_restore(flags);
+
+ ___free_pages_bulk(data.pages, nr_pages);
+ if (ret)
+ break;
+
+ start += nr_pages * PAGE_SIZE;
+ nr_total -= nr_pages;
+ }
+
+ free_page((unsigned long)data.pages);
+
+ return ret;
+}
+
+int kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mask)
{
unsigned long shadow_start, shadow_end;
int ret;
@@ -315,9 +398,7 @@ int kasan_populate_vmalloc(unsigned long addr, unsigned long size)
shadow_start = PAGE_ALIGN_DOWN(shadow_start);
shadow_end = PAGE_ALIGN(shadow_end);
- ret = apply_to_page_range(&init_mm, shadow_start,
- shadow_end - shadow_start,
- kasan_populate_vmalloc_pte, NULL);
+ ret = __kasan_populate_vmalloc(shadow_start, shadow_end, gfp_mask);
if (ret)
return ret;
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index ded9a00b20b5..4c3581364cb2 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -19,6 +19,7 @@
#include <linux/page_table_check.h>
#include <linux/swapops.h>
#include <linux/shmem_fs.h>
+#include <linux/dax.h>
#include <asm/tlb.h>
#include <asm/pgalloc.h>
@@ -1788,6 +1789,8 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr,
if (result != SCAN_SUCCEED)
goto out;
+ mapping_set_update(&xas, mapping);
+
/*
* Ensure we have slots for all the pages in the range. This is
* almost certainly a no-op because most of the pages must be present
diff --git a/mm/kmemleak.c b/mm/kmemleak.c
index d3c9a63d7a9d..2200f34a4f00 100644
--- a/mm/kmemleak.c
+++ b/mm/kmemleak.c
@@ -13,11 +13,12 @@
*
* The following locks and mutexes are used by kmemleak:
*
- * - kmemleak_lock (raw_spinlock_t): protects the object_list modifications and
- * accesses to the object_tree_root (or object_phys_tree_root). The
- * object_list is the main list holding the metadata (struct kmemleak_object)
- * for the allocated memory blocks. The object_tree_root and object_phys_tree_root
- * are red black trees used to look-up metadata based on a pointer to the
+ * - kmemleak_lock (raw_spinlock_t): protects the object_list as well as
+ * del_state modifications and accesses to the object_tree_root (or
+ * object_phys_tree_root). The object_list is the main list holding the
+ * metadata (struct kmemleak_object) for the allocated memory blocks.
+ * The object_tree_root and object_phys_tree_root are red
+ * black trees used to look-up metadata based on a pointer to the
* corresponding memory block. The object_phys_tree_root is for objects
* allocated with physical address. The kmemleak_object structures are
* added to the object_list and object_tree_root (or object_phys_tree_root)
@@ -79,6 +80,7 @@
#include <linux/mutex.h>
#include <linux/rcupdate.h>
#include <linux/stacktrace.h>
+#include <linux/stackdepot.h>
#include <linux/cache.h>
#include <linux/percpu.h>
#include <linux/memblock.h>
@@ -147,6 +149,7 @@ struct kmemleak_object {
struct rcu_head rcu; /* object_list lockless traversal */
/* object usage count; object freed when use_count == 0 */
atomic_t use_count;
+ unsigned int del_state; /* deletion state */
unsigned long pointer;
size_t size;
/* pass surplus references to this pointer */
@@ -159,8 +162,7 @@ struct kmemleak_object {
u32 checksum;
/* memory ranges to be scanned inside an object (empty for all) */
struct hlist_head area_list;
- unsigned long trace[MAX_TRACE];
- unsigned int trace_len;
+ depot_stack_handle_t trace_handle;
unsigned long jiffies; /* creation timestamp */
pid_t pid; /* pid of the current task */
char comm[TASK_COMM_LEN]; /* executable name */
@@ -177,6 +179,11 @@ struct kmemleak_object {
/* flag set for object allocated with physical address */
#define OBJECT_PHYS (1 << 4)
+/* set when __remove_object() called */
+#define DELSTATE_REMOVED (1 << 0)
+/* set to temporarily prevent deletion from object_list */
+#define DELSTATE_NO_DELETE (1 << 1)
+
#define HEX_PREFIX " "
/* number of bytes to print per line; must be 16 or 32 */
#define HEX_ROW_SIZE 16
@@ -346,18 +353,19 @@ static void print_unreferenced(struct seq_file *seq,
struct kmemleak_object *object)
{
int i;
- unsigned int msecs_age = jiffies_to_msecs(jiffies - object->jiffies);
+ unsigned long *entries;
+ unsigned int nr_entries;
+ nr_entries = stack_depot_fetch(object->trace_handle, &entries);
warn_or_seq_printf(seq, "unreferenced object 0x%08lx (size %zu):\n",
- object->pointer, object->size);
- warn_or_seq_printf(seq, " comm \"%s\", pid %d, jiffies %lu (age %d.%03ds)\n",
- object->comm, object->pid, object->jiffies,
- msecs_age / 1000, msecs_age % 1000);
+ object->pointer, object->size);
+ warn_or_seq_printf(seq, " comm \"%s\", pid %d, jiffies %lu\n",
+ object->comm, object->pid, object->jiffies);
hex_dump_object(seq, object);
warn_or_seq_printf(seq, " backtrace:\n");
- for (i = 0; i < object->trace_len; i++) {
- void *ptr = (void *)object->trace[i];
+ for (i = 0; i < nr_entries; i++) {
+ void *ptr = (void *)entries[i];
warn_or_seq_printf(seq, " [<%p>] %pS\n", ptr, ptr);
}
}
@@ -370,15 +378,16 @@ static void print_unreferenced(struct seq_file *seq,
static void dump_object_info(struct kmemleak_object *object)
{
pr_notice("Object 0x%08lx (size %zu):\n",
- object->pointer, object->size);
+ object->pointer, object->size);
pr_notice(" comm \"%s\", pid %d, jiffies %lu\n",
- object->comm, object->pid, object->jiffies);
+ object->comm, object->pid, object->jiffies);
pr_notice(" min_count = %d\n", object->min_count);
pr_notice(" count = %d\n", object->count);
pr_notice(" flags = 0x%x\n", object->flags);
pr_notice(" checksum = %u\n", object->checksum);
pr_notice(" backtrace:\n");
- stack_trace_print(object->trace, object->trace_len, 4);
+ if (object->trace_handle)
+ stack_depot_print(object->trace_handle);
}
/*
@@ -570,7 +579,9 @@ static void __remove_object(struct kmemleak_object *object)
rb_erase(&object->rb_node, object->flags & OBJECT_PHYS ?
&object_phys_tree_root :
&object_tree_root);
- list_del_rcu(&object->object_list);
+ if (!(object->del_state & DELSTATE_NO_DELETE))
+ list_del_rcu(&object->object_list);
+ object->del_state |= DELSTATE_REMOVED;
}
/*
@@ -594,12 +605,23 @@ static struct kmemleak_object *find_and_remove_object(unsigned long ptr, int ali
return object;
}
-/*
- * Save stack trace to the given array of MAX_TRACE size.
- */
-static int __save_stack_trace(unsigned long *trace)
+static noinline depot_stack_handle_t set_track_prepare(void)
{
- return stack_trace_save(trace, MAX_TRACE, 2);
+ depot_stack_handle_t trace_handle;
+ unsigned long entries[MAX_TRACE];
+ unsigned int nr_entries;
+
+ /*
+ * Use object_cache to determine whether kmemleak_init() has
+ * been invoked. stack_depot_early_init() is called before
+ * kmemleak_init() in mm_core_init().
+ */
+ if (!object_cache)
+ return 0;
+ nr_entries = stack_trace_save(entries, ARRAY_SIZE(entries), 3);
+ trace_handle = stack_depot_save(entries, nr_entries, GFP_NOWAIT);
+
+ return trace_handle;
}
/*
@@ -636,6 +658,7 @@ static void __create_object(unsigned long ptr, size_t size,
object->count = 0; /* white color initially */
object->jiffies = jiffies;
object->checksum = 0;
+ object->del_state = 0;
/* task information */
if (in_hardirq()) {
@@ -656,7 +679,7 @@ static void __create_object(unsigned long ptr, size_t size,
}
/* kernel backtrace */
- object->trace_len = __save_stack_trace(object->trace);
+ object->trace_handle = set_track_prepare();
raw_spin_lock_irqsave(&kmemleak_lock, flags);
@@ -695,7 +718,6 @@ static void __create_object(unsigned long ptr, size_t size,
rb_link_node(&object->rb_node, rb_parent, link);
rb_insert_color(&object->rb_node, is_phys ? &object_phys_tree_root :
&object_tree_root);
-
list_add_tail_rcu(&object->object_list, &object_list);
out:
raw_spin_unlock_irqrestore(&kmemleak_lock, flags);
@@ -1094,7 +1116,7 @@ void __ref kmemleak_update_trace(const void *ptr)
}
raw_spin_lock_irqsave(&object->lock, flags);
- object->trace_len = __save_stack_trace(object->trace);
+ object->trace_handle = set_track_prepare();
raw_spin_unlock_irqrestore(&object->lock, flags);
put_object(object);
@@ -1466,22 +1488,66 @@ static void scan_gray_list(void)
/*
* Conditionally call resched() in a object iteration loop while making sure
* that the given object won't go away without RCU read lock by performing a
- * get_object() if !pinned.
- *
- * Return: false if can't do a cond_resched() due to get_object() failure
- * true otherwise
+ * get_object() if necessaary.
*/
-static bool kmemleak_cond_resched(struct kmemleak_object *object, bool pinned)
+static void kmemleak_cond_resched(struct kmemleak_object *object)
{
- if (!pinned && !get_object(object))
- return false;
+ if (!get_object(object))
+ return; /* Try next object */
+
+ raw_spin_lock_irq(&kmemleak_lock);
+ if (object->del_state & DELSTATE_REMOVED)
+ goto unlock_put; /* Object removed */
+ object->del_state |= DELSTATE_NO_DELETE;
+ raw_spin_unlock_irq(&kmemleak_lock);
rcu_read_unlock();
cond_resched();
rcu_read_lock();
- if (!pinned)
- put_object(object);
- return true;
+
+ raw_spin_lock_irq(&kmemleak_lock);
+ if (object->del_state & DELSTATE_REMOVED)
+ list_del_rcu(&object->object_list);
+ object->del_state &= ~DELSTATE_NO_DELETE;
+unlock_put:
+ raw_spin_unlock_irq(&kmemleak_lock);
+ put_object(object);
+}
+
+/*
+ * Scan all task kernel stacks, rescheduling between tasks. Each task is looked
+ * up and pinned within its own RCU read-side section, so no lock is held across
+ * the scan and the walk cannot trip the soft lockup watchdog.
+ */
+static void kmemleak_scan_task_stacks(void)
+{
+ struct pid *pid;
+ int nr = 1;
+
+ do {
+ struct task_struct *p = NULL;
+
+ rcu_read_lock();
+ pid = find_ge_pid(nr, &init_pid_ns);
+ if (pid) {
+ nr = pid_nr(pid) + 1;
+ p = pid_task(pid, PIDTYPE_PID);
+ if (p)
+ get_task_struct(p);
+ }
+ rcu_read_unlock();
+
+ if (p) {
+ void *stack = try_get_task_stack(p);
+
+ if (stack) {
+ scan_block(stack, stack + THREAD_SIZE, NULL);
+ put_task_stack(p);
+ }
+ put_task_struct(p);
+ }
+ cond_resched();
+ } while (pid && !scan_should_stop());
}
/*
@@ -1495,15 +1561,12 @@ static void kmemleak_scan(void)
struct zone *zone;
int __maybe_unused i;
int new_leaks = 0;
- int loop_cnt = 0;
jiffies_last_scan = jiffies;
/* prepare the kmemleak_object's */
rcu_read_lock();
list_for_each_entry_rcu(object, &object_list, object_list) {
- bool obj_pinned = false;
-
raw_spin_lock_irq(&object->lock);
#ifdef DEBUG
/*
@@ -1529,19 +1592,13 @@ static void kmemleak_scan(void)
/* reset the reference count (whiten the object) */
object->count = 0;
- if (color_gray(object) && get_object(object)) {
+ if (color_gray(object) && get_object(object))
list_add_tail(&object->gray_list, &gray_list);
- obj_pinned = true;
- }
raw_spin_unlock_irq(&object->lock);
- /*
- * Do a cond_resched() every 64k objects to avoid soft lockup.
- */
- if (!(++loop_cnt & 0xffff) &&
- !kmemleak_cond_resched(object, obj_pinned))
- loop_cnt--; /* Try again on next object */
+ if (need_resched())
+ kmemleak_cond_resched(object);
}
rcu_read_unlock();
@@ -1583,19 +1640,8 @@ static void kmemleak_scan(void)
/*
* Scanning the task stacks (may introduce false negatives).
*/
- if (kmemleak_stack_scan) {
- struct task_struct *p, *g;
-
- rcu_read_lock();
- for_each_process_thread(g, p) {
- void *stack = try_get_task_stack(p);
- if (stack) {
- scan_block(stack, stack + THREAD_SIZE, NULL);
- put_task_stack(p);
- }
- }
- rcu_read_unlock();
- }
+ if (kmemleak_stack_scan)
+ kmemleak_scan_task_stacks();
/*
* Scan the objects already referenced from the sections scanned
@@ -1608,14 +1654,9 @@ static void kmemleak_scan(void)
* scan and color them gray until the next scan.
*/
rcu_read_lock();
- loop_cnt = 0;
list_for_each_entry_rcu(object, &object_list, object_list) {
- /*
- * Do a cond_resched() every 64k objects to avoid soft lockup.
- */
- if (!(++loop_cnt & 0xffff) &&
- !kmemleak_cond_resched(object, false))
- loop_cnt--; /* Try again on next object */
+ if (need_resched())
+ kmemleak_cond_resched(object);
/*
* This is racy but we can save the overhead of lock/unlock
@@ -1650,14 +1691,9 @@ static void kmemleak_scan(void)
* Scanning result reporting.
*/
rcu_read_lock();
- loop_cnt = 0;
list_for_each_entry_rcu(object, &object_list, object_list) {
- /*
- * Do a cond_resched() every 64k objects to avoid soft lockup.
- */
- if (!(++loop_cnt & 0xffff) &&
- !kmemleak_cond_resched(object, false))
- loop_cnt--; /* Try again on next object */
+ if (need_resched())
+ kmemleak_cond_resched(object);
/*
* This is racy but we can save the overhead of lock/unlock
@@ -2069,8 +2105,10 @@ static int __init kmemleak_boot_config(char *str)
return -EINVAL;
if (strcmp(str, "off") == 0)
kmemleak_disable();
- else if (strcmp(str, "on") == 0)
+ else if (strcmp(str, "on") == 0) {
kmemleak_skip_disable = 1;
+ stack_depot_want_early_init();
+ }
else
return -EINVAL;
return 0;
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 7b64dee7f386..e270dae84941 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -608,7 +608,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val)
{
unsigned int x;
- cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id());
+ __cgroup_rstat_updated(memcg->css.cgroup, smp_processor_id());
x = __this_cpu_add_return(stats_updates, abs(val));
if (x > MEMCG_CHARGE_BATCH) {
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 33b21fc6d90d..1a6cc0161fb7 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1185,7 +1185,8 @@ static pg_data_t __ref *hotadd_init_pgdat(int nid)
pgdat = NODE_DATA(nid);
/* init node's zones as empty zones, we don't have any present pages.*/
- free_area_init_core_hotplug(pgdat);
+ if (free_area_init_core_hotplug(pgdat))
+ return NULL;
/*
* The node we allocated has no zone fallback lists. For avoiding
diff --git a/mm/mempolicy.c b/mm/mempolicy.c
index 518adfe4ad85..c4a38d1b695f 100644
--- a/mm/mempolicy.c
+++ b/mm/mempolicy.c
@@ -1068,6 +1068,8 @@ static int migrate_to_node(struct mm_struct *mm, int source, int dest,
* space range and MPOL_MF_DISCONTIG_OK, this call can not fail.
*/
vma = find_vma(mm, 0);
+ if (unlikely(!vma))
+ return 0;
VM_BUG_ON(!(flags & (MPOL_MF_MOVE | MPOL_MF_MOVE_ALL)));
queue_pages_range(mm, vma->vm_start, mm->task_size, &nmask,
flags | MPOL_MF_DISCONTIG_OK, &pagelist);
diff --git a/mm/migrate.c b/mm/migrate.c
index 161ab6e2713d..72ee6896a2cb 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -1010,32 +1010,29 @@ static int move_to_new_folio(struct folio *dst, struct folio *src,
}
/*
- * To record some information during migration, we use some unused
- * fields (mapping and private) of struct folio of the newly allocated
- * destination folio. This is safe because nobody is using them
- * except us.
+ * To record some information during migration, we use unused private
+ * field of struct folio of the newly allocated destination folio.
+ * This is safe because nobody is using it except us.
*/
-union migration_ptr {
- struct anon_vma *anon_vma;
- struct address_space *mapping;
+enum {
+ PAGE_WAS_MAPPED = BIT(0),
};
+
static void __migrate_folio_record(struct folio *dst,
- unsigned long page_was_mapped,
+ int page_was_mapped,
struct anon_vma *anon_vma)
{
- union migration_ptr ptr = { .anon_vma = anon_vma };
- dst->mapping = ptr.mapping;
- dst->private = (void *)page_was_mapped;
+ dst->private = (void *)anon_vma + page_was_mapped;
}
static void __migrate_folio_extract(struct folio *dst,
int *page_was_mappedp,
struct anon_vma **anon_vmap)
{
- union migration_ptr ptr = { .mapping = dst->mapping };
- *anon_vmap = ptr.anon_vma;
- *page_was_mappedp = (unsigned long)dst->private;
- dst->mapping = NULL;
+ unsigned long private = (unsigned long)dst->private;
+
+ *anon_vmap = (struct anon_vma *)(private & ~PAGE_WAS_MAPPED);
+ *page_was_mappedp = private & PAGE_WAS_MAPPED;
dst->private = NULL;
}
diff --git a/mm/mlock.c b/mm/mlock.c
index 3bf9e1d263da..0124022f0f37 100644
--- a/mm/mlock.c
+++ b/mm/mlock.c
@@ -141,7 +141,7 @@ static struct lruvec *__munlock_page(struct page *page, struct lruvec *lruvec)
munlock:
if (TestClearPageMlocked(page)) {
- __mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages);
+ mod_zone_page_state(page_zone(page), NR_MLOCK, -nr_pages);
if (isolated || !PageUnevictable(page))
__count_vm_events(UNEVICTABLE_PGMUNLOCKED, nr_pages);
else
diff --git a/mm/mmap.c b/mm/mmap.c
index f93526b3df0c..1eb86aba7de5 100644
--- a/mm/mmap.c
+++ b/mm/mmap.c
@@ -1131,6 +1131,13 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct *
* acceptable for merging, so we can do all of this optimistically. But
* we do that READ_ONCE() to make sure that we never re-load the pointer.
*
+ * The READ_ONCE() establishes an address dependency between anon_vma and
+ * any access to its fields, which pairs with the assignment to
+ * vma->anon_vma performed with release semantics in __anon_vma_prepare().
+ *
+ * This is especially important as anon_vma's are SLAB_TYPESAFE_BY_RCU so
+ * accessing an uninitialised anon_vma's fields may result in a UAF.
+ *
* IOW: that the "list_is_singular()" test on the anon_vma_chain only
* matters for the 'stable anon_vma' case (ie the thing we want to avoid
* is to return an anon_vma that is "complex" due to having gone through
@@ -1143,6 +1150,7 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct *
static struct anon_vma *reusable_anon_vma(struct vm_area_struct *old, struct vm_area_struct *a, struct vm_area_struct *b)
{
if (anon_vma_compatible(a, b)) {
+ /* Paired with a memory barrier in __anon_vma_prepare(). */
struct anon_vma *anon_vma = READ_ONCE(old->anon_vma);
if (anon_vma && list_is_singular(&old->anon_vma_chain))
@@ -2974,6 +2982,7 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,
unsigned long populate = 0;
unsigned long ret = -EINVAL;
struct file *file;
+ vm_flags_t vm_flags;
pr_warn_once("%s (%d) uses deprecated remap_file_pages() syscall. See Documentation/mm/remap_file_pages.rst.\n",
current->comm, current->pid);
@@ -2990,12 +2999,60 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,
if (pgoff + (size >> PAGE_SHIFT) < pgoff)
return ret;
- if (mmap_write_lock_killable(mm))
+ if (mmap_read_lock_killable(mm))
return -EINTR;
+ /*
+ * Look up VMA under read lock first so we can perform the security
+ * without holding locks (which can be problematic). We reacquire a
+ * write lock later and check nothing changed underneath us.
+ */
vma = vma_lookup(mm, start);
- if (!vma || !(vma->vm_flags & VM_SHARED))
+ if (!vma || !(vma->vm_flags & VM_SHARED)) {
+ mmap_read_unlock(mm);
+ return -EINVAL;
+ }
+
+ prot |= vma->vm_flags & VM_READ ? PROT_READ : 0;
+ prot |= vma->vm_flags & VM_WRITE ? PROT_WRITE : 0;
+ prot |= vma->vm_flags & VM_EXEC ? PROT_EXEC : 0;
+
+ flags &= MAP_NONBLOCK;
+ flags |= MAP_SHARED | MAP_FIXED | MAP_POPULATE;
+ if (vma->vm_flags & VM_LOCKED)
+ flags |= MAP_LOCKED;
+
+ /* Save vm_flags used to calculate prot and flags, and recheck later. */
+ vm_flags = vma->vm_flags;
+ file = get_file(vma->vm_file);
+
+ mmap_read_unlock(mm);
+
+ /* Call outside mmap_lock to be consistent with other callers. */
+ ret = security_mmap_file(file, prot, flags);
+ if (ret) {
+ fput(file);
+ return ret;
+ }
+
+ ret = -EINVAL;
+
+ /* OK security check passed, take write lock + let it rip. */
+ if (mmap_write_lock_killable(mm)) {
+ fput(file);
+ return -EINTR;
+ }
+
+ vma = vma_lookup(mm, start);
+
+ if (!vma)
+ goto out;
+
+ /* Make sure things didn't change under us. */
+ if (vma->vm_flags != vm_flags)
+ goto out;
+ if (vma->vm_file != file)
goto out;
if (start + size > vma->vm_end) {
@@ -3023,25 +3080,11 @@ SYSCALL_DEFINE5(remap_file_pages, unsigned long, start, unsigned long, size,
goto out;
}
- prot |= vma->vm_flags & VM_READ ? PROT_READ : 0;
- prot |= vma->vm_flags & VM_WRITE ? PROT_WRITE : 0;
- prot |= vma->vm_flags & VM_EXEC ? PROT_EXEC : 0;
-
- flags &= MAP_NONBLOCK;
- flags |= MAP_SHARED | MAP_FIXED | MAP_POPULATE;
- if (vma->vm_flags & VM_LOCKED)
- flags |= MAP_LOCKED;
-
- file = get_file(vma->vm_file);
- ret = security_mmap_file(vma->vm_file, prot, flags);
- if (ret)
- goto out_fput;
ret = do_mmap(vma->vm_file, start, size,
prot, flags, pgoff, &populate, NULL);
-out_fput:
- fput(file);
out:
mmap_write_unlock(mm);
+ fput(file);
if (populate)
mm_populate(ret, populate);
if (!IS_ERR_VALUE(ret))
@@ -3329,13 +3372,14 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma)
*/
struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
unsigned long addr, unsigned long len, pgoff_t pgoff,
- bool *need_rmap_locks)
+ bool *need_rmap_locks, bool keep_source)
{
struct vm_area_struct *vma = *vmap;
unsigned long vma_start = vma->vm_start;
struct mm_struct *mm = vma->vm_mm;
struct vm_area_struct *new_vma, *prev;
bool faulted_in_anon_vma = true;
+ bool can_self_merge = false;
validate_mm_mt(mm);
/*
@@ -3345,15 +3389,29 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap,
if (unlikely(vma_is_anonymous(vma) && !vma->anon_vma)) {
pgoff = addr >> PAGE_SHIFT;
faulted_in_anon_vma = false;
+ can_self_merge = true;
}
new_vma = find_vma_prev(mm, addr, &prev);
if (new_vma && new_vma->vm_start < addr + len)
return NULL; /* should never get here */
- new_vma = vma_merge(mm, prev, addr, addr + len, vma->vm_flags,
- vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma),
- vma->vm_userfaultfd_ctx, anon_vma_name(vma));
+ /*
+ * If the original VMA is kept (MREMAP_DONTUNMAP), the source and
+ * destination VMA must be treated distinctly.
+ *
+ * A merge violates this, so in this case disallow a self-merge. A
+ * self-merge can only occur if the new range is immediately adjacent to
+ * the source VMA, so simply do not attempt to merge in that case.
+ */
+ if (can_self_merge && keep_source &&
+ (addr == vma->vm_end || vma->vm_start == addr + len))
+ new_vma = NULL;
+ else
+ new_vma = vma_merge(mm, prev, addr, addr + len, vma->vm_flags,
+ vma->anon_vma, vma->vm_file, pgoff,
+ vma_policy(vma), vma->vm_userfaultfd_ctx,
+ anon_vma_name(vma));
if (new_vma) {
/*
* Source vma may have been merged into new_vma
diff --git a/mm/mremap.c b/mm/mremap.c
index 6ed28eeae5a8..ec66e7be8afe 100644
--- a/mm/mremap.c
+++ b/mm/mremap.c
@@ -138,6 +138,7 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
struct vm_area_struct *new_vma, pmd_t *new_pmd,
unsigned long new_addr, bool need_rmap_locks)
{
+ bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma);
struct mm_struct *mm = vma->vm_mm;
pte_t *old_pte, *new_pte, pte;
spinlock_t *old_ptl, *new_ptl;
@@ -198,7 +199,18 @@ static void move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd,
force_flush = true;
pte = move_pte(pte, new_vma->vm_page_prot, old_addr, new_addr);
pte = move_soft_dirty_pte(pte);
- set_pte_at(mm, new_addr, new_pte, pte);
+
+ if (need_clear_uffd_wp && pte_marker_uffd_wp(pte))
+ pte_clear(mm, new_addr, new_pte);
+ else {
+ if (need_clear_uffd_wp) {
+ if (pte_present(pte))
+ pte = pte_clear_uffd_wp(pte);
+ else if (is_swap_pte(pte))
+ pte = pte_swp_clear_uffd_wp(pte);
+ }
+ set_pte_at(mm, new_addr, new_pte, pte);
+ }
}
arch_leave_lazy_mmu_mode();
@@ -257,6 +269,15 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr,
if (WARN_ON_ONCE(!pmd_none(*new_pmd)))
return false;
+ /* If this pmd belongs to a uffd vma with remap events disabled, we need
+ * to ensure that the uffd-wp state is cleared from all pgtables. This
+ * means recursing into lower page tables in move_page_tables(), and we
+ * can reuse the existing code if we simply treat the entry as "not
+ * moved".
+ */
+ if (vma_has_uffd_without_event_remap(vma))
+ return false;
+
/*
* We don't have to worry about the ordering of src and dst
* ptlocks because exclusive mmap_lock prevents deadlock.
@@ -306,6 +327,15 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr,
if (WARN_ON_ONCE(!pud_none(*new_pud)))
return false;
+ /* If this pud belongs to a uffd vma with remap events disabled, we need
+ * to ensure that the uffd-wp state is cleared from all pgtables. This
+ * means recursing into lower page tables in move_page_tables(), and we
+ * can reuse the existing code if we simply treat the entry as "not
+ * moved".
+ */
+ if (vma_has_uffd_without_event_remap(vma))
+ return false;
+
/*
* We don't have to worry about the ordering of src and dst
* ptlocks because exclusive mmap_lock prevents deadlock.
@@ -585,6 +615,8 @@ static unsigned long move_vma(struct vm_area_struct *vma,
int split = 0;
int err = 0;
bool need_rmap_locks;
+ const bool split_before = vma->vm_start != old_addr;
+ const bool split_after = vma->vm_end != old_addr + old_len;
/*
* We'd prefer to avoid failure later on in do_munmap:
@@ -597,9 +629,9 @@ static unsigned long move_vma(struct vm_area_struct *vma,
to_account = new_len;
if (vma->vm_ops && vma->vm_ops->may_split) {
- if (vma->vm_start != old_addr)
+ if (split_before)
err = vma->vm_ops->may_split(vma, old_addr);
- if (!err && vma->vm_end != old_addr + old_len)
+ if (!err && split_after)
err = vma->vm_ops->may_split(vma, old_addr + old_len);
if (err)
return err;
@@ -617,6 +649,19 @@ static unsigned long move_vma(struct vm_area_struct *vma,
if (err)
return err;
+ /*
+ * To account mlock()'d pages correctly in the MREMAP_DONTUNMAP
+ * case perform any split ahead of time for an mlock()'d VMA.
+ */
+ if ((flags & MREMAP_DONTUNMAP) && (vma->vm_flags & VM_LOCKED)) {
+ if (split_before)
+ err = split_vma(mm, vma, old_addr, 1);
+ if (!err && split_after)
+ err = split_vma(mm, vma, old_addr + old_len, 0);
+ if (err)
+ return err;
+ }
+
if (vm_flags & VM_ACCOUNT) {
if (security_vm_enough_memory_mm(mm, to_account >> PAGE_SHIFT))
return -ENOMEM;
@@ -624,7 +669,7 @@ static unsigned long move_vma(struct vm_area_struct *vma,
new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT);
new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff,
- &need_rmap_locks);
+ &need_rmap_locks, flags & MREMAP_DONTUNMAP);
if (!new_vma) {
if (vm_flags & VM_ACCOUNT)
vm_unacct_memory(to_account >> PAGE_SHIFT);
@@ -685,6 +730,9 @@ static unsigned long move_vma(struct vm_area_struct *vma,
untrack_pfn_clear(vma);
if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) {
+ /* Self-merge is disallowed. */
+ VM_WARN_ON_ONCE(new_vma == vma);
+
/* We always clear VM_LOCKED[ONFAULT] on the old vma */
vma->vm_flags &= VM_LOCKED_CLEAR_MASK;
@@ -692,9 +740,19 @@ static unsigned long move_vma(struct vm_area_struct *vma,
* anon_vma links of the old vma is no longer needed after its page
* table has been moved.
*/
- if (new_vma != vma && vma->vm_start == old_addr &&
- vma->vm_end == (old_addr + old_len))
+ if (vma->vm_start == old_addr &&
+ vma->vm_end == (old_addr + old_len)) {
+ const pgoff_t pgoff_unfaulted = vma->vm_start >> PAGE_SHIFT;
+
unlink_anon_vmas(vma);
+ /*
+ * The VMA is now unfaulted and it is an invariant that
+ * unfaulted anonymous VMAs have page offset equal to
+ * vma->vm_start >> PAGE_SHIFT.
+ */
+ if (vma_is_anonymous(vma) && !vma->vm_file)
+ vma->vm_pgoff = pgoff_unfaulted;
+ }
/* Because we won't unmap we don't need to touch locked_vm */
return new_addr;
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5f679f699477..85b189bcab7e 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -7730,7 +7730,7 @@ static void __meminit zone_init_internals(struct zone *zone, enum zone_type idx,
* NOTE: this function is only called during memory hotplug
*/
#ifdef CONFIG_MEMORY_HOTPLUG
-void __ref free_area_init_core_hotplug(struct pglist_data *pgdat)
+int __ref free_area_init_core_hotplug(struct pglist_data *pgdat)
{
int nid = pgdat->node_id;
enum zone_type z;
@@ -7738,8 +7738,14 @@ void __ref free_area_init_core_hotplug(struct pglist_data *pgdat)
pgdat_init_internals(pgdat);
- if (pgdat->per_cpu_nodestats == &boot_nodestats)
- pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat);
+ if (pgdat->per_cpu_nodestats == &boot_nodestats) {
+ struct per_cpu_nodestat __percpu *p;
+
+ p = alloc_percpu(struct per_cpu_nodestat);
+ if (!p)
+ return -ENOMEM;
+ pgdat->per_cpu_nodestats = p;
+ }
/*
* Reset the nr_zones, order and highest_zoneidx before reuse.
@@ -7767,6 +7773,8 @@ void __ref free_area_init_core_hotplug(struct pglist_data *pgdat)
for (z = 0; z < MAX_NR_ZONES; z++)
zone_init_internals(&pgdat->node_zones[z], z, nid, 0);
+
+ return 0;
}
#endif
diff --git a/mm/page_table_check.c b/mm/page_table_check.c
index 9392544d4754..451114bdc340 100644
--- a/mm/page_table_check.c
+++ b/mm/page_table_check.c
@@ -164,20 +164,31 @@ void __page_table_check_pte_clear(struct mm_struct *mm, unsigned long addr,
if (&init_mm == mm)
return;
- if (pte_user_accessible_page(pte)) {
+ if (pte_user_accessible_page(pte) && !pte_special(pte)) {
page_table_check_clear(mm, addr, pte_pfn(pte),
PAGE_SIZE >> PAGE_SHIFT);
}
}
EXPORT_SYMBOL(__page_table_check_pte_clear);
+static inline bool page_table_check_huge_zero_pmd(pmd_t pmd)
+{
+ unsigned long pfn = pmd_pfn(pmd);
+
+ if (!pfn_valid(pfn))
+ return false;
+
+ return is_huge_zero_page(pfn_to_page(pfn));
+}
+
void __page_table_check_pmd_clear(struct mm_struct *mm, unsigned long addr,
pmd_t pmd)
{
if (&init_mm == mm)
return;
- if (pmd_user_accessible_page(pmd)) {
+ if (pmd_user_accessible_page(pmd) &&
+ !page_table_check_huge_zero_pmd(pmd)) {
page_table_check_clear(mm, addr, pmd_pfn(pmd),
PMD_SIZE >> PAGE_SHIFT);
}
@@ -204,7 +215,7 @@ void __page_table_check_pte_set(struct mm_struct *mm, unsigned long addr,
return;
__page_table_check_pte_clear(mm, addr, *ptep);
- if (pte_user_accessible_page(pte)) {
+ if (pte_user_accessible_page(pte) && !pte_special(pte)) {
page_table_check_set(mm, addr, pte_pfn(pte),
PAGE_SIZE >> PAGE_SHIFT,
pte_write(pte));
@@ -219,7 +230,8 @@ void __page_table_check_pmd_set(struct mm_struct *mm, unsigned long addr,
return;
__page_table_check_pmd_clear(mm, addr, *pmdp);
- if (pmd_user_accessible_page(pmd)) {
+ if (pmd_user_accessible_page(pmd) &&
+ !page_table_check_huge_zero_pmd(pmd)) {
page_table_check_set(mm, addr, pmd_pfn(pmd),
PMD_SIZE >> PAGE_SHIFT,
pmd_write(pmd));
diff --git a/mm/pagewalk.c b/mm/pagewalk.c
index 09df5b12aef7..9c3eae516130 100644
--- a/mm/pagewalk.c
+++ b/mm/pagewalk.c
@@ -523,6 +523,14 @@ int walk_page_range_novma(struct mm_struct *mm, unsigned long start,
* to account for page table freeing on vmap huge page mapping.
*/
mmap_assert_write_locked(walk.mm);
+ /*
+ * x86, arm64 ptdump allow walks of efi mm's and x86 ptdump allows walks
+ * of arbitrary mm's.
+ *
+ * However, they both must also hold the init_mm lock to account for
+ * concurrent kernel page table freeing.
+ */
+ mmap_assert_write_locked(&init_mm);
return walk_pgd_range(start, end, &walk);
}
diff --git a/mm/ptdump.c b/mm/ptdump.c
index 8e29b90d6bc3..4c6cbbaf833f 100644
--- a/mm/ptdump.c
+++ b/mm/ptdump.c
@@ -154,11 +154,18 @@ void ptdump_walk_pgd(struct ptdump_state *st, struct mm_struct *mm, pgd_t *pgd)
get_online_mems();
mmap_write_lock(mm);
+ /* To stabilise kernel page tables we must hold the init_mm lock too. */
+ if (mm != &init_mm)
+ mmap_write_lock_nested(&init_mm, SINGLE_DEPTH_NESTING);
+
while (range->start != range->end) {
walk_page_range_novma(mm, range->start, range->end,
&ptdump_ops, pgd, st);
range++;
}
+
+ if (mm != &init_mm)
+ mmap_write_unlock(&init_mm);
mmap_write_unlock(mm);
put_online_mems();
diff --git a/mm/readahead.c b/mm/readahead.c
index e0bef7c0d7af..2ad297cfae7d 100644
--- a/mm/readahead.c
+++ b/mm/readahead.c
@@ -462,19 +462,6 @@ static int try_context_readahead(struct address_space *mapping,
return 1;
}
-/*
- * There are some parts of the kernel which assume that PMD entries
- * are exactly HPAGE_PMD_ORDER. Those should be fixed, but until then,
- * limit the maximum allocation order to PMD size. I'm not aware of any
- * assumptions about maximum order if THP are disabled, but 8 seems like
- * a good order (that's 1MB if you're using 4kB pages)
- */
-#ifdef CONFIG_TRANSPARENT_HUGEPAGE
-#define MAX_PAGECACHE_ORDER HPAGE_PMD_ORDER
-#else
-#define MAX_PAGECACHE_ORDER 8
-#endif
-
static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index,
pgoff_t mark, unsigned int order, gfp_t gfp)
{
diff --git a/mm/rmap.c b/mm/rmap.c
index 886000da89a6..8e74d143433a 100644
--- a/mm/rmap.c
+++ b/mm/rmap.c
@@ -210,7 +210,11 @@ int __anon_vma_prepare(struct vm_area_struct *vma)
/* page_table_lock to protect against threads */
spin_lock(&mm->page_table_lock);
if (likely(!vma->anon_vma)) {
- vma->anon_vma = anon_vma;
+ /*
+ * Make anon_vma fields visible before anon_vma is published.
+ * Paired with an address dependency in reusable_anon_vma().
+ */
+ smp_store_release(&vma->anon_vma, anon_vma);
anon_vma_chain_link(vma, avc, anon_vma);
anon_vma->num_active_vmas++;
allocated = NULL;
diff --git a/mm/secretmem.c b/mm/secretmem.c
index b570a6e25b6b..84d10092ecb0 100644
--- a/mm/secretmem.c
+++ b/mm/secretmem.c
@@ -18,6 +18,9 @@
#include <linux/secretmem.h>
#include <linux/set_memory.h>
#include <linux/sched/signal.h>
+#include <linux/sched/user.h>
+#include <linux/cred.h>
+#include <linux/slab.h>
#include <uapi/linux/magic.h>
@@ -47,10 +50,69 @@ bool secretmem_active(void)
return !!atomic_read(&secretmem_users);
}
+struct secretmem_inode_state {
+ struct user_struct *user;
+ atomic_long_t nr_pages_accounted;
+};
+
+static bool __secretmem_account_pages(struct user_struct *user,
+ unsigned long nr_pages)
+{
+ unsigned long page_limit, cur_pages, new_pages;
+
+ if (!nr_pages)
+ return true;
+
+ page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT;
+
+ cur_pages = atomic_long_read(&user->locked_vm);
+ do {
+ new_pages = cur_pages + nr_pages;
+ if (new_pages > page_limit)
+ return false;
+ } while (!atomic_long_try_cmpxchg(&user->locked_vm,
+ &cur_pages, new_pages));
+ return true;
+}
+
+static bool secretmem_account_folio(struct secretmem_inode_state *state,
+ struct folio *folio)
+{
+ const unsigned long nr_pages = folio_nr_pages(folio);
+
+ if (!__secretmem_account_pages(state->user, nr_pages))
+ return false;
+
+ atomic_long_add(nr_pages, &state->nr_pages_accounted);
+ return true;
+}
+
+static void __secretmem_unaccount_pages(struct secretmem_inode_state *state,
+ unsigned long nr_pages)
+{
+ atomic_long_sub(nr_pages, &state->user->locked_vm);
+ atomic_long_sub(nr_pages, &state->nr_pages_accounted);
+}
+
+static void secretmem_unaccount_folio(struct secretmem_inode_state *state,
+ struct folio *folio)
+{
+ __secretmem_unaccount_pages(state, folio_nr_pages(folio));
+}
+
+static void secretmem_unaccount_all_folios(struct secretmem_inode_state *state)
+{
+ const unsigned long nr_pages_accounted =
+ atomic_long_read(&state->nr_pages_accounted);
+
+ __secretmem_unaccount_pages(state, nr_pages_accounted);
+}
+
static vm_fault_t secretmem_fault(struct vm_fault *vmf)
{
struct address_space *mapping = vmf->vma->vm_file->f_mapping;
struct inode *inode = file_inode(vmf->vma->vm_file);
+ struct secretmem_inode_state *state = inode->i_private;
pgoff_t offset = vmf->pgoff;
gfp_t gfp = vmf->gfp_mask;
unsigned long addr;
@@ -72,8 +134,15 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf)
goto out;
}
+ if (!secretmem_account_folio(state, page_folio(page))) {
+ put_page(page);
+ ret = VM_FAULT_SIGBUS;
+ goto out;
+ }
+
err = set_direct_map_invalid_noflush(page);
if (err) {
+ secretmem_unaccount_folio(state, page_folio(page));
put_page(page);
ret = vmf_error(err);
goto out;
@@ -82,6 +151,7 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf)
__SetPageUptodate(page);
err = add_to_page_cache_lru(page, mapping, offset, gfp);
if (unlikely(err)) {
+ secretmem_unaccount_folio(state, page_folio(page));
/*
* If a split of large page was required, it
* already happened when we marked the page invalid
@@ -112,23 +182,30 @@ static const struct vm_operations_struct secretmem_vm_ops = {
.fault = secretmem_fault,
};
+static void secretmem_destroy_inode_priv(struct inode *inode)
+{
+ struct secretmem_inode_state *state = inode->i_private;
+
+ secretmem_unaccount_all_folios(state);
+ free_uid(state->user);
+ kfree(state);
+ inode->i_private = NULL;
+}
+
static int secretmem_release(struct inode *inode, struct file *file)
{
atomic_dec(&secretmem_users);
+ secretmem_destroy_inode_priv(inode);
+
return 0;
}
static int secretmem_mmap(struct file *file, struct vm_area_struct *vma)
{
- unsigned long len = vma->vm_end - vma->vm_start;
-
if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0)
return -EINVAL;
- if (mlock_future_check(vma->vm_mm, vma->vm_flags | VM_LOCKED, len))
- return -EAGAIN;
-
- vma->vm_flags |= VM_LOCKED | VM_DONTDUMP;
+ vma->vm_flags |= VM_DONTDUMP;
vma->vm_ops = &secretmem_vm_ops;
return 0;
@@ -188,19 +265,40 @@ static const struct inode_operations secretmem_iops = {
static struct vfsmount *secretmem_mnt;
+static int secretmem_init_inode_priv(struct inode *inode)
+{
+ struct secretmem_inode_state *state;
+
+ state = kzalloc(sizeof(*state), GFP_KERNEL);
+ if (!state)
+ return -ENOMEM;
+
+ state->user = get_uid(current_user());
+ inode->i_private = state;
+ return 0;
+}
+
static struct file *secretmem_file_create(unsigned long flags)
{
struct file *file = ERR_PTR(-ENOMEM);
struct inode *inode;
const char *anon_name = "[secretmem]";
+ int err;
inode = anon_inode_make_secure_inode(secretmem_mnt->mnt_sb, anon_name, NULL);
if (IS_ERR(inode))
return ERR_CAST(inode);
+
+ err = secretmem_init_inode_priv(inode);
+ if (err)
+ goto err_free_inode;
+
file = alloc_file_pseudo(inode, secretmem_mnt, "secretmem",
O_RDWR, &secretmem_fops);
- if (IS_ERR(file))
- goto err_free_inode;
+ if (IS_ERR(file)) {
+ err = PTR_ERR(file);
+ goto err_free_priv;
+ }
mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER);
mapping_set_unevictable(inode->i_mapping);
@@ -213,10 +311,11 @@ static struct file *secretmem_file_create(unsigned long flags)
inode->i_size = 0;
return file;
-
+err_free_priv:
+ secretmem_destroy_inode_priv(inode);
err_free_inode:
iput(inode);
- return file;
+ return ERR_PTR(err);
}
SYSCALL_DEFINE1(memfd_secret, unsigned int, flags)
diff --git a/mm/shmem.c b/mm/shmem.c
index 2034b9b3fb71..cfc573e04327 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -462,8 +462,8 @@ static bool shmem_confirm_swap(struct address_space *mapping,
static int shmem_huge __read_mostly = SHMEM_HUGE_NEVER;
-bool shmem_is_huge(struct vm_area_struct *vma, struct inode *inode,
- pgoff_t index, bool shmem_huge_force)
+static bool __shmem_is_huge(struct vm_area_struct *vma, struct inode *inode,
+ pgoff_t index, bool shmem_huge_force)
{
loff_t i_size;
@@ -495,6 +495,15 @@ bool shmem_is_huge(struct vm_area_struct *vma, struct inode *inode,
}
}
+bool shmem_is_huge(struct vm_area_struct *vma, struct inode *inode,
+ pgoff_t index, bool shmem_huge_force)
+{
+ if (HPAGE_PMD_ORDER > MAX_PAGECACHE_ORDER)
+ return false;
+
+ return __shmem_is_huge(vma, inode, index, shmem_huge_force);
+}
+
#if defined(CONFIG_SYSFS)
static int shmem_parse_huge(const char *str)
{
diff --git a/mm/slab.h b/mm/slab.h
index 0202a8c2f0d2..42eb31375fdf 100644
--- a/mm/slab.h
+++ b/mm/slab.h
@@ -40,6 +40,11 @@ struct slab {
struct {
unsigned inuse:16;
unsigned objects:15;
+ /*
+ * If slab debugging is enabled then the
+ * frozen bit can be reused to indicate
+ * that the slab was corrupted
+ */
unsigned frozen:1;
};
};
diff --git a/mm/slub.c b/mm/slub.c
index 5428a41d70eb..bafb09a3c00a 100644
--- a/mm/slub.c
+++ b/mm/slub.c
@@ -1196,6 +1196,11 @@ static int check_slab(struct kmem_cache *s, struct slab *slab)
slab->inuse, slab->objects);
return 0;
}
+ if (slab->frozen) {
+ slab_err(s, slab, "Slab disabled since SLUB metadata consistency check failed");
+ return 0;
+ }
+
/* Slab_pad_check fixes things up after itself */
slab_pad_check(s, slab);
return 1;
@@ -1392,6 +1397,7 @@ static noinline bool alloc_debug_processing(struct kmem_cache *s,
slab_fix(s, "Marking all objects used");
slab->inuse = slab->objects;
slab->freelist = NULL;
+ slab->frozen = 1; /* mark consistency-failed slab as frozen */
}
return false;
}
@@ -2088,7 +2094,8 @@ static void *alloc_single_from_partial(struct kmem_cache *s,
slab->inuse++;
if (!alloc_debug_processing(s, slab, object, orig_size)) {
- remove_partial(n, slab);
+ if (folio_test_slab(slab_folio(slab)))
+ remove_partial(n, slab);
return NULL;
}
diff --git a/mm/userfaultfd.c b/mm/userfaultfd.c
index 992a0a16846f..998c7075c62a 100644
--- a/mm/userfaultfd.c
+++ b/mm/userfaultfd.c
@@ -642,21 +642,23 @@ static __always_inline ssize_t __mcopy_atomic(struct mm_struct *dst_mm,
}
dst_pmdval = pmd_read_atomic(dst_pmd);
- /*
- * If the dst_pmd is mapped as THP don't
- * override it and just be strict.
- */
- if (unlikely(pmd_trans_huge(dst_pmdval))) {
- err = -EEXIST;
- break;
- }
if (unlikely(pmd_none(dst_pmdval)) &&
unlikely(__pte_alloc(dst_mm, dst_pmd))) {
err = -ENOMEM;
break;
}
- /* If an huge pmd materialized from under us fail */
- if (unlikely(pmd_trans_huge(*dst_pmd))) {
+ dst_pmdval = pmd_read_atomic(dst_pmd);
+ /*
+ * If the dst_pmd is THP don't override it and just be strict.
+ * (This includes the case where the PMD used to be THP and
+ * changed back to none after __pte_alloc().)
+ */
+ if (unlikely(!pmd_present(dst_pmdval) || pmd_trans_huge(dst_pmdval) ||
+ pmd_devmap(dst_pmdval))) {
+ err = -EEXIST;
+ break;
+ }
+ if (unlikely(pmd_bad(dst_pmdval))) {
err = -EFAULT;
break;
}
diff --git a/mm/vmalloc.c b/mm/vmalloc.c
index 6c8a9b748db1..32d172fa3e6c 100644
--- a/mm/vmalloc.c
+++ b/mm/vmalloc.c
@@ -1708,7 +1708,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size,
BUG_ON(va->va_start < vstart);
BUG_ON(va->va_end > vend);
- ret = kasan_populate_vmalloc(addr, size);
+ ret = kasan_populate_vmalloc(addr, size, gfp_mask);
if (ret) {
free_vmap_area(va);
return ERR_PTR(ret);
@@ -3997,7 +3997,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets,
/* populate the kasan shadow space */
for (area = 0; area < nr_vms; area++) {
- if (kasan_populate_vmalloc(vas[area]->va_start, sizes[area]))
+ if (kasan_populate_vmalloc(vas[area]->va_start, sizes[area], GFP_KERNEL))
goto err_free_shadow;
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index d39d3f88e0d2..0b87b66913bc 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4758,15 +4758,9 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio);
- /* unevictable */
- if (!folio_evictable(folio)) {
- success = lru_gen_del_folio(lruvec, folio, true);
- VM_WARN_ON_ONCE_FOLIO(!success, folio);
- folio_set_unevictable(folio);
- lruvec_add_folio(lruvec, folio);
- __count_vm_events(UNEVICTABLE_PGCULLED, delta);
- return true;
- }
+ /* unevictable: let it through and the generic path will cull it */
+ if (!folio_evictable(folio))
+ return false;
/* dirty lazyfree */
if (type == LRU_GEN_FILE && folio_test_anon(folio) && folio_test_dirty(folio)) {
@@ -5054,11 +5048,9 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap
sc->nr_reclaimed += reclaimed;
list_for_each_entry_safe_reverse(folio, next, &list, lru) {
- if (!folio_evictable(folio)) {
- list_del(&folio->lru);
- folio_putback_lru(folio);
+ /* move_folios_to_lru() culls unevictable folios via folio_putback_lru() */
+ if (!folio_evictable(folio))
continue;
- }
if (folio_test_reclaim(folio) &&
(folio_test_dirty(folio) || folio_test_writeback(folio))) {
diff --git a/net/8021q/vlan.c b/net/8021q/vlan.c
index 7c77482f3159..8b38466fb041 100644
--- a/net/8021q/vlan.c
+++ b/net/8021q/vlan.c
@@ -324,10 +324,9 @@ static void vlan_transfer_features(struct net_device *dev,
netif_inherit_tso_max(vlandev, dev);
- if (vlan_hw_offload_capable(dev->features, vlan->vlan_proto))
- vlandev->hard_header_len = dev->hard_header_len;
- else
- vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN;
+ vlandev->needed_headroom = dev->needed_headroom + VLAN_HLEN;
+ vlandev->needed_tailroom = dev->needed_tailroom;
+ vlandev->hard_header_len = dev->hard_header_len;
#if IS_ENABLED(CONFIG_FCOE)
vlandev->fcoe_ddp_xid = dev->fcoe_ddp_xid;
diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c
index a703d7f09f50..37ccd570419c 100644
--- a/net/8021q/vlan_dev.c
+++ b/net/8021q/vlan_dev.c
@@ -108,6 +108,8 @@ static netdev_tx_t vlan_dev_hard_start_xmit(struct sk_buff *skb,
unsigned int len;
int ret;
+ len = skb->len;
+
/* Handle non-VLAN frames if they are sent to us, for example by DHCP.
*
* NOTE: THIS ASSUMES DIX ETHERNET, SPECIFICALLY NOT SUPPORTING
@@ -119,10 +121,39 @@ static netdev_tx_t vlan_dev_hard_start_xmit(struct sk_buff *skb,
vlan_tci = vlan->vlan_id;
vlan_tci |= vlan_dev_get_egress_qos_mask(dev, skb->priority);
__vlan_hwaccel_put_tag(skb, vlan->vlan_proto, vlan_tci);
+ } else if (skb_headlen(skb) >= VLAN_ETH_HLEN + sizeof(__be16) &&
+ !skb_shared(skb) && !skb_vlan_tag_present(skb) &&
+ !skb_metadata_len(skb) &&
+ (skb->ip_summed != CHECKSUM_PARTIAL ||
+ skb_checksum_start_offset(skb) >= VLAN_ETH_HLEN) &&
+ vlan_hw_offload_capable(READ_ONCE(vlan->real_dev->features),
+ vlan->vlan_proto)) {
+ u16 vlan_tci;
+
+ /* The frame carries its outer tag in-band, either inserted
+ * by vlan_dev_hard_header() or provided by the sender.
+ * Let the real device insert it: some NICs can not offload
+ * checksum or TSO for frames carrying an in-band tag.
+ * Not all ndo_start_xmit() callers set the mac header.
+ * vlan_set_encap_proto() may read two bytes after the vlan
+ * header, so frames with a shorter linear part are sent
+ * unchanged, as are shared skbs (e.g. from pktgen), skbs
+ * already carrying a hwaccel tag (e.g. from generic XDP
+ * redirect), skbs carrying XDP metadata, and CHECKSUM_PARTIAL
+ * skbs whose checksum starts within the header to be popped.
+ * On allocation failure, drop the frame rather than handing
+ * it to the real device with an in-band tag.
+ */
+ skb_reset_mac_header(skb);
+ if (unlikely(__skb_vlan_pop(skb, &vlan_tci))) {
+ dev_kfree_skb_any(skb);
+ this_cpu_inc(vlan->vlan_pcpu_stats->tx_dropped);
+ return NETDEV_TX_OK;
+ }
+ __vlan_hwaccel_put_tag(skb, vlan->vlan_proto, vlan_tci);
}
skb->dev = vlan->real_dev;
- len = skb->len;
if (unlikely(netpoll_tx_running(dev)))
return vlan_netpoll_send_skb(vlan, skb);
@@ -505,26 +536,6 @@ static const struct header_ops vlan_header_ops = {
.parse_protocol = vlan_parse_protocol,
};
-static int vlan_passthru_hard_header(struct sk_buff *skb, struct net_device *dev,
- unsigned short type,
- const void *daddr, const void *saddr,
- unsigned int len)
-{
- struct vlan_dev_priv *vlan = vlan_dev_priv(dev);
- struct net_device *real_dev = vlan->real_dev;
-
- if (saddr == NULL)
- saddr = dev->dev_addr;
-
- return dev_hard_header(skb, real_dev, type, daddr, saddr, len);
-}
-
-static const struct header_ops vlan_passthru_header_ops = {
- .create = vlan_passthru_hard_header,
- .parse = eth_header_parse,
- .parse_protocol = vlan_parse_protocol,
-};
-
static struct device_type vlan_type = {
.name = "vlan",
};
@@ -577,14 +588,10 @@ static int vlan_dev_init(struct net_device *dev)
dev->fcoe_ddp_xid = real_dev->fcoe_ddp_xid;
#endif
- dev->needed_headroom = real_dev->needed_headroom;
- if (vlan_hw_offload_capable(real_dev->features, vlan->vlan_proto)) {
- dev->header_ops = &vlan_passthru_header_ops;
- dev->hard_header_len = real_dev->hard_header_len;
- } else {
- dev->header_ops = &vlan_header_ops;
- dev->hard_header_len = real_dev->hard_header_len + VLAN_HLEN;
- }
+ dev->needed_headroom = real_dev->needed_headroom + VLAN_HLEN;
+ dev->needed_tailroom = real_dev->needed_tailroom;
+ dev->header_ops = &vlan_header_ops;
+ dev->hard_header_len = real_dev->hard_header_len;
dev->netdev_ops = &vlan_netdev_ops;
diff --git a/net/9p/trans_fd.c b/net/9p/trans_fd.c
index 1c42360ff3e6..ed7e96c40948 100644
--- a/net/9p/trans_fd.c
+++ b/net/9p/trans_fd.c
@@ -875,6 +875,7 @@ static int p9_socket_open(struct p9_client *client, struct socket *csocket)
}
csocket->sk->sk_allocation = GFP_NOIO;
+ csocket->sk->sk_use_task_frag = false;
file = sock_alloc_file(csocket, 0, NULL);
if (IS_ERR(file)) {
pr_err("%s (%d): failed to map fd\n",
diff --git a/net/9p/trans_xen.c b/net/9p/trans_xen.c
index 36f432ca4cd5..7b3d18f96dce 100644
--- a/net/9p/trans_xen.c
+++ b/net/9p/trans_xen.c
@@ -273,45 +273,52 @@ static void xen_9pfs_front_free(struct xen_9pfs_front_priv *priv)
{
int i, j;
- write_lock(&xen_9pfs_lock);
- list_del(&priv->list);
- write_unlock(&xen_9pfs_lock);
-
- for (i = 0; i < priv->num_rings; i++) {
- struct xen_9pfs_dataring *ring = &priv->rings[i];
-
- cancel_work_sync(&ring->work);
-
- if (!priv->rings[i].intf)
- break;
- if (priv->rings[i].irq > 0)
- unbind_from_irqhandler(priv->rings[i].irq, ring);
- if (priv->rings[i].data.in) {
- for (j = 0;
- j < (1 << priv->rings[i].intf->ring_order);
- j++) {
- grant_ref_t ref;
-
- ref = priv->rings[i].intf->ref[j];
- gnttab_end_foreign_access(ref, NULL);
- }
- free_pages_exact(priv->rings[i].data.in,
+ if (priv->rings) {
+ for (i = 0; i < priv->num_rings; i++) {
+ struct xen_9pfs_dataring *ring = &priv->rings[i];
+
+ cancel_work_sync(&ring->work);
+
+ if (!priv->rings[i].intf)
+ break;
+ if (priv->rings[i].irq > 0)
+ unbind_from_irqhandler(priv->rings[i].irq, ring);
+ if (priv->rings[i].data.in) {
+ for (j = 0;
+ j < (1 << priv->rings[i].intf->ring_order);
+ j++) {
+ grant_ref_t ref;
+
+ ref = priv->rings[i].intf->ref[j];
+ gnttab_end_foreign_access(ref, NULL);
+ }
+ free_pages_exact(priv->rings[i].data.in,
1UL << (priv->rings[i].intf->ring_order +
XEN_PAGE_SHIFT));
+ }
+ gnttab_end_foreign_access(priv->rings[i].ref, NULL);
+ free_page((unsigned long)priv->rings[i].intf);
}
- gnttab_end_foreign_access(priv->rings[i].ref, NULL);
- free_page((unsigned long)priv->rings[i].intf);
+ kfree(priv->rings);
}
- kfree(priv->rings);
kfree(priv->tag);
kfree(priv);
}
static void xen_9pfs_front_remove(struct xenbus_device *dev)
{
- struct xen_9pfs_front_priv *priv = dev_get_drvdata(&dev->dev);
+ struct xen_9pfs_front_priv *priv;
+ write_lock(&xen_9pfs_lock);
+ priv = dev_get_drvdata(&dev->dev);
+ if (priv == NULL) {
+ write_unlock(&xen_9pfs_lock);
+ return;
+ }
dev_set_drvdata(&dev->dev, NULL);
+ list_del(&priv->list);
+ write_unlock(&xen_9pfs_lock);
+
xen_9pfs_front_free(priv);
}
@@ -378,7 +385,7 @@ static int xen_9pfs_front_init(struct xenbus_device *dev)
{
int ret, i;
struct xenbus_transaction xbt;
- struct xen_9pfs_front_priv *priv = dev_get_drvdata(&dev->dev);
+ struct xen_9pfs_front_priv *priv;
char *versions, *v;
unsigned int max_rings, max_ring_order, len = 0;
@@ -406,6 +413,10 @@ static int xen_9pfs_front_init(struct xenbus_device *dev)
if (p9_xen_trans.maxsize > XEN_FLEX_RING_SIZE(max_ring_order))
p9_xen_trans.maxsize = XEN_FLEX_RING_SIZE(max_ring_order) / 2;
+ priv = kzalloc(sizeof(*priv), GFP_KERNEL);
+ if (!priv)
+ return -ENOMEM;
+ priv->dev = dev;
priv->num_rings = XEN_9PFS_NUM_RINGS;
priv->rings = kcalloc(priv->num_rings, sizeof(*priv->rings),
GFP_KERNEL);
@@ -464,6 +475,11 @@ static int xen_9pfs_front_init(struct xenbus_device *dev)
goto error;
}
+ write_lock(&xen_9pfs_lock);
+ dev_set_drvdata(&dev->dev, priv);
+ list_add_tail(&priv->list, &xen_9pfs_devs);
+ write_unlock(&xen_9pfs_lock);
+
xenbus_switch_state(dev, XenbusStateInitialised);
return 0;
@@ -478,19 +494,6 @@ static int xen_9pfs_front_init(struct xenbus_device *dev)
static int xen_9pfs_front_probe(struct xenbus_device *dev,
const struct xenbus_device_id *id)
{
- struct xen_9pfs_front_priv *priv = NULL;
-
- priv = kzalloc(sizeof(*priv), GFP_KERNEL);
- if (!priv)
- return -ENOMEM;
-
- priv->dev = dev;
- dev_set_drvdata(&dev->dev, priv);
-
- write_lock(&xen_9pfs_lock);
- list_add_tail(&priv->list, &xen_9pfs_devs);
- write_unlock(&xen_9pfs_lock);
-
return 0;
}
diff --git a/net/ax25/af_ax25.c b/net/ax25/af_ax25.c
index 862e03493b7e..64b81a693e25 100644
--- a/net/ax25/af_ax25.c
+++ b/net/ax25/af_ax25.c
@@ -1270,28 +1270,18 @@ static int __must_check ax25_connect(struct socket *sock,
}
}
- /*
- * Must bind first - autobinding in this may or may not work. If
- * the socket is already bound, check to see if the device has
- * been filled in, error if it hasn't.
- */
+ /* Must bind first - autobinding does not work. */
if (sock_flag(sk, SOCK_ZAPPED)) {
- /* check if we can remove this feature. It is broken. */
- printk(KERN_WARNING "ax25_connect(): %s uses autobind, please contact jreuter@yaina.de\n",
- current->comm);
- if ((err = ax25_rt_autobind(ax25, &fsa->fsa_ax25.sax25_call)) < 0) {
- kfree(digi);
- goto out_release;
- }
+ kfree(digi);
+ err = -EINVAL;
+ goto out_release;
+ }
- ax25_fillin_cb(ax25, ax25->ax25_dev);
- ax25_cb_add(ax25);
- } else {
- if (ax25->ax25_dev == NULL) {
- kfree(digi);
- err = -EHOSTUNREACH;
- goto out_release;
- }
+ /* Check to see if the device has been filled in, error if it hasn't. */
+ if (ax25->ax25_dev == NULL) {
+ kfree(digi);
+ err = -EHOSTUNREACH;
+ goto out_release;
}
if (sk->sk_type == SOCK_SEQPACKET &&
diff --git a/net/ax25/ax25_route.c b/net/ax25/ax25_route.c
index 69de75db0c9c..10577434f40b 100644
--- a/net/ax25/ax25_route.c
+++ b/net/ax25/ax25_route.c
@@ -373,80 +373,6 @@ ax25_route *ax25_get_route(ax25_address *addr, struct net_device *dev)
return ax25_rt;
}
-/*
- * Adjust path: If you specify a default route and want to connect
- * a target on the digipeater path but w/o having a special route
- * set before, the path has to be truncated from your target on.
- */
-static inline void ax25_adjust_path(ax25_address *addr, ax25_digi *digipeat)
-{
- int k;
-
- for (k = 0; k < digipeat->ndigi; k++) {
- if (ax25cmp(addr, &digipeat->calls[k]) == 0)
- break;
- }
-
- digipeat->ndigi = k;
-}
-
-
-/*
- * Find which interface to use.
- */
-int ax25_rt_autobind(ax25_cb *ax25, ax25_address *addr)
-{
- ax25_uid_assoc *user;
- ax25_route *ax25_rt;
- int err = 0;
-
- ax25_route_lock_use();
- ax25_rt = ax25_get_route(addr, NULL);
- if (!ax25_rt) {
- ax25_route_lock_unuse();
- return -EHOSTUNREACH;
- }
- rcu_read_lock();
- if ((ax25->ax25_dev = ax25_dev_ax25dev(ax25_rt->dev)) == NULL) {
- err = -EHOSTUNREACH;
- goto put;
- }
-
- user = ax25_findbyuid(current_euid());
- if (user) {
- ax25->source_addr = user->call;
- ax25_uid_put(user);
- } else {
- if (ax25_uid_policy && !capable(CAP_NET_BIND_SERVICE)) {
- err = -EPERM;
- goto put;
- }
- ax25->source_addr = *(ax25_address *)ax25->ax25_dev->dev->dev_addr;
- }
-
- if (ax25_rt->digipeat != NULL) {
- ax25->digipeat = kmemdup(ax25_rt->digipeat, sizeof(ax25_digi),
- GFP_ATOMIC);
- if (ax25->digipeat == NULL) {
- err = -ENOMEM;
- goto put;
- }
- ax25_adjust_path(addr, ax25->digipeat);
- }
-
- if (ax25->sk != NULL) {
- local_bh_disable();
- bh_lock_sock(ax25->sk);
- sock_reset_flag(ax25->sk, SOCK_ZAPPED);
- bh_unlock_sock(ax25->sk);
- local_bh_enable();
- }
-
-put:
- rcu_read_unlock();
- ax25_route_lock_unuse();
- return err;
-}
struct sk_buff *ax25_rt_build_path(struct sk_buff *skb, ax25_address *src,
ax25_address *dest, ax25_digi *digi)
diff --git a/net/bluetooth/6lowpan.c b/net/bluetooth/6lowpan.c
index 723ead2c04f1..1b5a5bee3d07 100644
--- a/net/bluetooth/6lowpan.c
+++ b/net/bluetooth/6lowpan.c
@@ -998,16 +998,20 @@ static int get_l2cap_conn(char *buf, bdaddr_t *addr, u8 *addr_type,
hci_dev_lock(hdev);
hcon = hci_conn_hash_lookup_le(hdev, addr, le_addr_type);
- hci_dev_unlock(hdev);
- hci_dev_put(hdev);
-
- if (!hcon)
+ if (!hcon) {
+ hci_dev_unlock(hdev);
+ hci_dev_put(hdev);
return -ENOENT;
+ }
- *conn = (struct l2cap_conn *)hcon->l2cap_data;
+ *conn = hcon->l2cap_data ?
+ l2cap_conn_hold_unless_zero(hcon->l2cap_data) : NULL;
BT_DBG("conn %p dst %pMR type %u", *conn, &hcon->dst, hcon->dst_type);
+ hci_dev_unlock(hdev);
+ hci_dev_put(hdev);
+
return 0;
}
@@ -1062,23 +1066,15 @@ static void disconnect_all_peers(void)
} while (nchans);
}
-struct set_enable {
- struct work_struct work;
- bool flag;
-};
-
-static void do_enable_set(struct work_struct *work)
+static void do_enable_set(bool flag)
{
- struct set_enable *set_enable = container_of(work,
- struct set_enable, work);
-
- if (!set_enable->flag || enable_6lowpan != set_enable->flag)
+ if (!flag || enable_6lowpan != flag)
/* Disconnect existing connections if 6lowpan is
* disabled
*/
disconnect_all_peers();
- enable_6lowpan = set_enable->flag;
+ enable_6lowpan = flag;
mutex_lock(&set_lock);
if (listen_chan) {
@@ -1088,22 +1084,11 @@ static void do_enable_set(struct work_struct *work)
listen_chan = bt_6lowpan_listen();
mutex_unlock(&set_lock);
-
- kfree(set_enable);
}
static int lowpan_enable_set(void *data, u64 val)
{
- struct set_enable *set_enable;
-
- set_enable = kzalloc(sizeof(*set_enable), GFP_KERNEL);
- if (!set_enable)
- return -ENOMEM;
-
- set_enable->flag = !!val;
- INIT_WORK(&set_enable->work, do_enable_set);
-
- schedule_work(&set_enable->work);
+ do_enable_set(!!val);
return 0;
}
@@ -1150,18 +1135,22 @@ static ssize_t lowpan_control_write(struct file *fp,
if (conn) {
struct lowpan_peer *peer;
- if (!is_bt_6lowpan(conn->hcon))
+ if (!is_bt_6lowpan(conn->hcon)) {
+ l2cap_conn_put(conn);
return -EINVAL;
+ }
peer = lookup_peer(conn);
if (peer) {
BT_DBG("6LoWPAN connection already exists");
+ l2cap_conn_put(conn);
return -EALREADY;
}
BT_DBG("conn %p dst %pMR type %d user %u", conn,
&conn->hcon->dst, conn->hcon->dst_type,
addr_type);
+ l2cap_conn_put(conn);
}
ret = bt_6lowpan_connect(&addr, addr_type);
@@ -1177,6 +1166,8 @@ static ssize_t lowpan_control_write(struct file *fp,
return ret;
ret = bt_6lowpan_disconnect(conn, addr_type);
+ if (conn)
+ l2cap_conn_put(conn);
if (ret < 0)
return ret;
diff --git a/net/bluetooth/af_bluetooth.c b/net/bluetooth/af_bluetooth.c
index 2e37841e4835..212ac760f3d9 100644
--- a/net/bluetooth/af_bluetooth.c
+++ b/net/bluetooth/af_bluetooth.c
@@ -363,14 +363,11 @@ int bt_sock_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
if (flags & MSG_OOB)
return -EOPNOTSUPP;
- lock_sock(sk);
-
skb = skb_recv_datagram(sk, flags, &err);
if (!skb) {
if (sk->sk_shutdown & RCV_SHUTDOWN)
err = 0;
- release_sock(sk);
return err;
}
@@ -396,8 +393,6 @@ int bt_sock_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
skb_free_datagram(sk, skb);
- release_sock(sk);
-
if (flags & MSG_TRUNC)
copied = skblen;
@@ -630,10 +625,11 @@ int bt_sock_ioctl(struct socket *sock, unsigned int cmd, unsigned long arg)
if (sk->sk_state == BT_LISTEN)
return -EINVAL;
- lock_sock(sk);
+ spin_lock(&sk->sk_receive_queue.lock);
skb = skb_peek(&sk->sk_receive_queue);
amount = skb ? skb->len : 0;
- release_sock(sk);
+ spin_unlock(&sk->sk_receive_queue.lock);
+
err = put_user(amount, (int __user *)arg);
break;
diff --git a/net/bluetooth/hci_conn.c b/net/bluetooth/hci_conn.c
index 9d94b4f55574..0c0c052294fc 100644
--- a/net/bluetooth/hci_conn.c
+++ b/net/bluetooth/hci_conn.c
@@ -412,11 +412,13 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data)
cp.tx_bandwidth = cpu_to_le32(0x00001f40);
cp.rx_bandwidth = cpu_to_le32(0x00001f40);
+ hci_dev_lock(hdev);
+
switch (conn->codec.id) {
case BT_CODEC_MSBC:
if (!find_next_esco_param(conn, esco_param_msbc,
ARRAY_SIZE(esco_param_msbc)))
- return -EINVAL;
+ goto unlock;
param = &esco_param_msbc[conn->attempt - 1];
cp.tx_coding_format.id = 0x05;
@@ -442,7 +444,7 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data)
case BT_CODEC_TRANSPARENT:
if (!find_next_esco_param(conn, esco_param_msbc,
ARRAY_SIZE(esco_param_msbc)))
- return -EINVAL;
+ goto unlock;
param = &esco_param_msbc[conn->attempt - 1];
cp.tx_coding_format.id = 0x03;
@@ -469,11 +471,11 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data)
if (lmp_esco_capable(conn->link)) {
if (!find_next_esco_param(conn, esco_param_cvsd,
ARRAY_SIZE(esco_param_cvsd)))
- return -EINVAL;
+ goto unlock;
param = &esco_param_cvsd[conn->attempt - 1];
} else {
if (conn->attempt > ARRAY_SIZE(sco_param_cvsd))
- return -EINVAL;
+ goto unlock;
param = &sco_param_cvsd[conn->attempt - 1];
}
cp.tx_coding_format.id = 2;
@@ -496,9 +498,11 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data)
cp.out_transport_unit_size = 16;
break;
default:
- return -EINVAL;
+ goto unlock;
}
+ hci_dev_unlock(hdev);
+
cp.retrans_effort = param->retrans_effort;
cp.pkt_type = __cpu_to_le16(param->pkt_type);
cp.max_latency = __cpu_to_le16(param->max_latency);
@@ -507,6 +511,10 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data)
return -EIO;
return 0;
+
+unlock:
+ hci_dev_unlock(hdev);
+ return -EINVAL;
}
static bool hci_setup_sync_conn(struct hci_conn *conn, __u16 handle)
@@ -1006,11 +1014,37 @@ static struct hci_conn *__hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t
{
struct hci_conn *conn;
+ switch (type) {
+ case ACL_LINK:
+ if (!hdev->acl_mtu)
+ return ERR_PTR(-ECONNREFUSED);
+ break;
+ case ISO_LINK:
+ if (hdev->iso_mtu)
+ /* Dedicated ISO Buffer exists */
+ break;
+ fallthrough;
+ case LE_LINK:
+ if (hdev->le_mtu && hdev->le_mtu < HCI_MIN_LE_MTU)
+ return ERR_PTR(-ECONNREFUSED);
+ if (!hdev->le_mtu && hdev->acl_mtu < HCI_MIN_LE_MTU)
+ return ERR_PTR(-ECONNREFUSED);
+ break;
+ case SCO_LINK:
+ case ESCO_LINK:
+ if (!hdev->sco_pkts)
+ /* Controller does not support SCO or eSCO over HCI */
+ return ERR_PTR(-ECONNREFUSED);
+ break;
+ default:
+ return ERR_PTR(-ECONNREFUSED);
+ }
+
bt_dev_dbg(hdev, "dst %pMR handle 0x%4.4x", dst, handle);
conn = kzalloc(sizeof(*conn), GFP_KERNEL);
if (!conn)
- return NULL;
+ return ERR_PTR(-ENOMEM);
bacpy(&conn->dst, dst);
bacpy(&conn->src, &hdev->bdaddr);
@@ -1040,10 +1074,12 @@ static struct hci_conn *__hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t
switch (type) {
case ACL_LINK:
conn->pkt_type = hdev->pkt_type & ACL_PTYPE_MASK;
+ conn->mtu = hdev->acl_mtu;
break;
case LE_LINK:
/* conn->src should reflect the local identity address */
hci_copy_identity_address(hdev, &conn->src, &conn->src_type);
+ conn->mtu = hdev->le_mtu ? hdev->le_mtu : hdev->acl_mtu;
break;
case ISO_LINK:
/* conn->src should reflect the local identity address */
@@ -1055,6 +1091,8 @@ static struct hci_conn *__hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t
else if (conn->role == HCI_ROLE_MASTER)
conn->cleanup = cis_cleanup;
+ conn->mtu = hdev->iso_mtu ? hdev->iso_mtu :
+ hdev->le_mtu ? hdev->le_mtu : hdev->acl_mtu;
break;
case SCO_LINK:
if (lmp_esco_capable(hdev))
@@ -1062,9 +1100,12 @@ static struct hci_conn *__hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t
(hdev->esco_type & EDR_ESCO_MASK);
else
conn->pkt_type = hdev->pkt_type & SCO_PTYPE_MASK;
+
+ conn->mtu = hdev->sco_mtu;
break;
case ESCO_LINK:
conn->pkt_type = hdev->esco_type & ~EDR_ESCO_MASK;
+ conn->mtu = hdev->sco_mtu;
break;
}
@@ -1107,7 +1148,7 @@ struct hci_conn *hci_conn_add_unset(struct hci_dev *hdev, int type,
handle = hci_conn_hash_alloc_unset(hdev);
if (unlikely(handle < 0))
- return NULL;
+ return ERR_PTR(-ECONNREFUSED);
return __hci_conn_add(hdev, type, dst, role, handle);
}
@@ -1303,8 +1344,13 @@ static void create_le_conn_complete(struct hci_dev *hdev, void *data, int err)
goto done;
}
- /* Check if connection is still pending */
- if (conn != hci_lookup_le_connect(hdev))
+ /* Check if this connection is still pending.
+ *
+ * hci_lookup_le_connect() returns only the first LE connection
+ * in BT_CONNECT, which is not necessarily this one when two are
+ * pending at once, so ask the connection itself.
+ */
+ if (!hci_conn_valid(hdev, conn) || conn->state != BT_CONNECT)
goto done;
hci_conn_failed(conn, bt_status(err));
@@ -1378,8 +1424,8 @@ struct hci_conn *hci_connect_le(struct hci_dev *hdev, bdaddr_t *dst,
bacpy(&conn->dst, dst);
} else {
conn = hci_conn_add_unset(hdev, LE_LINK, dst, role);
- if (!conn)
- return ERR_PTR(-ENOMEM);
+ if (IS_ERR(conn))
+ return conn;
hci_conn_hold(conn);
conn->pending_sec_level = sec_level;
}
@@ -1545,8 +1591,8 @@ static struct hci_conn *hci_add_bis(struct hci_dev *hdev, bdaddr_t *dst,
return ERR_PTR(-EADDRINUSE);
conn = hci_conn_add_unset(hdev, ISO_LINK, dst, HCI_ROLE_MASTER);
- if (!conn)
- return ERR_PTR(-ENOMEM);
+ if (IS_ERR(conn))
+ return conn;
set_bit(HCI_CONN_PER_ADV, &conn->flags);
conn->state = BT_CONNECT;
@@ -1590,8 +1636,8 @@ struct hci_conn *hci_connect_le_scan(struct hci_dev *hdev, bdaddr_t *dst,
BT_DBG("requesting refresh of dst_addr");
conn = hci_conn_add_unset(hdev, LE_LINK, dst, HCI_ROLE_MASTER);
- if (!conn)
- return ERR_PTR(-ENOMEM);
+ if (IS_ERR(conn))
+ return conn;
if (hci_explicit_conn_params_set(hdev, dst, dst_type) < 0) {
hci_conn_del(conn);
@@ -1638,8 +1684,8 @@ struct hci_conn *hci_connect_acl(struct hci_dev *hdev, bdaddr_t *dst,
acl = hci_conn_hash_lookup_ba(hdev, ACL_LINK, dst);
if (!acl) {
acl = hci_conn_add_unset(hdev, ACL_LINK, dst, HCI_ROLE_MASTER);
- if (!acl)
- return ERR_PTR(-ENOMEM);
+ if (IS_ERR(acl))
+ return acl;
}
hci_conn_hold(acl);
@@ -1669,9 +1715,9 @@ struct hci_conn *hci_connect_sco(struct hci_dev *hdev, int type, bdaddr_t *dst,
sco = hci_conn_hash_lookup_ba(hdev, type, dst);
if (!sco) {
sco = hci_conn_add_unset(hdev, type, dst, HCI_ROLE_MASTER);
- if (!sco) {
+ if (IS_ERR(sco)) {
hci_conn_drop(acl);
- return ERR_PTR(-ENOMEM);
+ return sco;
}
}
@@ -1880,8 +1926,8 @@ struct hci_conn *hci_bind_cis(struct hci_dev *hdev, bdaddr_t *dst,
qos->cis);
if (!cis) {
cis = hci_conn_add_unset(hdev, ISO_LINK, dst, HCI_ROLE_MASTER);
- if (!cis)
- return ERR_PTR(-ENOMEM);
+ if (IS_ERR(cis))
+ return cis;
cis->cleanup = cis_cleanup;
cis->dst_type = dst_type;
}
@@ -2061,14 +2107,8 @@ static void hci_iso_qos_setup(struct hci_dev *hdev, struct hci_conn *conn,
struct bt_iso_io_qos *qos, __u8 phy)
{
/* Only set MTU if PHY is enabled */
- if (!qos->sdu && qos->phy) {
- if (hdev->iso_mtu > 0)
- qos->sdu = hdev->iso_mtu;
- else if (hdev->le_mtu > 0)
- qos->sdu = hdev->le_mtu;
- else
- qos->sdu = hdev->acl_mtu;
- }
+ if (!qos->sdu && qos->phy)
+ qos->sdu = conn->mtu;
/* Use the same PHY as ACL if set to any */
if (qos->phy == BT_ISO_PHY_ANY)
diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c
index 056de77b4adc..c07e7c2ad76a 100644
--- a/net/bluetooth/hci_core.c
+++ b/net/bluetooth/hci_core.c
@@ -877,7 +877,7 @@ int hci_get_dev_info(void __user *arg)
else
flags = hdev->flags;
- strcpy(di.name, hdev->name);
+ strscpy(di.name, hdev->name, sizeof(di.name));
di.bdaddr = hdev->bdaddr;
di.type = (hdev->bus & 0x0f);
di.flags = flags;
@@ -2596,8 +2596,10 @@ int hci_register_dev(struct hci_dev *hdev)
return id;
error = dev_set_name(&hdev->dev, "hci%u", id);
- if (error)
+ if (error) {
+ ida_free(&hci_index_ida, id);
return error;
+ }
hdev->name = dev_name(&hdev->dev);
hdev->id = id;
@@ -2716,6 +2718,8 @@ void hci_unregister_dev(struct hci_dev *hdev)
hci_unregister_suspend_notifier(hdev);
hci_dev_do_close(hdev);
+ cancel_delayed_work_sync(&hdev->cmd_timer);
+ cancel_delayed_work_sync(&hdev->ncmd_timer);
if (!test_bit(HCI_INIT, &hdev->flags) &&
!hci_dev_test_flag(hdev, HCI_SETUP) &&
@@ -3214,6 +3218,19 @@ static void hci_queue_acl(struct hci_chan *chan, struct sk_buff_head *queue,
spin_unlock_bh(&queue->lock);
}
+
+ bt_dev_dbg(hdev, "chan %p queued %d", chan, skb_queue_len(queue));
+}
+
+/* Queue hdev->tx_work, unless hdev->workqueue is being drained by
+ * hci_dev_close_sync(), which would otherwise WARN and drop the work.
+ */
+static void hci_sched_tx(struct hci_dev *hdev)
+{
+ rcu_read_lock();
+ if (!hci_dev_test_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE))
+ queue_work(hdev->workqueue, &hdev->tx_work);
+ rcu_read_unlock();
}
void hci_send_acl(struct hci_chan *chan, struct sk_buff *skb, __u16 flags)
@@ -3224,7 +3241,7 @@ void hci_send_acl(struct hci_chan *chan, struct sk_buff *skb, __u16 flags)
hci_queue_acl(chan, &chan->data_q, skb, flags);
- queue_work(hdev->workqueue, &hdev->tx_work);
+ hci_sched_tx(hdev);
}
/* Send SCO data */
@@ -3245,7 +3262,11 @@ void hci_send_sco(struct hci_conn *conn, struct sk_buff *skb)
hci_skb_pkt_type(skb) = HCI_SCODATA_PKT;
skb_queue_tail(&conn->data_q, skb);
- queue_work(hdev->workqueue, &hdev->tx_work);
+
+ bt_dev_dbg(hdev, "hcon %p queued %d", conn,
+ skb_queue_len(&conn->data_q));
+
+ hci_sched_tx(hdev);
}
/* Send ISO data */
@@ -3289,6 +3310,8 @@ static void hci_queue_iso(struct hci_conn *conn, struct sk_buff_head *queue,
skb_shinfo(skb)->frag_list = NULL;
+ spin_lock_bh(&queue->lock);
+
__skb_queue_tail(queue, skb);
do {
@@ -3303,7 +3326,11 @@ static void hci_queue_iso(struct hci_conn *conn, struct sk_buff_head *queue,
__skb_queue_tail(queue, skb);
} while (list);
+
+ spin_unlock_bh(&queue->lock);
}
+
+ bt_dev_dbg(hdev, "hcon %p queued %d", conn, skb_queue_len(queue));
}
void hci_send_iso(struct hci_conn *conn, struct sk_buff *skb)
@@ -3314,7 +3341,7 @@ void hci_send_iso(struct hci_conn *conn, struct sk_buff *skb)
hci_queue_iso(conn, &conn->data_q, skb);
- queue_work(hdev->workqueue, &hdev->tx_work);
+ hci_sched_tx(hdev);
}
/* ---- HCI TX task (outgoing data) ---- */
@@ -3602,6 +3629,8 @@ static void hci_sched_acl_pkt(struct hci_dev *hdev)
__check_timeout(hdev, cnt, ACL_LINK);
+ hci_dev_lock(hdev);
+
while (hdev->acl_cnt &&
(chan = hci_chan_sent(hdev, ACL_LINK, "e))) {
u32 priority = (skb_peek(&chan->data_q))->priority;
@@ -3633,6 +3662,8 @@ static void hci_sched_acl_pkt(struct hci_dev *hdev)
if (cnt != hdev->acl_cnt)
hci_prio_recalculate(hdev, ACL_LINK);
+
+ hci_dev_unlock(hdev);
}
static void hci_sched_acl(struct hci_dev *hdev)
@@ -3661,6 +3692,8 @@ static void hci_sched_le(struct hci_dev *hdev)
__check_timeout(hdev, *cnt, LE_LINK);
+ hci_dev_lock(hdev);
+
tmp = *cnt;
while (*cnt && (chan = hci_chan_sent(hdev, LE_LINK, "e))) {
u32 priority = (skb_peek(&chan->data_q))->priority;
@@ -3689,6 +3722,8 @@ static void hci_sched_le(struct hci_dev *hdev)
if (*cnt != tmp)
hci_prio_recalculate(hdev, LE_LINK);
+
+ hci_dev_unlock(hdev);
}
/* Schedule CIS */
diff --git a/net/bluetooth/hci_debugfs.c b/net/bluetooth/hci_debugfs.c
index c9400a7d93d7..aa71618f54e6 100644
--- a/net/bluetooth/hci_debugfs.c
+++ b/net/bluetooth/hci_debugfs.c
@@ -367,7 +367,7 @@ static int inquiry_cache_show(struct seq_file *f, void *p)
list_for_each_entry(e, &cache->all, all) {
struct inquiry_data *data = &e->data;
- seq_printf(f, "%pMR %d %d %d 0x%.2x%.2x%.2x 0x%.4x %d %d %u\n",
+ seq_printf(f, "%pMR %d %d %d 0x%.2x%.2x%.2x 0x%.4x %d %d %lu\n",
&data->bdaddr,
data->pscan_rep_mode, data->pscan_period_mode,
data->pscan_mode, data->dev_class[2],
diff --git a/net/bluetooth/hci_event.c b/net/bluetooth/hci_event.c
index aa879b0b64cf..e319d55dc006 100644
--- a/net/bluetooth/hci_event.c
+++ b/net/bluetooth/hci_event.c
@@ -80,6 +80,10 @@ static void *hci_le_ev_skb_pull(struct hci_dev *hdev, struct sk_buff *skb,
return data;
}
+static void hci_store_wake_reason(struct hci_dev *hdev,
+ const bdaddr_t *bdaddr, u8 addr_type)
+ __must_hold(&hdev->lock);
+
static u8 hci_cc_inquiry_cancel(struct hci_dev *hdev, void *data,
struct sk_buff *skb)
{
@@ -939,6 +943,9 @@ static u8 hci_cc_read_buffer_size(struct hci_dev *hdev, void *data,
BT_DBG("%s acl mtu %d:%d sco mtu %d:%d", hdev->name, hdev->acl_mtu,
hdev->acl_pkts, hdev->sco_mtu, hdev->sco_pkts);
+ if (!hdev->acl_mtu || !hdev->acl_pkts)
+ return HCI_ERROR_INVALID_PARAMETERS;
+
return rp->status;
}
@@ -1202,6 +1209,9 @@ static u8 hci_cc_le_read_buffer_size(struct hci_dev *hdev, void *data,
BT_DBG("%s le mtu %d:%d", hdev->name, hdev->le_mtu, hdev->le_pkts);
+ if (hdev->le_mtu && hdev->le_mtu < HCI_MIN_LE_MTU)
+ return HCI_ERROR_INVALID_PARAMETERS;
+
return rp->status;
}
@@ -1748,6 +1758,13 @@ static void le_set_scan_enable_complete(struct hci_dev *hdev, u8 enable)
hci_dev_clear_flag(hdev, HCI_LE_SCAN);
+ if (hdev->discovery.type == DISCOV_TYPE_INTERLEAVED &&
+ test_bit(HCI_QUIRK_SIMULTANEOUS_DISCOVERY, &hdev->quirks) &&
+ !test_bit(HCI_INQUIRY, &hdev->flags) &&
+ hdev->discovery.state == DISCOVERY_FINDING) {
+ hci_discovery_set_state(hdev, DISCOVERY_STOPPED);
+ }
+
/* The HCI_LE_SCAN_INTERRUPTED flag indicates that we
* interrupted scanning due to a connect request. Mark
* therefore discovery as stopped.
@@ -2256,8 +2273,8 @@ static void hci_cs_create_conn(struct hci_dev *hdev, __u8 status)
if (!conn) {
conn = hci_conn_add_unset(hdev, ACL_LINK, &cp->bdaddr,
HCI_ROLE_MASTER);
- if (!conn)
- bt_dev_err(hdev, "no memory for new connection");
+ if (IS_ERR(conn))
+ bt_dev_err(hdev, "connection err: %ld", PTR_ERR(conn));
}
}
@@ -3065,6 +3082,7 @@ static void hci_conn_complete_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
/* Check for existing connection:
*
@@ -3099,8 +3117,8 @@ static void hci_conn_complete_evt(struct hci_dev *hdev, void *data,
BDADDR_BREDR)) {
conn = hci_conn_add_unset(hdev, ev->link_type,
&ev->bdaddr, HCI_ROLE_SLAVE);
- if (!conn) {
- bt_dev_err(hdev, "no memory for new conn");
+ if (IS_ERR(conn)) {
+ bt_dev_err(hdev, "connection err: %ld", PTR_ERR(conn));
goto unlock;
}
} else {
@@ -3249,6 +3267,10 @@ static void hci_conn_request_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "bdaddr %pMR type 0x%x", &ev->bdaddr, ev->link_type);
+ hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
+ hci_dev_unlock(hdev);
+
/* Reject incoming connection from device with same BD ADDR against
* CVE-2020-26555
*/
@@ -3298,8 +3320,8 @@ static void hci_conn_request_evt(struct hci_dev *hdev, void *data,
if (!conn) {
conn = hci_conn_add_unset(hdev, ev->link_type, &ev->bdaddr,
HCI_ROLE_SLAVE);
- if (!conn) {
- bt_dev_err(hdev, "no memory for new connection");
+ if (IS_ERR(conn)) {
+ bt_dev_err(hdev, "connection err: %ld", PTR_ERR(conn));
goto unlock;
}
}
@@ -3778,6 +3800,9 @@ static u8 hci_cc_le_read_buffer_size_v2(struct hci_dev *hdev, void *data,
BT_DBG("%s acl mtu %d:%d iso mtu %d:%d", hdev->name, hdev->acl_mtu,
hdev->acl_pkts, hdev->iso_mtu, hdev->iso_pkts);
+ if (hdev->le_mtu && hdev->le_mtu < HCI_MIN_LE_MTU)
+ return HCI_ERROR_INVALID_PARAMETERS;
+
return rp->status;
}
@@ -4899,6 +4924,7 @@ static void hci_sync_conn_complete_evt(struct hci_dev *hdev, void *data,
bt_dev_dbg(hdev, "status 0x%2.2x", status);
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR);
conn = hci_conn_hash_lookup_ba(hdev, ev->link_type, &ev->bdaddr);
if (!conn) {
@@ -5607,6 +5633,7 @@ static void le_conn_complete_evt(struct hci_dev *hdev, u8 status,
u8 addr_type;
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, bdaddr, bdaddr_type);
/* Advertising stops when a connection is created. On a failed
* connection it keeps running, so leave the state bit alone.
@@ -5633,8 +5660,8 @@ static void le_conn_complete_evt(struct hci_dev *hdev, u8 status,
goto unlock;
conn = hci_conn_add_unset(hdev, LE_LINK, bdaddr, role);
- if (!conn) {
- bt_dev_err(hdev, "no memory for new connection");
+ if (IS_ERR(conn)) {
+ bt_dev_err(hdev, "connection err: %ld", PTR_ERR(conn));
goto unlock;
}
@@ -6235,6 +6262,8 @@ static void hci_le_adv_report_evt(struct hci_dev *hdev, void *data,
info->length + 1))
break;
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
if (info->length <= max_adv_len(hdev)) {
rssi = info->data[info->length];
process_adv_report(hdev, info->type, &info->bdaddr,
@@ -6318,6 +6347,8 @@ static void hci_le_ext_adv_report_evt(struct hci_dev *hdev, void *data,
info->length))
break;
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
evt_type = __le16_to_cpu(info->type);
legacy_evt_type = ext_evt_type_to_legacy(hdev, evt_type);
if (legacy_evt_type != LE_ADV_INVALID) {
@@ -6355,6 +6386,7 @@ static void hci_le_pa_sync_estabilished_evt(struct hci_dev *hdev, void *data,
return;
hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, &ev->bdaddr, ev->bdaddr_type);
hci_dev_clear_flag(hdev, HCI_PA_SYNC);
@@ -6592,6 +6624,8 @@ static void hci_le_direct_adv_report_evt(struct hci_dev *hdev, void *data,
for (i = 0; i < ev->num; i++) {
struct hci_ev_le_direct_adv_info *info = &ev->info[i];
+ hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type);
+
process_adv_report(hdev, info->type, &info->bdaddr,
info->bdaddr_type, &info->direct_addr,
info->direct_addr_type, info->rssi, NULL, 0,
@@ -6733,7 +6767,7 @@ static void hci_le_cis_req_evt(struct hci_dev *hdev, void *data,
if (!cis) {
cis = hci_conn_add(hdev, ISO_LINK, &acl->dst, HCI_ROLE_SLAVE,
cis_handle);
- if (!cis) {
+ if (IS_ERR(cis)) {
hci_le_reject_cis(hdev, ev->cis_handle);
goto unlock;
}
@@ -6829,7 +6863,7 @@ static void hci_le_big_sync_established_evt(struct hci_dev *hdev, void *data,
}
bis = hci_conn_add(hdev, ISO_LINK, BDADDR_ANY,
HCI_ROLE_SLAVE, handle);
- if (!bis)
+ if (IS_ERR(bis))
continue;
}
@@ -7047,73 +7081,29 @@ static bool hci_get_cmd_complete(struct hci_dev *hdev, u16 opcode,
return true;
}
-static void hci_store_wake_reason(struct hci_dev *hdev, u8 event,
- struct sk_buff *skb)
+static void hci_store_wake_reason(struct hci_dev *hdev,
+ const bdaddr_t *bdaddr, u8 addr_type)
+ __must_hold(&hdev->lock)
{
- struct hci_ev_le_advertising_info *adv;
- struct hci_ev_le_direct_adv_info *direct_adv;
- struct hci_ev_le_ext_adv_info *ext_adv;
- const struct hci_ev_conn_complete *conn_complete = (void *)skb->data;
- const struct hci_ev_conn_request *conn_request = (void *)skb->data;
-
- hci_dev_lock(hdev);
+ lockdep_assert_held(&hdev->lock);
/* If we are currently suspended and this is the first BT event seen,
* save the wake reason associated with the event.
*/
if (!hdev->suspended || hdev->wake_reason)
- goto unlock;
+ return;
+
+ if (!bdaddr) {
+ hdev->wake_reason = MGMT_WAKE_REASON_UNEXPECTED;
+ return;
+ }
/* Default to remote wake. Values for wake_reason are documented in the
* Bluez mgmt api docs.
*/
hdev->wake_reason = MGMT_WAKE_REASON_REMOTE_WAKE;
-
- /* Once configured for remote wakeup, we should only wake up for
- * reconnections. It's useful to see which device is waking us up so
- * keep track of the bdaddr of the connection event that woke us up.
- */
- if (event == HCI_EV_CONN_REQUEST) {
- bacpy(&hdev->wake_addr, &conn_request->bdaddr);
- hdev->wake_addr_type = BDADDR_BREDR;
- } else if (event == HCI_EV_CONN_COMPLETE) {
- bacpy(&hdev->wake_addr, &conn_complete->bdaddr);
- hdev->wake_addr_type = BDADDR_BREDR;
- } else if (event == HCI_EV_LE_META) {
- struct hci_ev_le_meta *le_ev = (void *)skb->data;
- u8 subevent = le_ev->subevent;
- u8 *ptr = &skb->data[sizeof(*le_ev)];
- u8 num_reports = *ptr;
-
- if ((subevent == HCI_EV_LE_ADVERTISING_REPORT ||
- subevent == HCI_EV_LE_DIRECT_ADV_REPORT ||
- subevent == HCI_EV_LE_EXT_ADV_REPORT) &&
- num_reports) {
- adv = (void *)(ptr + 1);
- direct_adv = (void *)(ptr + 1);
- ext_adv = (void *)(ptr + 1);
-
- switch (subevent) {
- case HCI_EV_LE_ADVERTISING_REPORT:
- bacpy(&hdev->wake_addr, &adv->bdaddr);
- hdev->wake_addr_type = adv->bdaddr_type;
- break;
- case HCI_EV_LE_DIRECT_ADV_REPORT:
- bacpy(&hdev->wake_addr, &direct_adv->bdaddr);
- hdev->wake_addr_type = direct_adv->bdaddr_type;
- break;
- case HCI_EV_LE_EXT_ADV_REPORT:
- bacpy(&hdev->wake_addr, &ext_adv->bdaddr);
- hdev->wake_addr_type = ext_adv->bdaddr_type;
- break;
- }
- }
- } else {
- hdev->wake_reason = MGMT_WAKE_REASON_UNEXPECTED;
- }
-
-unlock:
- hci_dev_unlock(hdev);
+ bacpy(&hdev->wake_addr, bdaddr);
+ hdev->wake_addr_type = addr_type;
}
#define HCI_EV_VL(_op, _func, _min_len, _max_len) \
@@ -7358,14 +7348,15 @@ void hci_event_packet(struct hci_dev *hdev, struct sk_buff *skb)
skb_pull(skb, HCI_EVENT_HDR_SIZE);
- /* Store wake reason if we're suspended */
- hci_store_wake_reason(hdev, event, skb);
-
bt_dev_dbg(hdev, "event 0x%2.2x", event);
hci_event_func(hdev, event, skb, &opcode, &status, &req_complete,
&req_complete_skb);
+ hci_dev_lock(hdev);
+ hci_store_wake_reason(hdev, NULL, 0);
+ hci_dev_unlock(hdev);
+
if (req_complete) {
req_complete(hdev, status, opcode);
} else if (req_complete_skb) {
diff --git a/net/bluetooth/hci_request.c b/net/bluetooth/hci_request.c
index cf69e973b724..303fb14fdfad 100644
--- a/net/bluetooth/hci_request.c
+++ b/net/bluetooth/hci_request.c
@@ -46,7 +46,7 @@ void hci_req_purge(struct hci_request *req)
bool hci_req_status_pend(struct hci_dev *hdev)
{
- return hdev->req_status == HCI_REQ_PEND;
+ return READ_ONCE(hdev->req_status) == HCI_REQ_PEND;
}
static int req_run(struct hci_request *req, hci_req_complete_t complete,
diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c
index b2a6441afb5f..6fa29333676b 100644
--- a/net/bluetooth/hci_sock.c
+++ b/net/bluetooth/hci_sock.c
@@ -1161,8 +1161,10 @@ static int hci_sock_bind(struct socket *sock, struct sockaddr *addr,
*/
hdev = hci_pi(sk)->hdev;
if (hdev && hci_dev_test_flag(hdev, HCI_UNREGISTER)) {
+ write_lock(&hci_sk_list.lock);
hci_pi(sk)->hdev = NULL;
sk->sk_state = BT_OPEN;
+ write_unlock(&hci_sk_list.lock);
hci_dev_put(hdev);
}
hdev = NULL;
diff --git a/net/bluetooth/hci_sync.c b/net/bluetooth/hci_sync.c
index 5825ab89eb22..c0c19abd9259 100644
--- a/net/bluetooth/hci_sync.c
+++ b/net/bluetooth/hci_sync.c
@@ -24,11 +24,11 @@ static void hci_cmd_sync_complete(struct hci_dev *hdev, u8 result, u16 opcode,
{
bt_dev_dbg(hdev, "result 0x%2.2x", result);
- if (hdev->req_status != HCI_REQ_PEND)
+ if (READ_ONCE(hdev->req_status) != HCI_REQ_PEND)
return;
hdev->req_result = result;
- hdev->req_status = HCI_REQ_DONE;
+ WRITE_ONCE(hdev->req_status, HCI_REQ_DONE);
if (skb) {
struct sock *sk = hci_skb_sk(skb);
@@ -163,20 +163,20 @@ struct sk_buff *__hci_cmd_sync_sk(struct hci_dev *hdev, u16 opcode, u32 plen,
hci_cmd_sync_add(&req, opcode, plen, param, event, sk);
- hdev->req_status = HCI_REQ_PEND;
+ WRITE_ONCE(hdev->req_status, HCI_REQ_PEND);
err = hci_req_sync_run(&req);
if (err < 0)
return ERR_PTR(err);
err = wait_event_interruptible_timeout(hdev->req_wait_q,
- hdev->req_status != HCI_REQ_PEND,
+ READ_ONCE(hdev->req_status) != HCI_REQ_PEND,
timeout);
if (err == -ERESTARTSYS)
return ERR_PTR(-EINTR);
- switch (hdev->req_status) {
+ switch (READ_ONCE(hdev->req_status)) {
case HCI_REQ_DONE:
err = -bt_to_errno(hdev->req_result);
break;
@@ -190,7 +190,7 @@ struct sk_buff *__hci_cmd_sync_sk(struct hci_dev *hdev, u16 opcode, u32 plen,
break;
}
- hdev->req_status = 0;
+ WRITE_ONCE(hdev->req_status, 0);
hdev->req_result = 0;
skb = hdev->req_skb;
hdev->req_skb = NULL;
@@ -308,6 +308,10 @@ static void hci_cmd_sync_work(struct work_struct *work)
while (1) {
struct hci_cmd_sync_work_entry *entry;
+ /* Leave the backlog to hci_cmd_sync_clear() */
+ if (hci_dev_test_flag(hdev, HCI_UNREGISTER))
+ break;
+
mutex_lock(&hdev->cmd_sync_work_lock);
entry = list_first_entry_or_null(&hdev->cmd_sync_work_list,
struct hci_cmd_sync_work_entry,
@@ -677,6 +681,8 @@ void hci_cmd_sync_clear(struct hci_dev *hdev)
{
struct hci_cmd_sync_work_entry *entry, *tmp;
+ /* cmd_work is disabled, the pending request can only time out */
+ hci_cmd_sync_cancel_sync(hdev, ENODEV);
cancel_work_sync(&hdev->cmd_sync_work);
cancel_work_sync(&hdev->reenable_adv_work);
@@ -690,9 +696,9 @@ void hci_cmd_sync_cancel(struct hci_dev *hdev, int err)
{
bt_dev_dbg(hdev, "err 0x%2.2x", err);
- if (hdev->req_status == HCI_REQ_PEND) {
+ if (READ_ONCE(hdev->req_status) == HCI_REQ_PEND) {
hdev->req_result = err;
- hdev->req_status = HCI_REQ_CANCELED;
+ WRITE_ONCE(hdev->req_status, HCI_REQ_CANCELED);
queue_work(hdev->workqueue, &hdev->cmd_sync_cancel_work);
}
@@ -708,12 +714,12 @@ void hci_cmd_sync_cancel_sync(struct hci_dev *hdev, int err)
{
bt_dev_dbg(hdev, "err 0x%2.2x", err);
- if (hdev->req_status == HCI_REQ_PEND) {
+ if (READ_ONCE(hdev->req_status) == HCI_REQ_PEND) {
/* req_result is __u32 so error must be positive to be properly
* propagated.
*/
hdev->req_result = err < 0 ? -err : err;
- hdev->req_status = HCI_REQ_CANCELED;
+ WRITE_ONCE(hdev->req_status, HCI_REQ_CANCELED);
wake_up_interruptible(&hdev->req_wait_q);
}
@@ -805,7 +811,7 @@ int hci_cmd_sync_queue_once(struct hci_dev *hdev, hci_cmd_sync_work_func_t func,
void *data, hci_cmd_sync_work_destroy_t destroy)
{
if (hci_cmd_sync_lookup_entry(hdev, func, data, destroy))
- return 0;
+ return -EEXIST;
return hci_cmd_sync_queue(hdev, func, data, destroy);
}
@@ -3320,6 +3326,8 @@ static int update_passive_scan_sync(struct hci_dev *hdev, void *data)
int hci_update_passive_scan(struct hci_dev *hdev)
{
+ int err;
+
/* Only queue if it would have any effect */
if (!test_bit(HCI_UP, &hdev->flags) ||
test_bit(HCI_INIT, &hdev->flags) ||
@@ -3329,8 +3337,9 @@ int hci_update_passive_scan(struct hci_dev *hdev)
hci_dev_test_flag(hdev, HCI_UNREGISTER))
return 0;
- return hci_cmd_sync_queue_once(hdev, update_passive_scan_sync, NULL,
- NULL);
+ err = hci_cmd_sync_queue_once(hdev, update_passive_scan_sync, NULL,
+ NULL);
+ return (err == -EEXIST) ? 0 : err;
}
int hci_write_sc_support_sync(struct hci_dev *hdev, u8 val)
@@ -5247,6 +5256,12 @@ int hci_dev_close_sync(struct hci_dev *hdev)
bt_dev_dbg(hdev, "");
+ /* Set HCI_DRAIN_WORKQUEUE flag to prevent queuing work during
+ * reset/close. See hci_cmd_work() and handle_cmd_cnt_and_timer().
+ */
+ hci_dev_set_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE);
+ synchronize_rcu();
+
cancel_delayed_work(&hdev->power_off);
cancel_delayed_work(&hdev->ncmd_timer);
cancel_delayed_work(&hdev->le_scan_disable);
@@ -5265,6 +5280,7 @@ int hci_dev_close_sync(struct hci_dev *hdev)
if (!test_and_clear_bit(HCI_UP, &hdev->flags)) {
cancel_delayed_work_sync(&hdev->cmd_timer);
+ hci_dev_clear_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE);
return err;
}
@@ -5358,6 +5374,7 @@ int hci_dev_close_sync(struct hci_dev *hdev)
/* Clear flags */
hdev->flags &= BIT(HCI_RAW);
hci_dev_clear_volatile_flags(hdev);
+ hci_dev_clear_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE);
memset(hdev->eir, 0, sizeof(hdev->eir));
memset(hdev->dev_class, 0, sizeof(hdev->dev_class));
diff --git a/net/bluetooth/iso.c b/net/bluetooth/iso.c
index 7ffb47a12dd3..5bf91e702330 100644
--- a/net/bluetooth/iso.c
+++ b/net/bluetooth/iso.c
@@ -49,6 +49,11 @@ static void iso_sock_kill(struct sock *sk);
#define BASE_MAX_LENGTH (HCI_MAX_PER_AD_LENGTH - EIR_SERVICE_DATA_LENGTH)
#define EIR_BAA_SERVICE_UUID 0x1851
+/* iso_pinfo flags values */
+enum {
+ BT_SK_CONNECTING,
+};
+
struct iso_pinfo {
struct bt_sock bt;
bdaddr_t src;
@@ -59,7 +64,7 @@ struct iso_pinfo {
__u8 bc_num_bis;
__u8 bc_bis[ISO_MAX_NUM_BIS];
__u16 sync_handle;
- __u32 flags;
+ unsigned long flags;
struct bt_iso_qos qos;
__u8 base_len;
__u8 base[BASE_MAX_LENGTH];
@@ -225,6 +230,9 @@ static int __iso_chan_add(struct iso_conn *conn, struct sock *sk,
return -EBUSY;
}
+ if (iso_pi(sk)->conn)
+ return -EISCONN;
+
iso_pi(sk)->conn = conn;
conn->sk = sk;
@@ -259,16 +267,29 @@ static int iso_connect_bis(struct sock *sk)
struct iso_conn *conn;
struct hci_conn *hcon;
struct hci_dev *hdev;
+ bdaddr_t src, dst;
+ u8 src_type;
int err;
- BT_DBG("%pMR", &iso_pi(sk)->src);
+ lock_sock(sk);
+ bacpy(&src, &iso_pi(sk)->src);
+ bacpy(&dst, &iso_pi(sk)->dst);
+ src_type = iso_pi(sk)->src_type;
+ release_sock(sk);
- hdev = hci_get_route(&iso_pi(sk)->dst, &iso_pi(sk)->src,
- iso_pi(sk)->src_type);
+ BT_DBG("%pMR", &src);
+
+ hdev = hci_get_route(&dst, &src, src_type);
if (!hdev)
return -EHOSTUNREACH;
hci_dev_lock(hdev);
+ lock_sock(sk);
+
+ if (iso_pi(sk)->conn) {
+ err = -EISCONN;
+ goto unlock;
+ }
if (!bis_capable(hdev)) {
err = -EOPNOTSUPP;
@@ -296,13 +317,9 @@ static int iso_connect_bis(struct sock *sk)
goto unlock;
}
- lock_sock(sk);
-
err = iso_chan_add(conn, sk, NULL);
- if (err) {
- release_sock(sk);
+ if (err)
goto unlock;
- }
/* Update source addr of the socket */
bacpy(&iso_pi(sk)->src, &hcon->src);
@@ -315,9 +332,8 @@ static int iso_connect_bis(struct sock *sk)
iso_sock_set_timer(sk, sk->sk_sndtimeo);
}
- release_sock(sk);
-
unlock:
+ release_sock(sk);
hci_dev_unlock(hdev);
hci_dev_put(hdev);
return err;
@@ -328,16 +344,24 @@ static int iso_connect_cis(struct sock *sk)
struct iso_conn *conn;
struct hci_conn *hcon;
struct hci_dev *hdev;
+ bdaddr_t src, dst;
+ u8 src_type;
int err;
- BT_DBG("%pMR -> %pMR", &iso_pi(sk)->src, &iso_pi(sk)->dst);
+ lock_sock(sk);
+ bacpy(&src, &iso_pi(sk)->src);
+ bacpy(&dst, &iso_pi(sk)->dst);
+ src_type = iso_pi(sk)->src_type;
+ release_sock(sk);
- hdev = hci_get_route(&iso_pi(sk)->dst, &iso_pi(sk)->src,
- iso_pi(sk)->src_type);
+ BT_DBG("%pMR -> %pMR", &src, &dst);
+
+ hdev = hci_get_route(&dst, &src, src_type);
if (!hdev)
return -EHOSTUNREACH;
hci_dev_lock(hdev);
+ lock_sock(sk);
if (!cis_central_capable(hdev)) {
err = -EOPNOTSUPP;
@@ -369,6 +393,13 @@ static int iso_connect_cis(struct sock *sk)
}
}
+ /* The socket lock keeps the current attachment and its hcon stable. */
+ if (iso_pi(sk)->conn && iso_pi(sk)->conn->hcon != hcon) {
+ hci_conn_drop(hcon);
+ err = -EISCONN;
+ goto unlock;
+ }
+
conn = iso_conn_add(hcon);
if (!conn) {
hci_conn_drop(hcon);
@@ -376,13 +407,9 @@ static int iso_connect_cis(struct sock *sk)
goto unlock;
}
- lock_sock(sk);
-
err = iso_chan_add(conn, sk, NULL);
- if (err) {
- release_sock(sk);
+ if (err)
goto unlock;
- }
/* Update source addr of the socket */
bacpy(&iso_pi(sk)->src, &hcon->src);
@@ -398,9 +425,8 @@ static int iso_connect_cis(struct sock *sk)
iso_sock_set_timer(sk, sk->sk_sndtimeo);
}
- release_sock(sk);
-
unlock:
+ release_sock(sk);
hci_dev_unlock(hdev);
hci_dev_put(hdev);
return err;
@@ -847,17 +873,26 @@ static int iso_sock_connect(struct socket *sock, struct sockaddr *addr,
addr->sa_family != AF_BLUETOOTH)
return -EINVAL;
- if (sk->sk_state != BT_OPEN && sk->sk_state != BT_BOUND)
- return -EBADFD;
+ lock_sock(sk);
+
+ if ((sk->sk_state != BT_OPEN && sk->sk_state != BT_BOUND) ||
+ test_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags)) {
+ err = -EBADFD;
+ goto done;
+ }
- if (sk->sk_type != SOCK_SEQPACKET)
- return -EINVAL;
+ if (sk->sk_type != SOCK_SEQPACKET) {
+ err = -EINVAL;
+ goto done;
+ }
/* Check if the address type is of LE type */
- if (!bdaddr_type_is_le(sa->iso_bdaddr_type))
- return -EINVAL;
+ if (!bdaddr_type_is_le(sa->iso_bdaddr_type)) {
+ err = -EINVAL;
+ goto done;
+ }
- lock_sock(sk);
+ set_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags);
bacpy(&iso_pi(sk)->dst, &sa->iso_bdaddr);
iso_pi(sk)->dst_type = sa->iso_bdaddr_type;
@@ -869,16 +904,18 @@ static int iso_sock_connect(struct socket *sock, struct sockaddr *addr,
else
err = iso_connect_bis(sk);
- if (err)
- return err;
-
lock_sock(sk);
+ clear_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags);
+ if (err)
+ goto done;
+
if (!test_bit(BT_SK_DEFER_SETUP, &bt_sk(sk)->flags)) {
err = bt_sock_wait_state(sk, BT_CONNECTED,
sock_sndtimeo(sk, flags & O_NONBLOCK));
}
+done:
release_sock(sk);
return err;
}
@@ -1043,6 +1080,7 @@ static int iso_sock_getname(struct socket *sock, struct sockaddr *addr,
BT_DBG("sock %p, sk %p", sock, sk);
+ memset(sa, 0, sizeof(struct sockaddr_iso));
addr->sa_family = AF_BLUETOOTH;
if (peer) {
@@ -1080,7 +1118,7 @@ static int iso_sock_sendmsg(struct socket *sock, struct msghdr *msg,
return -ENOTCONN;
}
- mtu = iso_pi(sk)->conn->hcon->hdev->iso_mtu;
+ mtu = iso_pi(sk)->conn->hcon->mtu;
release_sock(sk);
@@ -1426,14 +1464,11 @@ static void iso_sock_ready(struct sock *sk)
{
BT_DBG("sk %p", sk);
- if (!sk)
- return;
+ lockdep_assert(lockdep_sock_is_held(sk));
- lock_sock(sk);
iso_sock_clear_timer(sk);
sk->sk_state = BT_CONNECTED;
sk->sk_state_change(sk);
- release_sock(sk);
}
struct iso_list_data {
@@ -1451,14 +1486,32 @@ static bool iso_match_big(struct sock *sk, void *data)
static void iso_conn_ready(struct iso_conn *conn)
{
struct sock *parent;
- struct sock *sk = conn->sk;
+ struct sock *sk;
struct hci_ev_le_big_sync_estabilished *ev;
struct hci_conn *hcon;
BT_DBG("conn %p", conn);
+ iso_conn_lock(conn);
+ sk = iso_sock_hold(conn);
+ iso_conn_unlock(conn);
+
if (sk) {
- iso_sock_ready(conn->sk);
+ lock_sock(sk);
+
+ /* conn->sk may have become NULL if racing with sk close, but
+ * due to held hdev->lock, it can't become different sk.
+ */
+ if (!conn->sk) {
+ release_sock(sk);
+ sock_put(sk);
+ return;
+ }
+
+ iso_sock_ready(sk);
+
+ release_sock(sk);
+ sock_put(sk);
} else {
hcon = conn->hcon;
if (!hcon)
diff --git a/net/bluetooth/l2cap_core.c b/net/bluetooth/l2cap_core.c
index 88ddd8eabae9..ea41bf1efc6d 100644
--- a/net/bluetooth/l2cap_core.c
+++ b/net/bluetooth/l2cap_core.c
@@ -630,6 +630,10 @@ void __l2cap_chan_add(struct l2cap_conn *conn, struct l2cap_chan *chan)
BT_DBG("conn %p, psm 0x%2.2x, dcid 0x%4.4x", conn,
__le16_to_cpu(chan->psm), chan->dcid);
+ /* Caller must ensure l2cap_chan is linked to l2cap_conn only once */
+ if (WARN_ON_ONCE(chan->conn || test_bit(FLAG_DEL, &chan->flags)))
+ return;
+
conn->disc_reason = HCI_ERROR_REMOTE_USER_TERM;
chan->conn = l2cap_conn_get(conn);
@@ -998,6 +1002,16 @@ static int l2cap_get_ident(struct l2cap_conn *conn)
return ident;
}
+static void l2cap_send_acl(struct l2cap_conn *conn, struct sk_buff *skb,
+ u8 flags)
+{
+ /* Check if the hcon still valid before attempting to send */
+ if (hci_conn_valid(conn->hcon->hdev, conn->hcon))
+ hci_send_acl(conn->hchan, skb, flags);
+ else
+ kfree_skb(skb);
+}
+
static void l2cap_send_cmd(struct l2cap_conn *conn, u8 ident, u8 code, u16 len,
void *data)
{
@@ -1020,7 +1034,7 @@ static void l2cap_send_cmd(struct l2cap_conn *conn, u8 ident, u8 code, u16 len,
bt_cb(skb)->force_active = BT_POWER_FORCE_ACTIVE_ON;
skb->priority = HCI_PRIO_MAX;
- hci_send_acl(conn->hchan, skb, flags);
+ l2cap_send_acl(conn, skb, flags);
}
static void l2cap_do_send(struct l2cap_chan *chan, struct sk_buff *skb)
@@ -1843,10 +1857,7 @@ static void l2cap_conn_del(struct hci_conn *hcon, int err)
mutex_unlock(&conn->chan_lock);
- hci_chan_del(conn->hchan);
-
hcon->l2cap_data = NULL;
- conn->hchan = NULL;
l2cap_conn_put(conn);
}
@@ -1854,6 +1865,7 @@ static void l2cap_conn_free(struct kref *ref)
{
struct l2cap_conn *conn = container_of(ref, struct l2cap_conn, ref);
+ hci_chan_del(conn->hchan);
hci_conn_put(conn->hcon);
kfree(conn);
}
@@ -6547,7 +6559,7 @@ static int l2cap_finish_move(struct l2cap_chan *chan)
BT_DBG("chan %p", chan);
chan->rx_state = L2CAP_RX_STATE_RECV;
- chan->conn->mtu = chan->conn->hcon->hdev->acl_mtu;
+ chan->conn->mtu = chan->conn->hcon->mtu;
return l2cap_resegment(chan);
}
@@ -6614,7 +6626,7 @@ static int l2cap_rx_state_wait_f(struct l2cap_chan *chan,
*/
chan->next_tx_seq = control->reqseq;
chan->unacked_frames = 0;
- chan->conn->mtu = chan->conn->hcon->hdev->acl_mtu;
+ chan->conn->mtu = chan->conn->hcon->mtu;
err = l2cap_resegment(chan);
@@ -7213,18 +7225,7 @@ static struct l2cap_conn *l2cap_conn_add(struct hci_conn *hcon)
BT_DBG("hcon %p conn %p hchan %p", hcon, conn, hchan);
- switch (hcon->type) {
- case LE_LINK:
- if (hcon->hdev->le_mtu) {
- conn->mtu = hcon->hdev->le_mtu;
- break;
- }
- fallthrough;
- default:
- conn->mtu = hcon->hdev->acl_mtu;
- break;
- }
-
+ conn->mtu = hcon->mtu;
conn->feat_mask = 0;
conn->local_fixed_chan = L2CAP_FC_SIG_BREDR | L2CAP_FC_CONNLESS;
@@ -7431,7 +7432,8 @@ int l2cap_chan_connect(struct l2cap_chan *chan, __le16 psm, u16 cid,
mutex_lock(&conn->chan_lock);
l2cap_chan_lock(chan);
- if (cid && __l2cap_get_chan_by_dcid(conn, cid)) {
+ if ((cid && __l2cap_get_chan_by_dcid(conn, cid)) || chan->conn ||
+ test_bit(FLAG_DEL, &chan->flags)) {
hci_conn_drop(hcon);
err = -EBUSY;
goto chan_unlock;
@@ -7824,14 +7826,34 @@ static void l2cap_recv_reset(struct l2cap_conn *conn)
conn->rx_len = 0;
}
+struct l2cap_conn *l2cap_conn_hold_unless_zero(struct l2cap_conn *c)
+{
+ BT_DBG("conn %p orig refcnt %u", c, kref_read(&c->ref));
+
+ if (!kref_get_unless_zero(&c->ref))
+ return NULL;
+
+ return c;
+}
+EXPORT_SYMBOL(l2cap_conn_hold_unless_zero);
+
void l2cap_recv_acldata(struct hci_conn *hcon, struct sk_buff *skb, u16 flags)
{
- struct l2cap_conn *conn = hcon->l2cap_data;
+ struct l2cap_conn *conn;
int len;
+ /* Lock hdev to access l2cap_data to avoid race with l2cap_conn_del */
+ hci_dev_lock(hcon->hdev);
+
+ conn = hcon->l2cap_data;
+
if (!conn)
conn = l2cap_conn_add(hcon);
+ conn = l2cap_conn_hold_unless_zero(conn);
+
+ hci_dev_unlock(hcon->hdev);
+
if (!conn)
goto drop;
@@ -7923,6 +7945,8 @@ void l2cap_recv_acldata(struct hci_conn *hcon, struct sk_buff *skb, u16 flags)
break;
}
+ l2cap_conn_put(conn);
+
drop:
kfree_skb(skb);
}
diff --git a/net/bluetooth/l2cap_sock.c b/net/bluetooth/l2cap_sock.c
index 9b8bddefeb84..655922aa82f1 100644
--- a/net/bluetooth/l2cap_sock.c
+++ b/net/bluetooth/l2cap_sock.c
@@ -111,6 +111,7 @@ static int l2cap_sock_bind(struct socket *sock, struct sockaddr *addr, int alen)
return -EINVAL;
}
+ l2cap_chan_lock(chan);
lock_sock(sk);
if (sk->sk_state != BT_OPEN) {
@@ -176,6 +177,7 @@ static int l2cap_sock_bind(struct socket *sock, struct sockaddr *addr, int alen)
done:
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
@@ -437,11 +439,26 @@ static int l2cap_get_mode(struct l2cap_chan *chan)
return -EINVAL;
}
+static struct l2cap_conn *l2cap_chan_conn(struct l2cap_chan *chan)
+{
+ lockdep_assert_held(&chan->lock);
+
+ /* l2cap_conn_del() sets FLAG_DEL while holding chan->lock before
+ * conn->hcon is deleted. If not set and conn is non-NULL, conn->hcon
+ * remains alive during this chan->lock critical section.
+ */
+ if (test_bit(FLAG_DEL, &chan->flags))
+ return NULL;
+
+ return chan->conn;
+}
+
static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
char __user *optval, int __user *optlen)
{
struct sock *sk = sock->sk;
struct l2cap_chan *chan = l2cap_pi(sk)->chan;
+ struct l2cap_conn *conn;
struct l2cap_options opts;
struct l2cap_conninfo cinfo;
int err = 0;
@@ -453,6 +470,7 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
if (get_user(len, optlen))
return -EFAULT;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -538,9 +556,15 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
break;
}
+ conn = l2cap_chan_conn(chan);
+ if (!conn) {
+ err = -ENOTCONN;
+ break;
+ }
+
memset(&cinfo, 0, sizeof(cinfo));
- cinfo.hci_handle = chan->conn->hcon->handle;
- memcpy(cinfo.dev_class, chan->conn->hcon->dev_class, 3);
+ cinfo.hci_handle = conn->hcon->handle;
+ memcpy(cinfo.dev_class, conn->hcon->dev_class, 3);
len = min(len, sizeof(cinfo));
if (copy_to_user(optval, (char *) &cinfo, len))
@@ -554,6 +578,8 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
+
return err;
}
@@ -562,6 +588,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
{
struct sock *sk = sock->sk;
struct l2cap_chan *chan = l2cap_pi(sk)->chan;
+ struct l2cap_conn *conn;
struct bt_security sec;
struct bt_power pwr;
u32 phys;
@@ -578,6 +605,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
if (get_user(len, optlen))
return -EFAULT;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -589,12 +617,14 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
}
+ conn = l2cap_chan_conn(chan);
+
memset(&sec, 0, sizeof(sec));
- if (chan->conn) {
- sec.level = chan->conn->hcon->sec_level;
+ if (conn) {
+ sec.level = conn->hcon->sec_level;
if (sk->sk_state == BT_CONNECTED)
- sec.key_size = chan->conn->hcon->enc_key_size;
+ sec.key_size = conn->hcon->enc_key_size;
} else {
sec.level = chan->sec_level;
}
@@ -670,12 +700,14 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
break;
case BT_PHY:
- if (sk->sk_state != BT_CONNECTED) {
+ conn = l2cap_chan_conn(chan);
+
+ if (sk->sk_state != BT_CONNECTED || !conn) {
err = -ENOTCONN;
break;
}
- phys = hci_conn_get_phy(chan->conn->hcon);
+ phys = hci_conn_get_phy(conn->hcon);
if (put_user(phys, (u32 __user *) optval))
err = -EFAULT;
@@ -708,6 +740,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
@@ -738,6 +771,7 @@ static int l2cap_sock_setsockopt_old(struct socket *sock, int optname,
BT_DBG("sk %p", sk);
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -838,6 +872,7 @@ static int l2cap_sock_setsockopt_old(struct socket *sock, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
@@ -901,6 +936,7 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
if (level != SOL_BLUETOOTH)
return -ENOPROTOOPT;
+ l2cap_chan_lock(chan);
lock_sock(sk);
switch (optname) {
@@ -926,11 +962,10 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
chan->sec_level = sec.level;
- if (!chan->conn)
+ conn = l2cap_chan_conn(chan);
+ if (!conn)
break;
- conn = chan->conn;
-
/* change security for LE channels */
if (chan->scid == L2CAP_CID_ATT) {
if (smp_conn_security(conn->hcon, sec.level)) {
@@ -985,7 +1020,8 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
}
if (opt == BT_FLUSHABLE_OFF) {
- conn = chan->conn;
+ conn = l2cap_chan_conn(chan);
+
/* proceed further only when we have l2cap_conn and
No Flush support in the LM */
if (!conn || !lmp_no_flush_capable(conn->hcon->hdev)) {
@@ -1108,6 +1144,7 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname,
}
release_sock(sk);
+ l2cap_chan_unlock(chan);
return err;
}
diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c
index 251e0c0a062a..b09a8aaf0362 100644
--- a/net/bluetooth/mgmt.c
+++ b/net/bluetooth/mgmt.c
@@ -500,12 +500,10 @@ static int read_unconf_index_list(struct sock *sk, struct hci_dev *hdev,
read_lock(&hci_dev_list_lock);
count = 0;
- list_for_each_entry(d, &hci_dev_list, list) {
- if (hci_dev_test_flag(d, HCI_UNCONFIGURED))
- count++;
- }
+ list_for_each_entry(d, &hci_dev_list, list)
+ count++;
- rp_len = sizeof(*rp) + (2 * count);
+ rp_len = sizeof(*rp) + (sizeof(__le16) * count);
rp = kmalloc(rp_len, GFP_ATOMIC);
if (!rp) {
read_unlock(&hci_dev_list_lock);
@@ -1455,7 +1453,7 @@ static void cmd_complete_rsp(struct mgmt_pending_cmd *cmd, void *data)
return;
}
- cmd_status_rsp(cmd, data);
+ cmd_status_rsp(cmd, &match->mgmt_status);
}
static int generic_cmd_complete(struct mgmt_pending_cmd *cmd, u8 status)
diff --git a/net/bluetooth/rfcomm/core.c b/net/bluetooth/rfcomm/core.c
index 6324e03ff0d0..1e21e1e38543 100644
--- a/net/bluetooth/rfcomm/core.c
+++ b/net/bluetooth/rfcomm/core.c
@@ -63,10 +63,9 @@ static void rfcomm_make_uih(struct sk_buff *skb, u8 addr);
static void rfcomm_process_connect(struct rfcomm_session *s);
-static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src,
- bdaddr_t *dst,
- u8 sec_level,
- int *err);
+static struct socket *rfcomm_session_connect(bdaddr_t *src, bdaddr_t *dst,
+ u8 sec_level, int *err);
+static struct rfcomm_session *rfcomm_session_add(struct socket *sock, int state);
static struct rfcomm_session *rfcomm_session_get(bdaddr_t *src, bdaddr_t *dst);
static struct rfcomm_session *rfcomm_session_del(struct rfcomm_session *s);
@@ -364,28 +363,17 @@ static int rfcomm_check_channel(u8 channel)
return channel < 1 || channel > 30;
}
-static int __rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst, u8 channel)
+static int __rfcomm_dlc_open(struct rfcomm_dlc *d, struct rfcomm_session *s,
+ u8 channel)
{
- struct rfcomm_session *s;
- int err = 0;
u8 dlci;
- BT_DBG("dlc %p state %ld %pMR -> %pMR channel %d",
- d, d->state, src, dst, channel);
-
- if (rfcomm_check_channel(channel))
- return -EINVAL;
+ BT_DBG("dlc %p state %ld session %p channel %d",
+ d, d->state, s, channel);
if (d->state != BT_OPEN && d->state != BT_CLOSED)
return 0;
- s = rfcomm_session_get(src, dst);
- if (!s) {
- s = rfcomm_session_create(src, dst, d->sec_level, &err);
- if (!s)
- return err;
- }
-
dlci = __dlci(__session_dir(s), channel);
/* Check if DLCI already exists */
@@ -420,14 +408,84 @@ static int __rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst,
int rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst, u8 channel)
{
- int r;
+ struct rfcomm_session *s;
+ struct socket *sock;
+ int err;
+
+ BT_DBG("dlc %p state %ld %pMR -> %pMR channel %d",
+ d, d->state, src, dst, channel);
+
+ if (rfcomm_check_channel(channel))
+ return -EINVAL;
rfcomm_lock();
- r = __rfcomm_dlc_open(d, src, dst, channel);
+ clear_bit(RFCOMM_CLOSED, &d->flags);
+ /* Do not page the remote device for a DLC that cannot be opened
+ * anyway. __rfcomm_dlc_open() looks at the state again once the
+ * lock has been re-acquired below.
+ */
+ if (d->state != BT_OPEN && d->state != BT_CLOSED) {
+ rfcomm_unlock();
+ return 0;
+ }
+
+ s = rfcomm_session_get(src, dst);
+ if (s) {
+ err = __rfcomm_dlc_open(d, s, channel);
+ rfcomm_unlock();
+ return err;
+ }
rfcomm_unlock();
- return r;
+
+ /* There is no session for this pair yet. kernel_connect() ends up in
+ * l2cap_chan_connect(), which takes hdev->lock, and the HCI event
+ * path takes rfcomm_mutex while holding hdev->lock, so the socket has
+ * to be connected with rfcomm_mutex released.
+ */
+ sock = rfcomm_session_connect(src, dst, d->sec_level, &err);
+ if (!sock)
+ return err;
+
+ rfcomm_lock();
+
+ /* The DLC may have been closed while the socket was connecting. It
+ * was not on a session, so rfcomm_dlc_close() could only mark it;
+ * attaching it now would leave it with no owner.
+ */
+ if (test_bit(RFCOMM_CLOSED, &d->flags)) {
+ rfcomm_unlock();
+ sock_release(sock);
+ return -ECONNRESET;
+ }
+
+ /* Another opener may have added a session for the same pair in the
+ * meantime; that one is used and this socket is dropped.
+ */
+ s = rfcomm_session_get(src, dst);
+ if (!s) {
+ s = rfcomm_session_add(sock, BT_BOUND);
+ if (s) {
+ s->initiator = 1;
+ sock = NULL;
+ }
+ }
+
+ err = s ? __rfcomm_dlc_open(d, s, channel) : -ENOMEM;
+
+ rfcomm_unlock();
+
+ if (sock)
+ sock_release(sock);
+
+ /* Over an existing ACL link the connection can complete before the
+ * session reaches the list, and that wakeup is then lost, so let
+ * krfcommd look at the socket state now.
+ */
+ rfcomm_schedule();
+
+ return err;
}
static void __rfcomm_dlc_disconn(struct rfcomm_dlc *d)
@@ -507,8 +565,14 @@ int rfcomm_dlc_close(struct rfcomm_dlc *d, int err)
rfcomm_lock();
s = d->session;
- if (!s)
+ if (!s) {
+ /* Not on a session yet: rfcomm_dlc_open() may be connecting a
+ * socket for it with rfcomm_mutex released. Leave a mark so
+ * that it does not attach the DLC once it holds the lock again.
+ */
+ set_bit(RFCOMM_CLOSED, &d->flags);
goto no_session;
+ }
/* after waiting on the mutex check the session still exists
* then check the dlc still exists
@@ -756,12 +820,12 @@ static struct rfcomm_session *rfcomm_session_close(struct rfcomm_session *s,
return rfcomm_session_del(s);
}
-static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src,
- bdaddr_t *dst,
- u8 sec_level,
- int *err)
+/* Creates the L2CAP socket a new session will run on and starts connecting
+ * it. Must be called with rfcomm_mutex released.
+ */
+static struct socket *rfcomm_session_connect(bdaddr_t *src, bdaddr_t *dst,
+ u8 sec_level, int *err)
{
- struct rfcomm_session *s = NULL;
struct sockaddr_l2 addr;
struct socket *sock;
struct sock *sk;
@@ -791,24 +855,16 @@ static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src,
l2cap_pi(sk)->chan->mode = L2CAP_MODE_ERTM;
release_sock(sk);
- s = rfcomm_session_add(sock, BT_BOUND);
- if (!s) {
- *err = -ENOMEM;
- goto failed;
- }
-
- s->initiator = 1;
-
bacpy(&addr.l2_bdaddr, dst);
addr.l2_family = AF_BLUETOOTH;
addr.l2_psm = cpu_to_le16(L2CAP_PSM_RFCOMM);
addr.l2_cid = 0;
addr.l2_bdaddr_type = BDADDR_BREDR;
*err = kernel_connect(sock, (struct sockaddr *) &addr, sizeof(addr), O_NONBLOCK);
- if (*err == 0 || *err == -EINPROGRESS)
- return s;
+ if (*err && *err != -EINPROGRESS)
+ goto failed;
- return rfcomm_session_del(s);
+ return sock;
failed:
sock_release(sock);
diff --git a/net/bluetooth/rfcomm/sock.c b/net/bluetooth/rfcomm/sock.c
index 3c5cdd42622a..8e61991d97c0 100644
--- a/net/bluetooth/rfcomm/sock.c
+++ b/net/bluetooth/rfcomm/sock.c
@@ -50,8 +50,11 @@ static void rfcomm_sock_kill(struct sock *sk);
static void rfcomm_sk_data_ready(struct rfcomm_dlc *d, struct sk_buff *skb)
{
struct sock *sk = d->owner;
- if (!sk)
+
+ if (!sk) {
+ kfree_skb(skb);
return;
+ }
atomic_add(skb->len, &sk->sk_rmem_alloc);
skb_queue_tail(&sk->sk_receive_queue, skb);
diff --git a/net/bluetooth/rfcomm/tty.c b/net/bluetooth/rfcomm/tty.c
index 852137838d74..0578bb1ee24b 100644
--- a/net/bluetooth/rfcomm/tty.c
+++ b/net/bluetooth/rfcomm/tty.c
@@ -131,7 +131,7 @@ static void rfcomm_dev_shutdown(struct tty_port *port)
{
struct rfcomm_dev *dev = container_of(port, struct rfcomm_dev, port);
- if (dev->tty_dev->parent)
+ if (dev->tty_dev && dev->tty_dev->parent)
device_move(dev->tty_dev, NULL, DPM_ORDER_DEV_LAST);
/* close the dlc */
@@ -465,7 +465,7 @@ static int __rfcomm_release_dev(void __user *arg)
/* Shut down TTY synchronously before freeing rfcomm_dev */
tty_port_tty_vhangup(&dev->port);
- if (!test_bit(RFCOMM_TTY_OWNED, &dev->status))
+ if (!test_and_set_bit(RFCOMM_TTY_OWNED, &dev->status))
tty_port_put(&dev->port);
tty_port_put(&dev->port);
@@ -730,10 +730,9 @@ static int rfcomm_tty_install(struct tty_driver *driver, struct tty_struct *tty)
* when the last process closes the tty. The behaviour is expected by
* userspace.
*/
- if (test_bit(RFCOMM_RELEASE_ONHUP, &dev->flags)) {
- set_bit(RFCOMM_TTY_OWNED, &dev->status);
+ if (test_bit(RFCOMM_RELEASE_ONHUP, &dev->flags) &&
+ !test_and_set_bit(RFCOMM_TTY_OWNED, &dev->status))
tty_port_put(&dev->port);
- }
return 0;
}
diff --git a/net/bluetooth/sco.c b/net/bluetooth/sco.c
index f957a68aafee..178b1afa35ee 100644
--- a/net/bluetooth/sco.c
+++ b/net/bluetooth/sco.c
@@ -135,7 +135,6 @@ static void sco_sock_clear_timer(struct sock *sk)
/* ---- SCO connections ---- */
static struct sco_conn *sco_conn_add(struct hci_conn *hcon)
{
- struct hci_dev *hdev = hcon->hdev;
struct sco_conn *conn = hcon->sco_data;
if (conn) {
@@ -153,9 +152,10 @@ static struct sco_conn *sco_conn_add(struct hci_conn *hcon)
hcon->sco_data = conn;
conn->hcon = hcon;
+ conn->mtu = hcon->mtu;
- if (hdev->sco_mtu > 0)
- conn->mtu = hdev->sco_mtu;
+ if (hcon->mtu > 0)
+ conn->mtu = hcon->mtu;
else
conn->mtu = 60;
diff --git a/net/bridge/br_arp_nd_proxy.c b/net/bridge/br_arp_nd_proxy.c
index 97d0349189aa..9143bb51d83a 100644
--- a/net/bridge/br_arp_nd_proxy.c
+++ b/net/bridge/br_arp_nd_proxy.c
@@ -191,13 +191,15 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br,
n = neigh_lookup(&arp_tbl, &tip, vlandev);
if (n) {
struct net_bridge_fdb_entry *f;
+ u8 ha[ETH_ALEN] __aligned(2);
if (!(READ_ONCE(n->nud_state) & NUD_VALID)) {
neigh_release(n);
return;
}
- f = br_fdb_find_rcu(br, n->ha, vid);
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = br_fdb_find_rcu(br, ha, vid);
if (f) {
const struct net_bridge_port *dst = READ_ONCE(f->dst);
bool replied = false;
@@ -207,10 +209,10 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br,
BR_NEIGH_SUPPRESS)))) {
if (!vid)
br_arp_send(br, p, skb->dev, sip, tip,
- sha, n->ha, sha, 0, 0);
+ sha, ha, sha, 0, 0);
else
br_arp_send(br, p, skb->dev, sip, tip,
- sha, n->ha, sha,
+ sha, ha, sha,
skb->vlan_proto,
skb_vlan_tag_get(skb));
replied = true;
@@ -248,7 +250,7 @@ struct nd_msg *br_is_nd_neigh_msg(struct sk_buff *skb, struct nd_msg *msg)
}
static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
- struct sk_buff *request, struct neighbour *n,
+ struct sk_buff *request, struct neighbour *n, u8 *ha,
__be16 vlan_proto, u16 vlan_tci)
{
struct net_device *dev = request->dev;
@@ -306,7 +308,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
ipv6_eth_mc_map(&in6addr_linklocal_allnodes, eth_hdr(reply)->h_dest);
else
ether_addr_copy(eth_hdr(reply)->h_dest, daddr);
- ether_addr_copy(eth_hdr(reply)->h_source, n->ha);
+ ether_addr_copy(eth_hdr(reply)->h_source, ha);
eth_hdr(reply)->h_proto = htons(ETH_P_IPV6);
reply->protocol = htons(ETH_P_IPV6);
@@ -336,7 +338,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
na->icmph.icmp6_override = 1;
na->icmph.icmp6_solicited = dad ? 0 : 1;
na->target = ns->target;
- ether_addr_copy(&na->opt[2], n->ha);
+ ether_addr_copy(&na->opt[2], ha);
na->opt[0] = ND_OPT_TARGET_LL_ADDR;
na->opt[1] = na_olen >> 3;
@@ -365,7 +367,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p,
__vlan_hwaccel_put_tag(reply, vlan_proto, vlan_tci);
netdev_dbg(dev, "nd send dev %s dst %pI6 dst_hw %pM src %pI6 src_hw %pM\n",
- dev->name, &pip6->daddr, daddr, &pip6->saddr, n->ha);
+ dev->name, &pip6->daddr, daddr, &pip6->saddr, ha);
if (p) {
dev_queue_xmit(reply);
@@ -462,24 +464,26 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br,
n = neigh_lookup(ipv6_stub->nd_tbl, &msg->target, vlandev);
if (n) {
struct net_bridge_fdb_entry *f;
+ u8 ha[ETH_ALEN] __aligned(2);
if (!(READ_ONCE(n->nud_state) & NUD_VALID)) {
neigh_release(n);
return;
}
- f = br_fdb_find_rcu(br, n->ha, vid);
+ neigh_ha_snapshot(ha, n, n->dev);
+ f = br_fdb_find_rcu(br, ha, vid);
if (f) {
const struct net_bridge_port *dst = READ_ONCE(f->dst);
bool replied = false;
if (dst && (dst->flags & BR_NEIGH_SUPPRESS)) {
if (vid != 0)
- br_nd_send(br, p, skb, n,
+ br_nd_send(br, p, skb, n, ha,
skb->vlan_proto,
skb_vlan_tag_get(skb));
else
- br_nd_send(br, p, skb, n, 0, 0);
+ br_nd_send(br, p, skb, n, ha, 0, 0);
replied = true;
}
diff --git a/net/bridge/br_cfm.c b/net/bridge/br_cfm.c
index a89aec831416..b9a3546fb4b4 100644
--- a/net/bridge/br_cfm.c
+++ b/net/bridge/br_cfm.c
@@ -367,7 +367,7 @@ static u32 ccm_tlv_extract(struct sk_buff *skb, u32 index,
}
/* note: already called with rcu_read_lock */
-static int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb)
+int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb)
{
u32 mdlevel, interval, size, index, max;
const struct br_cfm_common_hdr *hdr;
@@ -489,11 +489,6 @@ static int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb)
return 1;
}
-static struct br_frame_type cfm_frame_type __read_mostly = {
- .type = cpu_to_be16(ETH_P_CFM),
- .frame_handler = br_cfm_frame_rx,
-};
-
int br_cfm_mep_create(struct net_bridge *br,
const u32 instance,
struct br_cfm_mep_create *const create,
@@ -559,7 +554,7 @@ int br_cfm_mep_create(struct net_bridge *br,
INIT_DELAYED_WORK(&mep->ccm_tx_dwork, ccm_tx_work_expired);
if (hlist_empty(&br->mep_list))
- br_add_frame(br, &cfm_frame_type);
+ br_opt_toggle(br, BROPT_CFM_ENABLED, true);
hlist_add_tail_rcu(&mep->head, &br->mep_list);
@@ -588,7 +583,7 @@ static void mep_delete_implementation(struct net_bridge *br,
kfree_rcu(mep, rcu);
if (hlist_empty(&br->mep_list))
- br_del_frame(br, &cfm_frame_type);
+ br_opt_toggle(br, BROPT_CFM_ENABLED, false);
}
int br_cfm_mep_delete(struct net_bridge *br,
diff --git a/net/bridge/br_device.c b/net/bridge/br_device.c
index 052986e05e62..9966a3e09323 100644
--- a/net/bridge/br_device.c
+++ b/net/bridge/br_device.c
@@ -508,7 +508,6 @@ void br_dev_setup(struct net_device *dev)
spin_lock_init(&br->lock);
INIT_LIST_HEAD(&br->port_list);
INIT_HLIST_HEAD(&br->fdb_list);
- INIT_HLIST_HEAD(&br->frame_type_list);
#if IS_ENABLED(CONFIG_BRIDGE_MRP)
INIT_HLIST_HEAD(&br->mrp_list);
#endif
diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c
index aca6db6f9535..bfc7de46a705 100644
--- a/net/bridge/br_input.c
+++ b/net/bridge/br_input.c
@@ -286,17 +286,25 @@ static int nf_hook_bridge_pre(struct sk_buff *skb, struct sk_buff **pskb)
return RX_HANDLER_CONSUMED;
}
+#define BR_CFM_MRP_OPTS \
+ ((IS_ENABLED(CONFIG_BRIDGE_CFM) ? BIT(BROPT_CFM_ENABLED) : 0UL) | \
+ (IS_ENABLED(CONFIG_BRIDGE_MRP) ? BIT(BROPT_MRP_ENABLED) : 0UL))
+
/* Return 0 if the frame was not processed otherwise 1
* note: already called with rcu_read_lock
*/
static int br_process_frame_type(struct net_bridge_port *p,
struct sk_buff *skb)
{
- struct br_frame_type *tmp;
+ struct net_bridge *br = p->br;
+
+ if (skb->protocol == htons(ETH_P_CFM) &&
+ br_opt_get(br, BROPT_CFM_ENABLED))
+ return br_cfm_frame_rx(p, skb);
- hlist_for_each_entry_rcu(tmp, &p->br->frame_type_list, list)
- if (unlikely(tmp->type == skb->protocol))
- return tmp->frame_handler(p, skb);
+ if (skb->protocol == htons(ETH_P_MRP) &&
+ br_opt_get(br, BROPT_MRP_ENABLED))
+ return br_mrp_process(p, skb);
return 0;
}
@@ -389,7 +397,8 @@ static rx_handler_result_t br_handle_frame(struct sk_buff **pskb)
}
}
- if (unlikely(br_process_frame_type(p, skb)))
+ if (unlikely((READ_ONCE(p->br->options) & BR_CFM_MRP_OPTS) &&
+ br_process_frame_type(p, skb)))
return RX_HANDLER_PASS;
forward:
@@ -430,19 +439,3 @@ rx_handler_func_t *br_get_rx_handler(const struct net_device *dev)
return br_handle_frame;
}
-
-void br_add_frame(struct net_bridge *br, struct br_frame_type *ft)
-{
- hlist_add_head_rcu(&ft->list, &br->frame_type_list);
-}
-
-void br_del_frame(struct net_bridge *br, struct br_frame_type *ft)
-{
- struct br_frame_type *tmp;
-
- hlist_for_each_entry(tmp, &br->frame_type_list, list)
- if (ft == tmp) {
- hlist_del_rcu(&ft->list);
- return;
- }
-}
diff --git a/net/bridge/br_mrp.c b/net/bridge/br_mrp.c
index a837ff7d6f36..79d6bc62179e 100644
--- a/net/bridge/br_mrp.c
+++ b/net/bridge/br_mrp.c
@@ -6,13 +6,6 @@
static const u8 mrp_test_dmac[ETH_ALEN] = { 0x1, 0x15, 0x4e, 0x0, 0x0, 0x1 };
static const u8 mrp_in_test_dmac[ETH_ALEN] = { 0x1, 0x15, 0x4e, 0x0, 0x0, 0x3 };
-static int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb);
-
-static struct br_frame_type mrp_frame_type __read_mostly = {
- .type = cpu_to_be16(ETH_P_MRP),
- .frame_handler = br_mrp_process,
-};
-
static bool br_mrp_is_ring_port(struct net_bridge_port *p_port,
struct net_bridge_port *s_port,
struct net_bridge_port *port)
@@ -486,7 +479,7 @@ static void br_mrp_del_impl(struct net_bridge *br, struct br_mrp *mrp)
kfree_rcu(mrp, rcu);
if (hlist_empty(&br->mrp_list))
- br_del_frame(br, &mrp_frame_type);
+ br_opt_toggle(br, BROPT_MRP_ENABLED, false);
}
/* Adds a new MRP instance.
@@ -536,7 +529,7 @@ int br_mrp_add(struct net_bridge *br, struct br_mrp_instance *instance)
rcu_assign_pointer(mrp->s_port, p);
if (hlist_empty(&br->mrp_list))
- br_add_frame(br, &mrp_frame_type);
+ br_opt_toggle(br, BROPT_MRP_ENABLED, true);
INIT_DELAYED_WORK(&mrp->test_work, br_mrp_test_work_expired);
INIT_DELAYED_WORK(&mrp->in_test_work, br_mrp_in_test_work_expired);
@@ -1241,7 +1234,7 @@ static int br_mrp_rcv(struct net_bridge_port *p,
* normal forwarding.
* note: already called with rcu_read_lock
*/
-static int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb)
+int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb)
{
/* If there is no MRP instance do normal forwarding */
if (likely(!(p->flags & BR_MRP_AWARE)))
diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c
index 4245d8d7767d..2e1bd7f06590 100644
--- a/net/bridge/br_multicast.c
+++ b/net/bridge/br_multicast.c
@@ -80,6 +80,10 @@ __br_multicast_add_group(struct net_bridge_mcast *brmctx,
bool blocked);
static void br_multicast_find_del_pg(struct net_bridge *br,
struct net_bridge_port_group *pg);
+static void __br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
+ struct net_bridge_port_group *pg,
+ struct net_bridge_port_group __rcu **pp,
+ bool sg_del_exclude_ports);
static void __br_multicast_stop(struct net_bridge_mcast *brmctx);
static int br_mc_disabled_update(struct net_device *dev, bool value,
@@ -436,7 +440,7 @@ static void br_multicast_sg_del_exclude_ports(struct net_bridge_mdb_entry *sgmp)
for (pp = &sgmp->ports;
(p = mlock_dereference(*pp, sgmp->br)) != NULL;) {
if (!(p->flags & MDB_PG_FLAGS_PERMANENT))
- br_multicast_del_pg(sgmp, p, pp);
+ __br_multicast_del_pg(sgmp, p, pp, false);
else
pp = &p->next;
}
@@ -675,16 +679,21 @@ static void br_multicast_destroy_port_group(struct net_bridge_mcast_gc *gc)
kfree_rcu(pg, rcu);
}
-void br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
- struct net_bridge_port_group *pg,
- struct net_bridge_port_group __rcu **pp)
+static void __br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
+ struct net_bridge_port_group *pg,
+ struct net_bridge_port_group __rcu **pp,
+ bool sg_del_exclude_ports)
{
struct net_bridge *br = pg->key.port->br;
struct net_bridge_group_src *ent;
struct hlist_node *tmp;
rcu_assign_pointer(*pp, pg->next);
- hlist_del_init(&pg->mglist);
+ /* Keep ->next (held under multicast_lock, freed later by the GC work):
+ * a port->mglist teardown walk may have latched this node as its next,
+ * and deleting other groups of the same port must not truncate it.
+ */
+ hlist_del_init_rcu(&pg->mglist);
br_multicast_eht_clean_sets(pg);
hlist_for_each_entry_safe(ent, tmp, &pg->src_list, node)
br_multicast_del_group_src(ent, false);
@@ -692,7 +701,8 @@ void br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
if (!br_multicast_is_star_g(&mp->addr)) {
rhashtable_remove_fast(&br->sg_port_tbl, &pg->rhnode,
br_sg_port_rht_params);
- br_multicast_sg_del_exclude_ports(mp);
+ if (sg_del_exclude_ports)
+ br_multicast_sg_del_exclude_ports(mp);
} else {
br_multicast_star_g_handle_mode(pg, MCAST_INCLUDE);
}
@@ -703,6 +713,13 @@ void br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
mod_timer(&mp->timer, jiffies);
}
+void br_multicast_del_pg(struct net_bridge_mdb_entry *mp,
+ struct net_bridge_port_group *pg,
+ struct net_bridge_port_group __rcu **pp)
+{
+ __br_multicast_del_pg(mp, pg, pp, true);
+}
+
static void br_multicast_find_del_pg(struct net_bridge *br,
struct net_bridge_port_group *pg)
{
@@ -710,6 +727,13 @@ static void br_multicast_find_del_pg(struct net_bridge *br,
struct net_bridge_mdb_entry *mp;
struct net_bridge_port_group *p;
+ /* A teardown walk over port->mglist can reach a group that an earlier
+ * iteration already deleted as a side effect. It is off mp->ports by
+ * now, so skip it instead of falling through to the WARN_ON() below.
+ */
+ if (hlist_unhashed(&pg->mglist))
+ return;
+
mp = br_mdb_ip_get(br, &pg->key.addr);
if (WARN_ON(!mp))
return;
@@ -1859,10 +1883,19 @@ void br_multicast_port_ctx_init(struct net_bridge_port *port,
void br_multicast_port_ctx_deinit(struct net_bridge_mcast_port *pmctx)
{
+ struct net_bridge *br = pmctx->port->br;
+ bool del = false;
+
#if IS_ENABLED(CONFIG_IPV6)
del_timer_sync(&pmctx->ip6_mc_router_timer);
#endif
del_timer_sync(&pmctx->ip4_mc_router_timer);
+
+ spin_lock_bh(&br->multicast_lock);
+ del |= br_ip6_multicast_rport_del(pmctx);
+ del |= br_ip4_multicast_rport_del(pmctx);
+ br_multicast_rport_del_notify(pmctx, del);
+ spin_unlock_bh(&br->multicast_lock);
}
int br_multicast_add_port(struct net_bridge_port *port)
diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h
index 617e636a8a4c..8c0912283df7 100644
--- a/net/bridge/br_private.h
+++ b/net/bridge/br_private.h
@@ -459,12 +459,13 @@ enum net_bridge_opts {
BROPT_VLAN_BRIDGE_BINDING,
BROPT_MCAST_VLAN_SNOOPING_ENABLED,
BROPT_MST_ENABLED,
+ BROPT_CFM_ENABLED,
+ BROPT_MRP_ENABLED,
};
struct net_bridge {
spinlock_t lock;
spinlock_t hash_lock;
- struct hlist_head frame_type_list;
struct net_device *dev;
unsigned long options;
/* These fields are accessed on each packet */
@@ -861,16 +862,6 @@ int nbp_backup_change(struct net_bridge_port *p, struct net_device *backup_dev);
int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb);
rx_handler_func_t *br_get_rx_handler(const struct net_device *dev);
-struct br_frame_type {
- __be16 type;
- int (*frame_handler)(struct net_bridge_port *port,
- struct sk_buff *skb);
- struct hlist_node list;
-};
-
-void br_add_frame(struct net_bridge *br, struct br_frame_type *ft);
-void br_del_frame(struct net_bridge *br, struct br_frame_type *ft);
-
static inline bool br_rx_handler_check_rcu(const struct net_device *dev)
{
return rcu_dereference(dev->rx_handler) == br_get_rx_handler(dev);
@@ -1931,6 +1922,7 @@ int br_mrp_parse(struct net_bridge *br, struct net_bridge_port *p,
bool br_mrp_enabled(struct net_bridge *br);
void br_mrp_port_del(struct net_bridge *br, struct net_bridge_port *p);
int br_mrp_fill_info(struct sk_buff *skb, struct net_bridge *br);
+int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb);
#else
static inline int br_mrp_parse(struct net_bridge *br, struct net_bridge_port *p,
struct nlattr *attr, int cmd,
@@ -1954,6 +1946,11 @@ static inline int br_mrp_fill_info(struct sk_buff *skb, struct net_bridge *br)
return 0;
}
+static inline int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb)
+{
+ return 0;
+}
+
#endif
/* br_cfm.c */
@@ -1962,6 +1959,7 @@ int br_cfm_parse(struct net_bridge *br, struct net_bridge_port *p,
struct nlattr *attr, int cmd, struct netlink_ext_ack *extack);
bool br_cfm_created(struct net_bridge *br);
void br_cfm_port_del(struct net_bridge *br, struct net_bridge_port *p);
+int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb);
int br_cfm_config_fill_info(struct sk_buff *skb, struct net_bridge *br);
int br_cfm_status_fill_info(struct sk_buff *skb,
struct net_bridge *br,
@@ -1986,6 +1984,12 @@ static inline void br_cfm_port_del(struct net_bridge *br,
{
}
+static inline int br_cfm_frame_rx(struct net_bridge_port *port,
+ struct sk_buff *skb)
+{
+ return 0;
+}
+
static inline int br_cfm_config_fill_info(struct sk_buff *skb, struct net_bridge *br)
{
return -EOPNOTSUPP;
diff --git a/net/bridge/netfilter/ebt_dnat.c b/net/bridge/netfilter/ebt_dnat.c
index 3fda71a8579d..73f185cccd63 100644
--- a/net/bridge/netfilter/ebt_dnat.c
+++ b/net/bridge/netfilter/ebt_dnat.c
@@ -39,7 +39,9 @@ ebt_dnat_tg(struct sk_buff *skb, const struct xt_action_param *par)
dev = xt_in(par);
break;
case NF_BR_PRE_ROUTING:
- dev = br_port_get_rcu(xt_in(par))->br->dev;
+ dev = netdev_master_upper_dev_get_rcu(xt_in(par));
+ if (!dev) /* bridge port removed? */
+ return EBT_DROP;
break;
default:
dev = NULL;
diff --git a/net/bridge/netfilter/ebt_redirect.c b/net/bridge/netfilter/ebt_redirect.c
index 307790562b49..83486cd4d564 100644
--- a/net/bridge/netfilter/ebt_redirect.c
+++ b/net/bridge/netfilter/ebt_redirect.c
@@ -24,12 +24,18 @@ ebt_redirect_tg(struct sk_buff *skb, const struct xt_action_param *par)
if (skb_ensure_writable(skb, 0))
return EBT_DROP;
- if (xt_hooknum(par) != NF_BR_BROUTING)
- /* rcu_read_lock()ed by nf_hook_thresh */
- ether_addr_copy(eth_hdr(skb)->h_dest,
- br_port_get_rcu(xt_in(par))->br->dev->dev_addr);
- else
+ if (xt_hooknum(par) != NF_BR_BROUTING) {
+ const struct net_device *dev;
+
+ dev = netdev_master_upper_dev_get_rcu(xt_in(par));
+ if (!dev)
+ return EBT_DROP;
+
+ ether_addr_copy(eth_hdr(skb)->h_dest, dev->dev_addr);
+ } else {
ether_addr_copy(eth_hdr(skb)->h_dest, xt_in(par)->dev_addr);
+ }
+
skb->pkt_type = PACKET_HOST;
return info->target;
}
diff --git a/net/bridge/netfilter/ebtable_broute.c b/net/bridge/netfilter/ebtable_broute.c
index 8f19253024b0..33d8640d21ac 100644
--- a/net/bridge/netfilter/ebtable_broute.c
+++ b/net/bridge/netfilter/ebtable_broute.c
@@ -128,8 +128,8 @@ static int __init ebtable_broute_init(void)
static void __exit ebtable_broute_fini(void)
{
- unregister_pernet_subsys(&broute_net_ops);
ebt_unregister_template(&broute_table);
+ unregister_pernet_subsys(&broute_net_ops);
}
module_init(ebtable_broute_init);
diff --git a/net/bridge/netfilter/ebtable_filter.c b/net/bridge/netfilter/ebtable_filter.c
index 278f324e6752..fdb988c24916 100644
--- a/net/bridge/netfilter/ebtable_filter.c
+++ b/net/bridge/netfilter/ebtable_filter.c
@@ -109,8 +109,8 @@ static int __init ebtable_filter_init(void)
static void __exit ebtable_filter_fini(void)
{
- unregister_pernet_subsys(&frame_filter_net_ops);
ebt_unregister_template(&frame_filter);
+ unregister_pernet_subsys(&frame_filter_net_ops);
}
module_init(ebtable_filter_init);
diff --git a/net/bridge/netfilter/ebtable_nat.c b/net/bridge/netfilter/ebtable_nat.c
index 9066f7f376d5..8b981b2041b5 100644
--- a/net/bridge/netfilter/ebtable_nat.c
+++ b/net/bridge/netfilter/ebtable_nat.c
@@ -109,8 +109,8 @@ static int __init ebtable_nat_init(void)
static void __exit ebtable_nat_fini(void)
{
- unregister_pernet_subsys(&frame_nat_net_ops);
ebt_unregister_template(&frame_nat);
+ unregister_pernet_subsys(&frame_nat_net_ops);
}
module_init(ebtable_nat_init);
diff --git a/net/bridge/netfilter/ebtables.c b/net/bridge/netfilter/ebtables.c
index 32e73f04d5b1..b1846d5c879a 100644
--- a/net/bridge/netfilter/ebtables.c
+++ b/net/bridge/netfilter/ebtables.c
@@ -42,6 +42,7 @@
struct ebt_pernet {
struct list_head tables;
+ struct list_head dead_tables;
};
struct ebt_template {
@@ -1164,11 +1165,6 @@ static int do_replace(struct net *net, sockptr_t arg, unsigned int len)
static void __ebt_unregister_table(struct net *net, struct ebt_table *table)
{
- mutex_lock(&ebt_mutex);
- list_del(&table->list);
- mutex_unlock(&ebt_mutex);
- audit_log_nfcfg(table->name, AF_BRIDGE, table->private->nentries,
- AUDIT_XT_OP_UNREGISTER, GFP_KERNEL);
EBT_ENTRY_ITERATE(table->private->entries, table->private->entries_size,
ebt_cleanup_entry, net, NULL);
if (table->private->nentries)
@@ -1269,13 +1265,15 @@ int ebt_register_table(struct net *net, const struct ebt_table *input_table,
for (i = 0; i < num_ops; i++)
ops[i].priv = table;
- list_add(&table->list, &ebt_net->tables);
- mutex_unlock(&ebt_mutex);
-
table->ops = ops;
ret = nf_register_net_hooks(net, ops, num_ops);
- if (ret)
+ if (ret) {
+ synchronize_rcu();
__ebt_unregister_table(net, table);
+ } else {
+ list_add(&table->list, &ebt_net->tables);
+ }
+ mutex_unlock(&ebt_mutex);
audit_log_nfcfg(repl->name, AF_BRIDGE, repl->nentries,
AUDIT_XT_OP_REGISTER, GFP_KERNEL);
@@ -1341,7 +1339,7 @@ void ebt_unregister_template(const struct ebt_table *t)
}
EXPORT_SYMBOL(ebt_unregister_template);
-static struct ebt_table *__ebt_find_table(struct net *net, const char *name)
+void ebt_unregister_table_pre_exit(struct net *net, const char *name)
{
struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id);
struct ebt_table *t;
@@ -1350,30 +1348,36 @@ static struct ebt_table *__ebt_find_table(struct net *net, const char *name)
list_for_each_entry(t, &ebt_net->tables, list) {
if (strcmp(t->name, name) == 0) {
+ list_move(&t->list, &ebt_net->dead_tables);
mutex_unlock(&ebt_mutex);
- return t;
+ nf_unregister_net_hooks(net, t->ops, hweight32(t->valid_hooks));
+ return;
}
}
mutex_unlock(&ebt_mutex);
- return NULL;
-}
-
-void ebt_unregister_table_pre_exit(struct net *net, const char *name)
-{
- struct ebt_table *table = __ebt_find_table(net, name);
-
- if (table)
- nf_unregister_net_hooks(net, table->ops, hweight32(table->valid_hooks));
}
EXPORT_SYMBOL(ebt_unregister_table_pre_exit);
void ebt_unregister_table(struct net *net, const char *name)
{
- struct ebt_table *table = __ebt_find_table(net, name);
+ struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id);
+ struct ebt_table *t;
- if (table)
- __ebt_unregister_table(net, table);
+ mutex_lock(&ebt_mutex);
+
+ list_for_each_entry(t, &ebt_net->dead_tables, list) {
+ if (strcmp(t->name, name) == 0) {
+ list_del(&t->list);
+ audit_log_nfcfg(t->name, AF_BRIDGE, t->private->nentries,
+ AUDIT_XT_OP_UNREGISTER, GFP_KERNEL);
+ __ebt_unregister_table(net, t);
+ mutex_unlock(&ebt_mutex);
+ return;
+ }
+ }
+
+ mutex_unlock(&ebt_mutex);
}
/* userspace just supplied us with counters */
@@ -2594,11 +2598,21 @@ static int __net_init ebt_pernet_init(struct net *net)
struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id);
INIT_LIST_HEAD(&ebt_net->tables);
+ INIT_LIST_HEAD(&ebt_net->dead_tables);
return 0;
}
+static void __net_exit ebt_pernet_exit(struct net *net)
+{
+ struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id);
+
+ WARN_ON_ONCE(!list_empty(&ebt_net->tables));
+ WARN_ON_ONCE(!list_empty(&ebt_net->dead_tables));
+}
+
static struct pernet_operations ebt_net_ops = {
.init = ebt_pernet_init,
+ .exit = ebt_pernet_exit,
.id = &ebt_pernet_id,
.size = sizeof(struct ebt_pernet),
};
@@ -2607,19 +2621,20 @@ static int __init ebtables_init(void)
{
int ret;
- ret = xt_register_target(&ebt_standard_target);
+ ret = register_pernet_subsys(&ebt_net_ops);
if (ret < 0)
return ret;
- ret = nf_register_sockopt(&ebt_sockopts);
+
+ ret = xt_register_target(&ebt_standard_target);
if (ret < 0) {
- xt_unregister_target(&ebt_standard_target);
+ unregister_pernet_subsys(&ebt_net_ops);
return ret;
}
- ret = register_pernet_subsys(&ebt_net_ops);
+ ret = nf_register_sockopt(&ebt_sockopts);
if (ret < 0) {
- nf_unregister_sockopt(&ebt_sockopts);
xt_unregister_target(&ebt_standard_target);
+ unregister_pernet_subsys(&ebt_net_ops);
return ret;
}
diff --git a/net/can/af_can.c b/net/can/af_can.c
index e7bafb2e013a..de6bb3484843 100644
--- a/net/can/af_can.c
+++ b/net/can/af_can.c
@@ -643,10 +643,8 @@ static int can_rcv_filter(struct can_dev_rcv_lists *dev_rcv_lists, struct sk_buf
void can_set_skb_uid(struct sk_buff *skb)
{
/* create non-zero unique skb identifier together with *skb */
- while (!(skb->hash))
- skb->hash = atomic_inc_return(&skbcounter);
-
- skb->sw_hash = 1;
+ while (!(can_skb_prv(skb)->skbcnt))
+ can_skb_prv(skb)->skbcnt = atomic_inc_return(&skbcounter);
}
EXPORT_SYMBOL(can_set_skb_uid);
diff --git a/net/can/bcm.c b/net/can/bcm.c
index 35f0037ac48b..16640650c433 100644
--- a/net/can/bcm.c
+++ b/net/can/bcm.c
@@ -338,6 +338,7 @@ static void bcm_can_tx(struct bcm_op *op, struct canfd_frame *cf)
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skb_put_data(skb, cf, op->cfsiz);
@@ -1581,6 +1582,7 @@ static int bcm_tx_send(struct msghdr *msg, int ifindex, struct sock *sk,
}
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skb->dev = dev;
can_skb_set_owner(skb, sk);
err = can_send(skb, 1); /* send with loopback */
diff --git a/net/can/isotp.c b/net/can/isotp.c
index 3a2eca45ef43..5a0ecbc4f6b9 100644
--- a/net/can/isotp.c
+++ b/net/can/isotp.c
@@ -287,6 +287,7 @@ static int isotp_send_fc(struct sock *sk, int ae, u8 flowstatus)
can_skb_reserve(nskb);
can_skb_prv(nskb)->ifindex = dev->ifindex;
+ can_skb_prv(nskb)->skbcnt = 0;
nskb->dev = dev;
can_skb_set_owner(nskb, sk);
@@ -848,6 +849,7 @@ static void isotp_send_cframe(struct isotp_sock *so)
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
/* set uid in tx skb to identify CF echo frames */
can_set_skb_uid(skb);
@@ -876,7 +878,7 @@ static void isotp_send_cframe(struct isotp_sock *so)
pr_notice_once("can-isotp: cfecho is %08X != 0\n", old_cfecho);
/* set consecutive frame echo tag */
- WRITE_ONCE(so->cfecho, skb->hash);
+ WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt);
/* send frame with local echo enabled */
can_send_ret = can_send(skb, 1);
@@ -939,7 +941,7 @@ static void isotp_rcv_echo(struct sk_buff *skb, void *data)
spin_lock(&so->rx_lock);
/* so->cfecho may since belong to a new transfer; recheck under lock */
- if (READ_ONCE(so->cfecho) != skb->hash)
+ if (READ_ONCE(so->cfecho) != can_skb_prv(skb)->skbcnt)
goto out_unlock;
/* cancel local echo timeout */
@@ -1161,6 +1163,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
/* set uid in tx skb to identify CF echo frames */
can_set_skb_uid(skb);
@@ -1202,7 +1205,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
cf->data[ae] |= size;
/* set CF echo tag for isotp_rcv_echo() (SF-mode) */
- WRITE_ONCE(so->cfecho, skb->hash);
+ WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt);
} else {
/* send first frame */
@@ -1219,7 +1222,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
so->txfc.bs = 0;
/* set CF echo tag for isotp_rcv_echo() (CF-mode) */
- WRITE_ONCE(so->cfecho, skb->hash);
+ WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt);
} else {
/* standard flow control check */
new_state = ISOTP_WAIT_FIRST_FC;
diff --git a/net/can/j1939/socket.c b/net/can/j1939/socket.c
index 1b0086198ba3..00a4ee90cbaa 100644
--- a/net/can/j1939/socket.c
+++ b/net/can/j1939/socket.c
@@ -895,6 +895,7 @@ static struct sk_buff *j1939_sk_alloc_skb(struct net_device *ndev,
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = ndev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skb_reserve(skb, offsetof(struct can_frame, data));
ret = memcpy_from_msg(skb_put(skb, size), msg, size);
diff --git a/net/can/j1939/transport.c b/net/can/j1939/transport.c
index a1a28240fec2..b11febe2b5fb 100644
--- a/net/can/j1939/transport.c
+++ b/net/can/j1939/transport.c
@@ -613,6 +613,7 @@ sk_buff *j1939_tp_tx_dat_new(struct j1939_priv *priv,
skb->dev = priv->ndev;
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = priv->ndev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
/* reserve CAN header */
skb_reserve(skb, offsetof(struct can_frame, data));
@@ -1552,6 +1553,7 @@ j1939_session *j1939_session_fresh_new(struct j1939_priv *priv,
skb->dev = priv->ndev;
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = priv->ndev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
skcb = j1939_skb_to_cb(skb);
memcpy(skcb, rel_skcb, sizeof(*skcb));
diff --git a/net/can/raw.c b/net/can/raw.c
index c54d3b8ee215..6d957f1e0fc6 100644
--- a/net/can/raw.c
+++ b/net/can/raw.c
@@ -75,8 +75,8 @@ MODULE_ALIAS("can-proto-1");
*/
struct uniqframe {
+ u32 skbcnt;
const struct sk_buff *skb;
- u32 hash;
unsigned int join_rx_count;
};
@@ -140,7 +140,7 @@ static void raw_rcv(struct sk_buff *oskb, void *data)
/* eliminate multiple filter matches for the same skb */
if (this_cpu_ptr(ro->uniq)->skb == oskb &&
- this_cpu_ptr(ro->uniq)->hash == oskb->hash) {
+ this_cpu_ptr(ro->uniq)->skbcnt == can_skb_prv(oskb)->skbcnt) {
if (!ro->join_filters)
return;
@@ -150,7 +150,7 @@ static void raw_rcv(struct sk_buff *oskb, void *data)
return;
} else {
this_cpu_ptr(ro->uniq)->skb = oskb;
- this_cpu_ptr(ro->uniq)->hash = oskb->hash;
+ this_cpu_ptr(ro->uniq)->skbcnt = can_skb_prv(oskb)->skbcnt;
this_cpu_ptr(ro->uniq)->join_rx_count = 1;
/* drop first frame to check all enabled filters? */
if (ro->join_filters && ro->count > 1)
@@ -870,6 +870,7 @@ static int raw_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
can_skb_reserve(skb);
can_skb_prv(skb)->ifindex = dev->ifindex;
+ can_skb_prv(skb)->skbcnt = 0;
/* fill the skb before testing for valid CAN frames */
err = memcpy_from_msg(skb_put(skb, size), msg, size);
diff --git a/net/ceph/auth_x.c b/net/ceph/auth_x.c
index c83559f015e4..f805a813308f 100644
--- a/net/ceph/auth_x.c
+++ b/net/ceph/auth_x.c
@@ -86,6 +86,11 @@ static int __ceph_x_decrypt(struct ceph_crypto_key *secret, void *p,
if (ret)
return ret;
+ if (plaintext_len < sizeof(*hdr)) {
+ pr_err("%s plaintext too small %d\n", __func__, plaintext_len);
+ return -EINVAL;
+ }
+
if (le64_to_cpu(hdr->magic) != CEPHX_ENC_MAGIC) {
pr_err("%s bad magic\n", __func__);
return -EINVAL;
diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
index db2794d50bdb..3009df467c8d 100644
--- a/net/ceph/messenger.c
+++ b/net/ceph/messenger.c
@@ -446,6 +446,7 @@ int ceph_tcp_connect(struct ceph_connection *con)
if (ret)
return ret;
sock->sk->sk_allocation = GFP_NOFS;
+ sock->sk->sk_use_task_frag = false;
#ifdef CONFIG_LOCKDEP
lockdep_set_class(&sock->sk->sk_lock, &socket_class);
diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c
index 0124536e8a9d..d4078e288f42 100644
--- a/net/core/bpf_sk_storage.c
+++ b/net/core/bpf_sk_storage.c
@@ -164,8 +164,6 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk)
struct bpf_local_storage_elem *selem;
int ret = 0;
- RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
-
rcu_read_lock();
sk_storage = rcu_dereference(sk->sk_bpf_storage);
diff --git a/net/core/dev.c b/net/core/dev.c
index f9597dbfd422..7c0d0905dd31 100644
--- a/net/core/dev.c
+++ b/net/core/dev.c
@@ -2121,6 +2121,11 @@ void net_dec_egress_queue(void)
EXPORT_SYMBOL_GPL(net_dec_egress_queue);
#endif
+#ifdef CONFIG_NET_CLS_ACT
+DEFINE_STATIC_KEY_FALSE(tcf_sw_enabled_key);
+EXPORT_SYMBOL(tcf_sw_enabled_key);
+#endif
+
DEFINE_STATIC_KEY_FALSE(netstamp_needed_key);
EXPORT_SYMBOL(netstamp_needed_key);
#ifdef CONFIG_JUMP_LABEL
@@ -3633,10 +3638,15 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb,
* IPv4 header has the potential to be fragmented.
*/
if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4) {
- struct iphdr *iph = skb->encapsulation ?
- inner_ip_hdr(skb) : ip_hdr(skb);
+ const struct iphdr *iph;
+ struct iphdr _iph;
+ int nhoff = skb->encapsulation ?
+ skb_inner_network_offset(skb) :
+ skb_network_offset(skb);
- if (!(iph->frag_off & htons(IP_DF)))
+ iph = skb_header_pointer(skb, nhoff, sizeof(_iph), &_iph);
+
+ if (!iph || !(iph->frag_off & htons(IP_DF)))
features &= ~NETIF_F_TSO_MANGLEID;
}
@@ -3948,6 +3958,10 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
return rc;
}
+ if (unlikely(READ_ONCE(q->owner) == smp_processor_id())) {
+ kfree_skb_reason(skb, SKB_DROP_REASON_TC_EGRESS);
+ return NET_XMIT_DROP;
+ }
/*
* Heuristic to force contended enqueues to serialize on a
* separate lock before trying to get qdisc main lock.
@@ -3987,7 +4001,9 @@ static inline int __dev_xmit_skb(struct sk_buff *skb, struct Qdisc *q,
qdisc_run_end(q);
rc = NET_XMIT_SUCCESS;
} else {
+ WRITE_ONCE(q->owner, smp_processor_id());
rc = dev_qdisc_enqueue(skb, q, &to_free, txq);
+ WRITE_ONCE(q->owner, -1);
if (qdisc_run_begin(q)) {
if (unlikely(contended)) {
spin_unlock(&q->busylock);
@@ -4064,6 +4080,7 @@ sch_handle_egress(struct sk_buff *skb, int *ret, struct net_device *dev)
/* qdisc_skb_cb(skb)->pkt_len was already set by the caller. */
tc_skb_cb(skb)->mru = 0;
tc_skb_cb(skb)->post_ct = false;
+ tcf_set_drop_reason(skb, SKB_DROP_REASON_TC_EGRESS);
mini_qdisc_bstats_cpu_update(miniq, skb);
switch (tcf_classify(skb, miniq->block, miniq->filter_list, &cl_res, false)) {
@@ -4074,7 +4091,7 @@ sch_handle_egress(struct sk_buff *skb, int *ret, struct net_device *dev)
case TC_ACT_SHOT:
mini_qdisc_qstats_cpu_drop(miniq);
*ret = NET_XMIT_DROP;
- kfree_skb_reason(skb, SKB_DROP_REASON_TC_EGRESS);
+ kfree_skb_reason(skb, tcf_get_drop_reason(skb) ?: SKB_DROP_REASON_TC_EGRESS);
return NULL;
case TC_ACT_STOLEN:
case TC_ACT_QUEUED:
@@ -5257,6 +5274,7 @@ sch_handle_ingress(struct sk_buff *skb, struct packet_type **pt_prev, int *ret,
tc_skb_cb(skb)->mru = 0;
tc_skb_cb(skb)->post_ct = false;
skb->tc_at_ingress = 1;
+ tcf_set_drop_reason(skb, SKB_DROP_REASON_TC_INGRESS);
mini_qdisc_bstats_cpu_update(miniq, skb);
switch (tcf_classify(skb, miniq->block, miniq->filter_list, &cl_res, false)) {
@@ -5266,7 +5284,7 @@ sch_handle_ingress(struct sk_buff *skb, struct packet_type **pt_prev, int *ret,
break;
case TC_ACT_SHOT:
mini_qdisc_qstats_cpu_drop(miniq);
- kfree_skb_reason(skb, SKB_DROP_REASON_TC_INGRESS);
+ kfree_skb_reason(skb, tcf_get_drop_reason(skb) ?: SKB_DROP_REASON_TC_INGRESS);
*ret = NET_RX_DROP;
return NULL;
case TC_ACT_STOLEN:
@@ -9231,6 +9249,29 @@ static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode,
struct netdev_bpf xdp;
int err;
+ if (prog) {
+ enum bpf_xdp_mode other_mode = mode == XDP_MODE_SKB
+ ? XDP_MODE_DRV : XDP_MODE_SKB;
+ bool offload = mode == XDP_MODE_HW;
+
+ if (!offload && dev_xdp_prog(dev, other_mode)) {
+ NL_SET_ERR_MSG(extack, "Native and generic XDP can't be active at the same time");
+ return -EEXIST;
+ }
+ if (!offload && bpf_prog_is_dev_bound(prog->aux)) {
+ NL_SET_ERR_MSG(extack, "Using device-bound program without HW_MODE flag is not supported");
+ return -EINVAL;
+ }
+ if (prog->expected_attach_type == BPF_XDP_DEVMAP) {
+ NL_SET_ERR_MSG(extack, "BPF_XDP_DEVMAP programs can not be attached to a device");
+ return -EINVAL;
+ }
+ if (prog->expected_attach_type == BPF_XDP_CPUMAP) {
+ NL_SET_ERR_MSG(extack, "BPF_XDP_CPUMAP programs can not be attached to a device");
+ return -EINVAL;
+ }
+ }
+
memset(&xdp, 0, sizeof(xdp));
xdp.command = mode == XDP_MODE_HW ? XDP_SETUP_PROG_HW : XDP_SETUP_PROG;
xdp.extack = extack;
@@ -9359,30 +9400,10 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack
new_prog = link->link.prog;
if (new_prog) {
- bool offload = mode == XDP_MODE_HW;
- enum bpf_xdp_mode other_mode = mode == XDP_MODE_SKB
- ? XDP_MODE_DRV : XDP_MODE_SKB;
-
if ((flags & XDP_FLAGS_UPDATE_IF_NOEXIST) && cur_prog) {
NL_SET_ERR_MSG(extack, "XDP program already attached");
return -EBUSY;
}
- if (!offload && dev_xdp_prog(dev, other_mode)) {
- NL_SET_ERR_MSG(extack, "Native and generic XDP can't be active at the same time");
- return -EEXIST;
- }
- if (!offload && bpf_prog_is_dev_bound(new_prog->aux)) {
- NL_SET_ERR_MSG(extack, "Using device-bound program without HW_MODE flag is not supported");
- return -EINVAL;
- }
- if (new_prog->expected_attach_type == BPF_XDP_DEVMAP) {
- NL_SET_ERR_MSG(extack, "BPF_XDP_DEVMAP programs can not be attached to a device");
- return -EINVAL;
- }
- if (new_prog->expected_attach_type == BPF_XDP_CPUMAP) {
- NL_SET_ERR_MSG(extack, "BPF_XDP_CPUMAP programs can not be attached to a device");
- return -EINVAL;
- }
}
/* don't call drivers if the effective program didn't change */
@@ -10060,7 +10081,8 @@ static int netif_alloc_netdev_queues(struct net_device *dev)
netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL);
spin_lock_init(&dev->tx_global_lock);
-
+ spin_lock_init(&dev->watchdog_lock);
+ dev->watchdog_ref_held = false;
return 0;
}
@@ -10854,6 +10876,7 @@ struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name,
free_percpu(dev->pcpu_refcnt);
free_dev:
#endif
+ ref_tracker_dir_exit(&dev->refcnt_tracker);
netdev_freemem(dev);
return NULL;
}
diff --git a/net/core/dst.c b/net/core/dst.c
index cbb6888aa32b..2ac7d3bbabf1 100644
--- a/net/core/dst.c
+++ b/net/core/dst.c
@@ -173,6 +173,14 @@ void dst_release(struct dst_entry *dst)
net_warn_ratelimited("%s: dst:%p refcnt:%d\n",
__func__, dst, newrefcnt);
if (!newrefcnt){
+#ifdef CONFIG_DST_CACHE
+ if (dst->flags & DST_METADATA) {
+ struct metadata_dst *md_dst = (struct metadata_dst *)dst;
+
+ if (md_dst->type == METADATA_IP_TUNNEL)
+ dst_cache_reset_now(&md_dst->u.tun_info.dst_cache);
+ }
+#endif
dst_count_dec(dst);
call_rcu(&dst->rcu_head, dst_destroy_rcu);
}
diff --git a/net/core/filter.c b/net/core/filter.c
index 678fd0c108c8..cb62f24cb4a1 100644
--- a/net/core/filter.c
+++ b/net/core/filter.c
@@ -7333,7 +7333,7 @@ BPF_CALL_5(bpf_tcp_check_syncookie, struct sock *, sk, void *, iph, u32, iph_len
return -EINVAL;
/* sk_listener() allows TCP_NEW_SYN_RECV, which makes no sense here. */
- if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN)
+ if (sk->sk_state != TCP_LISTEN || sk->sk_protocol != IPPROTO_TCP)
return -EINVAL;
if (!READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_syncookies))
@@ -7408,7 +7408,7 @@ BPF_CALL_5(bpf_tcp_gen_syncookie, struct sock *, sk, void *, iph, u32, iph_len,
if (unlikely(!sk || th_len < sizeof(*th) || th_len != th->doff * 4))
return -EINVAL;
- if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN)
+ if (sk->sk_state != TCP_LISTEN || sk->sk_protocol != IPPROTO_TCP)
return -EINVAL;
if (!READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_syncookies))
diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c
index e604c6a08cbd..511335741d68 100644
--- a/net/core/flow_dissector.c
+++ b/net/core/flow_dissector.c
@@ -921,6 +921,8 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx,
result = bpf_prog_run_pin_on_cpu(prog, ctx);
+ /* bpf_flow_keys offsets are u16: do not let @hlen be truncated. */
+ hlen = min_t(int, hlen, U16_MAX);
flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen);
flow_keys->thoff = clamp_t(u16, flow_keys->thoff,
flow_keys->nhoff, hlen);
@@ -969,6 +971,7 @@ bool __skb_flow_dissect(const struct net *net,
int mpls_lse = 0;
int num_hdrs = 0;
u8 ip_proto = 0;
+ u32 thoff;
bool ret;
if (!data) {
@@ -1575,7 +1578,13 @@ bool __skb_flow_dissect(const struct net *net,
ret = true;
out:
- key_control->thoff = min_t(u16, nhoff, skb ? skb->len : hlen);
+ thoff = min_t(u32, nhoff, skb ? skb->len : hlen);
+ if (unlikely(thoff > U16_MAX)) {
+ /* Cannot be represented in key_control->thoff. */
+ thoff = U16_MAX;
+ ret = false;
+ }
+ key_control->thoff = thoff;
key_basic->n_proto = proto;
key_basic->ip_proto = ip_proto;
@@ -1760,7 +1769,8 @@ u32 __skb_get_hash_symmetric(const struct sk_buff *skb)
EXPORT_SYMBOL_GPL(__skb_get_hash_symmetric);
/**
- * __skb_get_hash: calculate a flow hash
+ * __skb_get_hash_net: calculate a flow hash
+ * @net: associated network namespace, derived from @skb if NULL
* @skb: sk_buff to calculate flow hash from
*
* This function calculates a flow hash based on src/dst addresses
@@ -1768,18 +1778,24 @@ EXPORT_SYMBOL_GPL(__skb_get_hash_symmetric);
* on success, zero indicates no valid hash. Also, sets l4_hash in skb
* if hash is a canonical 4-tuple hash over transport ports.
*/
-void __skb_get_hash(struct sk_buff *skb)
+void __skb_get_hash_net(const struct net *net, struct sk_buff *skb)
{
struct flow_keys keys;
u32 hash;
+ memset(&keys, 0, sizeof(keys));
+
+ __skb_flow_dissect(net, skb, &flow_keys_dissector,
+ &keys, NULL, 0, 0, 0,
+ FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL);
+
__flow_hash_secret_init();
- hash = ___skb_get_hash(skb, &keys, &hashrnd);
+ hash = __flow_hash_from_keys(&keys, &hashrnd);
__skb_set_sw_hash(skb, hash, flow_keys_have_l4(&keys));
}
-EXPORT_SYMBOL(__skb_get_hash);
+EXPORT_SYMBOL(__skb_get_hash_net);
__u32 skb_get_hash_perturb(const struct sk_buff *skb,
const siphash_key_t *perturb)
diff --git a/net/core/netpoll.c b/net/core/netpoll.c
index a8957be28803..84ce00940bc8 100644
--- a/net/core/netpoll.c
+++ b/net/core/netpoll.c
@@ -830,14 +830,6 @@ static void rcu_cleanup_netpoll_info(struct rcu_head *rcu_head)
container_of(rcu_head, struct netpoll_info, rcu);
skb_queue_purge(&npinfo->txq);
-
- /* we can't call cancel_delayed_work_sync here, as we are in softirq */
- cancel_delayed_work(&npinfo->tx_work);
-
- /* clean after last, unfinished work */
- __skb_queue_purge(&npinfo->txq);
- /* now cancel it again */
- cancel_delayed_work(&npinfo->tx_work);
kfree(npinfo);
}
@@ -863,6 +855,8 @@ void __netpoll_cleanup(struct netpoll *np)
ops->ndo_netpoll_cleanup(np->dev);
RCU_INIT_POINTER(np->dev->npinfo, NULL);
+ synchronize_rcu();
+ cancel_delayed_work_sync(&npinfo->tx_work);
call_rcu(&npinfo->rcu, rcu_cleanup_netpoll_info);
}
}
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 4a3abd86f8ce..22d2eab4864b 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -727,7 +727,8 @@ struct page *page_pool_alloc_frag(struct page_pool *pool,
size > max_size))
return NULL;
- size = ALIGN(size, dma_get_cache_alignment());
+ size = ALIGN(size, max_t(unsigned int, dma_get_cache_alignment(),
+ __alignof__(struct skb_shared_info)));
*offset = pool->frag_offset;
if (page && *offset + size > max_size) {
diff --git a/net/core/pktgen.c b/net/core/pktgen.c
index 23ec71a88290..9756e9f9be64 100644
--- a/net/core/pktgen.c
+++ b/net/core/pktgen.c
@@ -114,6 +114,7 @@
#include <linux/sys.h>
#include <linux/types.h>
+#include <linux/cpu.h>
#include <linux/module.h>
#include <linux/moduleparam.h>
#include <linux/kernel.h>
@@ -3615,7 +3616,7 @@ static int pktgen_thread_worker(void *arg)
struct pktgen_dev *pkt_dev = NULL;
int cpu = t->cpu;
- WARN_ON(smp_processor_id() != cpu);
+ WARN_ON_ONCE(smp_processor_id() != cpu);
init_waitqueue_head(&t->queue);
complete(&t->start_done);
@@ -3951,6 +3952,7 @@ static int __net_init pg_net_init(struct net *net)
goto remove;
}
+ cpus_read_lock();
for_each_online_cpu(cpu) {
int err;
@@ -3959,6 +3961,7 @@ static int __net_init pg_net_init(struct net *net)
pr_warn("Cannot create thread for cpu %d (%d)\n",
cpu, err);
}
+ cpus_read_unlock();
if (list_empty(&pn->pktgen_threads)) {
pr_err("Initialization failed for all threads\n");
diff --git a/net/core/secure_seq.c b/net/core/secure_seq.c
index b0ff6153be62..740642aeaf76 100644
--- a/net/core/secure_seq.c
+++ b/net/core/secure_seq.c
@@ -20,7 +20,6 @@
#include <net/tcp.h>
static siphash_aligned_key_t net_secret;
-static siphash_aligned_key_t ts_secret;
#define EPHEMERAL_PORT_SHUFFLE_PERIOD (10 * HZ)
@@ -28,11 +27,6 @@ static __always_inline void net_secret_init(void)
{
net_get_random_once(&net_secret, sizeof(net_secret));
}
-
-static __always_inline void ts_secret_init(void)
-{
- net_get_random_once(&ts_secret, sizeof(ts_secret));
-}
#endif
#ifdef CONFIG_INET
@@ -53,28 +47,9 @@ static u32 seq_scale(u32 seq)
#endif
#if IS_ENABLED(CONFIG_IPV6)
-u32 secure_tcpv6_ts_off(const struct net *net,
- const __be32 *saddr, const __be32 *daddr)
-{
- const struct {
- struct in6_addr saddr;
- struct in6_addr daddr;
- } __aligned(SIPHASH_ALIGNMENT) combined = {
- .saddr = *(struct in6_addr *)saddr,
- .daddr = *(struct in6_addr *)daddr,
- };
-
- if (READ_ONCE(net->ipv4.sysctl_tcp_timestamps) != 1)
- return 0;
-
- ts_secret_init();
- return siphash(&combined, offsetofend(typeof(combined), daddr),
- &ts_secret);
-}
-EXPORT_SYMBOL(secure_tcpv6_ts_off);
-
-u32 secure_tcpv6_seq(const __be32 *saddr, const __be32 *daddr,
- __be16 sport, __be16 dport)
+union tcp_seq_and_ts_off
+secure_tcpv6_seq_and_ts_off(const struct net *net, const __be32 *saddr,
+ const __be32 *daddr, __be16 sport, __be16 dport)
{
const struct {
struct in6_addr saddr;
@@ -87,14 +62,20 @@ u32 secure_tcpv6_seq(const __be32 *saddr, const __be32 *daddr,
.sport = sport,
.dport = dport
};
- u32 hash;
+ union tcp_seq_and_ts_off st;
net_secret_init();
- hash = siphash(&combined, offsetofend(typeof(combined), dport),
- &net_secret);
- return seq_scale(hash);
+
+ st.hash64 = siphash(&combined, offsetofend(typeof(combined), dport),
+ &net_secret);
+
+ if (READ_ONCE(net->ipv4.sysctl_tcp_timestamps) != 1)
+ st.ts_off = 0;
+
+ st.seq = seq_scale(st.seq);
+ return st;
}
-EXPORT_SYMBOL(secure_tcpv6_seq);
+EXPORT_SYMBOL(secure_tcpv6_seq_and_ts_off);
u64 secure_ipv6_port_ephemeral(const __be32 *saddr, const __be32 *daddr,
__be16 dport)
@@ -118,33 +99,30 @@ EXPORT_SYMBOL(secure_ipv6_port_ephemeral);
#endif
#ifdef CONFIG_INET
-u32 secure_tcp_ts_off(const struct net *net, __be32 saddr, __be32 daddr)
-{
- if (READ_ONCE(net->ipv4.sysctl_tcp_timestamps) != 1)
- return 0;
-
- ts_secret_init();
- return siphash_2u32((__force u32)saddr, (__force u32)daddr,
- &ts_secret);
-}
-
/* secure_tcp_seq_and_tsoff(a, b, 0, d) == secure_ipv4_port_ephemeral(a, b, d),
* but fortunately, `sport' cannot be 0 in any circumstances. If this changes,
* it would be easy enough to have the former function use siphash_4u32, passing
* the arguments as separate u32.
*/
-u32 secure_tcp_seq(__be32 saddr, __be32 daddr,
- __be16 sport, __be16 dport)
+union tcp_seq_and_ts_off
+secure_tcp_seq_and_ts_off(const struct net *net, __be32 saddr, __be32 daddr,
+ __be16 sport, __be16 dport)
{
- u32 hash;
+ u32 ports = (__force u32)sport << 16 | (__force u32)dport;
+ union tcp_seq_and_ts_off st;
net_secret_init();
- hash = siphash_3u32((__force u32)saddr, (__force u32)daddr,
- (__force u32)sport << 16 | (__force u32)dport,
- &net_secret);
- return seq_scale(hash);
+
+ st.hash64 = siphash_3u32((__force u32)saddr, (__force u32)daddr,
+ ports, &net_secret);
+
+ if (READ_ONCE(net->ipv4.sysctl_tcp_timestamps) != 1)
+ st.ts_off = 0;
+
+ st.seq = seq_scale(st.seq);
+ return st;
}
-EXPORT_SYMBOL_GPL(secure_tcp_seq);
+EXPORT_SYMBOL_GPL(secure_tcp_seq_and_ts_off);
u64 secure_ipv4_port_ephemeral(__be32 saddr, __be32 daddr, __be16 dport)
{
diff --git a/net/core/skmsg.c b/net/core/skmsg.c
index 20e67399f0b6..02d3f2ddd25d 100644
--- a/net/core/skmsg.c
+++ b/net/core/skmsg.c
@@ -1266,15 +1266,27 @@ static int sk_psock_verdict_recv(struct sock *sk, struct sk_buff *skb)
static void sk_psock_verdict_data_ready(struct sock *sk)
{
- struct socket *sock = sk->sk_socket;
+ const struct proto_ops *ops = NULL;
+ struct sk_psock *psock;
+ struct socket *sock;
int copied;
- if (unlikely(!sock || !sock->ops || !sock->ops->read_skb))
+ rcu_read_lock();
+ psock = sk_psock(sk);
+ if (psock && tls_sw_has_ctx_rx(sk)) {
+ psock->saved_data_ready(sk);
+ rcu_read_unlock();
+ return;
+ }
+ sock = READ_ONCE(sk->sk_socket);
+ if (likely(sock))
+ ops = READ_ONCE(sock->ops);
+ rcu_read_unlock();
+ if (!ops || !ops->read_skb)
return;
- copied = sock->ops->read_skb(sk, sk_psock_verdict_recv);
- if (copied >= 0) {
- struct sk_psock *psock;
+ copied = ops->read_skb(sk, sk_psock_verdict_recv);
+ if (copied >= 0) {
rcu_read_lock();
psock = sk_psock(sk);
if (psock)
diff --git a/net/core/sock.c b/net/core/sock.c
index 899ddf069940..d05c0cb1e52e 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -2261,6 +2261,9 @@ struct sock *sk_clone_lock(const struct sock *sk, const gfp_t priority)
sock_copy(newsk, sk);
newsk->sk_prot_creator = prot;
+#ifdef CONFIG_BPF_SYSCALL
+ RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL);
+#endif
/* SANITY */
if (likely(newsk->sk_net_refcnt)) {
@@ -2395,9 +2398,12 @@ static void sk_trim_gso_size(struct sock *sk)
void sk_setup_caps(struct sock *sk, struct dst_entry *dst)
{
+ const struct net_device *dev;
u32 max_segs = 1;
- sk->sk_route_caps = dst_dev(dst)->features;
+ rcu_read_lock();
+ dev = dst_dev_rcu(dst);
+ sk->sk_route_caps = dev->features;
if (sk_is_tcp(sk))
sk->sk_route_caps |= NETIF_F_GSO;
if (sk->sk_route_caps & NETIF_F_GSO)
@@ -2410,15 +2416,16 @@ void sk_setup_caps(struct sock *sk, struct dst_entry *dst)
} else {
sk->sk_route_caps |= NETIF_F_SG | NETIF_F_HW_CSUM;
/* pairs with the WRITE_ONCE() in netif_set_gso_max_size() */
- sk->sk_gso_max_size = READ_ONCE(dst_dev(dst)->gso_max_size);
+ sk->sk_gso_max_size = READ_ONCE(dev->gso_max_size);
sk_trim_gso_size(sk);
sk->sk_gso_max_size -= (MAX_TCP_HEADER + 1);
/* pairs with the WRITE_ONCE() in netif_set_gso_max_segs() */
- max_segs = max_t(u32, READ_ONCE(dst_dev(dst)->gso_max_segs), 1);
+ max_segs = max_t(u32, READ_ONCE(dev->gso_max_segs), 1);
}
}
sk->sk_gso_max_segs = max_segs;
sk_dst_set(sk, dst);
+ rcu_read_unlock();
}
EXPORT_SYMBOL_GPL(sk_setup_caps);
@@ -2670,6 +2677,22 @@ void *sock_kmalloc(struct sock *sk, int size, gfp_t priority)
}
EXPORT_SYMBOL(sock_kmalloc);
+/*
+ * Duplicate the input "src" memory block using the socket's
+ * option memory buffer.
+ */
+void *sock_kmemdup(struct sock *sk, const void *src,
+ int size, gfp_t priority)
+{
+ void *mem;
+
+ mem = sock_kmalloc(sk, size, priority);
+ if (mem)
+ memcpy(mem, src, size);
+ return mem;
+}
+EXPORT_SYMBOL(sock_kmemdup);
+
/* Free an option memory block. Note, we actually want the inline
* here as this allows gcc to detect the nullify and fold away the
* condition entirely.
@@ -3417,6 +3440,7 @@ void sock_init_data_uid(struct socket *sock, struct sock *sk, kuid_t uid)
sk->sk_rcvbuf = READ_ONCE(sysctl_rmem_default);
sk->sk_sndbuf = READ_ONCE(sysctl_wmem_default);
sk->sk_state = TCP_CLOSE;
+ sk->sk_use_task_frag = true;
sk_set_socket(sk, sock);
sock_set_flag(sk, SOCK_ZAPPED);
diff --git a/net/dccp/timer.c b/net/dccp/timer.c
index 27a3b37acd2e..ff9c0184e3f6 100644
--- a/net/dccp/timer.c
+++ b/net/dccp/timer.c
@@ -160,7 +160,9 @@ static void dccp_write_timer(struct timer_list *t)
static void dccp_keepalive_timer(struct timer_list *t)
{
- struct sock *sk = from_timer(sk, t, sk_timer);
+ struct inet_connection_sock *icsk =
+ from_timer(icsk, t, icsk_keepalive_timer);
+ struct sock *sk = &icsk->icsk_inet.sk;
pr_err("dccp should not use a keepalive timer !\n");
sock_put(sk);
diff --git a/net/dsa/dsa2.c b/net/dsa/dsa2.c
index 9ecb5e34e484..51befa73bfd6 100644
--- a/net/dsa/dsa2.c
+++ b/net/dsa/dsa2.c
@@ -1541,14 +1541,25 @@ static int dsa_port_parse_of(struct dsa_port *dp, struct device_node *dn)
if (ethernet) {
struct net_device *master;
const char *user_protocol;
+ int err;
+ rtnl_lock();
master = of_find_net_device_by_node(ethernet);
of_node_put(ethernet);
- if (!master)
+ if (!master) {
+ rtnl_unlock();
return -EPROBE_DEFER;
+ }
+
+ netdev_hold(master, &dp->conduit_tracker, GFP_KERNEL);
+ put_device(&master->dev);
+ rtnl_unlock();
user_protocol = of_get_property(dn, "dsa-tag-protocol", NULL);
- return dsa_port_parse_cpu(dp, master, user_protocol);
+ err = dsa_port_parse_cpu(dp, master, user_protocol);
+ if (err)
+ netdev_put(master, &dp->conduit_tracker);
+ return err;
}
if (link)
@@ -1668,14 +1679,23 @@ static int dsa_port_parse(struct dsa_port *dp, const char *name,
{
if (!strcmp(name, "cpu")) {
struct net_device *master;
+ int err;
+ rtnl_lock();
master = dsa_dev_to_net_device(dev);
- if (!master)
+ if (!master) {
+ rtnl_unlock();
return -EPROBE_DEFER;
+ }
+ netdev_hold(master, &dp->conduit_tracker, GFP_KERNEL);
dev_put(master);
+ rtnl_unlock();
- return dsa_port_parse_cpu(dp, master, NULL);
+ err = dsa_port_parse_cpu(dp, master, NULL);
+ if (err)
+ netdev_put(master, &dp->conduit_tracker);
+ return err;
}
if (!strcmp(name, "dsa"))
@@ -1743,6 +1763,9 @@ static void dsa_switch_release_ports(struct dsa_switch *ds)
struct dsa_vlan *v, *n;
dsa_switch_for_each_port_safe(dp, next, ds) {
+ if (dsa_port_is_cpu(dp) && dp->master)
+ netdev_put(dp->master, &dp->conduit_tracker);
+
/* These are either entries that upper layers lost track of
* (probably due to bugs), or installed through interfaces
* where one does not necessarily have to remove them, like
@@ -1887,8 +1910,10 @@ void dsa_switch_shutdown(struct dsa_switch *ds)
/* Disconnect from further netdevice notifiers on the master,
* since netdev_uses_dsa() will now return false.
*/
- dsa_switch_for_each_cpu_port(dp, ds)
+ dsa_switch_for_each_cpu_port(dp, ds) {
dp->master->dsa_ptr = NULL;
+ netdev_put(dp->master, &dp->conduit_tracker);
+ }
rtnl_unlock();
out:
diff --git a/net/ethtool/channels.c b/net/ethtool/channels.c
index 403158862011..05f3068ec0fe 100644
--- a/net/ethtool/channels.c
+++ b/net/ethtool/channels.c
@@ -191,9 +191,12 @@ int ethnl_set_channels(struct sk_buff *skb, struct genl_info *info)
/* ensure the new Rx count fits within the configured Rx flow
* indirection table settings
*/
- if (netif_is_rxfh_configured(dev) &&
- !ethtool_get_max_rxfh_channel(dev, &max_rx_in_use) &&
- (channels.combined_count + channels.rx_count) <= max_rx_in_use) {
+ if (netif_is_rxfh_configured(dev)) {
+ ret = ethtool_get_max_rxfh_channel(dev, &max_rx_in_use);
+ if (ret)
+ goto out_ops;
+ }
+ if (channels.combined_count + channels.rx_count <= max_rx_in_use) {
ret = -EINVAL;
GENL_SET_ERR_MSG(info, "requested channel counts are too low for existing indirection table settings");
goto out_ops;
diff --git a/net/ethtool/ioctl.c b/net/ethtool/ioctl.c
index 1f64ff5f3f21..439871dd1ca8 100644
--- a/net/ethtool/ioctl.c
+++ b/net/ethtool/ioctl.c
@@ -1832,10 +1832,13 @@ static noinline_for_stack int ethtool_set_channels(struct net_device *dev,
/* ensure the new Rx count fits within the configured Rx flow
* indirection table settings */
- if (netif_is_rxfh_configured(dev) &&
- !ethtool_get_max_rxfh_channel(dev, &max_rx_in_use) &&
- (channels.combined_count + channels.rx_count) <= max_rx_in_use)
- return -EINVAL;
+ if (netif_is_rxfh_configured(dev)) {
+ ret = ethtool_get_max_rxfh_channel(dev, &max_rx_in_use);
+ if (ret)
+ return ret;
+ }
+ if (channels.combined_count + channels.rx_count <= max_rx_in_use)
+ return -EINVAL;
/* Disabling channels, query zero-copy AF_XDP sockets */
from_channel = channels.combined_count +
diff --git a/net/hsr/hsr_framereg.c b/net/hsr/hsr_framereg.c
index 60eab82ca5a0..35c2fae8d1f7 100644
--- a/net/hsr/hsr_framereg.c
+++ b/net/hsr/hsr_framereg.c
@@ -43,10 +43,8 @@ bool hsr_addr_is_self(struct hsr_priv *hsr, unsigned char *addr)
rcu_read_lock();
sn = rcu_dereference(hsr->self_node);
- if (!sn) {
- WARN_ONCE(1, "HSR: No self node\n");
+ if (!sn)
goto out;
- }
if (ether_addr_equal(addr, sn->macaddress_A) ||
ether_addr_equal(addr, sn->macaddress_B))
diff --git a/net/ieee802154/core.c b/net/ieee802154/core.c
index b3b89a1cd2a4..a5b179540883 100644
--- a/net/ieee802154/core.c
+++ b/net/ieee802154/core.c
@@ -206,8 +206,10 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev,
continue;
wpan_dev->netdev->features &= ~NETIF_F_NETNS_LOCAL;
err = dev_change_net_namespace(wpan_dev->netdev, net, "wpan%d");
- if (err)
+ if (err) {
+ WARN_ON(err && err != -ENOMEM);
break;
+ }
wpan_dev->netdev->features |= NETIF_F_NETNS_LOCAL;
}
@@ -223,7 +225,7 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev,
wpan_dev->netdev->features &= ~NETIF_F_NETNS_LOCAL;
err = dev_change_net_namespace(wpan_dev->netdev, net,
"wpan%d");
- WARN_ON(err);
+ WARN_ON(err && err != -ENOMEM);
wpan_dev->netdev->features |= NETIF_F_NETNS_LOCAL;
}
@@ -233,7 +235,7 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev,
wpan_phy_net_set(&rdev->wpan_phy, net);
err = device_rename(&rdev->wpan_phy.dev, dev_name(&rdev->wpan_phy.dev));
- WARN_ON(err);
+ WARN_ON(err && err != -ENOMEM);
return 0;
}
diff --git a/net/ife/ife.c b/net/ife/ife.c
index 7a75947a31e3..2ddf725d3389 100644
--- a/net/ife/ife.c
+++ b/net/ife/ife.c
@@ -37,7 +37,7 @@ void *ife_encode(struct sk_buff *skb, u16 metalen)
* where ORIGDATA = original ethernet header ...
*/
int hdrm = metalen + IFE_METAHDRLEN;
- int total_push = hdrm + skb->dev->hard_header_len;
+ int total_push = hdrm + ETH_HLEN;
struct ifeheadr *ifehdr;
struct ethhdr *iethh; /* inner ether header */
int skboff = 0;
@@ -50,9 +50,9 @@ void *ife_encode(struct sk_buff *skb, u16 metalen)
iethh = (struct ethhdr *) skb->data;
__skb_push(skb, total_push);
- memcpy(skb->data, iethh, skb->dev->hard_header_len);
+ memcpy(skb->data, iethh, ETH_HLEN);
skb_reset_mac_header(skb);
- skboff += skb->dev->hard_header_len;
+ skboff += ETH_HLEN;
/* total metadata length */
ifehdr = (struct ifeheadr *) (skb->data + skboff);
@@ -69,12 +69,12 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen)
int total_pull;
u16 ifehdrln;
- if (!pskb_may_pull(skb, skb->dev->hard_header_len + IFE_METAHDRLEN))
+ if (!pskb_may_pull(skb, ETH_HLEN + IFE_METAHDRLEN))
return NULL;
- ifehdr = (struct ifeheadr *) (skb->data + skb->dev->hard_header_len);
+ ifehdr = (struct ifeheadr *)(skb->data + ETH_HLEN);
ifehdrln = ntohs(ifehdr->metalen);
- total_pull = skb->dev->hard_header_len + ifehdrln;
+ total_pull = ETH_HLEN + ifehdrln;
if (unlikely(ifehdrln < 2))
return NULL;
@@ -82,7 +82,7 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen)
if (unlikely(!pskb_may_pull(skb, total_pull + ETH_HLEN)))
return NULL;
- ifehdr = (struct ifeheadr *)(skb->data + skb->dev->hard_header_len);
+ ifehdr = (struct ifeheadr *)(skb->data + ETH_HLEN);
skb_set_mac_header(skb, total_pull);
__skb_pull(skb, total_pull);
*metalen = ifehdrln - IFE_METAHDRLEN;
diff --git a/net/ipv4/af_inet.c b/net/ipv4/af_inet.c
index a09b90552cf6..73881ce0ca56 100644
--- a/net/ipv4/af_inet.c
+++ b/net/ipv4/af_inet.c
@@ -1700,6 +1700,7 @@ int inet_ctl_sock_create(struct sock **sk, unsigned short family,
if (rc == 0) {
*sk = sock->sk;
(*sk)->sk_allocation = GFP_ATOMIC;
+ (*sk)->sk_use_task_frag = false;
/*
* Unhash it so that IP input processing does not even see it,
* we do not wish this socket to see incoming packets.
diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c
index 09c2efa3a5b1..55b5c890b15a 100644
--- a/net/ipv4/esp4.c
+++ b/net/ipv4/esp4.c
@@ -95,7 +95,7 @@ static inline struct scatterlist *esp_req_sg(struct crypto_aead *aead,
__alignof__(struct scatterlist));
}
-static void esp_ssg_unref(struct xfrm_state *x, void *tmp)
+static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb)
{
struct crypto_aead *aead = x->data;
int extralen = 0;
@@ -114,7 +114,7 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp)
*/
if (req->src != req->dst)
for (sg = sg_next(req->src); sg; sg = sg_next(sg))
- put_page(sg_page(sg));
+ skb_page_unref(skb, sg_page(sg), false);
}
#ifdef CONFIG_INET_ESPINTCP
@@ -218,7 +218,7 @@ static void esp_output_done(struct crypto_async_request *base, int err)
}
tmp = ESP_SKB_CB(skb)->tmp;
- esp_ssg_unref(x, tmp);
+ esp_ssg_unref(x, tmp, skb);
kfree(tmp);
if (xo && (xo->flags & XFRM_DEV_RESUME)) {
@@ -608,7 +608,7 @@ int esp_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info *
}
if (sg != dsg)
- esp_ssg_unref(x, tmp);
+ esp_ssg_unref(x, tmp, skb);
if (!err && x->encap && x->encap->encap_type == TCP_ENCAP_ESPINTCP)
err = esp_output_tail_tcp(x, skb);
diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c
index 674062511ce0..e81abd42a835 100644
--- a/net/ipv4/fib_semantics.c
+++ b/net/ipv4/fib_semantics.c
@@ -514,6 +514,9 @@ size_t fib_nlmsg_size(struct fib_info *fi)
struct fib_nh_common *nhc = fib_info_nhc(fi, i);
size_t nhsize;
+ if (!nhc)
+ break;
+
nhsize = fib_nexthop_nlmsg_size(nhc, nhs != 1);
if (nhs != 1)
@@ -854,7 +857,7 @@ static void fib_rebalance(struct fib_info *fi)
change_nexthops(fi) {
int upper_bound;
- if (nexthop_nh->fib_nh_flags & RTNH_F_DEAD) {
+ if (!total || nexthop_nh->fib_nh_flags & RTNH_F_DEAD) {
upper_bound = -1;
} else if (ip_ignore_linkdown(nexthop_nh->fib_nh_dev) &&
nexthop_nh->fib_nh_flags & RTNH_F_LINKDOWN) {
diff --git a/net/ipv4/fib_trie.c b/net/ipv4/fib_trie.c
index f40a774686eb..29c84154a0d7 100644
--- a/net/ipv4/fib_trie.c
+++ b/net/ipv4/fib_trie.c
@@ -1084,8 +1084,6 @@ void fib_alias_hw_flags_set(struct net *net, const struct fib_rt_info *fri)
err = fib_dump_info(skb, 0, 0, RTM_NEWROUTE, fri, 0);
if (err < 0) {
- /* -EMSGSIZE implies BUG in fib_nlmsg_size() */
- WARN_ON(err == -EMSGSIZE);
kfree_skb(skb);
goto errout;
}
diff --git a/net/ipv4/igmp.c b/net/ipv4/igmp.c
index 1b8adf5fbf8a..f2c4dc7e24ea 100644
--- a/net/ipv4/igmp.c
+++ b/net/ipv4/igmp.c
@@ -201,13 +201,18 @@ static void ip_sf_list_clear_all(struct ip_sf_list *psf)
static void igmp_stop_timer(struct ip_mc_list *im)
{
+ bool put = false;
+
spin_lock_bh(&im->lock);
if (del_timer(&im->timer))
- refcount_dec(&im->refcnt);
+ put = true;
im->tm_running = 0;
im->reporter = 0;
im->unsolicit_count = 0;
spin_unlock_bh(&im->lock);
+
+ if (put)
+ ip_ma_put(im);
}
/* It must be called with locked im->lock */
@@ -250,6 +255,8 @@ static void igmp_ifc_start_timer(struct in_device *in_dev, int delay)
static void igmp_mod_timer(struct ip_mc_list *im, int max_delay)
{
+ bool put = false;
+
spin_lock_bh(&im->lock);
im->unsolicit_count = 0;
if (del_timer(&im->timer)) {
@@ -259,10 +266,13 @@ static void igmp_mod_timer(struct ip_mc_list *im, int max_delay)
spin_unlock_bh(&im->lock);
return;
}
- refcount_dec(&im->refcnt);
+ put = true;
}
igmp_start_timer(im, max_delay);
spin_unlock_bh(&im->lock);
+
+ if (put)
+ ip_ma_put(im);
}
diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c
index 6184534aa1bf..5e31302b4250 100644
--- a/net/ipv4/inet_connection_sock.c
+++ b/net/ipv4/inet_connection_sock.c
@@ -749,7 +749,7 @@ void inet_csk_init_xmit_timers(struct sock *sk,
timer_setup(&icsk->icsk_retransmit_timer, retransmit_handler, 0);
timer_setup(&icsk->icsk_delack_timer, delack_handler, 0);
- timer_setup(&sk->sk_timer, keepalive_handler, 0);
+ timer_setup(&icsk->icsk_keepalive_timer, keepalive_handler, 0);
icsk->icsk_pending = icsk->icsk_ack.pending = 0;
}
EXPORT_SYMBOL(inet_csk_init_xmit_timers);
@@ -762,7 +762,7 @@ void inet_csk_clear_xmit_timers(struct sock *sk)
sk_stop_timer(sk, &icsk->icsk_retransmit_timer);
sk_stop_timer(sk, &icsk->icsk_delack_timer);
- sk_stop_timer(sk, &sk->sk_timer);
+ sk_stop_timer(sk, &icsk->icsk_keepalive_timer);
}
EXPORT_SYMBOL(inet_csk_clear_xmit_timers);
@@ -777,18 +777,18 @@ void inet_csk_clear_xmit_timers_sync(struct sock *sk)
sk_stop_timer_sync(sk, &icsk->icsk_retransmit_timer);
sk_stop_timer_sync(sk, &icsk->icsk_delack_timer);
- sk_stop_timer_sync(sk, &sk->sk_timer);
+ sk_stop_timer_sync(sk, &icsk->icsk_keepalive_timer);
}
void inet_csk_delete_keepalive_timer(struct sock *sk)
{
- sk_stop_timer(sk, &sk->sk_timer);
+ sk_stop_timer(sk, &inet_csk(sk)->icsk_keepalive_timer);
}
EXPORT_SYMBOL(inet_csk_delete_keepalive_timer);
void inet_csk_reset_keepalive_timer(struct sock *sk, unsigned long len)
{
- sk_reset_timer(sk, &sk->sk_timer, jiffies + len);
+ sk_reset_timer(sk, &inet_csk(sk)->icsk_keepalive_timer, jiffies + len);
}
EXPORT_SYMBOL(inet_csk_reset_keepalive_timer);
diff --git a/net/ipv4/inet_diag.c b/net/ipv4/inet_diag.c
index 2bd5d0ed7a6f..02d8a63f39eb 100644
--- a/net/ipv4/inet_diag.c
+++ b/net/ipv4/inet_diag.c
@@ -310,11 +310,11 @@ int inet_sk_diag_fill(struct sock *sk, struct inet_connection_sock *icsk,
r->idiag_retrans = icsk->icsk_probes_out;
r->idiag_expires =
jiffies_delta_to_msecs(icsk->icsk_timeout - jiffies);
- } else if (timer_pending(&sk->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
r->idiag_timer = 2;
r->idiag_retrans = icsk->icsk_probes_out;
r->idiag_expires =
- jiffies_delta_to_msecs(sk->sk_timer.expires - jiffies);
+ jiffies_delta_to_msecs(icsk->icsk_keepalive_timer.expires - jiffies);
}
if ((ext & (1 << (INET_DIAG_INFO - 1))) && handler->idiag_info_size) {
diff --git a/net/ipv4/ipconfig.c b/net/ipv4/ipconfig.c
index e90bc0aa85c7..024c54a5b0e1 100644
--- a/net/ipv4/ipconfig.c
+++ b/net/ipv4/ipconfig.c
@@ -673,11 +673,40 @@ static const u8 ic_bootp_cookie[4] = { 99, 130, 83, 99 };
#ifdef IPCONFIG_DHCP
+static bool __init
+ic_dhcp_add_option(u8 **options, const u8 *end, u8 type, const void *value,
+ int len)
+{
+ u8 *e = *options;
+
+ /* leave room for the option header and the END marker */
+ if (len > U8_MAX || end - e < len + 3)
+ return false;
+
+ *e++ = type;
+ *e++ = len;
+ memcpy(e, value, len);
+ *options = e + len;
+
+ return true;
+}
+
static void __init
ic_dhcp_init_options(u8 *options, struct ic_device *d)
{
- u8 mt = ((ic_servaddr == NONE)
- ? DHCPDISCOVER : DHCPREQUEST);
+ static const u8 ic_req_params[] = {
+ 1, /* Subnet mask */
+ 3, /* Default gateway */
+ 6, /* DNS server */
+ 12, /* Host name */
+ 15, /* Domain name */
+ 17, /* Boot path */
+ 26, /* MTU */
+ 40, /* NIS domain name */
+ 42, /* NTP servers */
+ };
+ u8 mt = (ic_servaddr == NONE) ? DHCPDISCOVER : DHCPREQUEST;
+ u8 *end = options + sizeof(((struct bootp_pkt *)0)->exten);
u8 *e = options;
int len;
@@ -702,52 +731,25 @@ ic_dhcp_init_options(u8 *options, struct ic_device *d)
e += 4;
}
- /* always? */
- {
- static const u8 ic_req_params[] = {
- 1, /* Subnet mask */
- 3, /* Default gateway */
- 6, /* DNS server */
- 12, /* Host name */
- 15, /* Domain name */
- 17, /* Boot path */
- 26, /* MTU */
- 40, /* NIS domain name */
- 42, /* NTP servers */
- };
-
- *e++ = 55; /* Parameter request list */
- *e++ = sizeof(ic_req_params);
- memcpy(e, ic_req_params, sizeof(ic_req_params));
- e += sizeof(ic_req_params);
-
- if (ic_host_name_set) {
- *e++ = 12; /* host-name */
- len = strlen(utsname()->nodename);
- *e++ = len;
- memcpy(e, utsname()->nodename, len);
- e += len;
- }
- if (*vendor_class_identifier) {
+ *e++ = 55; /* Parameter request list */
+ *e++ = sizeof(ic_req_params);
+ memcpy(e, ic_req_params, sizeof(ic_req_params));
+ e += sizeof(ic_req_params);
+
+ if (ic_host_name_set) {
+ len = strlen(utsname()->nodename);
+ ic_dhcp_add_option(&e, end, 12, utsname()->nodename, len);
+ }
+ if (*vendor_class_identifier) {
+ len = strlen(vendor_class_identifier);
+ if (ic_dhcp_add_option(&e, end, 60, vendor_class_identifier, len))
pr_info("DHCP: sending class identifier \"%s\"\n",
vendor_class_identifier);
- *e++ = 60; /* Class-identifier */
- len = strlen(vendor_class_identifier);
- *e++ = len;
- memcpy(e, vendor_class_identifier, len);
- e += len;
- }
- len = strlen(dhcp_client_identifier + 1);
- /* the minimum length of identifier is 2, include 1 byte type,
- * and can not be larger than the length of options
- */
- if (len >= 1 && len < 312 - (e - options) - 1) {
- *e++ = 61;
- *e++ = len + 1;
- memcpy(e, dhcp_client_identifier, len + 1);
- e += len + 1;
- }
}
+ len = strlen(dhcp_client_identifier + 1);
+ /* the minimum length of identifier is 2, include 1 byte type */
+ if (len >= 1)
+ ic_dhcp_add_option(&e, end, 61, dhcp_client_identifier, len + 1);
*e++ = 255; /* End of the list */
}
diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c
index 8d4f556fced2..dd4e8897ee9c 100644
--- a/net/ipv4/ipmr.c
+++ b/net/ipv4/ipmr.c
@@ -2125,6 +2125,9 @@ int ip_mr_input(struct sk_buff *skb)
if (IPCB(skb)->flags & IPSKB_FORWARDED)
goto dont_forward;
+ if (!local)
+ skb_orphan(skb);
+
mrt = ipmr_rt_fib_lookup(net, skb);
if (IS_ERR(mrt)) {
kfree_skb(skb);
@@ -3145,7 +3148,8 @@ int __init ip_mr_init(void)
mrt_cachep = kmem_cache_create("ip_mrt_cache",
sizeof(struct mfc_cache),
- 0, SLAB_HWCACHE_ALIGN | SLAB_PANIC,
+ 0, SLAB_HWCACHE_ALIGN | SLAB_PANIC |
+ SLAB_ACCOUNT,
NULL);
err = register_pernet_subsys(&ipmr_net_ops);
diff --git a/net/ipv4/ipmr_base.c b/net/ipv4/ipmr_base.c
index de98ce66d38f..ae5a685daeea 100644
--- a/net/ipv4/ipmr_base.c
+++ b/net/ipv4/ipmr_base.c
@@ -38,7 +38,7 @@ mr_table_alloc(struct net *net, u32 id,
struct mr_table *mrt;
int err;
- mrt = kzalloc(sizeof(*mrt), GFP_KERNEL);
+ mrt = kzalloc(sizeof(*mrt), GFP_KERNEL_ACCOUNT);
if (!mrt)
return ERR_PTR(-ENOMEM);
mrt->id = id;
diff --git a/net/ipv4/netfilter/iptable_nat.c b/net/ipv4/netfilter/iptable_nat.c
index 12ca666d6e2c..a8fe357bc6ec 100644
--- a/net/ipv4/netfilter/iptable_nat.c
+++ b/net/ipv4/netfilter/iptable_nat.c
@@ -79,7 +79,7 @@ static int ipt_nat_register_lookups(struct net *net)
while (i)
nf_nat_ipv4_unregister_fn(net, &ops[--i]);
- kfree(ops);
+ kfree_rcu(ops, rcu);
return ret;
}
}
@@ -100,7 +100,7 @@ static void ipt_nat_unregister_lookups(struct net *net)
for (i = 0; i < ARRAY_SIZE(nf_nat_ipv4_ops); i++)
nf_nat_ipv4_unregister_fn(net, &ops[i]);
- kfree(ops);
+ kfree_rcu(ops, rcu);
}
static int iptable_nat_table_init(struct net *net)
diff --git a/net/ipv4/netfilter/nft_dup_ipv4.c b/net/ipv4/netfilter/nft_dup_ipv4.c
index 000539b94e1e..ef5dd88107dd 100644
--- a/net/ipv4/netfilter/nft_dup_ipv4.c
+++ b/net/ipv4/netfilter/nft_dup_ipv4.c
@@ -52,7 +52,8 @@ static int nft_dup_ipv4_init(const struct nft_ctx *ctx,
return err;
}
-static int nft_dup_ipv4_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_dup_ipv4_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_dup_ipv4 *priv = nft_expr_priv(expr);
diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c
index cd93dd74608a..790135fd80e7 100644
--- a/net/ipv4/nexthop.c
+++ b/net/ipv4/nexthop.c
@@ -92,6 +92,7 @@ __nh_notifier_single_info_init(struct nh_notifier_single_info *nh_info,
else if (nh_info->gw_family == AF_INET6)
nh_info->ipv6 = nhi->fib_nhc.nhc_gw.ipv6;
+ nh_info->id = nhi->nh_parent->id;
nh_info->is_reject = nhi->reject_nh;
nh_info->is_fdb = nhi->fdb_nh;
nh_info->has_encap = !!nhi->fib_nhc.nhc_lwtstate;
@@ -137,7 +138,6 @@ static int nh_notifier_mpath_info_init(struct nh_notifier_info *info,
struct nh_info *nhi;
nhi = rtnl_dereference(nhge->nh->nh_info);
- info->nh_grp->nh_entries[i].id = nhge->nh->id;
info->nh_grp->nh_entries[i].weight = nhge->weight;
__nh_notifier_single_info_init(&info->nh_grp->nh_entries[i].nh,
nhi);
@@ -660,6 +660,7 @@ static int nla_put_nh_group(struct sk_buff *skb, struct nh_group *nhg)
size_t len = nhg->num_nh * sizeof(*p);
struct nlattr *nla;
u16 group_type = 0;
+ u16 weight;
int i;
if (nhg->hash_threshold)
@@ -676,9 +677,12 @@ static int nla_put_nh_group(struct sk_buff *skb, struct nh_group *nhg)
p = nla_data(nla);
for (i = 0; i < nhg->num_nh; ++i) {
+ weight = nhg->nh_entries[i].weight - 1;
+
*p++ = (struct nexthop_grp) {
.id = nhg->nh_entries[i].nh->id,
- .weight = nhg->nh_entries[i].weight - 1,
+ .weight = weight,
+ .weight_high = weight >> 8,
};
}
@@ -786,7 +790,8 @@ static size_t nh_nlmsg_size_grp(struct nexthop *nh)
struct nh_group *nhg = rtnl_dereference(nh->nh_grp);
size_t sz = sizeof(struct nexthop_grp) * nhg->num_nh;
size_t tot = nla_total_size(sz) +
- nla_total_size(2); /* NHA_GROUP_TYPE */
+ nla_total_size(2) + /* NHA_GROUP_TYPE */
+ nla_total_size(0); /* NHA_FDB */
if (nhg->resilient)
tot += nh_nlmsg_size_grp_res(nhg);
@@ -1062,11 +1067,14 @@ static int nh_check_attr_group(struct net *net,
nhg = nla_data(tb[NHA_GROUP]);
for (i = 0; i < len; ++i) {
- if (nhg[i].resvd1 || nhg[i].resvd2) {
- NL_SET_ERR_MSG(extack, "Reserved fields in nexthop_grp must be 0");
+ if (nhg[i].resvd2) {
+ NL_SET_ERR_MSG(extack, "Reserved field in nexthop_grp must be 0");
return -EINVAL;
}
- if (nhg[i].weight > 254) {
+ if (nexthop_grp_weight(&nhg[i]) == 0) {
+ /* 0xffff got passed in, representing weight of 0x10000,
+ * which is too heavy.
+ */
NL_SET_ERR_MSG(extack, "Invalid value for weight");
return -EINVAL;
}
@@ -1153,41 +1161,64 @@ static bool ipv4_good_nh(const struct fib_nh *nh)
return !!(state & NUD_VALID);
}
-static struct nexthop *nexthop_select_path_hthr(struct nh_group *nhg, int hash)
+static bool nexthop_is_good_nh(const struct nexthop *nh)
+{
+ struct nh_info *nhi = rcu_dereference(nh->nh_info);
+
+ switch (nhi->family) {
+ case AF_INET:
+ return ipv4_good_nh(&nhi->fib_nh);
+ case AF_INET6:
+ return ipv6_good_nh(&nhi->fib6_nh);
+ }
+
+ return false;
+}
+
+static struct nexthop *nexthop_select_path_fdb(struct nh_group *nhg, int hash)
{
- struct nexthop *rc = NULL;
int i;
- for (i = 0; i < nhg->num_nh; ++i) {
+ for (i = 0; i < nhg->num_nh; i++) {
struct nh_grp_entry *nhge = &nhg->nh_entries[i];
- struct nh_info *nhi;
if (hash > atomic_read(&nhge->hthr.upper_bound))
continue;
- nhi = rcu_dereference(nhge->nh->nh_info);
- if (nhi->fdb_nh)
- return nhge->nh;
+ return nhge->nh;
+ }
+
+ WARN_ON_ONCE(1);
+ return NULL;
+}
+
+static struct nexthop *nexthop_select_path_hthr(struct nh_group *nhg, int hash)
+{
+ struct nexthop *rc = NULL;
+ int i;
+
+ if (nhg->fdb_nh)
+ return nexthop_select_path_fdb(nhg, hash);
+
+ for (i = 0; i < nhg->num_nh; ++i) {
+ struct nh_grp_entry *nhge = &nhg->nh_entries[i];
/* nexthops always check if it is good and does
* not rely on a sysctl for this behavior
*/
- switch (nhi->family) {
- case AF_INET:
- if (ipv4_good_nh(&nhi->fib_nh))
- return nhge->nh;
- break;
- case AF_INET6:
- if (ipv6_good_nh(&nhi->fib6_nh))
- return nhge->nh;
- break;
- }
+ if (!nexthop_is_good_nh(nhge->nh))
+ continue;
if (!rc)
rc = nhge->nh;
+
+ if (hash > atomic_read(&nhge->hthr.upper_bound))
+ continue;
+
+ return nhge->nh;
}
- return rc;
+ return rc ? : nhg->nh_entries[0].nh;
}
static struct nexthop *nexthop_select_path_res(struct nh_group *nhg, int hash)
@@ -1632,9 +1663,9 @@ static void nh_res_table_cancel_upkeep(struct nh_res_table *res_table)
static void nh_res_group_rebalance(struct nh_group *nhg,
struct nh_res_table *res_table)
{
- int prev_upper_bound = 0;
- int total = 0;
- int w = 0;
+ u16 prev_upper_bound = 0;
+ u32 total = 0;
+ u32 w = 0;
int i;
INIT_LIST_HEAD(&res_table->uw_nh_entries);
@@ -1644,11 +1675,12 @@ static void nh_res_group_rebalance(struct nh_group *nhg,
for (i = 0; i < nhg->num_nh; ++i) {
struct nh_grp_entry *nhge = &nhg->nh_entries[i];
- int upper_bound;
+ u16 upper_bound;
+ u64 btw;
w += nhge->weight;
- upper_bound = DIV_ROUND_CLOSEST(res_table->num_nh_buckets * w,
- total);
+ btw = ((u64)res_table->num_nh_buckets) * w;
+ upper_bound = DIV_ROUND_CLOSEST_ULL(btw, total);
nhge->res.wants_buckets = upper_bound - prev_upper_bound;
prev_upper_bound = upper_bound;
@@ -1714,8 +1746,8 @@ static void replace_nexthop_grp_res(struct nh_group *oldg,
static void nh_hthr_group_rebalance(struct nh_group *nhg)
{
- int total = 0;
- int w = 0;
+ u32 total = 0;
+ u32 w = 0;
int i;
for (i = 0; i < nhg->num_nh; ++i)
@@ -1723,7 +1755,7 @@ static void nh_hthr_group_rebalance(struct nh_group *nhg)
for (i = 0; i < nhg->num_nh; ++i) {
struct nh_grp_entry *nhge = &nhg->nh_entries[i];
- int upper_bound;
+ u32 upper_bound;
w += nhge->weight;
upper_bound = DIV_ROUND_CLOSEST_ULL((u64)w << 31, total) - 1;
@@ -2469,7 +2501,8 @@ static struct nexthop *nexthop_create_group(struct net *net,
nhg->has_v4 = true;
nhg->nh_entries[i].nh = nhe;
- nhg->nh_entries[i].weight = entry[i].weight + 1;
+ nhg->nh_entries[i].weight = nexthop_grp_weight(&entry[i]);
+
list_add(&nhg->nh_entries[i].nh_list, &nhe->grp_list);
nhg->nh_entries[i].nh_parent = nh;
}
@@ -3038,15 +3071,15 @@ static int rtm_get_nexthop(struct sk_buff *in_skb, struct nlmsghdr *nlh,
if (err)
return err;
- err = -ENOBUFS;
- skb = alloc_skb(NLMSG_GOODSIZE, GFP_KERNEL);
- if (!skb)
- goto out;
-
err = -ENOENT;
nh = nexthop_find_by_id(net, id);
if (!nh)
- goto errout_free;
+ goto out;
+
+ err = -ENOBUFS;
+ skb = nlmsg_new(nh_nlmsg_size(nh), GFP_KERNEL);
+ if (!skb)
+ goto out;
err = nh_fill_node(skb, nh, RTM_NEWNEXTHOP, NETLINK_CB(in_skb).portid,
nlh->nlmsg_seq, 0);
diff --git a/net/ipv4/route.c b/net/ipv4/route.c
index 5528a5543df5..bd8ea5b8fa58 100644
--- a/net/ipv4/route.c
+++ b/net/ipv4/route.c
@@ -1089,6 +1089,8 @@ static void __ip_rt_update_pmtu(struct rtable *rt, struct flowi4 *fl4, u32 mtu)
for (nhsel = 0; nhsel < fib_info_num_path(res.fi); nhsel++) {
nhc = fib_info_nhc(res.fi, nhsel);
+ if (!nhc)
+ break;
update_or_create_fnhe(nhc, fl4->daddr, 0, mtu, lock,
jiffies + net->ipv4.ip_rt_mtu_expires);
}
@@ -1573,9 +1575,9 @@ void rt_add_uncached_list(struct rtable *rt)
void rt_del_uncached_list(struct rtable *rt)
{
- if (!list_empty(&rt->rt_uncached)) {
- struct uncached_list *ul = rt->rt_uncached_list;
+ struct uncached_list *ul = rt->rt_uncached_list;
+ if (ul) {
spin_lock_bh(&ul->lock);
list_del_init(&rt->rt_uncached);
spin_unlock_bh(&ul->lock);
@@ -1701,6 +1703,7 @@ struct rtable *rt_dst_alloc(struct net_device *dev,
rt->rt_gw_family = 0;
rt->rt_gw4 = 0;
INIT_LIST_HEAD(&rt->rt_uncached);
+ rt->rt_uncached_list = NULL;
rt->dst.output = ip_output;
if (flags & RTCF_LOCAL)
@@ -1732,6 +1735,7 @@ struct rtable *rt_dst_clone(struct net_device *dev, struct rtable *rt)
else if (rt->rt_gw_family == AF_INET6)
new_rt->rt_gw6 = rt->rt_gw6;
INIT_LIST_HEAD(&new_rt->rt_uncached);
+ new_rt->rt_uncached_list = NULL;
new_rt->dst.input = rt->dst.input;
new_rt->dst.output = rt->dst.output;
@@ -2923,6 +2927,7 @@ struct dst_entry *ipv4_blackhole_route(struct net *net, struct dst_entry *dst_or
rt->rt_gw6 = ort->rt_gw6;
INIT_LIST_HEAD(&rt->rt_uncached);
+ rt->rt_uncached_list = NULL;
}
dst_release(dst_orig);
diff --git a/net/ipv4/syncookies.c b/net/ipv4/syncookies.c
index d7202f86566a..4576e7713925 100644
--- a/net/ipv4/syncookies.c
+++ b/net/ipv4/syncookies.c
@@ -353,9 +353,13 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb)
tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
- tsoff = secure_tcp_ts_off(sock_net(sk),
- ip_hdr(skb)->daddr,
- ip_hdr(skb)->saddr);
+ union tcp_seq_and_ts_off st;
+
+ st = secure_tcp_seq_and_ts_off(sock_net(sk),
+ ip_hdr(skb)->daddr,
+ ip_hdr(skb)->saddr,
+ th->dest, th->source);
+ tsoff = st.ts_off;
tcp_opt.rcv_tsecr -= tsoff;
}
diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c
index 82d6c01c7885..2c9b377c4b4d 100644
--- a/net/ipv4/tcp.c
+++ b/net/ipv4/tcp.c
@@ -4209,9 +4209,11 @@ int do_tcp_getsockopt(struct sock *sk, int level,
if (copy_from_sockptr(&len, optlen, sizeof(int)))
return -EFAULT;
- ca_ops = icsk->icsk_ca_ops;
+ rcu_read_lock();
+ ca_ops = READ_ONCE(icsk->icsk_ca_ops);
if (ca_ops && ca_ops->get_info)
sz = ca_ops->get_info(sk, ~0U, &attr, &info);
+ rcu_read_unlock();
len = min_t(unsigned int, len, sz);
if (copy_to_sockptr(optlen, &len, sizeof(int)))
@@ -4224,16 +4226,24 @@ int do_tcp_getsockopt(struct sock *sk, int level,
val = !inet_csk_in_pingpong_mode(sk);
break;
- case TCP_CONGESTION:
+ case TCP_CONGESTION: {
+ char ca_name[TCP_CA_NAME_MAX] = {};
+
if (copy_from_sockptr(&len, optlen, sizeof(int)))
return -EFAULT;
len = min_t(unsigned int, len, TCP_CA_NAME_MAX);
if (copy_to_sockptr(optlen, &len, sizeof(int)))
return -EFAULT;
- if (copy_to_sockptr(optval, icsk->icsk_ca_ops->name, len))
+
+ rcu_read_lock();
+ memcpy(ca_name, READ_ONCE(icsk->icsk_ca_ops)->name,
+ sizeof(ca_name));
+ rcu_read_unlock();
+
+ if (copy_to_sockptr(optval, ca_name, len))
return -EFAULT;
return 0;
-
+ }
case TCP_ULP:
if (copy_from_sockptr(&len, optlen, sizeof(int)))
return -EFAULT;
diff --git a/net/ipv4/tcp_cong.c b/net/ipv4/tcp_cong.c
index d3cae40749e8..ab12c7cdbb4b 100644
--- a/net/ipv4/tcp_cong.c
+++ b/net/ipv4/tcp_cong.c
@@ -174,7 +174,7 @@ void tcp_assign_congestion_control(struct sock *sk)
ca = rcu_dereference(net->ipv4.tcp_congestion_control);
if (unlikely(!bpf_try_module_get(ca, ca->owner)))
ca = &tcp_reno;
- icsk->icsk_ca_ops = ca;
+ WRITE_ONCE(icsk->icsk_ca_ops, ca);
rcu_read_unlock();
memset(icsk->icsk_ca_priv, 0, sizeof(icsk->icsk_ca_priv));
@@ -204,7 +204,7 @@ static void tcp_reinit_congestion_control(struct sock *sk,
struct inet_connection_sock *icsk = inet_csk(sk);
tcp_cleanup_congestion_control(sk);
- icsk->icsk_ca_ops = ca;
+ WRITE_ONCE(icsk->icsk_ca_ops, ca);
icsk->icsk_ca_setsockopt = 1;
memset(icsk->icsk_ca_priv, 0, sizeof(icsk->icsk_ca_priv));
diff --git a/net/ipv4/tcp_dctcp.c b/net/ipv4/tcp_dctcp.c
index 863aab186055..ae4d9bc049ec 100644
--- a/net/ipv4/tcp_dctcp.c
+++ b/net/ipv4/tcp_dctcp.c
@@ -108,7 +108,7 @@ static void dctcp_init(struct sock *sk)
/* No ECN support? Fall back to Reno. Also need to clear
* ECT from sk since it is set during 3WHS for DCTCP.
*/
- inet_csk(sk)->icsk_ca_ops = &dctcp_reno;
+ WRITE_ONCE(inet_csk(sk)->icsk_ca_ops, &dctcp_reno);
INET_ECN_dontxmit(sk);
}
@@ -203,7 +203,7 @@ static size_t dctcp_get_info(struct sock *sk, u32 ext, int *attr,
if (ext & (1 << (INET_DIAG_DCTCPINFO - 1)) ||
ext & (1 << (INET_DIAG_VEGASINFO - 1))) {
memset(&info->dctcp, 0, sizeof(info->dctcp));
- if (inet_csk(sk)->icsk_ca_ops != &dctcp_reno) {
+ if (READ_ONCE(inet_csk(sk)->icsk_ca_ops) != &dctcp_reno) {
info->dctcp.dctcp_enabled = 1;
info->dctcp.dctcp_ce_state = (u16) ca->ce_state;
info->dctcp.dctcp_alpha = ca->dctcp_alpha;
diff --git a/net/ipv4/tcp_fastopen.c b/net/ipv4/tcp_fastopen.c
index 156685b4ae68..bb40b42f13d7 100644
--- a/net/ipv4/tcp_fastopen.c
+++ b/net/ipv4/tcp_fastopen.c
@@ -575,11 +575,12 @@ void tcp_fastopen_active_disable_ofo_check(struct sock *sk)
}
} else if (tp->syn_fastopen_ch &&
atomic_read(&sock_net(sk)->ipv4.tfo_active_disable_times)) {
- dst = sk_dst_get(sk);
- dev = dst ? dst_dev(dst) : NULL;
+ rcu_read_lock();
+ dst = __sk_dst_get(sk);
+ dev = dst ? dst_dev_rcu(dst) : NULL;
if (!(dev && (dev->flags & IFF_LOOPBACK)))
atomic_set(&sock_net(sk)->ipv4.tfo_active_disable_times, 0);
- dst_release(dst);
+ rcu_read_unlock();
}
}
diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c
index 4ab6222ff180..9074af9da52d 100644
--- a/net/ipv4/tcp_input.c
+++ b/net/ipv4/tcp_input.c
@@ -3707,24 +3707,17 @@ bool tcp_oow_rate_limited(struct net *net, const struct sk_buff *skb,
return __tcp_oow_rate_limited(net, mib_idx, last_oow_ack_time);
}
-/* RFC 5961 7 [ACK Throttling] */
-static void tcp_send_challenge_ack(struct sock *sk)
+/* Consume one slot from the per-netns RFC 5961 challenge ACK quota.
+ * Returns true if a challenge ACK may be sent.
+ */
+static bool tcp_challenge_ack_allowed(struct net *net)
{
- struct tcp_sock *tp = tcp_sk(sk);
- struct net *net = sock_net(sk);
u32 count, now, ack_limit;
- /* First check our per-socket dupack rate limit. */
- if (__tcp_oow_rate_limited(net,
- LINUX_MIB_TCPACKSKIPPEDCHALLENGE,
- &tp->last_oow_ack_time))
- return;
-
ack_limit = READ_ONCE(net->ipv4.sysctl_tcp_challenge_ack_limit);
if (ack_limit == INT_MAX)
- goto send_ack;
+ return true;
- /* Then check host-wide RFC 5961 rate limit. */
now = jiffies / HZ;
if (now != READ_ONCE(net->ipv4.tcp_challenge_timestamp)) {
u32 half = (ack_limit + 1) >> 1;
@@ -3735,12 +3728,49 @@ static void tcp_send_challenge_ack(struct sock *sk)
count = READ_ONCE(net->ipv4.tcp_challenge_count);
if (count > 0) {
WRITE_ONCE(net->ipv4.tcp_challenge_count, count - 1);
-send_ack:
+ return true;
+ }
+ return false;
+}
+
+/* RFC 5961 7 [ACK Throttling] */
+static void tcp_send_challenge_ack(struct sock *sk)
+{
+ struct tcp_sock *tp = tcp_sk(sk);
+ struct net *net = sock_net(sk);
+
+ /* First check our per-socket dupack rate limit. */
+ if (__tcp_oow_rate_limited(net,
+ LINUX_MIB_TCPACKSKIPPEDCHALLENGE,
+ &tp->last_oow_ack_time))
+ return;
+
+ /* Then check the per-netns RFC 5961 rate limit. */
+ if (tcp_challenge_ack_allowed(net)) {
NET_INC_STATS(net, LINUX_MIB_TCPCHALLENGEACK);
tcp_send_ack(sk);
}
}
+/* Send a challenge ACK from a SYN-RECEIVED request socket. Uses
+ * __tcp_oow_rate_limited() directly so that an RST carrying payload
+ * cannot bypass the per-request rate limit.
+ */
+void tcp_reqsk_send_challenge_ack(struct sock *sk, struct sk_buff *skb,
+ struct request_sock *req)
+{
+ struct net *net = sock_net(sk);
+
+ if (__tcp_oow_rate_limited(net, LINUX_MIB_TCPACKSKIPPEDCHALLENGE,
+ &tcp_rsk(req)->last_oow_ack_time))
+ return;
+
+ if (tcp_challenge_ack_allowed(net)) {
+ NET_INC_STATS(net, LINUX_MIB_TCPCHALLENGEACK);
+ req->rsk_ops->send_ack(sk, skb, req);
+ }
+}
+
static void tcp_store_ts_recent(struct tcp_sock *tp)
{
tp->rx_opt.ts_recent = tp->rx_opt.rcv_tsval;
@@ -4024,6 +4054,12 @@ static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
return 1;
old_ack:
+ /* An old ACK can carry new data. Update TS.Recent before SACK
+ * processing can trigger a retransmission.
+ */
+ if (flag & FLAG_UPDATE_TS_RECENT)
+ tcp_replace_ts_recent(tp, TCP_SKB_CB(skb)->seq);
+
/* If data was SACKed, tag it and see if we should send more data.
* If data was DSACKed, see if we can undo a cwnd reduction.
*/
@@ -7027,6 +7063,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
struct tcp_sock *tp = tcp_sk(sk);
struct net *net = sock_net(sk);
struct sock *fastopen_sk = NULL;
+ union tcp_seq_and_ts_off st;
struct request_sock *req;
bool want_cookie = false;
struct dst_entry *dst;
@@ -7084,8 +7121,11 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
if (!dst)
goto drop_and_free;
+ if (tmp_opt.tstamp_ok || (!want_cookie && !isn))
+ st = af_ops->init_seq_and_ts_off(net, skb);
+
if (tmp_opt.tstamp_ok)
- tcp_rsk(req)->ts_off = af_ops->init_ts_off(net, skb);
+ tcp_rsk(req)->ts_off = st.ts_off;
if (!want_cookie && !isn) {
int max_syn_backlog = READ_ONCE(net->ipv4.sysctl_max_syn_backlog);
@@ -7107,7 +7147,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops,
goto drop_and_release;
}
- isn = af_ops->init_seq(skb);
+ isn = st.seq;
}
tcp_ecn_create_request(req, skb, sk, dst);
diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c
index 38ce407af8fd..0e6d03af09e5 100644
--- a/net/ipv4/tcp_ipv4.c
+++ b/net/ipv4/tcp_ipv4.c
@@ -98,17 +98,14 @@ static DEFINE_PER_CPU(struct sock *, ipv4_tcp_sk);
static DEFINE_MUTEX(tcp_exit_batch_mutex);
-static u32 tcp_v4_init_seq(const struct sk_buff *skb)
+static union tcp_seq_and_ts_off
+tcp_v4_init_seq_and_ts_off(const struct net *net, const struct sk_buff *skb)
{
- return secure_tcp_seq(ip_hdr(skb)->daddr,
- ip_hdr(skb)->saddr,
- tcp_hdr(skb)->dest,
- tcp_hdr(skb)->source);
-}
-
-static u32 tcp_v4_init_ts_off(const struct net *net, const struct sk_buff *skb)
-{
- return secure_tcp_ts_off(net, ip_hdr(skb)->daddr, ip_hdr(skb)->saddr);
+ return secure_tcp_seq_and_ts_off(net,
+ ip_hdr(skb)->daddr,
+ ip_hdr(skb)->saddr,
+ tcp_hdr(skb)->dest,
+ tcp_hdr(skb)->source);
}
int tcp_twsk_unique(struct sock *sk, struct sock *sktw, void *twp)
@@ -315,15 +312,16 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len)
rt = NULL;
if (likely(!tp->repair)) {
+ union tcp_seq_and_ts_off st;
+
+ st = secure_tcp_seq_and_ts_off(net,
+ inet->inet_saddr,
+ inet->inet_daddr,
+ inet->inet_sport,
+ usin->sin_port);
if (!tp->write_seq)
- WRITE_ONCE(tp->write_seq,
- secure_tcp_seq(inet->inet_saddr,
- inet->inet_daddr,
- inet->inet_sport,
- usin->sin_port));
- WRITE_ONCE(tp->tsoffset,
- secure_tcp_ts_off(net, inet->inet_saddr,
- inet->inet_daddr));
+ WRITE_ONCE(tp->write_seq, st.seq);
+ WRITE_ONCE(tp->tsoffset, st.ts_off);
}
atomic_set(&inet->inet_id, get_random_u16());
@@ -1481,8 +1479,7 @@ const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops = {
.cookie_init_seq = cookie_v4_init_sequence,
#endif
.route_req = tcp_v4_route_req,
- .init_seq = tcp_v4_init_seq,
- .init_ts_off = tcp_v4_init_ts_off,
+ .init_seq_and_ts_off = tcp_v4_init_seq_and_ts_off,
.send_synack = tcp_v4_send_synack,
};
@@ -1575,7 +1572,7 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb,
tcp_ca_openreq_child(newsk, dst);
tcp_sync_mss(newsk, dst_mtu(dst));
- newtp->advmss = tcp_mss_clamp(tcp_sk(sk), dst_metric_advmss(dst));
+ newtp->advmss = tcp_mss_clamp(tcp_sk(sk), tcp_dst_advmss(dst));
tcp_initialize_rcv_mss(newsk);
@@ -2647,9 +2644,9 @@ static void get_tcp4_sock(struct sock *sk, struct seq_file *f, int i)
} else if (icsk->icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = icsk->icsk_timeout;
- } else if (timer_pending(&sk->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
- timer_expires = sk->sk_timer.expires;
+ timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = jiffies;
diff --git a/net/ipv4/tcp_metrics.c b/net/ipv4/tcp_metrics.c
index 8bf1118e04c6..d3559a9cb233 100644
--- a/net/ipv4/tcp_metrics.c
+++ b/net/ipv4/tcp_metrics.c
@@ -170,7 +170,7 @@ static struct tcp_metrics_block *tcpm_new(struct dst_entry *dst,
struct net *net;
spin_lock_bh(&tcp_metrics_lock);
- net = dev_net(dst_dev(dst));
+ net = dst_dev_net_rcu(dst);
/* While waiting for the spin-lock the cache might have been populated
* with this entry and so we have to check again.
@@ -273,7 +273,7 @@ static struct tcp_metrics_block *__tcp_get_metrics_req(struct request_sock *req,
return NULL;
}
- net = dev_net(dst_dev(dst));
+ net = dst_dev_net_rcu(dst);
hash ^= net_hash_mix(net);
hash = hash_32(hash, tcp_metrics_hash_log);
@@ -318,7 +318,7 @@ static struct tcp_metrics_block *tcp_get_metrics(struct sock *sk,
else
return NULL;
- net = dev_net(dst_dev(dst));
+ net = dst_dev_net_rcu(dst);
hash ^= net_hash_mix(net);
hash = hash_32(hash, tcp_metrics_hash_log);
diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c
index 67735100f9ac..0b1fa4cccaae 100644
--- a/net/ipv4/tcp_minisocks.c
+++ b/net/ipv4/tcp_minisocks.c
@@ -378,7 +378,7 @@ void tcp_openreq_init_rwin(struct request_sock *req,
u32 rcv_wnd;
int mss;
- mss = tcp_mss_clamp(tp, dst_metric_advmss(dst));
+ mss = tcp_mss_clamp(tp, tcp_dst_advmss(dst));
window_clamp = READ_ONCE(tp->window_clamp);
/* Set this up on the first call only */
req->rsk_window_clamp = window_clamp ? : dst_metric(dst, RTAX_WINDOW);
@@ -425,7 +425,7 @@ void tcp_ca_openreq_child(struct sock *sk, const struct dst_entry *dst)
ca = tcp_ca_find_key(ca_key);
if (likely(ca && bpf_try_module_get(ca, ca->owner))) {
icsk->icsk_ca_dst_locked = tcp_ca_dst_locked(dst);
- icsk->icsk_ca_ops = ca;
+ WRITE_ONCE(icsk->icsk_ca_ops, ca);
ca_got_dst = true;
}
rcu_read_unlock();
@@ -709,7 +709,7 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
* elsewhere and is checked directly against the child socket rather
* than req because user data may have been sent out.
*/
- if ((flg & TCP_FLAG_ACK) && !fastopen &&
+ if ((flg & TCP_FLAG_ACK) && !(flg & TCP_FLAG_RST) && !fastopen &&
(TCP_SKB_CB(skb)->ack_seq !=
tcp_rsk(req)->snt_isn + 1))
return sk;
@@ -742,6 +742,16 @@ struct sock *tcp_check_req(struct sock *sk, struct sk_buff *skb,
flg &= ~TCP_FLAG_SYN;
}
+ /* RFC 5961 section 3.2, as clarified by RFC 9293 section
+ * 3.10.7.4, requires a challenge ACK for a non-exact
+ * in-window RST in SYN-RECEIVED.
+ */
+ if ((flg & TCP_FLAG_RST) &&
+ TCP_SKB_CB(skb)->seq != tcp_rsk(req)->rcv_nxt) {
+ tcp_reqsk_send_challenge_ack(sk, skb, req);
+ return NULL;
+ }
+
/* RFC793: "second check the RST bit" and
* "fourth, check the SYN bit"
*/
diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c
index 39f25b886678..5117de7afc24 100644
--- a/net/ipv4/tcp_output.c
+++ b/net/ipv4/tcp_output.c
@@ -125,7 +125,7 @@ static __u16 tcp_advertise_mss(struct sock *sk)
int mss = tp->advmss;
if (dst) {
- unsigned int metric = dst_metric_advmss(dst);
+ unsigned int metric = tcp_dst_advmss(dst);
if (metric < mss) {
mss = metric;
@@ -3639,7 +3639,7 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst,
}
skb_dst_set(skb, dst);
- mss = tcp_mss_clamp(tp, dst_metric_advmss(dst));
+ mss = tcp_mss_clamp(tp, tcp_dst_advmss(dst));
memset(&opts, 0, sizeof(opts));
now = tcp_clock_ns();
@@ -3720,7 +3720,7 @@ static void tcp_ca_dst_init(struct sock *sk, const struct dst_entry *dst)
if (likely(ca && bpf_try_module_get(ca, ca->owner))) {
bpf_module_put(icsk->icsk_ca_ops, icsk->icsk_ca_ops->owner);
icsk->icsk_ca_dst_locked = tcp_ca_dst_locked(dst);
- icsk->icsk_ca_ops = ca;
+ WRITE_ONCE(icsk->icsk_ca_ops, ca);
}
rcu_read_unlock();
}
@@ -3756,7 +3756,7 @@ static void tcp_connect_init(struct sock *sk)
if (!tp->window_clamp)
tp->window_clamp = dst_metric(dst, RTAX_WINDOW);
- tp->advmss = tcp_mss_clamp(tp, dst_metric_advmss(dst));
+ tp->advmss = tcp_mss_clamp(tp, tcp_dst_advmss(dst));
tcp_initialize_rcv_mss(sk);
diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c
index 3662b49ce71a..d086026018d6 100644
--- a/net/ipv4/tcp_timer.c
+++ b/net/ipv4/tcp_timer.c
@@ -706,8 +706,9 @@ EXPORT_SYMBOL_GPL(tcp_set_keepalive);
static void tcp_keepalive_timer (struct timer_list *t)
{
- struct sock *sk = from_timer(sk, t, sk_timer);
- struct inet_connection_sock *icsk = inet_csk(sk);
+ struct inet_connection_sock *icsk =
+ from_timer(icsk, t, icsk_keepalive_timer);
+ struct sock *sk = &icsk->icsk_inet.sk;
struct tcp_sock *tp = tcp_sk(sk);
u32 elapsed;
diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c
index 6f745efa0db4..3c41aed3788c 100644
--- a/net/ipv4/udp.c
+++ b/net/ipv4/udp.c
@@ -414,6 +414,49 @@ u32 udp_ehashfn(const struct net *net, const __be32 laddr, const __u16 lport,
udp_ehash_secret + net_hash_mix(net));
}
+/**
+ * udp4_lib_lookup1() - Simplified lookup using primary hash (destination port)
+ * @net: Network namespace
+ * @saddr: Source address, network order
+ * @sport: Source port, network order
+ * @daddr: Destination address, network order
+ * @hnum: Destination port, host order
+ * @dif: Destination interface index
+ * @sdif: Destination bridge port index, if relevant
+ * @udptable: Set of UDP hash tables
+ *
+ * Simplified lookup to be used as fallback if no sockets are found due to a
+ * potential race between (receive) address change, and lookup happening before
+ * the rehash operation. This function ignores SO_REUSEPORT groups while scoring
+ * result sockets, because if we have one, we don't need the fallback at all.
+ *
+ * Called under rcu_read_lock().
+ *
+ * Return: socket with highest matching score if any, NULL if none
+ */
+static struct sock *udp4_lib_lookup1(struct net *net,
+ __be32 saddr, __be16 sport,
+ __be32 daddr, unsigned int hnum,
+ int dif, int sdif,
+ const struct udp_table *udptable)
+{
+ unsigned int slot = udp_hashfn(net, hnum, udptable->mask);
+ struct udp_hslot *hslot = &udptable->hash[slot];
+ struct sock *sk, *result = NULL;
+ int score, badness = 0;
+
+ sk_for_each_rcu(sk, &hslot->head) {
+ score = compute_score(sk, net,
+ saddr, sport, daddr, hnum, dif, sdif);
+ if (score > badness) {
+ result = sk;
+ badness = score;
+ }
+ }
+
+ return result;
+}
+
/* called with rcu_read_lock() */
static struct sock *udp4_lib_lookup2(struct net *net,
__be32 saddr, __be16 sport,
@@ -541,6 +584,19 @@ struct sock *__udp4_lib_lookup(struct net *net, __be32 saddr,
result = udp4_lib_lookup2(net, saddr, sport,
htonl(INADDR_ANY), hnum, dif, sdif,
hslot2, skb);
+ if (!IS_ERR_OR_NULL(result))
+ goto done;
+
+ /* Primary hash (destination port) lookup as fallback for this race:
+ * 1. __ip4_datagram_connect() sets sk_rcv_saddr
+ * 2. lookup (this function): new sk_rcv_saddr, hashes not updated yet
+ * 3. rehash operation updating _secondary and four-tuple_ hashes
+ * The primary hash doesn't need an update after 1., so, thanks to this
+ * further step, 1. and 3. don't need to be atomic against the lookup.
+ */
+ result = udp4_lib_lookup1(net, saddr, sport, daddr, hnum, dif, sdif,
+ udptable);
+
done:
if (IS_ERR(result))
return NULL;
@@ -823,6 +879,15 @@ int __udp4_lib_err(struct sk_buff *skb, u32 info, struct udp_table *udptable)
int udp_err(struct sk_buff *skb, u32 info)
{
+ const int type = icmp_hdr(skb)->type;
+ const int code = icmp_hdr(skb)->code;
+ struct net *net = dev_net(skb->dev);
+
+ if (type == ICMP_DEST_UNREACH && code == ICMP_FRAG_NEEDED)
+ ipv4_update_pmtu(skb, net, info, 0, IPPROTO_UDP);
+ else if (type == ICMP_REDIRECT)
+ ipv4_redirect(skb, net, 0, IPPROTO_UDP);
+
return __udp4_lib_err(skb, info, &udp_table);
}
diff --git a/net/ipv4/udp_bpf.c b/net/ipv4/udp_bpf.c
index 44271ba1adec..bcb4378d5354 100644
--- a/net/ipv4/udp_bpf.c
+++ b/net/ipv4/udp_bpf.c
@@ -51,7 +51,9 @@ static int udp_msg_wait_data(struct sock *sk, struct sk_psock *psock,
sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk);
ret = udp_msg_has_data(sk, psock);
if (!ret) {
+ release_sock(sk);
wait_woken(&wait, TASK_INTERRUPTIBLE, timeo);
+ lock_sock(sk);
ret = udp_msg_has_data(sk, psock);
}
sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk);
@@ -80,6 +82,7 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
goto out;
}
+ lock_sock(sk);
msg_bytes_ready:
copied = sk_msg_recvmsg(sk, psock, msg, len, flags);
if (!copied) {
@@ -91,11 +94,17 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
if (data) {
if (psock_has_data(psock))
goto msg_bytes_ready;
+
+ release_sock(sk);
+
ret = sk_udp_recvmsg(sk, msg, len, flags, addr_len);
goto out;
}
copied = -EAGAIN;
}
+
+ release_sock(sk);
+
ret = copied;
out:
sk_psock_put(sk, psock);
diff --git a/net/ipv4/udp_tunnel_nic.c b/net/ipv4/udp_tunnel_nic.c
index 72b0210cdead..98f73221a6bf 100644
--- a/net/ipv4/udp_tunnel_nic.c
+++ b/net/ipv4/udp_tunnel_nic.c
@@ -31,19 +31,17 @@ struct udp_tunnel_nic_table_entry {
* @dev: netdev pointer
* @need_sync: at least one port start changed
* @need_replay: space was freed, we need a replay of all ports
- * @work_pending: @work is currently scheduled
* @n_tables: number of tables under @entries
* @missed: bitmap of tables which overflown
* @entries: table of tables of ports currently offloaded
*/
struct udp_tunnel_nic {
- struct work_struct work;
+ struct delayed_work work;
struct net_device *dev;
u8 need_sync:1;
u8 need_replay:1;
- u8 work_pending:1;
unsigned int n_tables;
unsigned long missed;
@@ -310,10 +308,8 @@ udp_tunnel_nic_device_sync(struct net_device *dev, struct udp_tunnel_nic *utn)
may_sleep = info->flags & UDP_TUNNEL_NIC_INFO_MAY_SLEEP;
if (!may_sleep)
__udp_tunnel_nic_device_sync(dev, utn);
- if (may_sleep || utn->need_replay) {
- queue_work(udp_tunnel_nic_workqueue, &utn->work);
- utn->work_pending = 1;
- }
+ if (may_sleep || utn->need_replay)
+ queue_delayed_work(udp_tunnel_nic_workqueue, &utn->work, 0);
}
static bool
@@ -705,10 +701,16 @@ udp_tunnel_nic_replay(struct net_device *dev, struct udp_tunnel_nic *utn)
static void udp_tunnel_nic_device_sync_work(struct work_struct *work)
{
struct udp_tunnel_nic *utn =
- container_of(work, struct udp_tunnel_nic, work);
+ container_of(work, struct udp_tunnel_nic, work.work);
- rtnl_lock();
- utn->work_pending = 0;
+ /* We cannot block on RTNL here, otherwise we would deadlock with
+ * udp_tunnel_nic_unregister() calling cancel_delayed_work_sync()
+ * while holding RTNL. Requeue with 1 jiffy delay if RTNL is contended.
+ */
+ if (!rtnl_trylock()) {
+ queue_delayed_work(udp_tunnel_nic_workqueue, &utn->work, 1);
+ return;
+ }
__udp_tunnel_nic_device_sync(utn->dev, utn);
if (utn->need_replay)
@@ -727,7 +729,7 @@ udp_tunnel_nic_alloc(const struct udp_tunnel_nic_info *info,
if (!utn)
return NULL;
utn->n_tables = n_tables;
- INIT_WORK(&utn->work, udp_tunnel_nic_device_sync_work);
+ INIT_DELAYED_WORK(&utn->work, udp_tunnel_nic_device_sync_work);
utn->entries = kmalloc_array(n_tables, sizeof(void *), GFP_KERNEL);
if (!utn->entries)
@@ -868,11 +870,11 @@ udp_tunnel_nic_unregister(struct net_device *dev, struct udp_tunnel_nic *utn)
*/
udp_tunnel_nic_flush(dev, utn);
- /* Wait for the work to be done using the state, netdev core will
- * retry unregister until we give up our reference on this device.
+ /* Make sure no work is running or queued before freeing @utn.
+ * The work handler uses rtnl_trylock(), so it will not deadlock
+ * against the RTNL we are holding here.
*/
- if (utn->work_pending)
- return;
+ cancel_delayed_work_sync(&utn->work);
udp_tunnel_nic_free(utn);
release_dev:
diff --git a/net/ipv4/xfrm4_input.c b/net/ipv4/xfrm4_input.c
index f6e90ba50b63..d05e8ff9c37b 100644
--- a/net/ipv4/xfrm4_input.c
+++ b/net/ipv4/xfrm4_input.c
@@ -48,6 +48,7 @@ int xfrm4_transport_finish(struct sk_buff *skb, int async)
{
struct xfrm_offload *xo = xfrm_offload(skb);
struct iphdr *iph = ip_hdr(skb);
+ struct net_device *dev = skb->dev;
iph->protocol = XFRM_MODE_SKB_CB(skb)->protocol;
@@ -71,8 +72,10 @@ int xfrm4_transport_finish(struct sk_buff *skb, int async)
}
NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING,
- dev_net(skb->dev), NULL, skb, skb->dev, NULL,
+ dev_net(dev), NULL, skb, dev, NULL,
xfrm4_rcv_encap_finish);
+ if (async)
+ dev_put(dev);
return 0;
}
diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c
index d404f7bec268..72490d27d15f 100644
--- a/net/ipv6/esp6.c
+++ b/net/ipv6/esp6.c
@@ -112,7 +112,7 @@ static inline struct scatterlist *esp_req_sg(struct crypto_aead *aead,
__alignof__(struct scatterlist));
}
-static void esp_ssg_unref(struct xfrm_state *x, void *tmp)
+static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb)
{
struct crypto_aead *aead = x->data;
int extralen = 0;
@@ -131,7 +131,7 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp)
*/
if (req->src != req->dst)
for (sg = sg_next(req->src); sg; sg = sg_next(sg))
- put_page(sg_page(sg));
+ skb_page_unref(skb, sg_page(sg), false);
}
#ifdef CONFIG_INET6_ESPINTCP
@@ -252,7 +252,7 @@ static void esp_output_done(struct crypto_async_request *base, int err)
}
tmp = ESP_SKB_CB(skb)->tmp;
- esp_ssg_unref(x, tmp);
+ esp_ssg_unref(x, tmp, skb);
kfree(tmp);
esp_output_encap_csum(skb);
@@ -646,7 +646,7 @@ int esp6_output_tail(struct xfrm_state *x, struct sk_buff *skb, struct esp_info
}
if (sg != dsg)
- esp_ssg_unref(x, tmp);
+ esp_ssg_unref(x, tmp, skb);
if (!err && x->encap && x->encap->encap_type == TCP_ENCAP_ESPINTCP)
err = esp_output_tail_tcp(x, skb);
diff --git a/net/ipv6/exthdrs.c b/net/ipv6/exthdrs.c
index afed094c8ac0..4bd1f9548bd7 100644
--- a/net/ipv6/exthdrs.c
+++ b/net/ipv6/exthdrs.c
@@ -373,23 +373,16 @@ static void seg6_update_csum(struct sk_buff *skb)
(__be32 *)addr);
}
-static int ipv6_srh_rcv(struct sk_buff *skb)
+static int ipv6_srh_rcv(struct sk_buff *skb, struct inet6_dev *idev)
{
struct inet6_skb_parm *opt = IP6CB(skb);
struct net *net = dev_net(skb->dev);
struct ipv6_sr_hdr *hdr;
- struct inet6_dev *idev;
struct in6_addr *addr;
int accept_seg6;
hdr = (struct ipv6_sr_hdr *)skb_transport_header(skb);
- idev = __in6_dev_get(skb->dev);
- if (!idev) {
- kfree_skb(skb);
- return -1;
- }
-
accept_seg6 = net->ipv6.devconf_all->seg6_enabled;
if (accept_seg6 > idev->cnf.seg6_enabled)
accept_seg6 = idev->cnf.seg6_enabled;
@@ -497,12 +490,11 @@ static int ipv6_srh_rcv(struct sk_buff *skb)
return -1;
}
-static int ipv6_rpl_srh_rcv(struct sk_buff *skb)
+static int ipv6_rpl_srh_rcv(struct sk_buff *skb, struct inet6_dev *idev)
{
struct ipv6_rpl_sr_hdr *hdr, *ohdr, *chdr;
struct inet6_skb_parm *opt = IP6CB(skb);
struct net *net = dev_net(skb->dev);
- struct inet6_dev *idev;
struct ipv6hdr *oldhdr;
unsigned int chdr_len;
unsigned char *buf;
@@ -511,8 +503,6 @@ static int ipv6_rpl_srh_rcv(struct sk_buff *skb)
u64 n = 0;
u32 r;
- idev = __in6_dev_get(skb->dev);
-
accept_rpl_seg = net->ipv6.devconf_all->rpl_seg_enabled;
if (accept_rpl_seg > idev->cnf.rpl_seg_enabled)
accept_rpl_seg = idev->cnf.rpl_seg_enabled;
@@ -719,10 +709,14 @@ static int ipv6_rthdr_rcv(struct sk_buff *skb)
switch (hdr->type) {
case IPV6_SRCRT_TYPE_4:
/* segment routing */
- return ipv6_srh_rcv(skb);
+ if (!idev)
+ goto disabled;
+ return ipv6_srh_rcv(skb, idev);
case IPV6_SRCRT_TYPE_3:
/* rpl segment routing */
- return ipv6_rpl_srh_rcv(skb);
+ if (!idev)
+ goto disabled;
+ return ipv6_rpl_srh_rcv(skb, idev);
default:
break;
}
@@ -870,6 +864,10 @@ static int ipv6_rthdr_rcv(struct sk_buff *skb)
icmpv6_param_prob(skb, ICMPV6_HDR_FIELD,
(&hdr->type) - skb_network_header(skb));
return -1;
+
+disabled:
+ kfree_skb_reason(skb, SKB_DROP_REASON_IPV6DISABLED);
+ return -1;
}
static const struct inet6_protocol rthdr_protocol = {
diff --git a/net/ipv6/ioam6.c b/net/ipv6/ioam6.c
index a1953cf6131b..05d8d9a5b353 100644
--- a/net/ipv6/ioam6.c
+++ b/net/ipv6/ioam6.c
@@ -650,6 +650,9 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
struct ioam6_schema *sc,
unsigned int sclen, bool is_input)
{
+ /* Note: skb_dst_dev_rcu() can't be NULL at this point. */
+ struct net_device *dev = skb_dst_dev_rcu(skb);
+ struct inet6_dev *i_skb_dev, *idev;
struct timespec64 ts;
ktime_t tstamp;
u64 raw64;
@@ -660,13 +663,16 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
data = trace->data + trace->remlen * 4 - trace->nodelen * 4 - sclen * 4;
+ i_skb_dev = skb->dev ? __in6_dev_get(skb->dev) : NULL;
+ idev = __in6_dev_get(dev);
+
/* hop_lim and node_id */
if (trace->type.bit0) {
byte = ipv6_hdr(skb)->hop_limit;
if (is_input)
byte--;
- raw32 = dev_net(skb_dst(skb)->dev)->ipv6.sysctl.ioam6_id;
+ raw32 = READ_ONCE(dev_net(dev)->ipv6.sysctl.ioam6_id);
*(__be32 *)data = cpu_to_be32((byte << 24) | raw32);
data += sizeof(__be32);
@@ -674,18 +680,18 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
/* ingress_if_id and egress_if_id */
if (trace->type.bit1) {
- if (!skb->dev)
+ if (!i_skb_dev)
raw16 = IOAM6_U16_UNAVAILABLE;
else
- raw16 = (__force u16)__in6_dev_get(skb->dev)->cnf.ioam6_id;
+ raw16 = (__force u16)READ_ONCE(i_skb_dev->cnf.ioam6_id);
*(__be16 *)data = cpu_to_be16(raw16);
data += sizeof(__be16);
- if (skb_dst(skb)->dev->flags & IFF_LOOPBACK)
+ if ((dev->flags & IFF_LOOPBACK) || !idev)
raw16 = IOAM6_U16_UNAVAILABLE;
else
- raw16 = (__force u16)__in6_dev_get(skb_dst(skb)->dev)->cnf.ioam6_id;
+ raw16 = (__force u16)READ_ONCE(idev->cnf.ioam6_id);
*(__be16 *)data = cpu_to_be16(raw16);
data += sizeof(__be16);
@@ -737,12 +743,16 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
struct Qdisc *qdisc;
__u32 qlen, backlog;
- if (skb_dst(skb)->dev->flags & IFF_LOOPBACK) {
+ if (skb_dst(skb)->dev->flags & IFF_LOOPBACK ||
+ skb_get_queue_mapping(skb) >= skb_dst(skb)->dev->num_tx_queues) {
*(__be32 *)data = cpu_to_be32(IOAM6_U32_UNAVAILABLE);
} else {
queue = skb_get_tx_queue(skb_dst(skb)->dev, skb);
qdisc = rcu_dereference(queue->qdisc);
+
+ spin_lock_bh(qdisc_lock(qdisc));
qdisc_qstats_qlen_backlog(qdisc, &qlen, &backlog);
+ spin_unlock_bh(qdisc_lock(qdisc));
*(__be32 *)data = cpu_to_be32(backlog);
}
@@ -761,7 +771,7 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
if (is_input)
byte--;
- raw64 = dev_net(skb_dst(skb)->dev)->ipv6.sysctl.ioam6_id_wide;
+ raw64 = READ_ONCE(dev_net(dev)->ipv6.sysctl.ioam6_id_wide);
*(__be64 *)data = cpu_to_be64(((u64)byte << 56) | raw64);
data += sizeof(__be64);
@@ -769,18 +779,18 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb,
/* ingress_if_id and egress_if_id (wide) */
if (trace->type.bit9) {
- if (!skb->dev)
+ if (!i_skb_dev)
raw32 = IOAM6_U32_UNAVAILABLE;
else
- raw32 = __in6_dev_get(skb->dev)->cnf.ioam6_id_wide;
+ raw32 = READ_ONCE(i_skb_dev->cnf.ioam6_id_wide);
*(__be32 *)data = cpu_to_be32(raw32);
data += sizeof(__be32);
- if (skb_dst(skb)->dev->flags & IFF_LOOPBACK)
+ if ((dev->flags & IFF_LOOPBACK) || !idev)
raw32 = IOAM6_U32_UNAVAILABLE;
else
- raw32 = __in6_dev_get(skb_dst(skb)->dev)->cnf.ioam6_id_wide;
+ raw32 = READ_ONCE(idev->cnf.ioam6_id_wide);
*(__be32 *)data = cpu_to_be32(raw32);
data += sizeof(__be32);
diff --git a/net/ipv6/ioam6_iptunnel.c b/net/ipv6/ioam6_iptunnel.c
index 3a6065faaf9b..b536eae433d7 100644
--- a/net/ipv6/ioam6_iptunnel.c
+++ b/net/ipv6/ioam6_iptunnel.c
@@ -365,9 +365,12 @@ static int ioam6_output(struct net *net, struct sock *sk, struct sk_buff *skb)
goto drop;
}
- local_bh_disable();
- dst_cache_set_ip6(&ilwt->cache, dst, &fl6.saddr);
- local_bh_enable();
+ /* cache only if we don't create a dst reference loop */
+ if (skb_dst(skb)->lwtstate != dst->lwtstate) {
+ local_bh_disable();
+ dst_cache_set_ip6(&ilwt->cache, dst, &fl6.saddr);
+ local_bh_enable();
+ }
}
skb_dst_drop(skb);
diff --git a/net/ipv6/ip6_flowlabel.c b/net/ipv6/ip6_flowlabel.c
index eb0d517aa6cb..ce441f7970a9 100644
--- a/net/ipv6/ip6_flowlabel.c
+++ b/net/ipv6/ip6_flowlabel.c
@@ -460,6 +460,22 @@ fl_create(struct net *net, struct sock *sk, struct in6_flowlabel_req *freq,
return NULL;
}
+static bool fl_sock_at_lease_limit(const struct sock *sk)
+{
+ const struct ipv6_pinfo *np = inet6_sk(sk);
+ const struct ipv6_fl_socklist *sfl;
+ int count = 0;
+
+ rcu_read_lock_bh();
+ for_each_sk_fl_rcu(np, sfl) {
+ if (++count >= FL_MAX_PER_SOCK)
+ break;
+ }
+ rcu_read_unlock_bh();
+
+ return count >= FL_MAX_PER_SOCK;
+}
+
static int mem_check(struct sock *sk)
{
struct ipv6_pinfo *np = inet6_sk(sk);
@@ -670,6 +686,10 @@ static int ipv6_flowlabel_get(struct sock *sk, struct in6_flowlabel_req *freq,
err = -ENOMEM;
if (!sfl1)
goto release;
+ err = -ENOBUFS;
+ if (fl_sock_at_lease_limit(sk) &&
+ !capable(CAP_NET_ADMIN))
+ goto release;
if (fl->linger > fl1->linger)
fl1->linger = fl->linger;
if ((long)(fl->expires - fl1->expires) > 0)
diff --git a/net/ipv6/ip6_gre.c b/net/ipv6/ip6_gre.c
index 950d5e74b094..73ec4641f220 100644
--- a/net/ipv6/ip6_gre.c
+++ b/net/ipv6/ip6_gre.c
@@ -962,7 +962,6 @@ static netdev_tx_t ip6erspan_tunnel_xmit(struct sk_buff *skb,
{
struct ip_tunnel_info *tun_info = NULL;
struct ip6_tnl *t = netdev_priv(dev);
- struct dst_entry *dst = skb_dst(skb);
bool truncate = false;
int encap_limit = -1;
__u8 dsfield = false;
@@ -1094,10 +1093,6 @@ static netdev_tx_t ip6erspan_tunnel_xmit(struct sk_buff *skb,
/* Push GRE header. */
gre_build_header(skb, 8, TUNNEL_SEQ, proto, 0, htonl(atomic_fetch_inc(&t->o_seqno)));
- /* TooBig packet may have updated dst->dev's mtu */
- if (!t->parms.collect_md && dst && dst_mtu(dst) > dst->dev->mtu)
- dst->ops->update_pmtu(dst, NULL, skb, dst->dev->mtu, false);
-
err = ip6_tnl_xmit(skb, dev, dsfield, &fl6, encap_limit, &mtu,
NEXTHDR_GRE);
if (err != 0) {
diff --git a/net/ipv6/ip6_input.c b/net/ipv6/ip6_input.c
index 1ba97933c74f..26f57852e81f 100644
--- a/net/ipv6/ip6_input.c
+++ b/net/ipv6/ip6_input.c
@@ -572,6 +572,7 @@ int ip6_mc_input(struct sk_buff *skb)
if (deliver)
skb2 = skb_clone(skb, GFP_ATOMIC);
else {
+ skb_orphan(skb);
skb2 = skb;
skb = NULL;
}
diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c
index 7ad60793c853..3ff6f43bd40a 100644
--- a/net/ipv6/ip6_output.c
+++ b/net/ipv6/ip6_output.c
@@ -59,7 +59,7 @@
static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff *skb)
{
struct dst_entry *dst = skb_dst(skb);
- struct net_device *dev = dst->dev;
+ struct net_device *dev = dst_dev_rcu(dst);
struct inet6_dev *idev = ip6_dst_idev(dst);
unsigned int hh_len = LL_RESERVED_SPACE(dev);
const struct in6_addr *daddr, *nexthop;
@@ -69,15 +69,12 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff *
/* Be paranoid, rather than too clever. */
if (unlikely(hh_len > skb_headroom(skb)) && dev->header_ops) {
- /* Make sure idev stays alive */
- rcu_read_lock();
+ /* idev stays alive because we hold rcu_read_lock(). */
skb = skb_expand_head(skb, hh_len);
if (!skb) {
IP6_INC_STATS(net, idev, IPSTATS_MIB_OUTDISCARDS);
- rcu_read_unlock();
return -ENOMEM;
}
- rcu_read_unlock();
}
hdr = ipv6_hdr(skb);
@@ -122,7 +119,6 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff *
daddr = &hdr->daddr;
}
- rcu_read_lock();
nexthop = rt6_nexthop(dst_rt6_info(dst), daddr);
neigh = __ipv6_neigh_lookup_noref(dev, nexthop);
@@ -130,7 +126,6 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff *
if (unlikely(!neigh))
neigh = __neigh_create(&nd_tbl, nexthop, dev, false);
if (IS_ERR(neigh)) {
- rcu_read_unlock();
IP6_INC_STATS(net, idev, IPSTATS_MIB_OUTNOROUTES);
kfree_skb_reason(skb, SKB_DROP_REASON_NEIGH_CREATEFAIL);
return -EINVAL;
@@ -138,7 +133,6 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff *
}
sock_confirm_neigh(skb, neigh);
ret = neigh_output(neigh, skb, false);
- rcu_read_unlock();
return ret;
}
@@ -224,22 +218,30 @@ static int ip6_finish_output(struct net *net, struct sock *sk, struct sk_buff *s
int ip6_output(struct net *net, struct sock *sk, struct sk_buff *skb)
{
- struct net_device *dev = skb_dst(skb)->dev, *indev = skb->dev;
- struct inet6_dev *idev = ip6_dst_idev(skb_dst(skb));
+ struct dst_entry *dst = skb_dst(skb);
+ struct net_device *dev, *indev = skb->dev;
+ struct inet6_dev *idev;
+ int ret;
skb->protocol = htons(ETH_P_IPV6);
+ rcu_read_lock();
+ dev = dst_dev_rcu(dst);
+ idev = ip6_dst_idev(dst);
skb->dev = dev;
if (unlikely(!idev || READ_ONCE(idev->cnf.disable_ipv6))) {
IP6_INC_STATS(net, idev, IPSTATS_MIB_OUTDISCARDS);
+ rcu_read_unlock();
kfree_skb_reason(skb, SKB_DROP_REASON_IPV6DISABLED);
return 0;
}
- return NF_HOOK_COND(NFPROTO_IPV6, NF_INET_POST_ROUTING,
- net, sk, skb, indev, dev,
- ip6_finish_output,
- !(IP6CB(skb)->flags & IP6SKB_REROUTED));
+ ret = NF_HOOK_COND(NFPROTO_IPV6, NF_INET_POST_ROUTING,
+ net, sk, skb, indev, dev,
+ ip6_finish_output,
+ !(IP6CB(skb)->flags & IP6SKB_REROUTED));
+ rcu_read_unlock();
+ return ret;
}
EXPORT_SYMBOL(ip6_output);
diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c
index 3c60a63c9f31..b677f48179c8 100644
--- a/net/ipv6/ip6mr.c
+++ b/net/ipv6/ip6mr.c
@@ -1128,7 +1128,6 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt,
msg->im6_src = ipv6_hdr(pkt)->saddr;
msg->im6_dst = ipv6_hdr(pkt)->daddr;
- skb_dst_set(skb, dst_clone(skb_dst(pkt)));
skb->ip_summed = CHECKSUM_UNNECESSARY;
}
@@ -1138,6 +1137,7 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt,
return -EINVAL;
}
+ skb_dst_drop(skb);
mrt6msg_netlink_event(mrt, skb);
/* Deliver to user space multicast routing algorithms */
@@ -1217,6 +1217,7 @@ static int ip6mr_cache_unresolved(struct mr_table *mrt, mifi_t mifi,
skb->dev = dev;
skb->skb_iif = dev->ifindex;
}
+ skb_dst_drop(skb);
skb_queue_tail(&c->_c.mfc_un.unres.unresolved, skb);
err = 0;
}
@@ -1389,7 +1390,7 @@ int __init ip6_mr_init(void)
mrt_cachep = kmem_cache_create("ip6_mrt_cache",
sizeof(struct mfc6_cache),
- 0, SLAB_HWCACHE_ALIGN,
+ 0, SLAB_HWCACHE_ALIGN | SLAB_ACCOUNT,
NULL);
if (!mrt_cachep)
return -ENOMEM;
diff --git a/net/ipv6/netfilter/ip6table_nat.c b/net/ipv6/netfilter/ip6table_nat.c
index 52d597b16b65..21957d53e9c9 100644
--- a/net/ipv6/netfilter/ip6table_nat.c
+++ b/net/ipv6/netfilter/ip6table_nat.c
@@ -81,7 +81,7 @@ static int ip6t_nat_register_lookups(struct net *net)
while (i)
nf_nat_ipv6_unregister_fn(net, &ops[--i]);
- kfree(ops);
+ kfree_rcu(ops, rcu);
return ret;
}
}
@@ -102,7 +102,7 @@ static void ip6t_nat_unregister_lookups(struct net *net)
for (i = 0; i < ARRAY_SIZE(nf_nat_ipv6_ops); i++)
nf_nat_ipv6_unregister_fn(net, &ops[i]);
- kfree(ops);
+ kfree_rcu(ops, rcu);
}
static int ip6table_nat_table_init(struct net *net)
diff --git a/net/ipv6/netfilter/nft_dup_ipv6.c b/net/ipv6/netfilter/nft_dup_ipv6.c
index 5a7abea7739c..492a811828a7 100644
--- a/net/ipv6/netfilter/nft_dup_ipv6.c
+++ b/net/ipv6/netfilter/nft_dup_ipv6.c
@@ -50,7 +50,8 @@ static int nft_dup_ipv6_init(const struct nft_ctx *ctx,
return err;
}
-static int nft_dup_ipv6_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_dup_ipv6_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_dup_ipv6 *priv = nft_expr_priv(expr);
diff --git a/net/ipv6/route.c b/net/ipv6/route.c
index 0a819dbd8e58..1c61081a857c 100644
--- a/net/ipv6/route.c
+++ b/net/ipv6/route.c
@@ -149,9 +149,9 @@ void rt6_uncached_list_add(struct rt6_info *rt)
void rt6_uncached_list_del(struct rt6_info *rt)
{
- if (!list_empty(&rt->rt6i_uncached)) {
- struct uncached_list *ul = rt->rt6i_uncached_list;
+ struct uncached_list *ul = rt->rt6i_uncached_list;
+ if (ul) {
spin_lock_bh(&ul->lock);
list_del_init(&rt->rt6i_uncached);
spin_unlock_bh(&ul->lock);
@@ -6333,8 +6333,6 @@ void fib6_info_hw_flags_set(struct net *net, struct fib6_info *f6i,
err = rt6_fill_node(net, skb, f6i, NULL, NULL, NULL, 0, RTM_NEWROUTE, 0,
0, 0);
if (err < 0) {
- /* -EMSGSIZE implies BUG in rt6_nlmsg_size() */
- WARN_ON(err == -EMSGSIZE);
kfree_skb(skb);
goto errout;
}
diff --git a/net/ipv6/rpl_iptunnel.c b/net/ipv6/rpl_iptunnel.c
index cde14e180a85..be38d40188ec 100644
--- a/net/ipv6/rpl_iptunnel.c
+++ b/net/ipv6/rpl_iptunnel.c
@@ -236,9 +236,12 @@ static int rpl_output(struct net *net, struct sock *sk, struct sk_buff *skb)
goto drop;
}
- local_bh_disable();
- dst_cache_set_ip6(&rlwt->cache, dst, &fl6.saddr);
- local_bh_enable();
+ /* cache only if we don't create a dst reference loop */
+ if (orig_dst->lwtstate != dst->lwtstate) {
+ local_bh_disable();
+ dst_cache_set_ip6(&rlwt->cache, dst, &fl6.saddr);
+ local_bh_enable();
+ }
err = skb_cow_head(skb, LL_RESERVED_SPACE(dst->dev));
if (unlikely(err))
diff --git a/net/ipv6/seg6_iptunnel.c b/net/ipv6/seg6_iptunnel.c
index 96d2ca23ee38..d0968a319635 100644
--- a/net/ipv6/seg6_iptunnel.c
+++ b/net/ipv6/seg6_iptunnel.c
@@ -116,7 +116,7 @@ static __be32 seg6_make_flowlabel(struct net *net, struct sk_buff *skb,
u32 hash;
if (do_flowlabel > 0) {
- hash = skb_get_hash(skb);
+ hash = skb_get_hash_net(net, skb);
hash = rol32(hash, 16);
flowlabel = (__force __be32)hash & IPV6_FLOWLABEL_MASK;
} else if (!do_flowlabel && skb->protocol == htons(ETH_P_IPV6)) {
diff --git a/net/ipv6/syncookies.c b/net/ipv6/syncookies.c
index 8698b49dfc8d..99c2ea8c7d76 100644
--- a/net/ipv6/syncookies.c
+++ b/net/ipv6/syncookies.c
@@ -161,9 +161,14 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb)
tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL);
if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) {
- tsoff = secure_tcpv6_ts_off(sock_net(sk),
- ipv6_hdr(skb)->daddr.s6_addr32,
- ipv6_hdr(skb)->saddr.s6_addr32);
+ union tcp_seq_and_ts_off st;
+
+ st = secure_tcpv6_seq_and_ts_off(sock_net(sk),
+ ipv6_hdr(skb)->daddr.s6_addr32,
+ ipv6_hdr(skb)->saddr.s6_addr32,
+ tcp_hdr(skb)->dest,
+ tcp_hdr(skb)->source);
+ tsoff = st.ts_off;
tcp_opt.rcv_tsecr -= tsoff;
}
diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c
index ec668c19cb97..62ed0886af74 100644
--- a/net/ipv6/tcp_ipv6.c
+++ b/net/ipv6/tcp_ipv6.c
@@ -112,18 +112,14 @@ static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb)
}
}
-static u32 tcp_v6_init_seq(const struct sk_buff *skb)
+static union tcp_seq_and_ts_off
+tcp_v6_init_seq_and_ts_off(const struct net *net, const struct sk_buff *skb)
{
- return secure_tcpv6_seq(ipv6_hdr(skb)->daddr.s6_addr32,
- ipv6_hdr(skb)->saddr.s6_addr32,
- tcp_hdr(skb)->dest,
- tcp_hdr(skb)->source);
-}
-
-static u32 tcp_v6_init_ts_off(const struct net *net, const struct sk_buff *skb)
-{
- return secure_tcpv6_ts_off(net, ipv6_hdr(skb)->daddr.s6_addr32,
- ipv6_hdr(skb)->saddr.s6_addr32);
+ return secure_tcpv6_seq_and_ts_off(net,
+ ipv6_hdr(skb)->daddr.s6_addr32,
+ ipv6_hdr(skb)->saddr.s6_addr32,
+ tcp_hdr(skb)->dest,
+ tcp_hdr(skb)->source);
}
static int tcp_v6_pre_connect(struct sock *sk, struct sockaddr *uaddr,
@@ -323,14 +319,16 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr,
sk_set_txhash(sk);
if (likely(!tp->repair)) {
+ union tcp_seq_and_ts_off st;
+
+ st = secure_tcpv6_seq_and_ts_off(net,
+ np->saddr.s6_addr32,
+ sk->sk_v6_daddr.s6_addr32,
+ inet->inet_sport,
+ inet->inet_dport);
if (!tp->write_seq)
- WRITE_ONCE(tp->write_seq,
- secure_tcpv6_seq(np->saddr.s6_addr32,
- sk->sk_v6_daddr.s6_addr32,
- inet->inet_sport,
- inet->inet_dport));
- tp->tsoffset = secure_tcpv6_ts_off(net, np->saddr.s6_addr32,
- sk->sk_v6_daddr.s6_addr32);
+ WRITE_ONCE(tp->write_seq, st.seq);
+ tp->tsoffset = st.ts_off;
}
if (tcp_fastopen_defer_connect(sk, &err))
@@ -837,8 +835,7 @@ const struct tcp_request_sock_ops tcp_request_sock_ipv6_ops = {
.cookie_init_seq = cookie_v6_init_sequence,
#endif
.route_req = tcp_v6_route_req,
- .init_seq = tcp_v6_init_seq,
- .init_ts_off = tcp_v6_init_ts_off,
+ .init_seq_and_ts_off = tcp_v6_init_seq_and_ts_off,
.send_synack = tcp_v6_send_synack,
};
@@ -1338,7 +1335,7 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff *
tcp_ca_openreq_child(newsk, dst);
tcp_sync_mss(newsk, dst_mtu(dst));
- newtp->advmss = tcp_mss_clamp(tcp_sk(sk), dst_metric_advmss(dst));
+ newtp->advmss = tcp_mss_clamp(tcp_sk(sk), tcp_dst_advmss(dst));
tcp_initialize_rcv_mss(newsk);
@@ -2001,9 +1998,9 @@ static void get_tcp6_sock(struct seq_file *seq, struct sock *sp, int i)
} else if (icsk->icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = icsk->icsk_timeout;
- } else if (timer_pending(&sp->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
- timer_expires = sp->sk_timer.expires;
+ timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = jiffies;
diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c
index 96a46dc47336..8697f473357c 100644
--- a/net/ipv6/udp.c
+++ b/net/ipv6/udp.c
@@ -160,6 +160,49 @@ static int compute_score(struct sock *sk, struct net *net,
return score;
}
+/**
+ * udp6_lib_lookup1() - Simplified lookup using primary hash (destination port)
+ * @net: Network namespace
+ * @saddr: Source address, network order
+ * @sport: Source port, network order
+ * @daddr: Destination address, network order
+ * @hnum: Destination port, host order
+ * @dif: Destination interface index
+ * @sdif: Destination bridge port index, if relevant
+ * @udptable: Set of UDP hash tables
+ *
+ * Simplified lookup to be used as fallback if no sockets are found due to a
+ * potential race between (receive) address change, and lookup happening before
+ * the rehash operation. This function ignores SO_REUSEPORT groups while scoring
+ * result sockets, because if we have one, we don't need the fallback at all.
+ *
+ * Called under rcu_read_lock().
+ *
+ * Return: socket with highest matching score if any, NULL if none
+ */
+static struct sock *udp6_lib_lookup1(struct net *net,
+ const struct in6_addr *saddr, __be16 sport,
+ const struct in6_addr *daddr,
+ unsigned int hnum, int dif, int sdif,
+ const struct udp_table *udptable)
+{
+ unsigned int slot = udp_hashfn(net, hnum, udptable->mask);
+ struct udp_hslot *hslot = &udptable->hash[slot];
+ struct sock *sk, *result = NULL;
+ int score, badness = 0;
+
+ sk_for_each_rcu(sk, &hslot->head) {
+ score = compute_score(sk, net,
+ saddr, sport, daddr, hnum, dif, sdif);
+ if (score > badness) {
+ result = sk;
+ badness = score;
+ }
+ }
+
+ return result;
+}
+
/* called with rcu_read_lock() */
static struct sock *udp6_lib_lookup2(struct net *net,
const struct in6_addr *saddr, __be16 sport,
@@ -288,6 +331,13 @@ struct sock *__udp6_lib_lookup(struct net *net,
result = udp6_lib_lookup2(net, saddr, sport,
&in6addr_any, hnum, dif, sdif,
hslot2, skb);
+ if (!IS_ERR_OR_NULL(result))
+ goto done;
+
+ /* Cover address change/lookup/rehash race: see __udp4_lib_lookup() */
+ result = udp6_lib_lookup1(net, saddr, sport, daddr, hnum, dif, sdif,
+ udptable);
+
done:
if (IS_ERR(result))
return NULL;
diff --git a/net/ipv6/xfrm6_input.c b/net/ipv6/xfrm6_input.c
index 00d2a1b8b679..cd96299dfe2f 100644
--- a/net/ipv6/xfrm6_input.c
+++ b/net/ipv6/xfrm6_input.c
@@ -41,6 +41,7 @@ static int xfrm6_transport_finish2(struct net *net, struct sock *sk,
int xfrm6_transport_finish(struct sk_buff *skb, int async)
{
struct xfrm_offload *xo = xfrm_offload(skb);
+ struct net_device *dev = skb->dev;
int nhlen = skb->data - skb_network_header(skb);
skb_network_header(skb)[IP6CB(skb)->nhoff] =
@@ -66,8 +67,10 @@ int xfrm6_transport_finish(struct sk_buff *skb, int async)
}
NF_HOOK(NFPROTO_IPV6, NF_INET_PRE_ROUTING,
- dev_net(skb->dev), NULL, skb, skb->dev, NULL,
+ dev_net(dev), NULL, skb, dev, NULL,
xfrm6_transport_finish2);
+ if (async)
+ dev_put(dev);
return 0;
}
diff --git a/net/iucv/af_iucv.c b/net/iucv/af_iucv.c
index b9ce76ccd054..c56bf6d96ac0 100644
--- a/net/iucv/af_iucv.c
+++ b/net/iucv/af_iucv.c
@@ -207,12 +207,6 @@ static int afiucv_hs_send(struct iucv_message *imsg, struct sock *sock,
phs_hdr->flags = flags;
if (flags == AF_IUCV_FLAG_SYN)
phs_hdr->window = iucv->msglimit;
- else if ((flags == AF_IUCV_FLAG_WIN) || !flags) {
- confirm_recv = atomic_read(&iucv->msg_recv);
- phs_hdr->window = confirm_recv;
- if (confirm_recv)
- phs_hdr->flags = phs_hdr->flags | AF_IUCV_FLAG_WIN;
- }
memcpy(phs_hdr->destUserID, iucv->dst_user_id, 8);
memcpy(phs_hdr->destAppName, iucv->dst_name, 8);
memcpy(phs_hdr->srcUserID, iucv->src_user_id, 8);
@@ -247,13 +241,22 @@ static int afiucv_hs_send(struct iucv_message *imsg, struct sock *sock,
}
skb->protocol = cpu_to_be16(ETH_P_AF_IUCV);
+ /* Claim the receive credit here, not while building the header: every
+ * way this frame can be dropped has now been ruled out, so the window
+ * is zeroed only for as long as the transmit itself takes.
+ */
+ if (flags == AF_IUCV_FLAG_WIN || !flags) {
+ confirm_recv = atomic_xchg(&iucv->msg_recv, 0);
+ phs_hdr->window = confirm_recv;
+ if (confirm_recv)
+ phs_hdr->flags = phs_hdr->flags | AF_IUCV_FLAG_WIN;
+ }
+
atomic_inc(&iucv->skbs_in_xmit);
err = dev_queue_xmit(skb);
if (net_xmit_eval(err)) {
atomic_dec(&iucv->skbs_in_xmit);
- } else {
- atomic_sub(confirm_recv, &iucv->msg_recv);
- WARN_ON(atomic_read(&iucv->msg_recv) < 0);
+ atomic_add(confirm_recv, &iucv->msg_recv);
}
return net_xmit_eval(err);
@@ -1239,6 +1242,7 @@ static int iucv_sock_recvmsg(struct socket *sock, struct msghdr *msg,
struct iucv_sock *iucv = iucv_sk(sk);
unsigned int copied, rlen;
struct sk_buff *skb, *rskb, *cskb;
+ bool send_win = false;
int err = 0;
u32 offset;
@@ -1329,16 +1333,20 @@ static int iucv_sock_recvmsg(struct socket *sock, struct msghdr *msg,
if (skb_queue_empty(&iucv->backlog_skb_q)) {
if (!list_empty(&iucv->message_q.list))
iucv_process_message_q(sk);
- if (atomic_read(&iucv->msg_recv) >=
- iucv->msglimit / 2) {
- err = iucv_send_ctrl(sk, AF_IUCV_FLAG_WIN);
- if (err) {
- sk->sk_state = IUCV_DISCONN;
- sk->sk_state_change(sk);
- }
- }
+ if (iucv->transport == AF_IUCV_TRANS_HIPER &&
+ atomic_read(&iucv->msg_recv) >=
+ iucv->msglimit / 2)
+ send_win = true;
}
spin_unlock_bh(&iucv->message_q.lock);
+
+ if (send_win) {
+ err = iucv_send_ctrl(sk, AF_IUCV_FLAG_WIN);
+ if (err) {
+ sk->sk_state = IUCV_DISCONN;
+ sk->sk_state_change(sk);
+ }
+ }
}
done:
diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c
index ab7629cee1bd..7778a444ccb1 100644
--- a/net/kcm/kcmsock.c
+++ b/net/kcm/kcmsock.c
@@ -634,7 +634,7 @@ static int kcm_write_msgs(struct kcm_sock *kcm)
sent = 0;
do_frag_list:
- if (WARN_ON(!skb_shinfo(skb)->nr_frags)) {
+ if (WARN_ON_ONCE(!skb_shinfo(skb)->nr_frags)) {
ret = -EINVAL;
goto out;
}
@@ -908,7 +908,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
{
struct sock *sk = sock->sk;
struct kcm_sock *kcm = kcm_sk(sk);
- struct sk_buff *skb = NULL, *head = NULL;
+ struct sk_buff *skb = NULL, *head = NULL, *frag_prev = NULL;
size_t copy, copied = 0;
long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT);
int eor = (sock->type == SOCK_DGRAM) ?
@@ -983,6 +983,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
else
skb->next = tskb;
+ frag_prev = skb;
skb = tskb;
skb->ip_summed = CHECKSUM_UNNECESSARY;
continue;
@@ -1070,6 +1071,22 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
out_error:
kcm_push(kcm);
+ /* When MAX_SKB_FRAGS was reached, a new skb was allocated and
+ * linked into the frag_list before data copy. If the copy
+ * subsequently failed, this skb has zero frags. Remove it from
+ * the frag_list to prevent kcm_write_msgs from later hitting
+ * WARN_ON(!skb_shinfo(skb)->nr_frags).
+ */
+ if (frag_prev && !skb_shinfo(skb)->nr_frags) {
+ if (head == frag_prev)
+ skb_shinfo(head)->frag_list = NULL;
+ else
+ frag_prev->next = NULL;
+ kfree_skb(skb);
+ /* Update skb as it may be saved in partial_message via goto */
+ skb = frag_prev;
+ }
+
if (sock->type == SOCK_SEQPACKET) {
/* Wrote some bytes before encountering an
* error, return partial success.
diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c
index 3c701795fa10..717559ead628 100644
--- a/net/l2tp/l2tp_core.c
+++ b/net/l2tp/l2tp_core.c
@@ -380,11 +380,13 @@ int l2tp_session_register(struct l2tp_session *session,
}
l2tp_tunnel_inc_refcount(tunnel);
+ WRITE_ONCE(session->tunnel, tunnel);
hlist_add_head_rcu(&session->global_hlist, g_head);
spin_unlock_bh(&pn->l2tp_session_hlist_lock);
} else {
l2tp_tunnel_inc_refcount(tunnel);
+ WRITE_ONCE(session->tunnel, tunnel);
}
hlist_add_head_rcu(&session->hlist, head);
@@ -698,7 +700,8 @@ void l2tp_recv_common(struct l2tp_session *session, struct sk_buff *skb,
if (!session->lns_mode && !session->send_seq) {
trace_session_seqnum_lns_enable(session);
session->send_seq = 1;
- l2tp_session_set_header_len(session, tunnel->version);
+ l2tp_session_set_header_len(session, tunnel->version,
+ tunnel->encap);
}
} else {
/* No sequence numbers.
@@ -719,7 +722,8 @@ void l2tp_recv_common(struct l2tp_session *session, struct sk_buff *skb,
if (!session->lns_mode && session->send_seq) {
trace_session_seqnum_lns_disable(session);
session->send_seq = 0;
- l2tp_session_set_header_len(session, tunnel->version);
+ l2tp_session_set_header_len(session, tunnel->version,
+ tunnel->encap);
} else if (session->send_seq) {
pr_debug_ratelimited("%s: recv data has no seq numbers when required. Discarding.\n",
session->name);
@@ -1579,7 +1583,8 @@ EXPORT_SYMBOL_GPL(l2tp_session_delete);
/* We come here whenever a session's send_seq, cookie_len or
* l2specific_type parameters are set.
*/
-void l2tp_session_set_header_len(struct l2tp_session *session, int version)
+void l2tp_session_set_header_len(struct l2tp_session *session, int version,
+ enum l2tp_encap_type encap)
{
if (version == L2TP_HDR_VER_2) {
session->hdr_len = 6;
@@ -1588,7 +1593,7 @@ void l2tp_session_set_header_len(struct l2tp_session *session, int version)
} else {
session->hdr_len = 4 + session->cookie_len;
session->hdr_len += l2tp_get_l2specific_len(session);
- if (session->tunnel->encap == L2TP_ENCAPTYPE_UDP)
+ if (encap == L2TP_ENCAPTYPE_UDP)
session->hdr_len += 4;
}
}
@@ -1602,7 +1607,6 @@ struct l2tp_session *l2tp_session_create(int priv_size, struct l2tp_tunnel *tunn
session = kzalloc(sizeof(*session) + priv_size, GFP_KERNEL);
if (session) {
session->magic = L2TP_SESSION_MAGIC;
- session->tunnel = tunnel;
session->session_id = session_id;
session->peer_session_id = peer_session_id;
@@ -1638,7 +1642,7 @@ struct l2tp_session *l2tp_session_create(int priv_size, struct l2tp_tunnel *tunn
memcpy(&session->peer_cookie[0], &cfg->peer_cookie[0], cfg->peer_cookie_len);
}
- l2tp_session_set_header_len(session, tunnel->version);
+ l2tp_session_set_header_len(session, tunnel->version, tunnel->encap);
refcount_set(&session->ref_count, 1);
diff --git a/net/l2tp/l2tp_core.h b/net/l2tp/l2tp_core.h
index a88e070b431d..9f7143d2dd31 100644
--- a/net/l2tp/l2tp_core.h
+++ b/net/l2tp/l2tp_core.h
@@ -261,7 +261,8 @@ void l2tp_recv_common(struct l2tp_session *session, struct sk_buff *skb,
int l2tp_udp_encap_recv(struct sock *sk, struct sk_buff *skb);
/* Transmit path helpers for sending packets over the tunnel socket. */
-void l2tp_session_set_header_len(struct l2tp_session *session, int version);
+void l2tp_session_set_header_len(struct l2tp_session *session, int version,
+ enum l2tp_encap_type encap);
int l2tp_xmit_skb(struct l2tp_session *session, struct sk_buff *skb);
/* Pseudowire management.
diff --git a/net/l2tp/l2tp_debugfs.c b/net/l2tp/l2tp_debugfs.c
index 4595b56d175d..fe56dbae43be 100644
--- a/net/l2tp/l2tp_debugfs.c
+++ b/net/l2tp/l2tp_debugfs.c
@@ -307,6 +307,10 @@ static int l2tp_dfs_seq_release(struct inode *inode, struct file *file)
seq = file->private_data;
pd = seq->private;
+ if (pd->session)
+ l2tp_session_dec_refcount(pd->session);
+ if (pd->tunnel)
+ l2tp_tunnel_dec_refcount(pd->tunnel);
if (pd->net)
put_net_track(pd->net, &pd->ns_tracker);
kfree(pd);
diff --git a/net/l2tp/l2tp_netlink.c b/net/l2tp/l2tp_netlink.c
index e27e00cb16c6..25800de8b482 100644
--- a/net/l2tp/l2tp_netlink.c
+++ b/net/l2tp/l2tp_netlink.c
@@ -247,8 +247,8 @@ static int l2tp_nl_cmd_tunnel_create(struct sk_buff *skb, struct genl_info *info
kfree(tunnel);
goto out;
}
- ret = l2tp_tunnel_notify(&l2tp_nl_family, info, tunnel,
- L2TP_CMD_TUNNEL_CREATE);
+ l2tp_tunnel_notify(&l2tp_nl_family, info, tunnel,
+ L2TP_CMD_TUNNEL_CREATE);
l2tp_tunnel_dec_refcount(tunnel);
out:
@@ -304,8 +304,8 @@ static int l2tp_nl_cmd_tunnel_modify(struct sk_buff *skb, struct genl_info *info
goto out;
}
- ret = l2tp_tunnel_notify(&l2tp_nl_family, info,
- tunnel, L2TP_CMD_TUNNEL_MODIFY);
+ l2tp_tunnel_notify(&l2tp_nl_family, info,
+ tunnel, L2TP_CMD_TUNNEL_MODIFY);
l2tp_tunnel_dec_refcount(tunnel);
@@ -637,8 +637,8 @@ static int l2tp_nl_cmd_session_create(struct sk_buff *skb, struct genl_info *inf
if (ret >= 0) {
session = l2tp_tunnel_get_session(tunnel, session_id);
if (session) {
- ret = l2tp_session_notify(&l2tp_nl_family, info, session,
- L2TP_CMD_SESSION_CREATE);
+ l2tp_session_notify(&l2tp_nl_family, info, session,
+ L2TP_CMD_SESSION_CREATE);
l2tp_session_dec_refcount(session);
}
}
@@ -690,8 +690,10 @@ static int l2tp_nl_cmd_session_modify(struct sk_buff *skb, struct genl_info *inf
session->recv_seq = nla_get_u8(info->attrs[L2TP_ATTR_RECV_SEQ]);
if (info->attrs[L2TP_ATTR_SEND_SEQ]) {
+ struct l2tp_tunnel *tunnel = session->tunnel;
+
session->send_seq = nla_get_u8(info->attrs[L2TP_ATTR_SEND_SEQ]);
- l2tp_session_set_header_len(session, session->tunnel->version);
+ l2tp_session_set_header_len(session, tunnel->version, tunnel->encap);
}
if (info->attrs[L2TP_ATTR_LNS_MODE])
@@ -700,8 +702,8 @@ static int l2tp_nl_cmd_session_modify(struct sk_buff *skb, struct genl_info *inf
if (info->attrs[L2TP_ATTR_RECV_TIMEOUT])
session->reorder_timeout = nla_get_msecs(info->attrs[L2TP_ATTR_RECV_TIMEOUT]);
- ret = l2tp_session_notify(&l2tp_nl_family, info,
- session, L2TP_CMD_SESSION_MODIFY);
+ l2tp_session_notify(&l2tp_nl_family, info,
+ session, L2TP_CMD_SESSION_MODIFY);
l2tp_session_dec_refcount(session);
diff --git a/net/l2tp/l2tp_ppp.c b/net/l2tp/l2tp_ppp.c
index 2e856a83a750..21b1016020f3 100644
--- a/net/l2tp/l2tp_ppp.c
+++ b/net/l2tp/l2tp_ppp.c
@@ -1056,64 +1056,76 @@ static int pppol2tp_ioctl(struct socket *sock, unsigned int cmd,
{
struct pppol2tp_ioc_stats stats;
struct l2tp_session *session;
+ int err = 0;
+
+ session = pppol2tp_sock_to_session(sock->sk);
+ /* Validate session presence and magic integrity ONLY for commands
+ * that belong to L2TP and require a valid session.
+ */
switch (cmd) {
case PPPIOCGMRU:
case PPPIOCGFLAGS:
- session = sock->sk->sk_user_data;
+ case PPPIOCSMRU:
+ case PPPIOCSFLAGS:
+ case PPPIOCGL2TPSTATS:
if (!session)
return -ENOTCONN;
- if (WARN_ON(session->magic != L2TP_SESSION_MAGIC))
+ if (session->magic != L2TP_SESSION_MAGIC) {
+ sock_put(sock->sk);
return -EBADF;
+ }
+ break;
+ default:
+ break;
+ }
+ switch (cmd) {
+ case PPPIOCGMRU:
+ case PPPIOCGFLAGS:
/* Not defined for tunnels */
- if (!session->session_id && !session->peer_session_id)
- return -ENOSYS;
+ if (!session->session_id && !session->peer_session_id) {
+ err = -ENOSYS;
+ break;
+ }
- if (put_user(0, (int __user *)arg))
- return -EFAULT;
+ if (put_user(0, (int __user *)arg)) {
+ err = -EFAULT;
+ break;
+ }
break;
case PPPIOCSMRU:
case PPPIOCSFLAGS:
- session = sock->sk->sk_user_data;
- if (!session)
- return -ENOTCONN;
-
- if (WARN_ON(session->magic != L2TP_SESSION_MAGIC))
- return -EBADF;
-
/* Not defined for tunnels */
- if (!session->session_id && !session->peer_session_id)
- return -ENOSYS;
+ if (!session->session_id && !session->peer_session_id) {
+ err = -ENOSYS;
+ break;
+ }
- if (!access_ok((int __user *)arg, sizeof(int)))
- return -EFAULT;
+ if (!access_ok((int __user *)arg, sizeof(int))) {
+ err = -EFAULT;
+ break;
+ }
break;
case PPPIOCGL2TPSTATS:
- session = sock->sk->sk_user_data;
- if (!session)
- return -ENOTCONN;
-
- if (WARN_ON(session->magic != L2TP_SESSION_MAGIC))
- return -EBADF;
-
/* Session 0 represents the parent tunnel */
if (!session->session_id && !session->peer_session_id) {
u32 session_id;
- int err;
if (copy_from_user(&stats, (void __user *)arg,
- sizeof(stats)))
- return -EFAULT;
+ sizeof(stats))) {
+ err = -EFAULT;
+ break;
+ }
session_id = stats.session_id;
err = pppol2tp_tunnel_copy_stats(&stats,
session->tunnel);
if (err < 0)
- return err;
+ break;
stats.session_id = session_id;
} else {
@@ -1123,15 +1135,21 @@ static int pppol2tp_ioctl(struct socket *sock, unsigned int cmd,
stats.tunnel_id = session->tunnel->tunnel_id;
stats.using_ipsec = l2tp_tunnel_uses_xfrm(session->tunnel);
- if (copy_to_user((void __user *)arg, &stats, sizeof(stats)))
- return -EFAULT;
+ if (copy_to_user((void __user *)arg, &stats, sizeof(stats))) {
+ err = -EFAULT;
+ break;
+ }
break;
default:
- return -ENOIOCTLCMD;
+ err = -ENOIOCTLCMD;
+ break;
}
- return 0;
+ if (session)
+ sock_put(sock->sk);
+
+ return err;
}
/*****************************************************************************
@@ -1194,7 +1212,8 @@ static int pppol2tp_session_setsockopt(struct sock *sk,
po->chan.hdrlen = val ? PPPOL2TP_L2TP_HDR_SIZE_SEQ :
PPPOL2TP_L2TP_HDR_SIZE_NOSEQ;
}
- l2tp_session_set_header_len(session, session->tunnel->version);
+ l2tp_session_set_header_len(session, session->tunnel->version,
+ session->tunnel->encap);
break;
case PPPOL2TP_SO_LNSMODE:
@@ -1601,7 +1620,53 @@ static const struct seq_operations pppol2tp_seq_ops = {
.stop = pppol2tp_seq_stop,
.show = pppol2tp_seq_show,
};
-#endif /* CONFIG_PROC_FS */
+
+static int pppol2tp_proc_open(struct inode *inode, struct file *file)
+{
+ struct net *net = pde_data(inode);
+ struct pppol2tp_seq_data *pd;
+
+ net = maybe_get_net(net);
+ if (!net)
+ return -ENXIO;
+
+ pd = __seq_open_private(file, &pppol2tp_seq_ops, sizeof(*pd));
+ if (!pd) {
+ put_net(net);
+ return -ENOMEM;
+ }
+
+#ifdef CONFIG_NET_NS
+ pd->p.net = net;
+ netns_tracker_alloc(net, &pd->p.ns_tracker, GFP_KERNEL);
+#endif
+ return 0;
+}
+
+static int pppol2tp_proc_release(struct inode *inode, struct file *file)
+{
+ struct seq_file *seq = file->private_data;
+ struct pppol2tp_seq_data *pd = seq->private;
+
+ if (pd->session)
+ l2tp_session_dec_refcount(pd->session);
+ if (pd->tunnel)
+ l2tp_tunnel_dec_refcount(pd->tunnel);
+
+#ifdef CONFIG_NET_NS
+ put_net_track(pd->p.net, &pd->p.ns_tracker);
+#else
+ put_net(&init_net);
+#endif
+ return seq_release_private(inode, file);
+}
+
+static const struct proc_ops pppol2tp_proc_ops = {
+ .proc_open = pppol2tp_proc_open,
+ .proc_read = seq_read,
+ .proc_lseek = seq_lseek,
+ .proc_release = pppol2tp_proc_release,
+};
/*****************************************************************************
* Network namespace
@@ -1612,8 +1677,8 @@ static __net_init int pppol2tp_init_net(struct net *net)
struct proc_dir_entry *pde;
int err = 0;
- pde = proc_create_net("pppol2tp", 0444, net->proc_net,
- &pppol2tp_seq_ops, sizeof(struct pppol2tp_seq_data));
+ pde = proc_create_data("pppol2tp", 0444, net->proc_net,
+ &pppol2tp_proc_ops, net);
if (!pde) {
err = -ENOMEM;
goto out;
@@ -1628,9 +1693,13 @@ static __net_exit void pppol2tp_exit_net(struct net *net)
remove_proc_entry("pppol2tp", net->proc_net);
}
+#endif /* CONFIG_PROC_FS */
+
static struct pernet_operations pppol2tp_net_ops = {
+#ifdef CONFIG_PROC_FS
.init = pppol2tp_init_net,
.exit = pppol2tp_exit_net,
+#endif
.id = &pppol2tp_net_id,
};
diff --git a/net/mac80211/agg-rx.c b/net/mac80211/agg-rx.c
index 9414d3bbd65f..acb295bdb71c 100644
--- a/net/mac80211/agg-rx.c
+++ b/net/mac80211/agg-rx.c
@@ -103,7 +103,7 @@ void ___ieee80211_stop_rx_ba_session(struct sta_info *sta, u16 tid,
if (!tid_rx)
return;
- del_timer_sync(&tid_rx->session_timer);
+ timer_shutdown_sync(&tid_rx->session_timer);
/* make sure ieee80211_sta_reorder_release() doesn't re-arm the timer */
spin_lock_bh(&tid_rx->reorder_lock);
diff --git a/net/mac80211/chan.c b/net/mac80211/chan.c
index 26e692a853d9..5838d391421e 100644
--- a/net/mac80211/chan.c
+++ b/net/mac80211/chan.c
@@ -1062,6 +1062,7 @@ int ieee80211_link_unreserve_chanctx(struct ieee80211_link_data *link)
{
struct ieee80211_sub_if_data *sdata = link->sdata;
struct ieee80211_chanctx *ctx = link->reserved_chanctx;
+ struct ieee80211_chanctx *old_ctx = NULL;
lockdep_assert_held(&sdata->local->chanctx_mtx);
@@ -1076,6 +1077,7 @@ int ieee80211_link_unreserve_chanctx(struct ieee80211_link_data *link)
if (WARN_ON(!ctx->replace_ctx))
return -EINVAL;
+ old_ctx = ctx->replace_ctx;
WARN_ON(ctx->replace_ctx->replace_state !=
IEEE80211_CHANCTX_WILL_BE_REPLACED);
WARN_ON(ctx->replace_ctx->replace_ctx != ctx);
@@ -1086,6 +1088,10 @@ int ieee80211_link_unreserve_chanctx(struct ieee80211_link_data *link)
list_del_rcu(&ctx->list);
kfree_rcu(ctx, rcu_head);
+
+ if (ieee80211_chanctx_refcount(sdata->local,
+ old_ctx) == 0)
+ ieee80211_free_chanctx(sdata->local, old_ctx);
} else {
ieee80211_free_chanctx(sdata->local, ctx);
}
@@ -1317,7 +1323,9 @@ ieee80211_link_use_reserved_reassign(struct ieee80211_link_data *link)
ieee80211_check_fast_xmit_iface(sdata);
- if (ieee80211_chanctx_refcount(local, old_ctx) == 0)
+ if (ieee80211_chanctx_refcount(local, old_ctx) == 0 &&
+ !(old_ctx->replace_state == IEEE80211_CHANCTX_WILL_BE_REPLACED &&
+ old_ctx->replace_ctx))
ieee80211_free_chanctx(local, old_ctx);
ieee80211_recalc_chanctx_min_def(local, new_ctx, NULL);
@@ -1381,7 +1389,8 @@ ieee80211_link_use_reserved_assign(struct ieee80211_link_data *link)
}
static bool
-ieee80211_link_has_in_place_reservation(struct ieee80211_link_data *link)
+ieee80211_link_has_in_place_reservation(struct ieee80211_link_data *link,
+ struct ieee80211_chanctx *ctx)
{
struct ieee80211_sub_if_data *sdata = link->sdata;
struct ieee80211_chanctx *old_ctx, *new_ctx;
@@ -1391,6 +1400,9 @@ ieee80211_link_has_in_place_reservation(struct ieee80211_link_data *link)
new_ctx = link->reserved_chanctx;
old_ctx = ieee80211_link_get_chanctx(link);
+ if (new_ctx != ctx)
+ return false;
+
if (!old_ctx)
return false;
@@ -1403,6 +1415,12 @@ ieee80211_link_has_in_place_reservation(struct ieee80211_link_data *link)
if (new_ctx->replace_state != IEEE80211_CHANCTX_REPLACES_OTHER)
return false;
+ if (new_ctx->replace_ctx != old_ctx)
+ return false;
+
+ if (old_ctx->replace_ctx != new_ctx)
+ return false;
+
return true;
}
@@ -1452,7 +1470,7 @@ static int ieee80211_chsw_switch_vifs(struct ieee80211_local *local,
list_for_each_entry(link, &ctx->reserved_links,
reserved_chanctx_list) {
- if (!ieee80211_link_has_in_place_reservation(link))
+ if (!ieee80211_link_has_in_place_reservation(link, ctx))
continue;
old_ctx = ieee80211_link_get_chanctx(link);
@@ -1561,7 +1579,7 @@ static int ieee80211_vif_use_reserved_switch(struct ieee80211_local *local)
list_for_each_entry(link, &ctx->replace_ctx->assigned_links,
assigned_chanctx_list) {
n_assigned++;
- if (link->reserved_chanctx) {
+ if (ieee80211_link_has_in_place_reservation(link, ctx)) {
n_reserved++;
if (link->reserved_ready)
n_ready++;
@@ -1582,7 +1600,7 @@ static int ieee80211_vif_use_reserved_switch(struct ieee80211_local *local)
ctx->conf.radar_enabled = false;
list_for_each_entry(link, &ctx->reserved_links,
reserved_chanctx_list) {
- if (ieee80211_link_has_in_place_reservation(link) &&
+ if (ieee80211_link_has_in_place_reservation(link, ctx) &&
!link->reserved_ready)
return -EAGAIN;
@@ -1656,7 +1674,7 @@ static int ieee80211_vif_use_reserved_switch(struct ieee80211_local *local)
struct ieee80211_bss_conf *link_conf = link->conf;
u32 changed = 0;
- if (!ieee80211_link_has_in_place_reservation(link))
+ if (!ieee80211_link_has_in_place_reservation(link, ctx))
continue;
rcu_assign_pointer(link_conf->chanctx_conf,
@@ -1709,7 +1727,8 @@ static int ieee80211_vif_use_reserved_switch(struct ieee80211_local *local)
*/
list_for_each_entry_safe(link, link_tmp, &ctx->reserved_links,
reserved_chanctx_list) {
- if (WARN_ON(ieee80211_link_has_in_place_reservation(link)))
+ if (WARN_ON(ieee80211_link_has_in_place_reservation(link,
+ ctx)))
continue;
if (WARN_ON(link->reserved_chanctx != ctx))
@@ -1797,7 +1816,9 @@ static void __ieee80211_link_release_channel(struct ieee80211_link_data *link)
}
ieee80211_assign_link_chanctx(link, NULL);
- if (ieee80211_chanctx_refcount(local, ctx) == 0)
+ if (ieee80211_chanctx_refcount(local, ctx) == 0 &&
+ !(ctx->replace_state == IEEE80211_CHANCTX_WILL_BE_REPLACED &&
+ ctx->replace_ctx))
ieee80211_free_chanctx(local, ctx);
link->radar_required = false;
diff --git a/net/mac80211/fils_aead.c b/net/mac80211/fils_aead.c
index e1d4cfd99128..5b50f01034f1 100644
--- a/net/mac80211/fils_aead.c
+++ b/net/mac80211/fils_aead.c
@@ -95,6 +95,14 @@ static int aes_siv_encrypt(const u8 *key, size_t key_len,
if (res)
return res;
+ /* With an empty plaintext there is no ciphertext to produce and
+ * the S2V result is the complete output.
+ */
+ if (!plain_len) {
+ memcpy(out, v, AES_BLOCK_SIZE);
+ return 0;
+ }
+
/* Use a temporary buffer of the plaintext to handle need for
* overwriting this during AES-CTR.
*/
diff --git a/net/mac80211/ibss.c b/net/mac80211/ibss.c
index 363e7e4fdd02..f2d4c68cf3aa 100644
--- a/net/mac80211/ibss.c
+++ b/net/mac80211/ibss.c
@@ -692,7 +692,9 @@ static void ieee80211_ibss_disconnect(struct ieee80211_sub_if_data *sdata)
ifibss->state = IEEE80211_IBSS_MLME_SEARCH;
- sta_info_flush(sdata);
+ netif_carrier_off(sdata->dev);
+ if (!sta_info_flush(sdata))
+ synchronize_net();
spin_lock_bh(&ifibss->incomplete_lock);
while (!list_empty(&ifibss->incomplete_stations)) {
@@ -706,8 +708,6 @@ static void ieee80211_ibss_disconnect(struct ieee80211_sub_if_data *sdata)
}
spin_unlock_bh(&ifibss->incomplete_lock);
- netif_carrier_off(sdata->dev);
-
sdata->vif.cfg.ibss_joined = false;
sdata->vif.cfg.ibss_creator = false;
sdata->vif.bss_conf.enable_beacon = false;
@@ -737,7 +737,6 @@ static void ieee80211_csa_connection_drop_work(struct work_struct *work)
sdata_lock(sdata);
ieee80211_ibss_disconnect(sdata);
- synchronize_rcu();
skb_queue_purge(&sdata->skb_queue);
/* trigger a scan to find another IBSS network to join */
@@ -1878,8 +1877,6 @@ int ieee80211_ibss_leave(struct ieee80211_sub_if_data *sdata)
memset(&ifibss->ht_capa, 0, sizeof(ifibss->ht_capa));
memset(&ifibss->ht_capa_mask, 0, sizeof(ifibss->ht_capa_mask));
- synchronize_rcu();
-
skb_queue_purge(&sdata->skb_queue);
del_timer_sync(&sdata->u.ibss.timer);
diff --git a/net/mac80211/mesh_pathtbl.c b/net/mac80211/mesh_pathtbl.c
index e7b9dcf30adc..f2970f33f96c 100644
--- a/net/mac80211/mesh_pathtbl.c
+++ b/net/mac80211/mesh_pathtbl.c
@@ -407,8 +407,10 @@ struct mesh_path *mesh_path_add(struct ieee80211_sub_if_data *sdata,
return ERR_PTR(-ENOSPC);
new_mpath = mesh_path_new(sdata, dst, GFP_ATOMIC);
- if (!new_mpath)
+ if (!new_mpath) {
+ atomic_dec(&sdata->u.mesh.mpaths);
return ERR_PTR(-ENOMEM);
+ }
tbl = &sdata->u.mesh.mesh_paths;
spin_lock_bh(&tbl->walk_lock);
@@ -421,6 +423,7 @@ struct mesh_path *mesh_path_add(struct ieee80211_sub_if_data *sdata,
if (mpath) {
kfree(new_mpath);
+ atomic_dec(&sdata->u.mesh.mpaths);
if (IS_ERR(mpath))
return mpath;
@@ -446,10 +449,15 @@ int mpp_path_add(struct ieee80211_sub_if_data *sdata,
if (is_multicast_ether_addr(dst))
return -ENOTSUPP;
+ if (!atomic_add_unless(&sdata->u.mesh.mpaths, 1, MESH_MAX_MPATHS))
+ return -ENOSPC;
+
new_mpath = mesh_path_new(sdata, dst, GFP_ATOMIC);
- if (!new_mpath)
+ if (!new_mpath) {
+ atomic_dec(&sdata->u.mesh.mpaths);
return -ENOMEM;
+ }
memcpy(new_mpath->mpp, mpp, ETH_ALEN);
tbl = &sdata->u.mesh.mpp_paths;
@@ -462,8 +470,10 @@ int mpp_path_add(struct ieee80211_sub_if_data *sdata,
hlist_add_head_rcu(&new_mpath->walk_list, &tbl->walk_head);
spin_unlock_bh(&tbl->walk_lock);
- if (ret)
+ if (ret) {
kfree(new_mpath);
+ atomic_dec(&sdata->u.mesh.mpaths);
+ }
sdata->u.mesh.mpp_paths_generation++;
return ret;
diff --git a/net/mac80211/mlme.c b/net/mac80211/mlme.c
index 32f24dca40ab..05c7eda18196 100644
--- a/net/mac80211/mlme.c
+++ b/net/mac80211/mlme.c
@@ -452,7 +452,10 @@ ieee80211_determine_chantype(struct ieee80211_sub_if_data *sdata,
while (!cfg80211_chandef_usable(sdata->local->hw.wiphy, chandef,
tracking ? 0 :
IEEE80211_CHAN_DISABLED)) {
- if (WARN_ON(chandef->width == NL80211_CHAN_WIDTH_20_NOHT)) {
+ if (chandef->width == NL80211_CHAN_WIDTH_20_NOHT) {
+ sdata_info(sdata,
+ "unusable channel (%d MHz) for connection\n",
+ chandef->chan->center_freq);
ret = IEEE80211_CONN_DISABLE_HT |
IEEE80211_CONN_DISABLE_VHT |
IEEE80211_CONN_DISABLE_HE |
@@ -4023,6 +4026,7 @@ static bool ieee80211_assoc_config_link(struct ieee80211_link_data *link,
bool is_6ghz = cbss->channel->band == NL80211_BAND_6GHZ;
bool is_s1g = cbss->channel->band == NL80211_BAND_S1GHZ;
const struct cfg80211_bss_ies *bss_ies = NULL;
+ struct ieee802_11_elems *bss_elems = NULL;
struct ieee80211_supported_band *sband;
struct ieee802_11_elems *elems;
u16 capab_info;
@@ -4059,7 +4063,6 @@ static bool ieee80211_assoc_config_link(struct ieee80211_link_data *link,
(!(link->u.mgd.conn_flags & IEEE80211_CONN_DISABLE_VHT) &&
(!elems->vht_cap_elem || !elems->vht_operation)))) {
const struct cfg80211_bss_ies *ies;
- struct ieee802_11_elems *bss_elems;
rcu_read_lock();
ies = rcu_dereference(cbss->ies);
@@ -4115,8 +4118,6 @@ static bool ieee80211_assoc_config_link(struct ieee80211_link_data *link,
sdata_info(sdata,
"AP bug: VHT operation missing from AssocResp\n");
}
-
- kfree(bss_elems);
}
/*
@@ -4343,6 +4344,7 @@ static bool ieee80211_assoc_config_link(struct ieee80211_link_data *link,
ret = true;
out:
kfree(elems);
+ kfree(bss_elems);
kfree(bss_ies);
return ret;
}
diff --git a/net/mac80211/rc80211_minstrel_ht.c b/net/mac80211/rc80211_minstrel_ht.c
index 3d91b98db099..e1cd5799a455 100644
--- a/net/mac80211/rc80211_minstrel_ht.c
+++ b/net/mac80211/rc80211_minstrel_ht.c
@@ -7,6 +7,7 @@
#include <linux/types.h>
#include <linux/skbuff.h>
#include <linux/debugfs.h>
+#include <linux/limits.h>
#include <linux/random.h>
#include <linux/moduleparam.h>
#include <linux/ieee80211.h>
@@ -1977,14 +1978,39 @@ minstrel_ht_alloc(struct ieee80211_hw *hw)
}
#ifdef CONFIG_MAC80211_DEBUGFS
+static int minstrel_ht_fixed_rate_idx_get(void *data, u64 *val)
+{
+ *val = *(u32 *)data;
+ return 0;
+}
+
+static int minstrel_ht_fixed_rate_idx_set(void *data, u64 val)
+{
+ u32 idx = val;
+
+ /* U32_MAX is the default and keeps fixed rate processing disabled */
+ if (val != U32_MAX &&
+ (val > U16_MAX ||
+ MI_RATE_GROUP(idx) >= ARRAY_SIZE(minstrel_mcs_groups) ||
+ MI_RATE_IDX(idx) >= MCS_GROUP_RATES))
+ return -EINVAL;
+
+ *(u32 *)data = idx;
+ return 0;
+}
+
+DEFINE_DEBUGFS_ATTRIBUTE(minstrel_ht_fixed_rate_idx_fops,
+ minstrel_ht_fixed_rate_idx_get,
+ minstrel_ht_fixed_rate_idx_set, "%llu\n");
+
static void minstrel_ht_add_debugfs(struct ieee80211_hw *hw, void *priv,
struct dentry *debugfsdir)
{
struct minstrel_priv *mp = priv;
mp->fixed_rate_idx = (u32) -1;
- debugfs_create_u32("fixed_rate_idx", S_IRUGO | S_IWUGO, debugfsdir,
- &mp->fixed_rate_idx);
+ debugfs_create_file("fixed_rate_idx", S_IRUGO | S_IWUGO, debugfsdir,
+ &mp->fixed_rate_idx, &minstrel_ht_fixed_rate_idx_fops);
}
#endif
diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c
index 8a92077da153..23e66d04f6d7 100644
--- a/net/mac80211/sta_info.c
+++ b/net/mac80211/sta_info.c
@@ -122,6 +122,8 @@ static void __cleanup_single_sta(struct sta_info *sta)
struct ieee80211_local *local = sdata->local;
struct ps_data *ps;
+ cancel_work_sync(&sta->drv_deliver_wk);
+
if (test_sta_flag(sta, WLAN_STA_PS_STA) ||
test_sta_flag(sta, WLAN_STA_PS_DRIVER) ||
test_sta_flag(sta, WLAN_STA_PS_DELIVER)) {
@@ -162,8 +164,6 @@ static void __cleanup_single_sta(struct sta_info *sta)
if (ieee80211_vif_is_mesh(&sdata->vif))
mesh_sta_cleanup(sta);
- cancel_work_sync(&sta->drv_deliver_wk);
-
/*
* Destroy aggregation state here. It would be nice to wait for the
* driver to finish aggregation stop and then clean up, but for now
@@ -351,6 +351,15 @@ static void sta_info_free_link(struct link_sta_info *link_sta)
free_percpu(link_sta->pcpu_rx_stats);
}
+static void sta_link_free_rcu(struct rcu_head *head)
+{
+ struct sta_link_alloc *alloc =
+ container_of(head, struct sta_link_alloc, rcu_head);
+
+ sta_info_free_link(&alloc->info);
+ kfree(alloc);
+}
+
static void sta_remove_link(struct sta_info *sta, unsigned int link_id,
bool unhash)
{
@@ -373,10 +382,8 @@ static void sta_remove_link(struct sta_info *sta, unsigned int link_id,
sta->sta.valid_links &= ~BIT(link_id);
RCU_INIT_POINTER(sta->link[link_id], NULL);
RCU_INIT_POINTER(sta->sta.link[link_id], NULL);
- if (alloc) {
- sta_info_free_link(&alloc->info);
- kfree_rcu(alloc, rcu_head);
- }
+ if (alloc)
+ call_rcu(&alloc->rcu_head, sta_link_free_rcu);
ieee80211_sta_recalc_aggregates(&sta->sta);
}
@@ -1303,6 +1310,8 @@ static void __sta_info_destroy_part2(struct sta_info *sta)
sta->dead = true;
+ cancel_work_sync(&sta->drv_deliver_wk);
+
local->num_sta--;
local->sta_generation++;
@@ -2496,6 +2505,39 @@ static inline u64 sta_get_stats_bytes(struct ieee80211_sta_rx_stats *rxstats)
return value;
}
+#ifdef CONFIG_MAC80211_MESH
+static void sta_set_mesh_sinfo(struct sta_info *sta,
+ struct station_info *sinfo)
+{
+ struct ieee80211_local *local = sta->sdata->local;
+
+ sinfo->filled |= BIT_ULL(NL80211_STA_INFO_LLID) |
+ BIT_ULL(NL80211_STA_INFO_PLID) |
+ BIT_ULL(NL80211_STA_INFO_PLINK_STATE) |
+ BIT_ULL(NL80211_STA_INFO_LOCAL_PM) |
+ BIT_ULL(NL80211_STA_INFO_PEER_PM) |
+ BIT_ULL(NL80211_STA_INFO_NONPEER_PM) |
+ BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_GATE) |
+ BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_AS);
+
+ sinfo->llid = sta->mesh->llid;
+ sinfo->plid = sta->mesh->plid;
+ sinfo->plink_state = sta->mesh->plink_state;
+ if (test_sta_flag(sta, WLAN_STA_TOFFSET_KNOWN)) {
+ sinfo->filled |= BIT_ULL(NL80211_STA_INFO_T_OFFSET);
+ sinfo->t_offset = sta->mesh->t_offset;
+ }
+ sinfo->local_pm = sta->mesh->local_pm;
+ sinfo->peer_pm = sta->mesh->peer_pm;
+ sinfo->nonpeer_pm = sta->mesh->nonpeer_pm;
+ sinfo->connected_to_gate = sta->mesh->connected_to_gate;
+ sinfo->connected_to_as = sta->mesh->connected_to_as;
+
+ sinfo->filled |= BIT_ULL(NL80211_STA_INFO_AIRTIME_LINK_METRIC);
+ sinfo->airtime_link_metric = airtime_link_metric_get(local, sta);
+}
+#endif
+
void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo,
bool tidstats)
{
@@ -2679,31 +2721,10 @@ void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo,
sta_set_tidstats(sta, &sinfo->pertid[i], i);
}
- if (ieee80211_vif_is_mesh(&sdata->vif)) {
#ifdef CONFIG_MAC80211_MESH
- sinfo->filled |= BIT_ULL(NL80211_STA_INFO_LLID) |
- BIT_ULL(NL80211_STA_INFO_PLID) |
- BIT_ULL(NL80211_STA_INFO_PLINK_STATE) |
- BIT_ULL(NL80211_STA_INFO_LOCAL_PM) |
- BIT_ULL(NL80211_STA_INFO_PEER_PM) |
- BIT_ULL(NL80211_STA_INFO_NONPEER_PM) |
- BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_GATE) |
- BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_AS);
-
- sinfo->llid = sta->mesh->llid;
- sinfo->plid = sta->mesh->plid;
- sinfo->plink_state = sta->mesh->plink_state;
- if (test_sta_flag(sta, WLAN_STA_TOFFSET_KNOWN)) {
- sinfo->filled |= BIT_ULL(NL80211_STA_INFO_T_OFFSET);
- sinfo->t_offset = sta->mesh->t_offset;
- }
- sinfo->local_pm = sta->mesh->local_pm;
- sinfo->peer_pm = sta->mesh->peer_pm;
- sinfo->nonpeer_pm = sta->mesh->nonpeer_pm;
- sinfo->connected_to_gate = sta->mesh->connected_to_gate;
- sinfo->connected_to_as = sta->mesh->connected_to_as;
+ if (ieee80211_vif_is_mesh(&sdata->vif))
+ sta_set_mesh_sinfo(sta, sinfo);
#endif
- }
sinfo->bss_param.flags = 0;
if (sdata->vif.bss_conf.use_cts_prot)
@@ -2759,12 +2780,6 @@ void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo,
sinfo->filled |=
BIT_ULL(NL80211_STA_INFO_ACK_SIGNAL_AVG);
}
-
- if (ieee80211_vif_is_mesh(&sdata->vif)) {
- sinfo->filled |= BIT_ULL(NL80211_STA_INFO_AIRTIME_LINK_METRIC);
- sinfo->airtime_link_metric =
- airtime_link_metric_get(local, sta);
- }
}
u32 sta_get_expected_throughput(struct sta_info *sta)
diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c
index 00e8eda67f13..51cf8c4e0e90 100644
--- a/net/mac80211/tx.c
+++ b/net/mac80211/tx.c
@@ -4545,6 +4545,7 @@ static void ieee80211_8023_xmit(struct ieee80211_sub_if_data *sdata,
{
struct ieee80211_tx_info *info;
struct ieee80211_local *local = sdata->local;
+ struct ieee80211_chanctx_conf *chanctx_conf;
struct tid_ampdu_tx *tid_tx;
u8 tid;
@@ -4602,6 +4603,16 @@ static void ieee80211_8023_xmit(struct ieee80211_sub_if_data *sdata,
sdata = container_of(sdata->bss,
struct ieee80211_sub_if_data, u.ap);
+ /* MLD transmissions must not rely on the band */
+ if (!sdata->vif.valid_links) {
+ chanctx_conf = rcu_dereference(sdata->vif.bss_conf.chanctx_conf);
+ if (unlikely(!chanctx_conf)) {
+ kfree_skb(skb);
+ return;
+ }
+ info->band = chanctx_conf->def.chan->band;
+ }
+
info->flags |= IEEE80211_TX_CTL_HW_80211_ENCAP;
info->control.vif = &sdata->vif;
diff --git a/net/mctp/device.c b/net/mctp/device.c
index fd368249246d..365726010340 100644
--- a/net/mctp/device.c
+++ b/net/mctp/device.c
@@ -120,12 +120,19 @@ static int mctp_dump_addrinfo(struct sk_buff *skb, struct netlink_callback *cb)
struct net_device *dev;
struct ifaddrmsg *hdr;
struct mctp_dev *mdev;
- int ifindex;
+ int ifindex = 0;
int idx = 0, rc;
- hdr = nlmsg_data(cb->nlh);
- // filter by ifindex if requested
- ifindex = hdr->ifa_index;
+ /* Filter by ifindex if a header is provided */
+ if (cb->nlh->nlmsg_len >= nlmsg_msg_size(sizeof(*hdr))) {
+ hdr = nlmsg_data(cb->nlh);
+ ifindex = hdr->ifa_index;
+ } else {
+ if (cb->strict_check) {
+ NL_SET_ERR_MSG(cb->extack, "mctp: Invalid header for addr dump request");
+ return -EINVAL;
+ }
+ }
rcu_read_lock();
for (; mcb->h < NETDEV_HASHENTRIES; mcb->h++, mcb->idx = 0) {
@@ -352,13 +359,13 @@ static struct mctp_dev *mctp_add_dev(struct net_device *dev)
mdev->net = mctp_default_net(dev_net(dev));
- /* associate to net_device */
refcount_set(&mdev->refs, 1);
- rcu_assign_pointer(dev->mctp_ptr, mdev);
-
dev_hold(dev);
mdev->dev = dev;
+ /* Associate to net_device, publish the mdev. */
+ rcu_assign_pointer(dev->mctp_ptr, mdev);
+
return mdev;
}
diff --git a/net/mctp/route.c b/net/mctp/route.c
index 4d649ba66cfd..00fc79c05b4a 100644
--- a/net/mctp/route.c
+++ b/net/mctp/route.c
@@ -298,8 +298,14 @@ static int mctp_frag_queue(struct mctp_sk_key *key, struct sk_buff *skb)
& MCTP_HDR_SEQ_MASK;
if (!key->reasm_head) {
- key->reasm_head = skb;
- key->reasm_tailp = &(skb_shinfo(skb)->frag_list);
+ /* Since we're manipulating the shared frag_list, ensure it isn't
+ * shared with any other SKBs.
+ */
+ key->reasm_head = skb_unshare(skb, GFP_ATOMIC);
+ if (!key->reasm_head)
+ return -ENOMEM;
+
+ key->reasm_tailp = &(skb_shinfo(key->reasm_head)->frag_list);
key->last_seq = this_seq;
return 0;
}
@@ -429,18 +435,26 @@ static int mctp_route_input(struct mctp_route *route, struct sk_buff *skb)
/* we can queue without the key lock here, as the
* key isn't observable yet
*/
- mctp_frag_queue(key, skb);
-
- /* if the key_add fails, we've raced with another
- * SOM packet with the same src, dest and tag. There's
- * no way to distinguish future packets, so all we
- * can do is drop; we'll free the skb on exit from
- * this function.
- */
- rc = mctp_key_add(key, msk);
- if (!rc) {
- trace_mctp_key_acquire(key);
+ rc = mctp_frag_queue(key, skb);
+ if (rc == -ENOMEM) {
+ /* skb_unshare() has already freed the skb */
+ skb = NULL;
+ } else if (!rc) {
+ /* reasm_head owns the (possibly copied) skb */
skb = NULL;
+
+ /* if the key_add fails, we've raced with
+ * another SOM packet with the same src, dest
+ * and tag. There's no way to distinguish
+ * future packets, so all we can do is drop.
+ */
+ rc = mctp_key_add(key, msk);
+ if (!rc) {
+ trace_mctp_key_acquire(key);
+ } else {
+ kfree_skb(key->reasm_head);
+ key->reasm_head = NULL;
+ }
}
/* we don't need to release key->lock on exit, so
@@ -459,7 +473,8 @@ static int mctp_route_input(struct mctp_route *route, struct sk_buff *skb)
key = NULL;
} else {
rc = mctp_frag_queue(key, skb);
- if (!rc)
+ /* -ENOMEM: skb_unshare() already freed the skb */
+ if (!rc || rc == -ENOMEM)
skb = NULL;
}
}
diff --git a/net/mpls/af_mpls.c b/net/mpls/af_mpls.c
index 5e0900bd48b6..73443cf8b69c 100644
--- a/net/mpls/af_mpls.c
+++ b/net/mpls/af_mpls.c
@@ -74,16 +74,31 @@ static void rtmsg_lfib(int event, u32 label, struct mpls_route *rt,
struct nlmsghdr *nlh, struct net *net, u32 portid,
unsigned int nlm_flags);
-static struct mpls_route *mpls_route_input_rcu(struct net *net, unsigned index)
+static struct mpls_route __rcu **mpls_platform_label_rcu(struct net *net, size_t *platform_labels)
{
- struct mpls_route *rt = NULL;
+ struct mpls_route __rcu **platform_label;
+ unsigned int sequence;
- if (index < net->mpls.platform_labels) {
- struct mpls_route __rcu **platform_label =
- rcu_dereference_rtnl(net->mpls.platform_label);
- rt = rcu_dereference_rtnl(platform_label[index]);
- }
- return rt;
+ do {
+ sequence = read_seqcount_begin(&net->mpls.platform_label_seq);
+ platform_label = rcu_dereference_rtnl(net->mpls.platform_label);
+ *platform_labels = net->mpls.platform_labels;
+ } while (read_seqcount_retry(&net->mpls.platform_label_seq, sequence));
+
+ return platform_label;
+}
+
+static struct mpls_route *mpls_route_input_rcu(struct net *net, unsigned int index)
+{
+ struct mpls_route __rcu **platform_label;
+ size_t platform_labels;
+
+ platform_label = mpls_platform_label_rcu(net, &platform_labels);
+
+ if (index >= platform_labels)
+ return NULL;
+
+ return rcu_dereference_rtnl(platform_label[index]);
}
bool mpls_output_possible(const struct net_device *dev)
@@ -2209,8 +2224,7 @@ static int mpls_dump_routes(struct sk_buff *skb, struct netlink_callback *cb)
if (index < MPLS_LABEL_FIRST_UNRESERVED)
index = MPLS_LABEL_FIRST_UNRESERVED;
- platform_label = rtnl_dereference(net->mpls.platform_label);
- platform_labels = net->mpls.platform_labels;
+ platform_label = mpls_platform_label_rcu(net, &platform_labels);
if (filter.filter_set)
flags |= NLM_F_DUMP_FILTERED;
@@ -2597,8 +2611,12 @@ static int resize_platform_label_table(struct net *net, size_t limit)
}
/* Update the global pointers */
+ local_bh_disable();
+ write_seqcount_begin(&net->mpls.platform_label_seq);
net->mpls.platform_labels = limit;
rcu_assign_pointer(net->mpls.platform_label, labels);
+ write_seqcount_end(&net->mpls.platform_label_seq);
+ local_bh_enable();
rtnl_unlock();
@@ -2679,6 +2697,8 @@ static int mpls_net_init(struct net *net)
struct ctl_table *table;
int i;
+ seqcount_init(&net->mpls.platform_label_seq);
+
net->mpls.platform_labels = 0;
net->mpls.platform_label = NULL;
net->mpls.ip_ttl_propagate = 1;
diff --git a/net/mptcp/options.c b/net/mptcp/options.c
index 6f82517b8f6f..79c5f731a5e1 100644
--- a/net/mptcp/options.c
+++ b/net/mptcp/options.c
@@ -26,6 +26,7 @@ static void mptcp_parse_option(const struct sk_buff *skb,
{
u8 subtype = *ptr >> 4;
int expected_opsize;
+ u16 subopt;
u8 version;
u8 flags;
u8 i;
@@ -38,11 +39,23 @@ static void mptcp_parse_option(const struct sk_buff *skb,
expected_opsize = TCPOLEN_MPTCP_MPC_ACK_DATA;
else
expected_opsize = TCPOLEN_MPTCP_MPC_ACK;
+ subopt = OPTION_MPTCP_MPC_ACK;
} else {
- if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_ACK)
+ if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_ACK) {
expected_opsize = TCPOLEN_MPTCP_MPC_SYNACK;
- else
+ subopt = OPTION_MPTCP_MPC_SYNACK;
+ } else {
expected_opsize = TCPOLEN_MPTCP_MPC_SYN;
+ subopt = OPTION_MPTCP_MPC_SYN;
+ }
+ }
+
+ /* Only the MPC + ACK can be used with a RM_ADDR */
+ if (subopt == OPTION_MPTCP_MPC_ACK) {
+ if ((mp_opt->suboptions & ~OPTION_MPTCP_RM_ADDR) != 0)
+ break;
+ } else if (mp_opt->suboptions != 0) {
+ break;
}
/* Cfr RFC 8684 Section 3.3.0:
@@ -86,7 +99,7 @@ static void mptcp_parse_option(const struct sk_buff *skb,
mp_opt->deny_join_id0 = !!(flags & MPTCP_CAP_DENY_JOIN_ID0);
- mp_opt->suboptions |= OPTIONS_MPTCP_MPC;
+ mp_opt->suboptions |= subopt;
if (opsize >= TCPOLEN_MPTCP_MPC_SYNACK) {
mp_opt->sndr_key = get_unaligned_be64(ptr);
ptr += 8;
@@ -118,6 +131,11 @@ static void mptcp_parse_option(const struct sk_buff *skb,
break;
case MPTCPOPT_MP_JOIN:
+ /* Can be used with a restricted number of other options */
+ if ((mp_opt->suboptions & ~(OPTION_MPTCP_RM_ADDR |
+ OPTION_MPTCP_PRIO)) != 0)
+ break;
+
if (opsize == TCPOLEN_MPTCP_MPJ_SYN) {
mp_opt->suboptions |= OPTION_MPTCP_MPJ_SYN;
mp_opt->backup = *ptr++ & MPTCPOPT_BACKUP;
@@ -149,6 +167,14 @@ static void mptcp_parse_option(const struct sk_buff *skb,
break;
case MPTCPOPT_DSS:
+ /* Can be used with a restricted number of other options */
+ if ((mp_opt->suboptions & ~(OPTION_MPTCP_ADD_ADDR |
+ OPTION_MPTCP_RM_ADDR |
+ OPTION_MPTCP_PRIO |
+ OPTION_MPTCP_FASTCLOSE |
+ OPTION_MPTCP_FAIL)) != 0)
+ break;
+
pr_debug("DSS\n");
ptr++;
@@ -236,6 +262,12 @@ static void mptcp_parse_option(const struct sk_buff *skb,
break;
case MPTCPOPT_ADD_ADDR:
+ /* Can be used with a restricted number of other options */
+ if ((mp_opt->suboptions & ~(OPTIONS_MPTCP_DSS |
+ OPTION_MPTCP_RM_ADDR |
+ OPTION_MPTCP_PRIO)) != 0)
+ break;
+
mp_opt->echo = (*ptr++) & MPTCP_ADDR_ECHO;
if (!mp_opt->echo) {
if (opsize == TCPOLEN_MPTCP_ADD_ADDR ||
@@ -295,6 +327,14 @@ static void mptcp_parse_option(const struct sk_buff *skb,
break;
case MPTCPOPT_RM_ADDR:
+ /* Can be used with a restricted number of other options */
+ if ((mp_opt->suboptions & ~(OPTION_MPTCP_MPC_ACK |
+ OPTIONS_MPTCP_MPJ |
+ OPTIONS_MPTCP_DSS |
+ OPTION_MPTCP_ADD_ADDR |
+ OPTION_MPTCP_PRIO)) != 0)
+ break;
+
if (opsize < TCPOLEN_MPTCP_RM_ADDR_BASE + 1 ||
opsize > TCPOLEN_MPTCP_RM_ADDR_BASE + MPTCP_RM_IDS_MAX)
break;
@@ -309,6 +349,13 @@ static void mptcp_parse_option(const struct sk_buff *skb,
break;
case MPTCPOPT_MP_PRIO:
+ /* Can be used with a restricted number of other options */
+ if ((mp_opt->suboptions & ~(OPTIONS_MPTCP_MPJ |
+ OPTIONS_MPTCP_DSS |
+ OPTION_MPTCP_ADD_ADDR |
+ OPTION_MPTCP_RM_ADDR)) != 0)
+ break;
+
if (opsize != TCPOLEN_MPTCP_PRIO)
break;
@@ -318,6 +365,11 @@ static void mptcp_parse_option(const struct sk_buff *skb,
break;
case MPTCPOPT_MP_FASTCLOSE:
+ /* Can be used with a restricted number of other options */
+ if ((mp_opt->suboptions & ~(OPTIONS_MPTCP_DSS |
+ OPTION_MPTCP_RST)) != 0)
+ break;
+
if (opsize != TCPOLEN_MPTCP_FASTCLOSE)
break;
@@ -329,6 +381,11 @@ static void mptcp_parse_option(const struct sk_buff *skb,
break;
case MPTCPOPT_RST:
+ /* Can be used with a restricted number of other options */
+ if ((mp_opt->suboptions & ~(OPTION_MPTCP_FAIL |
+ OPTION_MPTCP_FASTCLOSE)) != 0)
+ break;
+
if (opsize != TCPOLEN_MPTCP_RST)
break;
@@ -344,6 +401,11 @@ static void mptcp_parse_option(const struct sk_buff *skb,
break;
case MPTCPOPT_MP_FAIL:
+ /* Can be used with a restricted number of other options */
+ if ((mp_opt->suboptions & ~(OPTIONS_MPTCP_DSS |
+ OPTION_MPTCP_RST)) != 0)
+ break;
+
if (opsize != TCPOLEN_MPTCP_FAIL)
break;
@@ -1416,7 +1478,7 @@ void mptcp_write_options(struct tcphdr *th, __be32 *ptr, struct tcp_sock *tp,
* RM | C | C | C | P |------|------|------|------|
* PRIO | X | C | C | C | C |------|------|------|
* FAIL | X | X | C | X | X | X |------|------|
- * FC | X | X | X | X | X | X | X |------|
+ * FC | X | X | P | X | X | X | X |------|
* RST | X | X | X | X | X | X | O | O |
* ------|------|------|------|------|------|------|------|------|
*
diff --git a/net/mptcp/pm_netlink.c b/net/mptcp/pm_netlink.c
index 0d8316bc5077..05fb4d43187a 100644
--- a/net/mptcp/pm_netlink.c
+++ b/net/mptcp/pm_netlink.c
@@ -428,10 +428,10 @@ bool mptcp_pm_alloc_anno_list(struct mptcp_sock *msk,
add_entry->addr = *addr;
add_entry->sock = msk;
- add_entry->retrans_times = 0;
timer_setup(&add_entry->add_timer, mptcp_pm_add_timer, 0);
reset_timer:
+ add_entry->retrans_times = 0;
add_entry->timer_done = false;
timeout = mptcp_get_add_addr_timeout(net);
if (timeout)
@@ -1721,6 +1721,8 @@ static int mptcp_nl_remove_id_zero_address(struct net *net,
while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) {
struct sock *sk = (struct sock *)msk;
struct mptcp_addr_info msk_local;
+ struct mptcp_addr_info anno_addr;
+ bool announced;
if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk))
goto next;
@@ -1730,7 +1732,13 @@ static int mptcp_nl_remove_id_zero_address(struct net *net,
goto next;
lock_sock(sk);
+ /* Drop a possibly pending ADD_ADDR for this address. */
+ anno_addr = msk_local;
+ anno_addr.port = 0;
+ announced = remove_anno_list_by_saddr(msk, &anno_addr);
spin_lock_bh(&msk->pm.lock);
+ if (announced)
+ msk->pm.add_addr_signaled--;
mptcp_pm_remove_addr(msk, &list);
mptcp_pm_nl_rm_subflow_received(msk, &list);
__mark_subflow_endp_available(msk, 0);
diff --git a/net/mptcp/pm_userspace.c b/net/mptcp/pm_userspace.c
index 52ab9a3f070a..a455303d2ed6 100644
--- a/net/mptcp/pm_userspace.c
+++ b/net/mptcp/pm_userspace.c
@@ -43,7 +43,6 @@ static int mptcp_userspace_pm_append_new_local_addr(struct mptcp_sock *msk,
bool needs_id)
{
DECLARE_BITMAP(id_bitmap, MPTCP_PM_MAX_ADDR_ID + 1);
- struct mptcp_pm_addr_entry *match = NULL;
struct sock *sk = (struct sock *)msk;
struct mptcp_pm_addr_entry *e;
bool addr_match = false;
@@ -62,37 +61,43 @@ static int mptcp_userspace_pm_append_new_local_addr(struct mptcp_sock *msk,
if (addr_match && entry->addr.id == 0 && needs_id)
entry->addr.id = e->addr.id;
id_match = (e->addr.id == entry->addr.id);
- if (addr_match && id_match) {
- match = e;
+ if (addr_match || id_match)
break;
- } else if (addr_match || id_match) {
- break;
- }
__set_bit(e->addr.id, id_bitmap);
}
- if (!match && !addr_match && !id_match) {
+ if (!addr_match && !id_match) {
+ unsigned int id;
+
+ if (!entry->addr.id && needs_id) {
+ id = find_next_zero_bit(id_bitmap,
+ MPTCP_PM_MAX_ADDR_ID + 1, 1);
+ if (id > MPTCP_PM_MAX_ADDR_ID) {
+ ret = -ENOSPC;
+ goto append_err;
+ }
+ } else {
+ id = entry->addr.id;
+ }
+
/* Memory for the entry is allocated from the
* sock option buffer.
*/
- e = sock_kmalloc(sk, sizeof(*e), GFP_ATOMIC);
+ e = sock_kmemdup(sk, entry, sizeof(*entry), GFP_ATOMIC);
if (!e) {
- spin_unlock_bh(&msk->pm.lock);
- return -ENOMEM;
+ ret = -ENOMEM;
+ goto append_err;
}
- *e = *entry;
- if (!e->addr.id && needs_id)
- e->addr.id = find_next_zero_bit(id_bitmap,
- MPTCP_PM_MAX_ADDR_ID + 1,
- 1);
+ e->addr.id = id;
list_add_tail_rcu(&e->list, &msk->pm.userspace_pm_local_addr_list);
msk->pm.local_addr_used++;
ret = e->addr.id;
- } else if (match) {
+ } else if (addr_match && id_match) {
ret = entry->addr.id;
}
+append_err:
spin_unlock_bh(&msk->pm.lock);
return ret;
}
diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c
index 10be901f8c02..bdace644dbc6 100644
--- a/net/mptcp/protocol.c
+++ b/net/mptcp/protocol.c
@@ -426,9 +426,7 @@ static bool __mptcp_move_skb(struct mptcp_sock *msk, struct sock *ssk,
static void mptcp_stop_rtx_timer(struct sock *sk)
{
- struct inet_connection_sock *icsk = inet_csk(sk);
-
- sk_stop_timer(sk, &icsk->icsk_retransmit_timer);
+ sk_stop_timer(sk, &sk->mptcp_retransmit_timer);
mptcp_sk(sk)->timer_ival = 0;
}
@@ -932,13 +930,6 @@ static bool __mptcp_finish_join(struct mptcp_sock *msk, struct sock *ssk)
if (sk->sk_state != TCP_ESTABLISHED)
return false;
- /* The caller possibly is not holding the msk socket lock, but
- * in the fallback case only the current subflow is touching
- * the OoO queue.
- */
- if (!RB_EMPTY_ROOT(&msk->out_of_order_queue))
- return false;
-
spin_lock_bh(&msk->fallback_lock);
if (!msk->allow_subflows) {
spin_unlock_bh(&msk->fallback_lock);
@@ -976,12 +967,11 @@ static void __mptcp_flush_join_list(struct sock *sk, struct list_head *join_list
static bool mptcp_rtx_timer_pending(struct sock *sk)
{
- return timer_pending(&inet_csk(sk)->icsk_retransmit_timer);
+ return timer_pending(&sk->mptcp_retransmit_timer);
}
static void mptcp_reset_rtx_timer(struct sock *sk)
{
- struct inet_connection_sock *icsk = inet_csk(sk);
unsigned long tout;
/* prevent rescheduling on close */
@@ -989,7 +979,7 @@ static void mptcp_reset_rtx_timer(struct sock *sk)
return;
tout = mptcp_sk(sk)->timer_ival;
- sk_reset_timer(sk, &icsk->icsk_retransmit_timer, jiffies + tout);
+ sk_reset_timer(sk, &sk->mptcp_retransmit_timer, jiffies + tout);
}
bool mptcp_schedule_work(struct sock *sk)
@@ -2349,9 +2339,7 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len,
static void mptcp_retransmit_timer(struct timer_list *t)
{
- struct inet_connection_sock *icsk = from_timer(icsk, t,
- icsk_retransmit_timer);
- struct sock *sk = &icsk->icsk_inet.sk;
+ struct sock *sk = from_timer(sk, t, mptcp_retransmit_timer);
struct mptcp_sock *msk = mptcp_sk(sk);
bh_lock_sock(sk);
@@ -2369,7 +2357,9 @@ static void mptcp_retransmit_timer(struct timer_list *t)
static void mptcp_tout_timer(struct timer_list *t)
{
- struct sock *sk = from_timer(sk, t, sk_timer);
+ struct inet_connection_sock *icsk =
+ from_timer(icsk, t, mptcp_tout_timer);
+ struct sock *sk = &icsk->icsk_inet.sk;
mptcp_schedule_work(sk);
sock_put(sk);
@@ -2801,7 +2791,7 @@ void mptcp_reset_tout_timer(struct mptcp_sock *msk, unsigned long fail_tout)
*/
timeout = inet_csk(sk)->icsk_mtup.probe_timestamp ? close_timeout : fail_tout;
- sk_reset_timer(sk, &sk->sk_timer, timeout);
+ sk_reset_timer(sk, &inet_csk(sk)->mptcp_tout_timer, timeout);
}
static void mptcp_mp_fail_no_response(struct mptcp_sock *msk)
@@ -2919,6 +2909,8 @@ static int __mptcp_init_sock(struct sock *sk)
msk->first_pending = NULL;
msk->rmem_fwd_alloc = 0;
WRITE_ONCE(msk->rmem_released, 0);
+ /* MPTCP sockets do not go through tcp_init_sock(). */
+ inet_csk(sk)->icsk_rto_min = TCP_RTO_MIN;
msk->timer_ival = TCP_RTO_MIN;
WRITE_ONCE(msk->first, NULL);
@@ -2931,9 +2923,8 @@ static int __mptcp_init_sock(struct sock *sk)
mptcp_pm_data_init(msk);
spin_lock_init(&msk->fallback_lock);
- /* re-use the csk retrans timer for MPTCP-level retrans */
- timer_setup(&msk->sk.icsk_retransmit_timer, mptcp_retransmit_timer, 0);
- timer_setup(&sk->sk_timer, mptcp_tout_timer, 0);
+ timer_setup(&sk->mptcp_retransmit_timer, mptcp_retransmit_timer, 0);
+ timer_setup(&msk->sk.mptcp_tout_timer, mptcp_tout_timer, 0);
return 0;
}
@@ -3110,7 +3101,7 @@ static void __mptcp_destroy_sock(struct sock *sk)
might_sleep();
mptcp_stop_rtx_timer(sk);
- sk_stop_timer(sk, &sk->sk_timer);
+ sk_stop_timer(sk, &inet_csk(sk)->mptcp_tout_timer);
msk->pm.status = 0;
sk->sk_prot->destroy(sk);
@@ -3271,6 +3262,7 @@ static void mptcp_copy_inaddrs(struct sock *msk, const struct sock *ssk)
static int mptcp_disconnect(struct sock *sk, int flags)
{
+ struct inet_connection_sock *icsk = inet_csk(sk);
struct mptcp_sock *msk = mptcp_sk(sk);
/* We are on the fastopen error path. We can't call straight into the
@@ -3283,8 +3275,13 @@ static int mptcp_disconnect(struct sock *sk, int flags)
mptcp_check_listen_stop(sk);
inet_sk_state_store(sk, TCP_CLOSE);
- mptcp_stop_rtx_timer(sk);
- mptcp_stop_tout_timer(sk);
+ /* The later subflow close can not kick again the tout timer,
+ * as the msk is already in closed status.
+ */
+ msk->timer_ival = TCP_RTO_MIN;
+ sk_stop_timer_sync(sk, &sk->mptcp_retransmit_timer);
+ icsk->icsk_mtup.probe_timestamp = 0;
+ sk_stop_timer_sync(sk, &icsk->mptcp_tout_timer);
if (mptcp_sk(sk)->token)
mptcp_event(MPTCP_EVENT_CLOSED, mptcp_sk(sk), NULL, GFP_KERNEL);
diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h
index 1de257b86839..31ea91b45363 100644
--- a/net/mptcp/protocol.h
+++ b/net/mptcp/protocol.h
@@ -36,6 +36,7 @@
OPTION_MPTCP_MPC_ACK)
#define OPTIONS_MPTCP_MPJ (OPTION_MPTCP_MPJ_SYN | OPTION_MPTCP_MPJ_SYNACK | \
OPTION_MPTCP_MPJ_ACK)
+#define OPTIONS_MPTCP_DSS (OPTION_MPTCP_DSS | OPTION_MPTCP_CSUMREQD)
/* MPTCP option subtypes */
#define MPTCPOPT_MP_CAPABLE 0
@@ -698,7 +699,7 @@ static inline void mptcp_stop_tout_timer(struct sock *sk)
if (!inet_csk(sk)->icsk_mtup.probe_timestamp)
return;
- sk_stop_timer(sk, &sk->sk_timer);
+ sk_stop_timer(sk, &inet_csk(sk)->mptcp_tout_timer);
inet_csk(sk)->icsk_mtup.probe_timestamp = 0;
}
@@ -1004,6 +1005,14 @@ static inline bool __mptcp_try_fallback(struct mptcp_sock *msk)
pr_debug("TCP fallback already done (msk=%p)\n", msk);
return true;
}
+
+ /* The caller possibly is not holding the msk socket lock, but
+ * in the fallback case only the current subflow is touching
+ * the OoO queue.
+ */
+ if (!RB_EMPTY_ROOT(&msk->out_of_order_queue))
+ return false;
+
spin_lock_bh(&msk->fallback_lock);
if (!msk->allow_infinite_fallback) {
spin_unlock_bh(&msk->fallback_lock);
diff --git a/net/ncsi/ncsi-manage.c b/net/ncsi/ncsi-manage.c
index 6a58ae8b5d03..82030281e60a 100644
--- a/net/ncsi/ncsi-manage.c
+++ b/net/ncsi/ncsi-manage.c
@@ -1967,6 +1967,8 @@ void ncsi_unregister_dev(struct ncsi_dev *nd)
list_del_rcu(&ndp->node);
spin_unlock_irqrestore(&ncsi_dev_lock, flags);
+ cancel_work_sync(&ndp->work);
+
kfree(ndp);
}
EXPORT_SYMBOL_GPL(ncsi_unregister_dev);
diff --git a/net/netfilter/core.c b/net/netfilter/core.c
index edf92074221e..50490bae3623 100644
--- a/net/netfilter/core.c
+++ b/net/netfilter/core.c
@@ -627,10 +627,10 @@ int nf_hook_slow(struct sk_buff *skb, struct nf_hook_state *state,
if (ret == 1)
continue;
return ret;
+ case NF_STOLEN:
+ return NF_DROP_GETERR(verdict);
default:
- /* Implicit handling for NF_STOLEN, as well as any other
- * non conventional verdicts.
- */
+ WARN_ON_ONCE(1);
return 0;
}
}
diff --git a/net/netfilter/ipset/ip_set_bitmap_gen.h b/net/netfilter/ipset/ip_set_bitmap_gen.h
index 605d54d2bd7e..3dc2ebbe4800 100644
--- a/net/netfilter/ipset/ip_set_bitmap_gen.h
+++ b/net/netfilter/ipset/ip_set_bitmap_gen.h
@@ -157,7 +157,7 @@ mtype_add(struct ip_set *set, void *value, const struct ip_set_ext *ext,
if (SET_WITH_COUNTER(set))
ip_set_init_counter(ext_counter(x, set), ext);
- if (SET_WITH_COMMENT(set))
+ if (SET_WITH_COMMENT(set) && !ext->target)
ip_set_init_comment(set, ext_comment(x, set), ext);
if (SET_WITH_SKBINFO(set))
ip_set_init_skbinfo(ext_skbinfo(x, set), ext);
diff --git a/net/netfilter/ipvs/ip_vs_conn.c b/net/netfilter/ipvs/ip_vs_conn.c
index 44ebc7b1c813..50a768bf83bc 100644
--- a/net/netfilter/ipvs/ip_vs_conn.c
+++ b/net/netfilter/ipvs/ip_vs_conn.c
@@ -591,6 +591,9 @@ ip_vs_bind_dest(struct ip_vs_conn *cp, struct ip_vs_dest *dest)
if (cp->protocol != IPPROTO_UDP)
conn_flags &= ~IP_VS_CONN_F_ONE_PACKET;
flags = cp->flags;
+ /* Only visible templates can control multiple connections */
+ if (flags & IP_VS_CONN_F_TEMPLATE)
+ conn_flags &= ~IP_VS_CONN_F_ONE_PACKET;
/* Bind with the destination and its corresponding transmitter */
if (flags & IP_VS_CONN_F_SYNC) {
/* Synced conns are hashed, so they can not get this flag */
diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c
index 41f09e29d221..dec212507ba9 100644
--- a/net/netfilter/ipvs/ip_vs_core.c
+++ b/net/netfilter/ipvs/ip_vs_core.c
@@ -745,7 +745,7 @@ static int ip_vs_route_me_harder(struct netns_ipvs *ipvs, int af,
* Packet has been made sufficiently writable in caller
* - inout: 1=in->out, 0=out->in
*/
-void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
+bool ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct ip_vs_conn *cp, int inout, unsigned int toff,
bool has_ports, struct ip_vs_iphdr *ciph)
{
@@ -753,6 +753,11 @@ void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
struct icmphdr *icmph = (struct icmphdr *)(skb->data + toff);
struct iphdr *cih = (struct iphdr *)(icmph + 1);
+ /* Before now we may used ihl from skb frag, revalidate it after
+ * copying it into skb head to prevent out-of-bounds access
+ */
+ if (cih->ihl * 4 != ciph->len - ciph->off)
+ return false;
if (inout) {
iph->saddr = cp->vaddr.ip;
ip_send_check(iph);
@@ -786,6 +791,7 @@ void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp,
else
IP_VS_DBG_PKT(11, AF_INET, pp, skb, ciph->off,
"Forwarding altered incoming ICMP");
+ return true;
}
#ifdef CONFIG_IP_VS_IPV6
@@ -877,7 +883,8 @@ static int handle_response_icmp(int af, struct sk_buff *skb,
ip_vs_nat_icmp_v6(skb, pp, cp, 1, toff, has_ports, ciph);
else
#endif
- ip_vs_nat_icmp(skb, pp, cp, 1, toff, has_ports, ciph);
+ if (!ip_vs_nat_icmp(skb, pp, cp, 1, toff, has_ports, ciph))
+ goto out;
if (ip_vs_route_me_harder(cp->ipvs, af, skb, hooknum))
goto out;
diff --git a/net/netfilter/ipvs/ip_vs_lblc.c b/net/netfilter/ipvs/ip_vs_lblc.c
index e2480f33a196..49ff1f9fe147 100644
--- a/net/netfilter/ipvs/ip_vs_lblc.c
+++ b/net/netfilter/ipvs/ip_vs_lblc.c
@@ -205,7 +205,11 @@ ip_vs_lblc_new(struct ip_vs_lblc_table *tbl, const union nf_inet_addr *daddr,
if (en->dest == dest)
return en;
ip_vs_lblc_del(en);
+ atomic_dec(&tbl->entries);
}
+ if (atomic_read(&tbl->entries) >= tbl->max_size * 3 / 2)
+ return NULL;
+
en = kmalloc(sizeof(*en), GFP_ATOMIC);
if (!en)
return NULL;
diff --git a/net/netfilter/ipvs/ip_vs_lblcr.c b/net/netfilter/ipvs/ip_vs_lblcr.c
index 7469547c8a44..6feab18861e1 100644
--- a/net/netfilter/ipvs/ip_vs_lblcr.c
+++ b/net/netfilter/ipvs/ip_vs_lblcr.c
@@ -365,6 +365,9 @@ ip_vs_lblcr_new(struct ip_vs_lblcr_table *tbl, const union nf_inet_addr *daddr,
en = ip_vs_lblcr_get(af, tbl, daddr);
if (!en) {
+ if (atomic_read(&tbl->entries) >= tbl->max_size * 3 / 2)
+ return NULL;
+
en = kmalloc(sizeof(*en), GFP_ATOMIC);
if (!en)
return NULL;
diff --git a/net/netfilter/ipvs/ip_vs_sync.c b/net/netfilter/ipvs/ip_vs_sync.c
index 56e371b18fc2..deac3c1715da 100644
--- a/net/netfilter/ipvs/ip_vs_sync.c
+++ b/net/netfilter/ipvs/ip_vs_sync.c
@@ -950,6 +950,21 @@ static void ip_vs_proc_conn(struct netns_ipvs *ipvs, struct ip_vs_conn_param *pa
ip_vs_conn_put(cp);
}
+/* Check for incompatible flags */
+static bool ip_vs_sync_validate_flags(u32 flags)
+{
+ /* We do not expect NO_CPORT, especially to allow lookups
+ * to hit templates
+ */
+ if (flags & IP_VS_CONN_F_NO_CPORT) {
+ if (flags & IP_VS_CONN_F_TEMPLATE)
+ return false;
+ }
+ if (flags & IP_VS_CONN_F_ONE_PACKET)
+ return false;
+ return true;
+}
+
/*
* Process received multicast message for Version 0
*/
@@ -973,8 +988,7 @@ static void ip_vs_process_message_v0(struct netns_ipvs *ipvs, const char *buffer
return;
}
s = (struct ip_vs_sync_conn_v0 *) p;
- flags = ntohs(s->flags) | IP_VS_CONN_F_SYNC;
- flags &= ~IP_VS_CONN_F_HASHED;
+ flags = ntohs(s->flags);
if (flags & IP_VS_CONN_F_SEQ_MASK) {
opt = (struct ip_vs_sync_conn_options *)&s[1];
p += FULL_CONN_SIZE;
@@ -987,6 +1001,13 @@ static void ip_vs_process_message_v0(struct netns_ipvs *ipvs, const char *buffer
p += SIMPLE_CONN_SIZE;
}
+ if (!ip_vs_sync_validate_flags(flags)) {
+ IP_VS_DBG(2, "BACKUP v0, Invalid flags 0x%X\n", flags);
+ continue;
+ }
+ flags &= IP_VS_CONN_F_BACKUP_MASK;
+ flags |= IP_VS_CONN_F_SYNC;
+
state = ntohs(s->state);
if (!(flags & IP_VS_CONN_F_TEMPLATE)) {
pp = ip_vs_proto_get(s->protocol);
@@ -1142,7 +1163,13 @@ static inline int ip_vs_proc_sync_conn(struct netns_ipvs *ipvs, __u8 *p, __u8 *m
}
/* Get flags and Mask off unsupported */
- flags = ntohl(s->v4.flags) & IP_VS_CONN_F_BACKUP_MASK;
+ flags = ntohl(s->v4.flags);
+ if (!ip_vs_sync_validate_flags(flags)) {
+ IP_VS_DBG(3, "BACKUP, Invalid flags 0x%X\n", flags);
+ retc = 25;
+ goto out;
+ }
+ flags &= IP_VS_CONN_F_BACKUP_MASK;
flags |= IP_VS_CONN_F_SYNC;
state = ntohs(s->v4.state);
diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c
index 79d6e3b4e2c0..cbef6ca5088a 100644
--- a/net/netfilter/ipvs/ip_vs_xmit.c
+++ b/net/netfilter/ipvs/ip_vs_xmit.c
@@ -306,6 +306,12 @@ static inline bool decrement_ttl(struct netns_ipvs *ipvs,
return true;
}
+/* rt has device that is down */
+static bool rt_dev_is_down(const struct net_device *dev)
+{
+ return dev && !netif_running(dev);
+}
+
/* Get route to destination or remote server */
static int
__ip_vs_get_out_rt(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb,
@@ -321,9 +327,11 @@ __ip_vs_get_out_rt(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb,
if (dest) {
dest_dst = __ip_vs_dst_check(dest);
- if (likely(dest_dst))
+ if (likely(dest_dst)) {
rt = (struct rtable *) dest_dst->dst_cache;
- else {
+ if (ret_saddr)
+ *ret_saddr = dest_dst->dst_saddr.ip;
+ } else {
dest_dst = ip_vs_dest_dst_alloc();
spin_lock_bh(&dest->dst_lock);
if (!dest_dst) {
@@ -339,14 +347,22 @@ __ip_vs_get_out_rt(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb,
ip_vs_dest_dst_free(dest_dst);
goto err_unreach;
}
- __ip_vs_dst_set(dest, dest_dst, &rt->dst, 0);
+ /* It is forbidden to attach dest->dest_dst if
+ * device is going down.
+ */
+ if (!rt_dev_is_down(dst_dev_rcu(&rt->dst)))
+ __ip_vs_dst_set(dest, dest_dst, &rt->dst, 0);
+ else
+ noref = 0;
spin_unlock_bh(&dest->dst_lock);
IP_VS_DBG(10, "new dst %pI4, src %pI4, refcnt=%d\n",
&dest->addr.ip, &dest_dst->dst_saddr.ip,
atomic_read(&rt->dst.__refcnt));
+ if (ret_saddr)
+ *ret_saddr = dest_dst->dst_saddr.ip;
+ if (!noref)
+ ip_vs_dest_dst_free(dest_dst);
}
- if (ret_saddr)
- *ret_saddr = dest_dst->dst_saddr.ip;
} else {
noref = 0;
@@ -483,9 +499,11 @@ __ip_vs_get_out_rt_v6(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb,
if (dest) {
dest_dst = __ip_vs_dst_check(dest);
- if (likely(dest_dst))
+ if (likely(dest_dst)) {
rt = dst_rt6_info(dest_dst->dst_cache);
- else {
+ if (ret_saddr)
+ *ret_saddr = dest_dst->dst_saddr.in6;
+ } else {
u32 cookie;
dest_dst = ip_vs_dest_dst_alloc();
@@ -506,14 +524,22 @@ __ip_vs_get_out_rt_v6(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb,
}
rt = dst_rt6_info(dst);
cookie = rt6_get_cookie(rt);
- __ip_vs_dst_set(dest, dest_dst, &rt->dst, cookie);
+ /* It is forbidden to attach dest->dest_dst if
+ * device is going down.
+ */
+ if (!rt_dev_is_down(dst_dev_rcu(&rt->dst)))
+ __ip_vs_dst_set(dest, dest_dst, &rt->dst, cookie);
+ else
+ noref = 0;
spin_unlock_bh(&dest->dst_lock);
IP_VS_DBG(10, "new dst %pI6, src %pI6, refcnt=%d\n",
&dest->addr.in6, &dest_dst->dst_saddr.in6,
atomic_read(&rt->dst.__refcnt));
+ if (ret_saddr)
+ *ret_saddr = dest_dst->dst_saddr.in6;
+ if (!noref)
+ ip_vs_dest_dst_free(dest_dst);
}
- if (ret_saddr)
- *ret_saddr = dest_dst->dst_saddr.in6;
} else {
noref = 0;
dst = __ip_vs_route_output_v6(net, daddr, ret_saddr, do_xfrm,
@@ -1589,7 +1615,8 @@ ip_vs_icmp_xmit(struct sk_buff *skb, struct ip_vs_conn *cp,
if (skb_cow(skb, rt->dst.dev->hard_header_len))
goto tx_error;
- ip_vs_nat_icmp(skb, pp, cp, 0, toff, has_ports, ciph);
+ if (!ip_vs_nat_icmp(skb, pp, cp, 0, toff, has_ports, ciph))
+ goto tx_error;
/* Another hack: avoid icmp_send in ip_fragment */
skb->ignore_df = 1;
diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c
index 816283f0aa59..2cea115c9c9d 100644
--- a/net/netfilter/nf_conntrack_bpf.c
+++ b/net/netfilter/nf_conntrack_bpf.c
@@ -57,6 +57,15 @@ enum {
NF_BPF_CT_OPTS_SZ = 12,
};
+static void *bpf_ct_opts_result(struct bpf_ct_opts *opts, u32 opts__sz, void *ret)
+{
+ if (!IS_ERR(ret))
+ return ret;
+ if (opts && opts__sz >= offsetofend(struct bpf_ct_opts, error))
+ opts->error = PTR_ERR(ret);
+ return NULL;
+}
+
static int bpf_nf_ct_tuple_parse(struct bpf_sock_tuple *bpf_tuple,
u32 tuple_len, u8 protonum, u8 dir,
struct nf_conntrack_tuple *tuple)
@@ -104,27 +113,32 @@ __bpf_nf_ct_alloc_entry(struct net *net, struct bpf_sock_tuple *bpf_tuple,
{
struct nf_conntrack_tuple otuple, rtuple;
struct nf_conn *ct;
+ s32 netns_id;
+ u8 l4proto;
int err;
- if (!opts || !bpf_tuple || opts->reserved[0] || opts->reserved[1] ||
- opts_len != NF_BPF_CT_OPTS_SZ)
+ if (!opts || !bpf_tuple || READ_ONCE(opts->reserved[0]) ||
+ READ_ONCE(opts->reserved[1]) || opts_len != NF_BPF_CT_OPTS_SZ)
return ERR_PTR(-EINVAL);
- if (unlikely(opts->netns_id < BPF_F_CURRENT_NETNS))
+ netns_id = READ_ONCE(opts->netns_id);
+ l4proto = READ_ONCE(opts->l4proto);
+
+ if (unlikely(netns_id < BPF_F_CURRENT_NETNS))
return ERR_PTR(-EINVAL);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_ORIGINAL, &otuple);
if (err < 0)
return ERR_PTR(err);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_REPLY, &rtuple);
if (err < 0)
return ERR_PTR(err);
- if (opts->netns_id >= 0) {
- net = get_net_ns_by_id(net, opts->netns_id);
+ if (netns_id >= 0) {
+ net = get_net_ns_by_id(net, netns_id);
if (unlikely(!net))
return ERR_PTR(-ENONET);
}
@@ -138,7 +152,7 @@ __bpf_nf_ct_alloc_entry(struct net *net, struct bpf_sock_tuple *bpf_tuple,
__nf_ct_set_timeout(ct, timeout * HZ);
out:
- if (opts->netns_id >= 0)
+ if (netns_id >= 0)
put_net(net);
return ct;
@@ -152,29 +166,34 @@ static struct nf_conn *__bpf_nf_ct_lookup(struct net *net,
struct nf_conntrack_tuple_hash *hash;
struct nf_conntrack_tuple tuple;
struct nf_conn *ct;
+ s32 netns_id;
+ u8 l4proto;
int err;
- if (!opts || !bpf_tuple || opts->reserved[0] || opts->reserved[1] ||
- opts_len != NF_BPF_CT_OPTS_SZ)
+ if (!opts || !bpf_tuple || READ_ONCE(opts->reserved[0]) ||
+ READ_ONCE(opts->reserved[1]) || opts_len != NF_BPF_CT_OPTS_SZ)
return ERR_PTR(-EINVAL);
- if (unlikely(opts->l4proto != IPPROTO_TCP && opts->l4proto != IPPROTO_UDP))
+
+ netns_id = READ_ONCE(opts->netns_id);
+ l4proto = READ_ONCE(opts->l4proto);
+ if (unlikely(l4proto != IPPROTO_TCP && l4proto != IPPROTO_UDP))
return ERR_PTR(-EPROTO);
- if (unlikely(opts->netns_id < BPF_F_CURRENT_NETNS))
+ if (unlikely(netns_id < BPF_F_CURRENT_NETNS))
return ERR_PTR(-EINVAL);
- err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto,
+ err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto,
IP_CT_DIR_ORIGINAL, &tuple);
if (err < 0)
return ERR_PTR(err);
- if (opts->netns_id >= 0) {
- net = get_net_ns_by_id(net, opts->netns_id);
+ if (netns_id >= 0) {
+ net = get_net_ns_by_id(net, netns_id);
if (unlikely(!net))
return ERR_PTR(-ENONET);
}
hash = nf_conntrack_find_get(net, &nf_ct_zone_dflt, &tuple);
- if (opts->netns_id >= 0)
+ if (netns_id >= 0)
put_net(net);
if (!hash)
return ERR_PTR(-ENOENT);
@@ -261,13 +280,7 @@ bpf_xdp_ct_alloc(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple,
nfct = __bpf_nf_ct_alloc_entry(dev_net(ctx->rxq->dev), bpf_tuple, tuple__sz,
opts, opts__sz, 10);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
-
- return (struct nf_conn___init *)nfct;
+ return (struct nf_conn___init *)bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_xdp_ct_lookup - Lookup CT entry for the given tuple, and acquire a
@@ -296,12 +309,7 @@ bpf_xdp_ct_lookup(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple,
caller_net = dev_net(ctx->rxq->dev);
nfct = __bpf_nf_ct_lookup(caller_net, bpf_tuple, tuple__sz, opts, opts__sz);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
- return nfct;
+ return bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_skb_ct_alloc - Allocate a new CT entry
@@ -329,13 +337,7 @@ bpf_skb_ct_alloc(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple,
net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk);
nfct = __bpf_nf_ct_alloc_entry(net, bpf_tuple, tuple__sz, opts, opts__sz, 10);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
-
- return (struct nf_conn___init *)nfct;
+ return (struct nf_conn___init *)bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_skb_ct_lookup - Lookup CT entry for the given tuple, and acquire a
@@ -364,12 +366,7 @@ bpf_skb_ct_lookup(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple,
caller_net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk);
nfct = __bpf_nf_ct_lookup(caller_net, bpf_tuple, tuple__sz, opts, opts__sz);
- if (IS_ERR(nfct)) {
- if (opts)
- opts->error = PTR_ERR(nfct);
- return NULL;
- }
- return nfct;
+ return bpf_ct_opts_result(opts, opts__sz, nfct);
}
/* bpf_ct_insert_entry - Add the provided entry into a CT map
diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c
index 342627b0d32b..2b9943f4d88c 100644
--- a/net/netfilter/nf_conntrack_core.c
+++ b/net/netfilter/nf_conntrack_core.c
@@ -1493,6 +1493,31 @@ static bool gc_worker_can_early_drop(const struct nf_conn *ct)
return false;
}
+static void nf_ct_help_gc(struct nf_conn *ct)
+{
+ struct nf_conn_help *help;
+
+ if (!refcount_inc_not_zero(&ct->ct_general.use))
+ return;
+
+ /* load ->status after refcount increase */
+ smp_acquire__after_ctrl_dep();
+
+ if (!nf_ct_is_confirmed(ct) || nf_ct_is_dying(ct)) {
+ nf_ct_put(ct);
+ return;
+ }
+
+ /* re-check helper due to SLAB_TYPESAFE_BY_RCU */
+ if (test_bit(IPS_HELPER_BIT, &ct->status)) {
+ help = nfct_help(ct);
+ if (help)
+ nf_ct_expectation_gc(help);
+ }
+
+ nf_ct_put(ct);
+}
+
static void gc_worker(struct work_struct *work)
{
unsigned int i, hashsz, nf_conntrack_max95 = 0;
@@ -1570,7 +1595,13 @@ static void gc_worker(struct work_struct *work)
expires = (expires - (long)next_run) / ++count;
next_run += expires;
- if (nf_conntrack_max95 == 0 || gc_worker_skip_ct(tmp))
+ if (gc_worker_skip_ct(tmp))
+ continue;
+
+ if (test_bit(IPS_HELPER_BIT, &tmp->status))
+ nf_ct_help_gc(tmp);
+
+ if (nf_conntrack_max95 == 0)
continue;
net = nf_ct_net(tmp);
@@ -1717,16 +1748,18 @@ void nf_conntrack_free(struct nf_conn *ct)
*/
WARN_ON(refcount_read(&ct->ct_general.use) != 0);
+ rcu_read_lock();
if (ct->status & IPS_SRC_NAT_DONE) {
const struct nf_nat_hook *nat_hook;
- rcu_read_lock();
nat_hook = rcu_dereference(nf_nat_hook);
if (nat_hook)
nat_hook->remove_nat_bysrc(ct);
- rcu_read_unlock();
}
+ nf_ct_timeout_put(ct);
+ rcu_read_unlock();
+
kfree(ct->ext);
kmem_cache_free(nf_conntrack_cachep, ct);
cnet = nf_ct_pernet(net);
diff --git a/net/netfilter/nf_conntrack_expect.c b/net/netfilter/nf_conntrack_expect.c
index 70d06ae0a379..a0b9cb8b00a9 100644
--- a/net/netfilter/nf_conntrack_expect.c
+++ b/net/netfilter/nf_conntrack_expect.c
@@ -43,6 +43,24 @@ unsigned int nf_ct_expect_max __read_mostly;
static struct kmem_cache *nf_ct_expect_cachep __read_mostly;
static siphash_aligned_key_t nf_ct_expect_hashrnd;
+void nf_ct_expectation_gc(struct nf_conn_help *master_help)
+{
+ struct nf_conntrack_expect *exp;
+ struct hlist_node *next;
+
+ if (hlist_empty(&master_help->expectations))
+ return;
+
+ spin_lock_bh(&nf_conntrack_expect_lock);
+ hlist_for_each_entry_safe(exp, next, &master_help->expectations, lnode) {
+ if (!nf_ct_exp_is_expired(exp))
+ continue;
+
+ nf_ct_unlink_expect(exp);
+ }
+ spin_unlock_bh(&nf_conntrack_expect_lock);
+}
+
/* nf_conntrack_expect helper functions */
void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp,
u32 portid, int report)
@@ -53,7 +71,6 @@ void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp,
lockdep_nfct_expect_lock_held();
WARN_ON(!master_help);
- WARN_ON(timer_pending(&exp->timeout));
hlist_del_rcu(&exp->hnode);
@@ -70,16 +87,6 @@ void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp,
}
EXPORT_SYMBOL_GPL(nf_ct_unlink_expect_report);
-static void nf_ct_expectation_timed_out(struct timer_list *t)
-{
- struct nf_conntrack_expect *exp = from_timer(exp, t, timeout);
-
- spin_lock_bh(&nf_conntrack_expect_lock);
- nf_ct_unlink_expect(exp);
- spin_unlock_bh(&nf_conntrack_expect_lock);
- nf_ct_expect_put(exp);
-}
-
static unsigned int nf_ct_expect_dst_hash(const struct net *n, const struct nf_conntrack_tuple *tuple)
{
struct {
@@ -117,19 +124,6 @@ nf_ct_exp_equal(const struct nf_conntrack_tuple *tuple,
nf_ct_exp_zone_equal_any(i, zone);
}
-bool nf_ct_remove_expect(struct nf_conntrack_expect *exp)
-{
- lockdep_nfct_expect_lock_held();
-
- if (del_timer(&exp->timeout)) {
- nf_ct_unlink_expect(exp);
- nf_ct_expect_put(exp);
- return true;
- }
- return false;
-}
-EXPORT_SYMBOL_GPL(nf_ct_remove_expect);
-
struct nf_conntrack_expect *
__nf_ct_expect_find(struct net *net,
const struct nf_conntrack_zone *zone,
@@ -144,6 +138,8 @@ __nf_ct_expect_find(struct net *net,
h = nf_ct_expect_dst_hash(net, tuple);
hlist_for_each_entry_rcu(i, &nf_ct_expect_hash[h], hnode) {
+ if (nf_ct_exp_is_expired(i))
+ continue;
if (nf_ct_exp_equal(tuple, i, zone, net))
return i;
}
@@ -178,6 +174,7 @@ nf_ct_find_expectation(struct net *net,
{
struct nf_conntrack_net *cnet = nf_ct_pernet(net);
struct nf_conntrack_expect *i, *exp = NULL;
+ struct hlist_node *next;
unsigned int h;
lockdep_nfct_expect_lock_held();
@@ -186,7 +183,11 @@ nf_ct_find_expectation(struct net *net,
return NULL;
h = nf_ct_expect_dst_hash(net, tuple);
- hlist_for_each_entry(i, &nf_ct_expect_hash[h], hnode) {
+ hlist_for_each_entry_safe(i, next, &nf_ct_expect_hash[h], hnode) {
+ if (nf_ct_exp_is_expired(i)) {
+ nf_ct_unlink_expect(i);
+ continue;
+ }
if (!(i->flags & NF_CT_EXPECT_INACTIVE) &&
nf_ct_exp_equal(tuple, i, zone, net)) {
exp = i;
@@ -196,13 +197,16 @@ nf_ct_find_expectation(struct net *net,
if (!exp)
return NULL;
+ if (!refcount_inc_not_zero(&exp->use))
+ return NULL;
+
/* If master is not in hash table yet (ie. packet hasn't left
this machine yet), how can other end know about expected?
Hence these are not the droids you are looking for (if
master ct never got confirmed, we'd hold a reference to it
and weird things would happen to future packets). */
if (!nf_ct_is_confirmed(exp->master))
- return NULL;
+ goto err_release_exp;
/* Avoid race with other CPUs, that for exp->master ct, is
* about to invoke ->destroy(), or nf_ct_delete() via timeout
@@ -214,18 +218,17 @@ nf_ct_find_expectation(struct net *net,
*/
if (unlikely(nf_ct_is_dying(exp->master) ||
!refcount_inc_not_zero(&exp->master->ct_general.use)))
- return NULL;
+ goto err_release_exp;
- if (exp->flags & NF_CT_EXPECT_PERMANENT || !unlink) {
- refcount_inc(&exp->use);
- return exp;
- } else if (del_timer(&exp->timeout)) {
- nf_ct_unlink_expect(exp);
+ if (exp->flags & NF_CT_EXPECT_PERMANENT || !unlink)
return exp;
- }
- /* Undo exp->master refcnt increase, if del_timer() failed */
- nf_ct_put(exp->master);
+ nf_ct_unlink_expect(exp);
+
+ return exp;
+
+err_release_exp:
+ nf_ct_expect_put(exp);
return NULL;
}
@@ -241,9 +244,8 @@ void nf_ct_remove_expectations(struct nf_conn *ct)
return;
spin_lock_bh(&nf_conntrack_expect_lock);
- hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) {
- nf_ct_remove_expect(exp);
- }
+ hlist_for_each_entry_safe(exp, next, &help->expectations, lnode)
+ nf_ct_unlink_expect(exp);
spin_unlock_bh(&nf_conntrack_expect_lock);
}
EXPORT_SYMBOL_GPL(nf_ct_remove_expectations);
@@ -292,7 +294,7 @@ static bool master_matches(const struct nf_conntrack_expect *a,
void nf_ct_unexpect_related(struct nf_conntrack_expect *exp)
{
spin_lock_bh(&nf_conntrack_expect_lock);
- nf_ct_remove_expect(exp);
+ WRITE_ONCE(exp->flags, exp->flags | NF_CT_EXPECT_DEAD);
spin_unlock_bh(&nf_conntrack_expect_lock);
}
EXPORT_SYMBOL_GPL(nf_ct_unexpect_related);
@@ -308,6 +310,7 @@ struct nf_conntrack_expect *nf_ct_expect_alloc(struct nf_conn *me)
if (!new)
return NULL;
+ new->timeout = nfct_time_stamp;
new->master = me;
refcount_set(&new->use, 1);
return new;
@@ -414,17 +417,12 @@ static void nf_ct_expect_insert(struct nf_conntrack_expect *exp)
struct net *net = nf_ct_exp_net(exp);
unsigned int h = nf_ct_expect_dst_hash(net, &exp->tuple);
- /* two references : one for hash insert, one for the timer */
- refcount_add(2, &exp->use);
+ refcount_inc(&exp->use);
- timer_setup(&exp->timeout, nf_ct_expectation_timed_out, 0);
helper = rcu_dereference_protected(master_help->helper,
lockdep_is_held(&nf_conntrack_expect_lock));
- if (helper) {
- exp->timeout.expires = jiffies +
- helper->expect_policy[exp->class].timeout * HZ;
- }
- add_timer(&exp->timeout);
+ if (helper)
+ exp->timeout += helper->expect_policy[exp->class].timeout * HZ;
hlist_add_head_rcu(&exp->lnode, &master_help->expectations);
master_help->expecting[exp->class]++;
@@ -436,20 +434,26 @@ static void nf_ct_expect_insert(struct nf_conntrack_expect *exp)
NF_CT_STAT_INC(net, expect_create);
}
-/* Race with expectations being used means we could have none to find; OK. */
-static void evict_oldest_expect(struct nf_conn *master,
- struct nf_conntrack_expect *new)
+static void evict_oldest_expect(struct nf_conn_help *master_help,
+ struct nf_conntrack_expect *new,
+ const struct nf_conntrack_expect_policy *p)
{
- struct nf_conn_help *master_help = nfct_help(master);
struct nf_conntrack_expect *exp, *last = NULL;
+ struct hlist_node *next;
- hlist_for_each_entry(exp, &master_help->expectations, lnode) {
+ hlist_for_each_entry_safe(exp, next, &master_help->expectations, lnode) {
+ if (nf_ct_exp_is_expired(exp)) {
+ nf_ct_unlink_expect(exp);
+ continue;
+ }
if (exp->class == new->class)
last = exp;
}
- if (last)
- nf_ct_remove_expect(last);
+ /* Still worth to evict oldest expectation after garbage collection? */
+ if (last &&
+ master_help->expecting[last->class] >= p->max_expected)
+ nf_ct_unlink_expect(last);
}
static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect,
@@ -474,14 +478,18 @@ static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect,
}
h = nf_ct_expect_dst_hash(net, &expect->tuple);
hlist_for_each_entry_safe(i, next, &nf_ct_expect_hash[h], hnode) {
+ if (nf_ct_exp_is_expired(i)) {
+ nf_ct_unlink_expect(i);
+ continue;
+ }
if (master_matches(i, expect, flags) &&
expect_matches(i, expect)) {
if (i->class != expect->class ||
i->master != expect->master)
return -EALREADY;
- if (nf_ct_remove_expect(i))
- break;
+ nf_ct_unlink_expect(i);
+ break;
} else if (expect_clash(i, expect)) {
ret = -EBUSY;
goto out;
@@ -493,14 +501,8 @@ static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect,
if (helper) {
p = &helper->expect_policy[expect->class];
if (p->max_expected &&
- master_help->expecting[expect->class] >= p->max_expected) {
- evict_oldest_expect(master, expect);
- if (master_help->expecting[expect->class]
- >= p->max_expected) {
- ret = -EMFILE;
- goto out;
- }
- }
+ master_help->expecting[expect->class] >= p->max_expected)
+ evict_oldest_expect(master_help, expect, p);
}
cnet = nf_ct_pernet(net);
@@ -547,10 +549,8 @@ void nf_ct_expect_iterate_destroy(bool (*iter)(struct nf_conntrack_expect *e, vo
hlist_for_each_entry_safe(exp, next,
&nf_ct_expect_hash[i],
hnode) {
- if (iter(exp, data) && del_timer(&exp->timeout)) {
+ if (iter(exp, data))
nf_ct_unlink_expect(exp);
- nf_ct_expect_put(exp);
- }
}
}
@@ -577,10 +577,8 @@ void nf_ct_expect_iterate_net(struct net *net,
if (!net_eq(nf_ct_exp_net(exp), net))
continue;
- if (iter(exp, data) && del_timer(&exp->timeout)) {
+ if (iter(exp, data))
nf_ct_unlink_expect_report(exp, portid, report);
- nf_ct_expect_put(exp);
- }
}
}
@@ -657,17 +655,17 @@ static int exp_seq_show(struct seq_file *s, void *v)
struct net *net = seq_file_net(s);
struct hlist_node *n = v;
char *delim = "";
+ __s32 timeout;
expect = hlist_entry(n, struct nf_conntrack_expect, hnode);
if (!net_eq(nf_ct_exp_net(expect), net))
return 0;
+ if (nf_ct_exp_is_expired(expect))
+ return 0;
- if (expect->timeout.function)
- seq_printf(s, "%ld ", timer_pending(&expect->timeout)
- ? (long)(expect->timeout.expires - jiffies)/HZ : 0);
- else
- seq_puts(s, "- ");
+ timeout = (__s32)(READ_ONCE(expect->timeout) - nfct_time_stamp) / HZ;
+ seq_printf(s, "%d ", timeout > 0 ? timeout : 0);
seq_printf(s, "l3proto = %u proto=%u ",
expect->tuple.src.l3num,
expect->tuple.dst.protonum);
diff --git a/net/netfilter/nf_conntrack_h323_main.c b/net/netfilter/nf_conntrack_h323_main.c
index c42547284f35..cf1c911a0941 100644
--- a/net/netfilter/nf_conntrack_h323_main.c
+++ b/net/netfilter/nf_conntrack_h323_main.c
@@ -1396,8 +1396,8 @@ static int process_rcf(struct sk_buff *skb, struct nf_conn *ct,
"timeout to %u seconds for",
info->timeout);
nf_ct_dump_tuple(&exp->tuple);
- mod_timer_pending(&exp->timeout,
- jiffies + info->timeout * HZ);
+ WRITE_ONCE(exp->timeout,
+ nfct_time_stamp + (info->timeout * HZ));
}
spin_unlock_bh(&nf_conntrack_expect_lock);
}
diff --git a/net/netfilter/nf_conntrack_helper.c b/net/netfilter/nf_conntrack_helper.c
index efa080cb1709..b16a45856bdc 100644
--- a/net/netfilter/nf_conntrack_helper.c
+++ b/net/netfilter/nf_conntrack_helper.c
@@ -178,10 +178,10 @@ nf_ct_helper_ext_add(struct nf_conn *ct, gfp_t gfp)
struct nf_conn_help *help;
help = nf_ct_ext_add(ct, NF_CT_EXT_HELPER, gfp);
- if (help)
+ if (help) {
+ __set_bit(IPS_HELPER_BIT, &ct->status);
INIT_HLIST_HEAD(&help->expectations);
- else
- pr_debug("failed to add helper extension area");
+ }
return help;
}
EXPORT_SYMBOL_GPL(nf_ct_helper_ext_add);
@@ -205,10 +205,8 @@ int __nf_ct_try_assign_helper(struct nf_conn *ct, struct nf_conn *tmpl,
return 0;
help = nfct_help(tmpl);
- if (help != NULL) {
+ if (help)
helper = rcu_dereference(help->helper);
- set_bit(IPS_HELPER_BIT, &ct->status);
- }
help = nfct_help(ct);
diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c
index 963efcdfca39..b0ec7f7790ba 100644
--- a/net/netfilter/nf_conntrack_netlink.c
+++ b/net/netfilter/nf_conntrack_netlink.c
@@ -2999,8 +2999,8 @@ static int
ctnetlink_exp_dump_expect(struct sk_buff *skb,
const struct nf_conntrack_expect *exp)
{
+ __s32 timeout = (__s32)(READ_ONCE(exp->timeout) - nfct_time_stamp) / HZ;
struct nf_conn *master = exp->master;
- long timeout = ((long)exp->timeout.expires - (long)jiffies) / HZ;
struct nf_conntrack_helper *helper;
#if IS_ENABLED(CONFIG_NF_NAT)
struct nlattr *nest_parms;
@@ -3043,7 +3043,7 @@ ctnetlink_exp_dump_expect(struct sk_buff *skb,
#endif
if (nla_put_be32(skb, CTA_EXPECT_TIMEOUT, htonl(timeout)) ||
nla_put_be32(skb, CTA_EXPECT_ID, nf_expect_get_id(exp)) ||
- nla_put_be32(skb, CTA_EXPECT_FLAGS, htonl(exp->flags)) ||
+ nla_put_be32(skb, CTA_EXPECT_FLAGS, htonl(exp->flags & NF_CT_EXPECT_MASK)) ||
nla_put_be32(skb, CTA_EXPECT_CLASS, htonl(exp->class)))
goto nla_put_failure;
@@ -3163,6 +3163,9 @@ ctnetlink_exp_dump_table(struct sk_buff *skb, struct netlink_callback *cb)
restart:
hlist_for_each_entry_rcu(exp, &nf_ct_expect_hash[cb->args[0]],
hnode) {
+ if (nf_ct_exp_is_expired(exp))
+ continue;
+
if (l3proto && exp->tuple.src.l3num != l3proto)
continue;
@@ -3442,11 +3445,8 @@ static int ctnetlink_del_expect(struct sk_buff *skb,
}
/* after list removal, usage count == 1 */
- if (del_timer(&exp->timeout)) {
- nf_ct_unlink_expect_report(exp, NETLINK_CB(skb).portid,
- nlmsg_report(info->nlh));
- nf_ct_expect_put(exp);
- }
+ nf_ct_unlink_expect_report(exp, NETLINK_CB(skb).portid,
+ nlmsg_report(info->nlh));
spin_unlock_bh(&nf_conntrack_expect_lock);
/* have to put what we 'get' above.
* after this line usage count == 0 */
@@ -3470,14 +3470,10 @@ static int
ctnetlink_change_expect(struct nf_conntrack_expect *x,
const struct nlattr * const cda[])
{
- if (cda[CTA_EXPECT_TIMEOUT]) {
- if (!del_timer(&x->timeout))
- return -ETIME;
+ if (cda[CTA_EXPECT_TIMEOUT])
+ WRITE_ONCE(x->timeout, nfct_time_stamp +
+ ntohl(nla_get_be32(cda[CTA_EXPECT_TIMEOUT])) * HZ);
- x->timeout.expires = jiffies +
- ntohl(nla_get_be32(cda[CTA_EXPECT_TIMEOUT])) * HZ;
- add_timer(&x->timeout);
- }
return 0;
}
diff --git a/net/netfilter/nf_conntrack_sip.c b/net/netfilter/nf_conntrack_sip.c
index 2b5f807436e2..3cab167224ec 100644
--- a/net/netfilter/nf_conntrack_sip.c
+++ b/net/netfilter/nf_conntrack_sip.c
@@ -894,11 +894,10 @@ static int refresh_signalling_expectation(struct nf_conn *ct,
exp->tuple.dst.protonum != proto ||
exp->tuple.dst.u.udp.port != port)
continue;
- if (mod_timer_pending(&exp->timeout, jiffies + expires * HZ)) {
- exp->flags &= ~NF_CT_EXPECT_INACTIVE;
- found = 1;
- break;
- }
+ WRITE_ONCE(exp->timeout, nfct_time_stamp + (expires * HZ));
+ WRITE_ONCE(exp->flags, exp->flags & ~NF_CT_EXPECT_INACTIVE);
+ found = 1;
+ break;
}
spin_unlock_bh(&nf_conntrack_expect_lock);
return found;
@@ -914,8 +913,7 @@ static void flush_expectations(struct nf_conn *ct, bool media)
hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) {
if ((exp->class != SIP_EXPECT_SIGNALLING) ^ media)
continue;
- if (!nf_ct_remove_expect(exp))
- continue;
+ nf_ct_unlink_expect(exp);
if (!media)
break;
}
@@ -1385,7 +1383,6 @@ static int process_register_request(struct sk_buff *skb, unsigned int protoff,
nf_ct_expect_init(exp, SIP_EXPECT_SIGNALLING, nf_ct_l3num(ct),
saddr, &daddr, proto, NULL, &port);
- exp->timeout.expires = sip_timeout * HZ;
rcu_assign_pointer(exp->assign_helper, helper);
exp->flags = NF_CT_EXPECT_PERMANENT | NF_CT_EXPECT_INACTIVE;
diff --git a/net/netfilter/nf_conntrack_timeout.c b/net/netfilter/nf_conntrack_timeout.c
index 0cc584d3dbb1..c81becde2afa 100644
--- a/net/netfilter/nf_conntrack_timeout.c
+++ b/net/netfilter/nf_conntrack_timeout.c
@@ -25,17 +25,32 @@
const struct nf_ct_timeout_hooks __rcu *nf_ct_timeout_hook __read_mostly;
EXPORT_SYMBOL_GPL(nf_ct_timeout_hook);
+/* nf_ct_iterate_cleanup() holds the conntrack lock. */
static int untimeout(struct nf_conn *ct, void *timeout)
{
struct nf_conn_timeout *timeout_ext = nf_ct_timeout_find(ct);
if (timeout_ext) {
- const struct nf_ct_timeout *t;
+ struct nf_ct_timeout *t;
- t = rcu_access_pointer(timeout_ext->timeout);
+ rcu_read_lock();
+ t = rcu_dereference(timeout_ext->timeout);
+ if (!t) {
+ rcu_read_unlock();
+ return 0;
+ }
- if (!timeout || t == timeout)
+ if (!timeout || t == timeout) {
RCU_INIT_POINTER(timeout_ext->timeout, NULL);
+
+ /* No race with nf_conntrack_free() which is called
+ * only after the conntrack has been removed from
+ * the hashes.
+ */
+ if (refcount_dec_and_test(&t->refcnt))
+ kfree_rcu(t, rcu);
+ }
+ rcu_read_unlock();
}
/* We are not intended to delete this conntrack. */
@@ -70,6 +85,8 @@ int nf_ct_set_timeout(struct net *net, struct nf_conn *ct,
const char *errmsg = NULL;
int ret = 0;
+ WARN_ON_ONCE(!nf_ct_is_template(ct));
+
rcu_read_lock();
h = rcu_dereference(nf_ct_timeout_hook);
if (!h) {
@@ -127,6 +144,8 @@ void nf_ct_destroy_timeout(struct nf_conn *ct)
struct nf_conn_timeout *timeout_ext;
const struct nf_ct_timeout_hooks *h;
+ WARN_ON_ONCE(!nf_ct_is_template(ct));
+
rcu_read_lock();
h = rcu_dereference(nf_ct_timeout_hook);
@@ -139,6 +158,8 @@ void nf_ct_destroy_timeout(struct nf_conn *ct)
if (t)
h->timeout_put(t);
RCU_INIT_POINTER(timeout_ext->timeout, NULL);
+ if (t && refcount_dec_and_test(&t->refcnt))
+ kfree_rcu(t, rcu);
}
}
rcu_read_unlock();
diff --git a/net/netfilter/nf_flow_table_ip.c b/net/netfilter/nf_flow_table_ip.c
index 523228e969ab..cd744ac065cd 100644
--- a/net/netfilter/nf_flow_table_ip.c
+++ b/net/netfilter/nf_flow_table_ip.c
@@ -262,6 +262,7 @@ static unsigned int nf_flow_xmit_xfrm(struct sk_buff *skb,
struct dst_entry *dst)
{
skb_orphan(skb);
+ skb_dst_drop(skb);
skb_dst_set_noref(skb, dst);
dst_output(state->net, state->sk, skb);
return NF_STOLEN;
@@ -417,6 +418,7 @@ nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb,
outdev = rt->dst.dev;
skb->dev = outdev;
nexthop = rt_nexthop(rt, flow->tuplehash[!dir].tuple.src_v4.s_addr);
+ skb_dst_drop(skb);
skb_dst_set_noref(skb, &rt->dst);
neigh_xmit(NEIGH_ARP_TABLE, outdev, &nexthop, skb);
ret = NF_STOLEN;
@@ -678,6 +680,7 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb,
outdev = rt->dst.dev;
skb->dev = outdev;
nexthop = rt6_nexthop(rt, &flow->tuplehash[!dir].tuple.src_v6);
+ skb_dst_drop(skb);
skb_dst_set_noref(skb, &rt->dst);
neigh_xmit(NEIGH_ND_TABLE, outdev, nexthop, skb);
ret = NF_STOLEN;
diff --git a/net/netfilter/nf_nat_bpf.c b/net/netfilter/nf_nat_bpf.c
index 0fa5a0bbb0ff..1d89cce3f7ca 100644
--- a/net/netfilter/nf_nat_bpf.c
+++ b/net/netfilter/nf_nat_bpf.c
@@ -41,6 +41,9 @@ int bpf_ct_set_nat_info(struct nf_conn___init *nfct,
if (proto != NFPROTO_IPV4 && proto != NFPROTO_IPV6)
return -EINVAL;
+ if (manip != NF_NAT_MANIP_SRC && manip != NF_NAT_MANIP_DST)
+ return -EINVAL;
+
memset(&range, 0, sizeof(struct nf_nat_range2));
range.flags = NF_NAT_RANGE_MAP_IPS;
range.min_addr = *addr;
diff --git a/net/netfilter/nf_nat_core.c b/net/netfilter/nf_nat_core.c
index 310449731a02..47416bafd2c0 100644
--- a/net/netfilter/nf_nat_core.c
+++ b/net/netfilter/nf_nat_core.c
@@ -588,8 +588,9 @@ nf_nat_setup_info(struct nf_conn *ct,
if (nf_ct_is_confirmed(ct))
return NF_ACCEPT;
- WARN_ON(maniptype != NF_NAT_MANIP_SRC &&
- maniptype != NF_NAT_MANIP_DST);
+ if (WARN_ON(maniptype != NF_NAT_MANIP_SRC &&
+ maniptype != NF_NAT_MANIP_DST))
+ return NF_DROP;
if (WARN_ON(nf_nat_initialized(ct, maniptype)))
return NF_DROP;
@@ -1049,9 +1050,11 @@ int nf_nat_register_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops,
ret = nf_register_net_hooks(net, nat_ops, ops_count);
if (ret < 0) {
mutex_unlock(&nf_nat_proto_mutex);
- for (i = 0; i < ops_count; i++)
- kfree(nat_ops[i].priv);
- kfree(nat_ops);
+ for (i = 0; i < ops_count; i++) {
+ priv = nat_ops[i].priv;
+ kfree_rcu(priv, rcu_head);
+ }
+ kfree_rcu(nat_ops, rcu);
return ret;
}
@@ -1115,7 +1118,7 @@ void nf_nat_unregister_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops,
}
nat_proto_net->nat_hook_ops = NULL;
- kfree(nat_ops);
+ kfree_rcu(nat_ops, rcu);
}
unlock:
mutex_unlock(&nf_nat_proto_mutex);
diff --git a/net/netfilter/nf_synproxy_core.c b/net/netfilter/nf_synproxy_core.c
index 500a90311ed5..35c38c3268a6 100644
--- a/net/netfilter/nf_synproxy_core.c
+++ b/net/netfilter/nf_synproxy_core.c
@@ -183,23 +183,23 @@ synproxy_check_timestamp_cookie(struct synproxy_options *opts)
opts->options |= opts->tsecr & (1 << 5) ? NF_SYNPROXY_OPT_ECN : 0;
}
-static unsigned int
+static bool
synproxy_tstamp_adjust(struct sk_buff *skb, unsigned int protoff,
struct tcphdr *th, struct nf_conn *ct,
enum ip_conntrack_info ctinfo,
const struct nf_conn_synproxy *synproxy)
{
unsigned int optoff, optend;
- __be32 *ptr, old;
+ u32 new, old;
if (synproxy->tsoff == 0)
- return 1;
+ return true;
optoff = protoff + sizeof(struct tcphdr);
optend = protoff + th->doff * 4;
if (skb_ensure_writable(skb, optend))
- return 0;
+ return false;
th = (struct tcphdr *)(skb->data + protoff);
@@ -208,7 +208,7 @@ synproxy_tstamp_adjust(struct sk_buff *skb, unsigned int protoff,
switch (op[0]) {
case TCPOPT_EOL:
- return 1;
+ return true;
case TCPOPT_NOP:
optoff++;
continue;
@@ -216,28 +216,26 @@ synproxy_tstamp_adjust(struct sk_buff *skb, unsigned int protoff,
if (optoff + 1 == optend ||
optoff + op[1] > optend ||
op[1] < 2)
- return 0;
+ return true;
if (op[0] == TCPOPT_TIMESTAMP &&
op[1] == TCPOLEN_TIMESTAMP) {
if (CTINFO2DIR(ctinfo) == IP_CT_DIR_REPLY) {
- ptr = (__be32 *)&op[2];
- old = *ptr;
- *ptr = htonl(ntohl(*ptr) -
- synproxy->tsoff);
+ old = get_unaligned_be32(&op[2]);
+ new = old - synproxy->tsoff;
+ put_unaligned_be32(new, &op[2]);
} else {
- ptr = (__be32 *)&op[6];
- old = *ptr;
- *ptr = htonl(ntohl(*ptr) +
- synproxy->tsoff);
+ old = get_unaligned_be32(&op[6]);
+ new = old + synproxy->tsoff;
+ put_unaligned_be32(new, &op[6]);
}
inet_proto_csum_replace4(&th->check, skb,
- old, *ptr, false);
- return 1;
+ cpu_to_be32(old),
+ cpu_to_be32(new), false);
}
optoff += op[1];
}
}
- return 1;
+ return true;
}
#ifdef CONFIG_PROC_FS
@@ -748,7 +746,9 @@ ipv4_synproxy_hook(void *priv, struct sk_buff *skb,
break;
}
- synproxy_tstamp_adjust(skb, thoff, th, ct, ctinfo, synproxy);
+ if (!synproxy_tstamp_adjust(skb, thoff, th, ct, ctinfo, synproxy))
+ return NF_DROP_REASON(skb, SKB_DROP_REASON_NETFILTER_DROP, ENOMEM);
+
return NF_ACCEPT;
}
EXPORT_SYMBOL_GPL(ipv4_synproxy_hook);
@@ -1176,7 +1176,9 @@ ipv6_synproxy_hook(void *priv, struct sk_buff *skb,
break;
}
- synproxy_tstamp_adjust(skb, thoff, th, ct, ctinfo, synproxy);
+ if (!synproxy_tstamp_adjust(skb, thoff, th, ct, ctinfo, synproxy))
+ return NF_DROP_REASON(skb, SKB_DROP_REASON_NETFILTER_DROP, ENOMEM);
+
return NF_ACCEPT;
}
EXPORT_SYMBOL_GPL(ipv6_synproxy_hook);
diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c
index a373a0969e99..99fba693ab4f 100644
--- a/net/netfilter/nf_tables_api.c
+++ b/net/netfilter/nf_tables_api.c
@@ -308,7 +308,7 @@ static void nft_netdev_unregister_hooks(struct net *net,
list_for_each_entry_safe(hook, next, hook_list, list) {
nf_unregister_net_hook(net, &hook->ops);
if (release_netdev) {
- list_del(&hook->list);
+ list_del_rcu(&hook->list);
kfree_rcu(hook, rcu);
}
}
@@ -628,6 +628,11 @@ static void nft_map_catchall_deactivate(const struct nft_ctx *ctx,
}
}
+/* Use NFT_ITER_UPDATE iterator even if this may be called from the preparation
+ * phase, the set clone might already exist from a previous command, or it might
+ * be a set that is going away and does not require a clone. The netns and
+ * netlink release paths also need to work on the live set.
+ */
static void nft_map_deactivate(const struct nft_ctx *ctx, struct nft_set *set)
{
struct nft_set_iter iter = {
@@ -2976,7 +2981,7 @@ static int nf_tables_fill_expr_info(struct sk_buff *skb,
NFTA_EXPR_DATA);
if (data == NULL)
goto nla_put_failure;
- if (expr->ops->dump(skb, expr) < 0)
+ if (expr->ops->dump(skb, expr, false) < 0)
goto nla_put_failure;
nla_nest_end(skb, data);
}
@@ -6126,39 +6131,11 @@ void nf_tables_set_elem_destroy(const struct nft_ctx *ctx,
kfree(elem);
}
-int nft_set_elem_expr_clone(const struct nft_ctx *ctx, struct nft_set *set,
- struct nft_expr *expr_array[])
-{
- struct nft_expr *expr;
- int err, i, k;
-
- for (i = 0; i < set->num_exprs; i++) {
- expr = kzalloc(set->exprs[i]->ops->size, GFP_KERNEL_ACCOUNT);
- if (!expr)
- goto err_expr;
-
- err = nft_expr_clone(expr, set->exprs[i], GFP_KERNEL_ACCOUNT);
- if (err < 0) {
- kfree(expr);
- goto err_expr;
- }
- expr_array[i] = expr;
- }
-
- return 0;
-
-err_expr:
- for (k = i - 1; k >= 0; k--)
- nft_expr_destroy(ctx, expr_array[k]);
-
- return -ENOMEM;
-}
-
static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
const struct nft_set_ext_tmpl *tmpl,
const struct nft_set_ext *ext,
struct nft_expr *expr_array[],
- u32 num_exprs)
+ u32 num_exprs, bool override_exprs)
{
struct nft_set_elem_expr *elem_expr = nft_set_ext_expr(ext);
u32 len = sizeof(struct nft_set_elem_expr);
@@ -6181,7 +6158,8 @@ static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
goto err_elem_expr_setup;
elem_expr->size += expr_array[i]->ops->size;
- nft_expr_destroy(ctx, expr_array[i]);
+ if (override_exprs)
+ nft_expr_destroy(ctx, expr_array[i]);
expr_array[i] = NULL;
}
@@ -6189,7 +6167,9 @@ static int nft_set_elem_expr_setup(struct nft_ctx *ctx,
err_elem_expr_setup:
for (; i < num_exprs; i++) {
- nft_expr_destroy(ctx, expr_array[i]);
+ if (override_exprs)
+ nft_expr_destroy(ctx, expr_array[i]);
+
expr_array[i] = NULL;
}
@@ -6218,16 +6198,19 @@ EXPORT_SYMBOL_GPL(nft_set_catchall_lookup);
static int nft_setelem_catchall_insert(const struct net *net,
struct nft_set *set,
const struct nft_set_elem *elem,
- struct nft_set_ext **pext)
+ void **priv)
{
struct nft_set_elem_catchall *catchall;
u8 genmask = nft_genmask_next(net);
+ u64 tstamp = nft_net_tstamp(net);
struct nft_set_ext *ext;
list_for_each_entry(catchall, &set->catchall_list, list) {
ext = nft_set_elem_ext(set, catchall->elem);
- if (nft_set_elem_active(ext, genmask)) {
- *pext = ext;
+ if (nft_set_elem_active(ext, genmask) &&
+ !__nft_set_elem_expired(ext, tstamp) &&
+ !nft_set_elem_is_dead(ext)) {
+ *priv = catchall->elem;
return -EEXIST;
}
}
@@ -6245,14 +6228,15 @@ static int nft_setelem_catchall_insert(const struct net *net,
static int nft_setelem_insert(const struct net *net,
struct nft_set *set,
const struct nft_set_elem *elem,
- struct nft_set_ext **ext, unsigned int flags)
+ void **elem_priv,
+ unsigned int flags)
{
int ret;
if (flags & NFT_SET_ELEM_CATCHALL)
- ret = nft_setelem_catchall_insert(net, set, elem, ext);
+ ret = nft_setelem_catchall_insert(net, set, elem, elem_priv);
else
- ret = set->ops->insert(net, set, elem, ext);
+ ret = set->ops->insert(net, set, elem, elem_priv);
return ret;
}
@@ -6286,11 +6270,14 @@ static int nft_setelem_catchall_deactivate(const struct net *net,
struct nft_set_elem *elem)
{
struct nft_set_elem_catchall *catchall;
+ u64 tstamp = nft_net_tstamp(net);
struct nft_set_ext *ext;
list_for_each_entry(catchall, &set->catchall_list, list) {
ext = nft_set_elem_ext(set, catchall->elem);
- if (!nft_is_active_next(net, ext))
+ if (!nft_is_active_next(net, ext) ||
+ __nft_set_elem_expired(ext, tstamp) ||
+ nft_set_elem_is_dead(ext))
continue;
kfree(elem->priv);
@@ -6415,11 +6402,13 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
struct nft_set_elem elem;
struct nft_set_binding *binding;
struct nft_object *obj = NULL;
+ bool override_exprs = false;
struct nft_userdata *udata;
struct nft_data_desc desc;
enum nft_registers dreg;
struct nft_trans *trans;
bool set_full = false;
+ void *elem_priv;
u64 timeout;
u64 expiration;
int err, i;
@@ -6520,6 +6509,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
expr_array[0] = expr;
num_exprs = 1;
+ override_exprs = true;
if (set->num_exprs && set->exprs[0]->ops != expr->ops) {
err = -EOPNOTSUPP;
@@ -6548,6 +6538,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
}
expr_array[i] = expr;
num_exprs++;
+ override_exprs = true;
if (set->num_exprs && expr->ops != set->exprs[i]->ops) {
err = -EOPNOTSUPP;
@@ -6561,9 +6552,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
}
} else if (set->num_exprs > 0 &&
!(flags & NFT_SET_ELEM_INTERVAL_END)) {
- err = nft_set_elem_expr_clone(ctx, set, expr_array);
- if (err < 0)
- goto err_set_elem_expr_clone;
+ for (i = 0; i < set->num_exprs; i++)
+ expr_array[i] = set->exprs[i];
num_exprs = set->num_exprs;
}
@@ -6708,7 +6698,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
udata->len = ulen - 1;
nla_memcpy(&udata->data, nla[NFTA_SET_ELEM_USERDATA], ulen);
}
- err = nft_set_elem_expr_setup(ctx, &tmpl, ext, expr_array, num_exprs);
+ err = nft_set_elem_expr_setup(ctx, &tmpl, ext, expr_array, num_exprs,
+ override_exprs);
if (err < 0)
goto err_elem_free;
@@ -6728,9 +6719,10 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
ext->genmask = nft_genmask_cur(ctx->net);
- err = nft_setelem_insert(ctx->net, set, &elem, &ext2, flags);
+ err = nft_setelem_insert(ctx->net, set, &elem, &elem_priv, flags);
if (err) {
if (err == -EEXIST) {
+ ext2 = nft_set_elem_ext(set, elem_priv);
if (nft_set_ext_exists(ext, NFT_SET_EXT_DATA) ^
nft_set_ext_exists(ext2, NFT_SET_EXT_DATA) ||
nft_set_ext_exists(ext, NFT_SET_EXT_OBJREF) ^
@@ -6778,9 +6770,11 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set,
err_parse_key:
nft_data_release(&elem.key.val, NFT_DATA_VALUE);
err_set_elem_expr:
- for (i = 0; i < num_exprs && expr_array[i]; i++)
- nft_expr_destroy(ctx, expr_array[i]);
-err_set_elem_expr_clone:
+ if (override_exprs) {
+ for (i = 0; i < num_exprs && expr_array[i]; i++)
+ nft_expr_destroy(ctx, expr_array[i]);
+ }
+
return err;
}
@@ -6860,6 +6854,16 @@ void nft_data_hold(const struct nft_data *data, enum nft_data_types type)
}
}
+static int nft_setelem_active_next(const struct net *net,
+ const struct nft_set *set,
+ struct nft_set_elem *elem)
+{
+ const struct nft_set_ext *ext = nft_set_elem_ext(set, elem->priv);
+ u8 genmask = nft_genmask_next(net);
+
+ return nft_set_elem_active(ext, genmask);
+}
+
static void nft_setelem_data_activate(const struct net *net,
const struct nft_set *set,
struct nft_set_elem *elem)
@@ -7057,9 +7061,12 @@ static int nft_set_catchall_flush(const struct nft_ctx *ctx,
static int nft_set_flush(struct nft_ctx *ctx, struct nft_set *set, u8 genmask)
{
+ /* The set backend might need to clone the set, do it now from the
+ * preparation phase, use NFT_ITER_UPDATE_CLONE iterator type.
+ */
struct nft_set_iter iter = {
.genmask = genmask,
- .type = NFT_ITER_UPDATE,
+ .type = NFT_ITER_UPDATE_CLONE,
.fn = nft_setelem_flush,
};
@@ -7812,18 +7819,17 @@ static int nf_tables_delobj(struct sk_buff *skb, const struct nfnl_info *info,
return nft_delobj(&ctx, obj);
}
-static void
-__nft_obj_notify(struct net *net, const struct nft_table *table,
- struct nft_object *obj, u32 portid, u32 seq, int event,
- u16 flags, int family, int report, gfp_t gfp)
+static struct sk_buff *
+nft_obj_notify_alloc(struct net *net, const struct nft_table *table,
+ struct nft_object *obj, u32 portid, u32 seq, int event,
+ u16 flags, int family, int report, gfp_t gfp)
{
- struct nftables_pernet *nft_net = nft_pernet(net);
struct sk_buff *skb;
int err;
if (!report &&
!nfnetlink_has_listeners(net, NFNLGRP_NFTABLES))
- return;
+ return NULL;
skb = nlmsg_new(NLMSG_GOODSIZE, gfp);
if (skb == NULL)
@@ -7837,10 +7843,10 @@ __nft_obj_notify(struct net *net, const struct nft_table *table,
goto err;
}
- nft_notify_enqueue(skb, report, &nft_net->notify_list);
- return;
+ return skb;
err:
nfnetlink_set_err(net, portid, NFNLGRP_NFTABLES, -ENOBUFS);
+ return NULL;
}
void nft_obj_notify(struct net *net, const struct nft_table *table,
@@ -7850,6 +7856,7 @@ void nft_obj_notify(struct net *net, const struct nft_table *table,
struct nftables_pernet *nft_net = nft_pernet(net);
char *buf = kasprintf(gfp, "%s:%u",
table->name, nft_net->base_seq);
+ struct sk_buff *skb;
audit_log_nfcfg(buf,
family,
@@ -7860,17 +7867,27 @@ void nft_obj_notify(struct net *net, const struct nft_table *table,
gfp);
kfree(buf);
- __nft_obj_notify(net, table, obj, portid, seq, event,
- flags, family, report, gfp);
+ /* Called from the packet path, holding no mutex: notify_list is
+ * serialised by commit_mutex, so send this notification directly.
+ */
+ skb = nft_obj_notify_alloc(net, table, obj, portid, seq, event,
+ flags, family, report, gfp);
+ if (skb)
+ nfnetlink_send(skb, net, portid, NFNLGRP_NFTABLES, report, gfp);
}
EXPORT_SYMBOL_GPL(nft_obj_notify);
static void nf_tables_obj_notify(const struct nft_ctx *ctx,
struct nft_object *obj, int event)
{
- __nft_obj_notify(ctx->net, ctx->table, obj, ctx->portid,
- ctx->seq, event, ctx->flags, ctx->family,
- ctx->report, GFP_KERNEL);
+ struct nftables_pernet *nft_net = nft_pernet(ctx->net);
+ struct sk_buff *skb;
+
+ skb = nft_obj_notify_alloc(ctx->net, ctx->table, obj, ctx->portid,
+ ctx->seq, event, ctx->flags, ctx->family,
+ ctx->report, GFP_KERNEL);
+ if (skb)
+ nft_notify_enqueue(skb, ctx->report, &nft_net->notify_list);
}
/*
@@ -8082,7 +8099,7 @@ static void __nft_unregister_flowtable_net_hooks(struct net *net,
flowtable->data.type->setup(&flowtable->data, hook->ops.dev,
FLOW_BLOCK_UNBIND);
if (release_netdev) {
- list_del(&hook->list);
+ list_del_rcu(&hook->list);
kfree_rcu(hook, rcu);
}
}
@@ -10224,8 +10241,10 @@ static int __nf_tables_abort(struct net *net, enum nfnl_abort_action action)
case NFT_MSG_DELSETELEM:
te = (struct nft_trans_elem *)trans->data;
- nft_setelem_data_activate(net, te->set, &te->elem);
- nft_setelem_activate(net, te->set, &te->elem);
+ if (!nft_setelem_active_next(net, te->set, &te->elem)) {
+ nft_setelem_data_activate(net, te->set, &te->elem);
+ nft_setelem_activate(net, te->set, &te->elem);
+ }
if (!nft_setelem_is_catchall(te->set, &te->elem))
te->set->ndeact--;
diff --git a/net/netfilter/nf_tables_core.c b/net/netfilter/nf_tables_core.c
index e0c117229ee9..1df0908cc886 100644
--- a/net/netfilter/nf_tables_core.c
+++ b/net/netfilter/nf_tables_core.c
@@ -284,10 +284,11 @@ nft_do_chain(struct nft_pktinfo *pkt, void *priv)
switch (regs.verdict.code & NF_VERDICT_MASK) {
case NF_ACCEPT:
- case NF_DROP:
case NF_QUEUE:
case NF_STOLEN:
return regs.verdict.code;
+ case NF_DROP:
+ return NF_DROP_REASON(pkt->skb, SKB_DROP_REASON_NETFILTER_DROP, EPERM);
}
switch (regs.verdict.code) {
@@ -322,6 +323,9 @@ nft_do_chain(struct nft_pktinfo *pkt, void *priv)
if (static_branch_unlikely(&nft_counters_enabled))
nft_update_chain_stats(basechain, pkt);
+ if (nft_base_chain(basechain)->policy == NF_DROP)
+ return NF_DROP_REASON(pkt->skb, SKB_DROP_REASON_NETFILTER_DROP, EPERM);
+
return nft_base_chain(basechain)->policy;
}
EXPORT_SYMBOL_GPL(nft_do_chain);
diff --git a/net/netfilter/nf_tables_trace.c b/net/netfilter/nf_tables_trace.c
index 1163ba9c1401..68916b5297e7 100644
--- a/net/netfilter/nf_tables_trace.c
+++ b/net/netfilter/nf_tables_trace.c
@@ -289,7 +289,7 @@ void nft_trace_init(struct nft_traceinfo *info, const struct nft_pktinfo *pkt,
net_get_random_once(&trace_key, sizeof(trace_key));
info->skbid = (u32)siphash_3u32(hash32_ptr(skb),
- skb_get_hash(skb),
+ skb_get_hash_net(nft_net(pkt), skb),
skb->skb_iif,
&trace_key);
}
diff --git a/net/netfilter/nfnetlink_cttimeout.c b/net/netfilter/nfnetlink_cttimeout.c
index f466af4f8531..6067e99830e4 100644
--- a/net/netfilter/nfnetlink_cttimeout.c
+++ b/net/netfilter/nfnetlink_cttimeout.c
@@ -37,11 +37,8 @@ struct ctnl_timeout {
struct list_head head;
struct list_head free_head;
struct rcu_head rcu_head;
- refcount_t refcnt;
char name[CTNL_TIMEOUT_NAME_MAX];
-
- /* must be at the end */
- struct nf_ct_timeout timeout;
+ struct nf_ct_timeout *timeout;
};
struct nfct_timeout_pernet {
@@ -133,12 +130,12 @@ static int cttimeout_new_timeout(struct sk_buff *skb,
/* You cannot replace one timeout policy by another of
* different kind, sorry.
*/
- if (matching->timeout.l3num != l3num ||
- matching->timeout.l4proto->l4proto != l4num)
+ if (matching->timeout->l3num != l3num ||
+ matching->timeout->l4proto->l4proto != l4num)
return -EINVAL;
- return ctnl_timeout_parse_policy(&matching->timeout.data,
- matching->timeout.l4proto,
+ return ctnl_timeout_parse_policy(&matching->timeout->data,
+ matching->timeout->l4proto,
info->net,
cda[CTA_TIMEOUT_DATA]);
}
@@ -154,26 +151,35 @@ static int cttimeout_new_timeout(struct sk_buff *skb,
goto err_proto_put;
}
- timeout = kzalloc(sizeof(struct ctnl_timeout) +
- l4proto->ctnl_timeout.obj_size, GFP_KERNEL);
+ timeout = kzalloc(sizeof(*timeout), GFP_KERNEL);
if (timeout == NULL) {
ret = -ENOMEM;
goto err_proto_put;
}
- ret = ctnl_timeout_parse_policy(&timeout->timeout.data, l4proto,
+ timeout->timeout = kzalloc(sizeof(*timeout->timeout) +
+ l4proto->ctnl_timeout.obj_size, GFP_KERNEL);
+ if (!timeout->timeout) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ ret = ctnl_timeout_parse_policy(&timeout->timeout->data, l4proto,
info->net, cda[CTA_TIMEOUT_DATA]);
if (ret < 0)
- goto err;
+ goto err_free_timeout_policy;
strcpy(timeout->name, nla_data(cda[CTA_TIMEOUT_NAME]));
- timeout->timeout.l3num = l3num;
- timeout->timeout.l4proto = l4proto;
- refcount_set(&timeout->refcnt, 1);
+ timeout->timeout->l3num = l3num;
+ timeout->timeout->l4proto = l4proto;
+ refcount_set(&timeout->timeout->refcnt, 1);
__module_get(THIS_MODULE);
list_add_tail_rcu(&timeout->head, &pernet->nfct_timeout_list);
return 0;
+
+err_free_timeout_policy:
+ kfree(timeout->timeout);
err:
kfree(timeout);
err_proto_put:
@@ -186,7 +192,7 @@ ctnl_timeout_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type,
{
struct nlmsghdr *nlh;
unsigned int flags = portid ? NLM_F_MULTI : 0;
- const struct nf_conntrack_l4proto *l4proto = timeout->timeout.l4proto;
+ const struct nf_conntrack_l4proto *l4proto = timeout->timeout->l4proto;
struct nlattr *nest_parms;
int ret;
@@ -198,17 +204,17 @@ ctnl_timeout_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type,
if (nla_put_string(skb, CTA_TIMEOUT_NAME, timeout->name) ||
nla_put_be16(skb, CTA_TIMEOUT_L3PROTO,
- htons(timeout->timeout.l3num)) ||
+ htons(timeout->timeout->l3num)) ||
nla_put_u8(skb, CTA_TIMEOUT_L4PROTO, l4proto->l4proto) ||
nla_put_be32(skb, CTA_TIMEOUT_USE,
- htonl(refcount_read(&timeout->refcnt))))
+ htonl(refcount_read(&timeout->timeout->refcnt))))
goto nla_put_failure;
nest_parms = nla_nest_start(skb, CTA_TIMEOUT_DATA);
if (!nest_parms)
goto nla_put_failure;
- ret = l4proto->ctnl_timeout.obj_to_nlattr(skb, &timeout->timeout.data);
+ ret = l4proto->ctnl_timeout.obj_to_nlattr(skb, &timeout->timeout->data);
if (ret < 0)
goto nla_put_failure;
@@ -308,23 +314,17 @@ static int cttimeout_get_timeout(struct sk_buff *skb,
return ret;
}
-/* try to delete object, fail if it is still in use. */
-static int ctnl_timeout_try_del(struct net *net, struct ctnl_timeout *timeout)
+static void ctnl_timeout_del(struct net *net, struct ctnl_timeout *timeout)
{
- int ret = 0;
+ /* We are protected by nfnl mutex. */
+ list_del_rcu(&timeout->head);
+ nf_ct_untimeout(net, timeout->timeout);
- /* We want to avoid races with ctnl_timeout_put. So only when the
- * current refcnt is 1, we decrease it to 0.
- */
- if (refcount_dec_if_one(&timeout->refcnt)) {
- /* We are protected by nfnl mutex. */
- list_del_rcu(&timeout->head);
- nf_ct_untimeout(net, &timeout->timeout);
- kfree_rcu(timeout, rcu_head);
- } else {
- ret = -EBUSY;
- }
- return ret;
+ if (refcount_dec_and_test(&timeout->timeout->refcnt))
+ kfree_rcu(timeout->timeout, rcu);
+
+ kfree_rcu(timeout, rcu_head);
+ module_put(THIS_MODULE);
}
static int cttimeout_del_timeout(struct sk_buff *skb,
@@ -339,7 +339,7 @@ static int cttimeout_del_timeout(struct sk_buff *skb,
if (!cda[CTA_TIMEOUT_NAME]) {
list_for_each_entry_safe(cur, tmp, &pernet->nfct_timeout_list,
head)
- ctnl_timeout_try_del(info->net, cur);
+ ctnl_timeout_del(info->net, cur);
return 0;
}
@@ -349,10 +349,8 @@ static int cttimeout_del_timeout(struct sk_buff *skb,
if (strncmp(cur->name, name, CTNL_TIMEOUT_NAME_MAX) != 0)
continue;
- ret = ctnl_timeout_try_del(info->net, cur);
- if (ret < 0)
- return ret;
-
+ ctnl_timeout_del(info->net, cur);
+ ret = 0;
break;
}
return ret;
@@ -519,24 +517,22 @@ static struct nf_ct_timeout *ctnl_timeout_find_get(struct net *net,
if (strncmp(timeout->name, name, CTNL_TIMEOUT_NAME_MAX) != 0)
continue;
- if (!refcount_inc_not_zero(&timeout->refcnt))
+ if (!refcount_inc_not_zero(&timeout->timeout->refcnt))
goto err;
matching = timeout;
+ __module_get(THIS_MODULE);
break;
}
err:
- return matching ? &matching->timeout : NULL;
+ return matching ? matching->timeout : NULL;
}
-static void ctnl_timeout_put(struct nf_ct_timeout *t)
+static void ctnl_timeout_put(struct nf_ct_timeout *timeout)
{
- struct ctnl_timeout *timeout =
- container_of(t, struct ctnl_timeout, timeout);
+ if (refcount_dec_and_test(&timeout->refcnt))
+ kfree_rcu(timeout, rcu);
- if (refcount_dec_and_test(&timeout->refcnt)) {
- kfree_rcu(timeout, rcu_head);
- module_put(THIS_MODULE);
- }
+ module_put(THIS_MODULE);
}
static const struct nfnl_callback cttimeout_cb[IPCTNL_MSG_TIMEOUT_MAX] = {
@@ -617,8 +613,11 @@ static void __net_exit cttimeout_net_exit(struct net *net)
list_for_each_entry_safe(cur, tmp, &pernet->nfct_timeout_freelist, free_head) {
list_del(&cur->free_head);
- if (refcount_dec_and_test(&cur->refcnt))
- kfree_rcu(cur, rcu_head);
+ if (refcount_dec_and_test(&cur->timeout->refcnt))
+ kfree_rcu(cur->timeout, rcu);
+
+ kfree_rcu(cur, rcu_head);
+ module_put(THIS_MODULE);
}
}
@@ -657,24 +656,13 @@ static int __init cttimeout_init(void)
return ret;
}
-static int untimeout(struct nf_conn *ct, void *timeout)
-{
- struct nf_conn_timeout *timeout_ext = nf_ct_timeout_find(ct);
-
- if (timeout_ext)
- RCU_INIT_POINTER(timeout_ext->timeout, NULL);
-
- return 0;
-}
-
static void __exit cttimeout_exit(void)
{
nfnetlink_subsys_unregister(&cttimeout_subsys);
- unregister_pernet_subsys(&cttimeout_ops);
RCU_INIT_POINTER(nf_ct_timeout_hook, NULL);
-
- nf_ct_iterate_destroy(untimeout, NULL);
+ synchronize_net();
+ unregister_pernet_subsys(&cttimeout_ops);
}
module_init(cttimeout_init);
diff --git a/net/netfilter/nfnetlink_log.c b/net/netfilter/nfnetlink_log.c
index af7fe55526bb..dd3d6f8e4feb 100644
--- a/net/netfilter/nfnetlink_log.c
+++ b/net/netfilter/nfnetlink_log.c
@@ -445,6 +445,23 @@ static int nfulnl_put_bridge(struct nfulnl_instance *inst, const struct sk_buff
return -1;
}
+#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
+static int nflog_put_master_ifindex(struct sk_buff *nlskb, int attr,
+ const struct net_device *dev)
+{
+ const struct net_device *upper;
+
+ if (dev && !netif_is_bridge_port(dev))
+ return 0;
+
+ upper = netdev_master_upper_dev_get_rcu((struct net_device *)dev);
+ if (upper && nla_put_be32(nlskb, attr, htonl(upper->ifindex)))
+ return -EMSGSIZE;
+
+ return 0;
+}
+#endif
+
/* This is an inline function, we don't really care about a long
* list of arguments */
static inline int
@@ -500,8 +517,7 @@ __build_packet_message(struct nfnl_log_net *log,
/* rcu_read_lock()ed by nf_hook_thresh or
* nf_log_packet.
*/
- nla_put_be32(inst->skb, NFULA_IFINDEX_INDEV,
- htonl(br_port_get_rcu(indev)->br->dev->ifindex)))
+ nflog_put_master_ifindex(inst->skb, NFULA_IFINDEX_INDEV, indev))
goto nla_put_failure;
} else {
int physinif;
@@ -537,8 +553,7 @@ __build_packet_message(struct nfnl_log_net *log,
/* rcu_read_lock()ed by nf_hook_thresh or
* nf_log_packet.
*/
- nla_put_be32(inst->skb, NFULA_IFINDEX_OUTDEV,
- htonl(br_port_get_rcu(outdev)->br->dev->ifindex)))
+ nflog_put_master_ifindex(inst->skb, NFULA_IFINDEX_OUTDEV, outdev))
goto nla_put_failure;
} else {
struct net_device *physoutdev;
diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c
index 015ffb11cb65..31ead68bc6a0 100644
--- a/net/netfilter/nfnetlink_queue.c
+++ b/net/netfilter/nfnetlink_queue.c
@@ -223,11 +223,76 @@ find_dequeue_entry(struct nfqnl_instance *queue, unsigned int id)
return entry;
}
+/* return true if the entry has an unconfirmed conntrack attached that isn't owned by us
+ * exclusively.
+ */
+static bool nf_ct_drop_unconfirmed(const struct nf_queue_entry *entry, bool *is_unconfirmed)
+{
+#if IS_ENABLED(CONFIG_NF_CONNTRACK)
+ struct nf_conn *ct = (void *)skb_nfct(entry->skb);
+
+ if (!ct || nf_ct_is_confirmed(ct))
+ return false;
+
+ if (is_unconfirmed)
+ *is_unconfirmed = true;
+
+ /* in some cases skb_clone() can occur after initial conntrack
+ * pickup, but conntrack assumes exclusive skb->_nfct ownership for
+ * unconfirmed entries.
+ *
+ * This happens for br_netfilter and with ip multicast routing.
+ * This can't be solved with serialization here because one clone
+ * could have been queued for local delivery or could be transmitted
+ * in parallel on another CPU.
+ */
+ return refcount_read(&ct->ct_general.use) > 1;
+#endif
+ return false;
+}
+
+static bool nf_bridge_port_valid(const struct net_device *dev)
+{
+ if (!dev)
+ return true;
+
+ return netif_is_bridge_port(dev);
+}
+
+/* queued skbs leave rcu protection. We bump device refcount so that
+ * the device cannot go away. However, while packet was out the port
+ * could have been removed from the bridge.
+ *
+ * Ensure in+outdev are still part of a bridge at reinject time.
+ *
+ * The device rx_handler_data could even be pointing at data that is
+ * not a net_bridge_port structure.
+ */
+static bool nf_bridge_ports_valid(const struct nf_queue_entry *entry)
+{
+#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
+ if (!nf_bridge_port_valid(entry->physin) ||
+ !nf_bridge_port_valid(entry->physout))
+ return false;
+#endif
+ if (entry->state.pf != PF_BRIDGE)
+ return true;
+
+ if (!nf_bridge_port_valid(entry->state.in) ||
+ !nf_bridge_port_valid(entry->state.out))
+ return false;
+
+ return true;
+}
+
static void nfqnl_reinject(struct nf_queue_entry *entry, unsigned int verdict)
{
const struct nf_ct_hook *ct_hook;
int err;
+ if (!nf_bridge_ports_valid(entry))
+ verdict = NF_DROP;
+
if (verdict == NF_ACCEPT ||
verdict == NF_REPEAT ||
verdict == NF_STOP) {
@@ -240,6 +305,24 @@ static void nfqnl_reinject(struct nf_queue_entry *entry, unsigned int verdict)
}
rcu_read_unlock();
}
+
+ if (verdict != NF_DROP && entry->nf_ct_is_unconfirmed) {
+ /* If first queued segment was already reinjected then
+ * there is a good chance the ct entry is now confirmed.
+ *
+ * Handle the rare cases:
+ * - out-of-order verdict
+ * - threaded userspace reinjecting in parallel
+ * - first segment was dropped
+ *
+ * In all of those cases we can't handle this packet
+ * because we can't be sure that another CPU won't modify
+ * nf_conn->ext in parallel which isn't allowed.
+ */
+ if (nf_ct_drop_unconfirmed(entry, NULL))
+ verdict = NF_DROP;
+ }
+
nf_reinject(entry, verdict);
}
@@ -385,6 +468,23 @@ static int nf_queue_checksum_help(struct sk_buff *entskb)
return skb_checksum_help(entskb);
}
+#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER)
+static int nfqnl_put_master_ifindex(struct sk_buff *nlskb, int attr,
+ const struct net_device *dev)
+{
+ const struct net_device *upper;
+
+ if (dev && !netif_is_bridge_port(dev))
+ return 0;
+
+ upper = netdev_master_upper_dev_get_rcu((struct net_device *)dev);
+ if (upper && nla_put_be32(nlskb, attr, htonl(upper->ifindex)))
+ return -EMSGSIZE;
+
+ return 0;
+}
+#endif
+
static struct sk_buff *
nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
struct nf_queue_entry *entry,
@@ -515,10 +615,7 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
* netfilter_bridge) */
if (nla_put_be32(skb, NFQA_IFINDEX_PHYSINDEV,
htonl(indev->ifindex)) ||
- /* this is the bridge group "brX" */
- /* rcu_read_lock()ed by __nf_queue */
- nla_put_be32(skb, NFQA_IFINDEX_INDEV,
- htonl(br_port_get_rcu(indev)->br->dev->ifindex)))
+ nfqnl_put_master_ifindex(skb, NFQA_IFINDEX_INDEV, indev))
goto nla_put_failure;
} else {
int physinif;
@@ -549,10 +646,7 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
* netfilter_bridge) */
if (nla_put_be32(skb, NFQA_IFINDEX_PHYSOUTDEV,
htonl(outdev->ifindex)) ||
- /* this is the bridge group "brX" */
- /* rcu_read_lock()ed by __nf_queue */
- nla_put_be32(skb, NFQA_IFINDEX_OUTDEV,
- htonl(br_port_get_rcu(outdev)->br->dev->ifindex)))
+ nfqnl_put_master_ifindex(skb, NFQA_IFINDEX_OUTDEV, outdev))
goto nla_put_failure;
} else {
int physoutif;
@@ -655,49 +749,6 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue,
return NULL;
}
-static bool nf_ct_drop_unconfirmed(const struct nf_queue_entry *entry)
-{
-#if IS_ENABLED(CONFIG_NF_CONNTRACK)
- static const unsigned long flags = IPS_CONFIRMED | IPS_DYING;
- struct nf_conn *ct = (void *)skb_nfct(entry->skb);
- unsigned long status;
- unsigned int use;
-
- if (!ct)
- return false;
-
- status = READ_ONCE(ct->status);
- if ((status & flags) == IPS_DYING)
- return true;
-
- if (status & IPS_CONFIRMED)
- return false;
-
- /* in some cases skb_clone() can occur after initial conntrack
- * pickup, but conntrack assumes exclusive skb->_nfct ownership for
- * unconfirmed entries.
- *
- * This happens for br_netfilter and with ip multicast routing.
- * We can't be solved with serialization here because one clone could
- * have been queued for local delivery.
- */
- use = refcount_read(&ct->ct_general.use);
- if (likely(use == 1))
- return false;
-
- /* Can't decrement further? Exclusive ownership. */
- if (!refcount_dec_not_one(&ct->ct_general.use))
- return false;
-
- skb_set_nfct(entry->skb, 0);
- /* No nf_ct_put(): we already decremented .use and it cannot
- * drop down to 0.
- */
- return true;
-#endif
- return false;
-}
-
static int
__nfqnl_enqueue_packet(struct net *net, struct nfqnl_instance *queue,
struct nf_queue_entry *entry)
@@ -714,9 +765,6 @@ __nfqnl_enqueue_packet(struct net *net, struct nfqnl_instance *queue,
}
spin_lock_bh(&queue->lock);
- if (nf_ct_drop_unconfirmed(entry))
- goto err_out_free_nskb;
-
if (queue->queue_total >= queue->queue_maxlen) {
if (queue->flags & NFQA_CFG_F_FAIL_OPEN) {
failopen = 1;
@@ -827,9 +875,10 @@ __nfqnl_enqueue_packet_gso(struct net *net, struct nfqnl_instance *queue,
static int
nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum)
{
- unsigned int queued;
- struct nfqnl_instance *queue;
struct sk_buff *skb, *segs, *nskb;
+ bool ct_is_unconfirmed = false;
+ struct nfqnl_instance *queue;
+ unsigned int queued;
int err = -ENOBUFS;
struct net *net = entry->state.net;
struct nfnl_queue_net *q = nfnl_queue_pernet(net);
@@ -853,6 +902,15 @@ nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum)
break;
}
+ /* Check if someone already holds another reference to
+ * unconfirmed ct. If so, we cannot queue the skb:
+ * concurrent modifications of nf_conn->ext are not
+ * allowed and we can't know if another CPU isn't
+ * processing the same nf_conn entry in parallel.
+ */
+ if (nf_ct_drop_unconfirmed(entry, &ct_is_unconfirmed))
+ return -EINVAL;
+
if (!skb_is_gso(skb) || ((queue->flags & NFQA_CFG_F_GSO) && !skb_is_gso_sctp(skb)))
return __nfqnl_enqueue_packet(net, queue, entry);
@@ -866,7 +924,23 @@ nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum)
goto out_err;
queued = 0;
err = 0;
+
skb_list_walk_safe(segs, segs, nskb) {
+ if (ct_is_unconfirmed && queued > 0) {
+ /* skb_gso_segment() increments the ct refcount.
+ * This is a problem for unconfirmed (not in hash)
+ * entries, those can race when reinjections happen
+ * in parallel.
+ *
+ * Annotate this for all queued entries except the
+ * first one.
+ *
+ * As long as the first one is reinjected first it
+ * will do the confirmation for us.
+ */
+ entry->nf_ct_is_unconfirmed = ct_is_unconfirmed;
+ }
+
if (err == 0)
err = __nfqnl_enqueue_packet_gso(net, queue,
segs, entry);
diff --git a/net/netfilter/nft_bitwise.c b/net/netfilter/nft_bitwise.c
index 5b96f5a490dd..ca478f2bee8b 100644
--- a/net/netfilter/nft_bitwise.c
+++ b/net/netfilter/nft_bitwise.c
@@ -42,8 +42,10 @@ static void nft_bitwise_eval_lshift(u32 *dst, const u32 *src,
u32 carry = 0;
for (i = DIV_ROUND_UP(priv->len, sizeof(u32)); i > 0; i--) {
- dst[i - 1] = (src[i - 1] << shift) | carry;
- carry = src[i - 1] >> (BITS_PER_TYPE(u32) - shift);
+ u32 tmp_src = src[i - 1];
+
+ dst[i - 1] = (tmp_src << shift) | carry;
+ carry = tmp_src >> (BITS_PER_TYPE(u32) - shift);
}
}
@@ -55,8 +57,10 @@ static void nft_bitwise_eval_rshift(u32 *dst, const u32 *src,
u32 carry = 0;
for (i = 0; i < DIV_ROUND_UP(priv->len, sizeof(u32)); i++) {
- dst[i] = carry | (src[i] >> shift);
- carry = src[i] << (BITS_PER_TYPE(u32) - shift);
+ u32 tmp_src = src[i];
+
+ dst[i] = carry | (tmp_src >> shift);
+ carry = tmp_src << (BITS_PER_TYPE(u32) - shift);
}
}
@@ -183,6 +187,9 @@ static int nft_bitwise_init(const struct nft_ctx *ctx,
if (err < 0)
return err;
+ if (nft_reg_overlap(priv->sreg, priv->dreg, priv->len))
+ return -EINVAL;
+
if (tb[NFTA_BITWISE_OP]) {
priv->op = ntohl(nla_get_be32(tb[NFTA_BITWISE_OP]));
switch (priv->op) {
@@ -233,7 +240,8 @@ static int nft_bitwise_dump_shift(struct sk_buff *skb,
return 0;
}
-static int nft_bitwise_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_bitwise_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_bitwise *priv = nft_expr_priv(expr);
int err = 0;
@@ -394,7 +402,8 @@ static int nft_bitwise_fast_init(const struct nft_ctx *ctx,
}
static int
-nft_bitwise_fast_dump(struct sk_buff *skb, const struct nft_expr *expr)
+nft_bitwise_fast_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_bitwise_fast_expr *priv = nft_expr_priv(expr);
struct nft_data data;
diff --git a/net/netfilter/nft_byteorder.c b/net/netfilter/nft_byteorder.c
index 60b18efadf5c..7218f8e3c6d4 100644
--- a/net/netfilter/nft_byteorder.c
+++ b/net/netfilter/nft_byteorder.c
@@ -144,12 +144,20 @@ static int nft_byteorder_init(const struct nft_ctx *ctx,
if (err < 0)
return err;
- return nft_parse_register_store(ctx, tb[NFTA_BYTEORDER_DREG],
- &priv->dreg, NULL, NFT_DATA_VALUE,
- priv->len);
+ err = nft_parse_register_store(ctx, tb[NFTA_BYTEORDER_DREG],
+ &priv->dreg, NULL, NFT_DATA_VALUE,
+ priv->len);
+ if (err < 0)
+ return err;
+
+ if (nft_reg_overlap(priv->sreg, priv->dreg, priv->len))
+ return -EINVAL;
+
+ return 0;
}
-static int nft_byteorder_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_byteorder_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_byteorder *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_cmp.c b/net/netfilter/nft_cmp.c
index 30181e12409d..4cdcbb74c74d 100644
--- a/net/netfilter/nft_cmp.c
+++ b/net/netfilter/nft_cmp.c
@@ -92,7 +92,8 @@ static int nft_cmp_init(const struct nft_ctx *ctx, const struct nft_expr *expr,
return 0;
}
-static int nft_cmp_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_cmp_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_cmp_expr *priv = nft_expr_priv(expr);
@@ -253,7 +254,8 @@ static int nft_cmp_fast_offload(struct nft_offload_ctx *ctx,
return __nft_cmp_offload(ctx, flow, &cmp);
}
-static int nft_cmp_fast_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_cmp_fast_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_cmp_fast_expr *priv = nft_expr_priv(expr);
enum nft_cmp_ops op = priv->inv ? NFT_CMP_NEQ : NFT_CMP_EQ;
@@ -347,7 +349,8 @@ static int nft_cmp16_fast_offload(struct nft_offload_ctx *ctx,
return __nft_cmp_offload(ctx, flow, &cmp);
}
-static int nft_cmp16_fast_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_cmp16_fast_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_cmp16_fast_expr *priv = nft_expr_priv(expr);
enum nft_cmp_ops op = priv->inv ? NFT_CMP_NEQ : NFT_CMP_EQ;
diff --git a/net/netfilter/nft_compat.c b/net/netfilter/nft_compat.c
index e1ca7f453ed7..67a138b9be3a 100644
--- a/net/netfilter/nft_compat.c
+++ b/net/netfilter/nft_compat.c
@@ -332,7 +332,8 @@ static int nft_extension_dump_info(struct sk_buff *skb, int attr,
return 0;
}
-static int nft_target_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_target_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct xt_target *target = expr->ops->data;
void *info = nft_expr_priv(expr);
@@ -388,6 +389,22 @@ static int nft_target_validate(const struct nft_ctx *ctx,
return 0;
}
+static int nft_target_bridge_validate(const struct nft_ctx *ctx,
+ const struct nft_expr *expr)
+{
+ struct xt_target *target = expr->ops->data;
+
+ /* Do not allow UNSPEC to stand-in for NFPROTO_BRIDGE
+ * targets: they are incompatible. ebtables targets return
+ * EBT_ACCEPT, DROP and so on which are not compatible with
+ * NF_ACCEPT, NF_DROP and so on.
+ */
+ if (target->family != NFPROTO_BRIDGE)
+ return -ENOENT;
+
+ return nft_target_validate(ctx, expr);
+}
+
static void __nft_match_eval(const struct nft_expr *expr,
struct nft_regs *regs,
const struct nft_pktinfo *pkt,
@@ -596,12 +613,14 @@ static int __nft_match_dump(struct sk_buff *skb, const struct nft_expr *expr,
return -1;
}
-static int nft_match_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_match_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
return __nft_match_dump(skb, expr, nft_expr_priv(expr));
}
-static int nft_match_large_dump(struct sk_buff *skb, const struct nft_expr *e)
+static int nft_match_large_dump(struct sk_buff *skb,
+ const struct nft_expr *e, bool reset)
{
struct nft_xt_match_priv *priv = nft_expr_priv(e);
@@ -913,14 +932,16 @@ nft_target_select_ops(const struct nft_ctx *ctx,
ops->init = nft_target_init;
ops->destroy = nft_target_destroy;
ops->dump = nft_target_dump;
- ops->validate = nft_target_validate;
ops->data = target;
ops->reduce = NFT_REDUCE_READONLY;
- if (family == NFPROTO_BRIDGE)
+ if (family == NFPROTO_BRIDGE) {
ops->eval = nft_target_eval_bridge;
- else
+ ops->validate = nft_target_bridge_validate;
+ } else {
ops->eval = nft_target_eval_xt;
+ ops->validate = nft_target_validate;
+ }
return ops;
err:
diff --git a/net/netfilter/nft_connlimit.c b/net/netfilter/nft_connlimit.c
index b3411dec16fd..5dd50b3ab5a4 100644
--- a/net/netfilter/nft_connlimit.c
+++ b/net/netfilter/nft_connlimit.c
@@ -179,7 +179,8 @@ static void nft_connlimit_eval(const struct nft_expr *expr,
nft_connlimit_do_eval(priv, regs, pkt, NULL);
}
-static int nft_connlimit_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_connlimit_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_connlimit *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_counter.c b/net/netfilter/nft_counter.c
index 224c7487b7a3..a578afc57c00 100644
--- a/net/netfilter/nft_counter.c
+++ b/net/netfilter/nft_counter.c
@@ -222,7 +222,8 @@ void nft_counter_eval(const struct nft_expr *expr, struct nft_regs *regs,
nft_counter_do_eval(priv, regs, pkt);
}
-static int nft_counter_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_counter_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_counter_percpu_priv *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_ct.c b/net/netfilter/nft_ct.c
index e2ef510ab048..f789c6c8f146 100644
--- a/net/netfilter/nft_ct.c
+++ b/net/netfilter/nft_ct.c
@@ -88,7 +88,7 @@ static void nft_ct_get_eval(const struct nft_expr *expr,
break;
}
- if (ct == NULL)
+ if (!ct || nf_ct_is_template(ct))
goto err;
switch (priv->key) {
@@ -190,12 +190,10 @@ static void nft_ct_get_eval(const struct nft_expr *expr,
tuple = &ct->tuplehash[priv->dir].tuple;
switch (priv->key) {
case NFT_CT_SRC:
- memcpy(dest, tuple->src.u3.all,
- nf_ct_l3num(ct) == NFPROTO_IPV4 ? 4 : 16);
+ memcpy(dest, tuple->src.u3.all, priv->len);
return;
case NFT_CT_DST:
- memcpy(dest, tuple->dst.u3.all,
- nf_ct_l3num(ct) == NFPROTO_IPV4 ? 4 : 16);
+ memcpy(dest, tuple->dst.u3.all, priv->len);
return;
case NFT_CT_PROTO_SRC:
nft_reg_store16(dest, (__force u16)tuple->src.u.all);
@@ -651,7 +649,8 @@ static void nft_ct_set_destroy(const struct nft_ctx *ctx,
nf_ct_netns_put(ctx->net, ctx->family);
}
-static int nft_ct_get_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_ct_get_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_ct *priv = nft_expr_priv(expr);
@@ -713,7 +712,8 @@ static bool nft_ct_get_reduce(struct nft_regs_track *track,
return nft_expr_reduce_bitwise(track, expr);
}
-static int nft_ct_set_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_ct_set_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_ct *priv = nft_expr_priv(expr);
@@ -910,8 +910,6 @@ static void nft_ct_timeout_obj_eval(struct nft_object *obj,
}
}
- rcu_assign_pointer(timeout->timeout, priv->timeout);
-
/* adjust the timeout as per 'new' state. ct is unconfirmed,
* so the current timestamp must not be added.
*/
@@ -962,6 +960,7 @@ static int nft_ct_timeout_obj_init(const struct nft_ctx *ctx,
timeout->l3num = l3num;
timeout->l4proto = l4proto;
+ refcount_set(&timeout->refcnt, 1);
ret = nf_ct_netns_get(ctx->net, ctx->family);
if (ret < 0)
@@ -982,10 +981,10 @@ static void nft_ct_timeout_obj_destroy(const struct nft_ctx *ctx,
struct nft_ct_timeout_obj *priv = nft_obj_data(obj);
struct nf_ct_timeout *timeout = priv->timeout;
- nf_queue_nf_hook_drop(ctx->net);
nf_ct_untimeout(ctx->net, timeout);
nf_ct_netns_put(ctx->net, ctx->family);
- kfree_rcu(priv->timeout, rcu);
+ if (refcount_dec_and_test(&timeout->refcnt))
+ kfree_rcu(priv->timeout, rcu);
}
static int nft_ct_timeout_obj_dump(struct sk_buff *skb,
@@ -1160,7 +1159,6 @@ static void nft_ct_helper_obj_eval(struct nft_object *obj,
help = nf_ct_helper_ext_add(ct, GFP_ATOMIC);
if (help) {
rcu_assign_pointer(help->helper, to_assign);
- set_bit(IPS_HELPER_BIT, &ct->status);
if ((ct->status & IPS_NAT_MASK) && !nfct_seqadj(ct))
if (!nfct_seqadj_ext_add(ct))
@@ -1342,7 +1340,7 @@ static void nft_ct_expect_obj_eval(struct nft_object *obj,
&ct->tuplehash[!dir].tuple.src.u3,
&ct->tuplehash[!dir].tuple.dst.u3,
priv->l4proto, NULL, &priv->dport);
- exp->timeout.expires = jiffies + priv->timeout * HZ;
+ exp->timeout += priv->timeout * HZ;
if (nf_ct_expect_related(exp, 0) != 0)
regs->verdict.code = NF_DROP;
diff --git a/net/netfilter/nft_dup_netdev.c b/net/netfilter/nft_dup_netdev.c
index 6e0a66588fb3..0573f96ce079 100644
--- a/net/netfilter/nft_dup_netdev.c
+++ b/net/netfilter/nft_dup_netdev.c
@@ -44,7 +44,8 @@ static int nft_dup_netdev_init(const struct nft_ctx *ctx,
sizeof(int));
}
-static int nft_dup_netdev_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_dup_netdev_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_dup_netdev *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_dynset.c b/net/netfilter/nft_dynset.c
index 6e71b937fb35..a1dfce36b539 100644
--- a/net/netfilter/nft_dynset.c
+++ b/net/netfilter/nft_dynset.c
@@ -19,7 +19,8 @@ struct nft_dynset {
u8 sreg_key;
u8 sreg_data;
bool invert;
- bool expr;
+ bool expr:1,
+ override_exprs:1;
u8 num_exprs;
u64 timeout;
struct nft_expr *expr_array[NFT_SET_EXPR_MAX];
@@ -254,6 +255,7 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
priv->num_exprs++;
priv->expr_array[0] = dynset_expr;
+ priv->override_exprs = true;
if (set->num_exprs > 1 ||
(set->num_exprs == 1 &&
@@ -286,6 +288,7 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
}
priv->expr_array[i] = dynset_expr;
priv->num_exprs++;
+ priv->override_exprs = true;
if (set->num_exprs) {
if (i >= set->num_exprs) {
@@ -304,9 +307,8 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
goto err_expr_free;
}
} else if (set->num_exprs > 0) {
- err = nft_set_elem_expr_clone(ctx, set, priv->expr_array);
- if (err < 0)
- return err;
+ for (i = 0; i < set->num_exprs; i++)
+ priv->expr_array[i] = set->exprs[i];
priv->num_exprs = set->num_exprs;
}
@@ -339,8 +341,10 @@ static int nft_dynset_init(const struct nft_ctx *ctx,
return 0;
err_expr_free:
- for (i = 0; i < priv->num_exprs; i++)
- nft_expr_destroy(ctx, priv->expr_array[i]);
+ if (priv->override_exprs) {
+ for (i = 0; i < priv->num_exprs; i++)
+ nft_expr_destroy(ctx, priv->expr_array[i]);
+ }
return err;
}
@@ -367,13 +371,15 @@ static void nft_dynset_destroy(const struct nft_ctx *ctx,
struct nft_dynset *priv = nft_expr_priv(expr);
int i;
- for (i = 0; i < priv->num_exprs; i++)
- nft_expr_destroy(ctx, priv->expr_array[i]);
-
+ if (priv->override_exprs) {
+ for (i = 0; i < priv->num_exprs; i++)
+ nft_expr_destroy(ctx, priv->expr_array[i]);
+ }
nf_tables_destroy_set(ctx, priv->set);
}
-static int nft_dynset_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_dynset_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_dynset *priv = nft_expr_priv(expr);
u32 flags = priv->invert ? NFT_DYNSET_F_INV : 0;
@@ -392,7 +398,7 @@ static int nft_dynset_dump(struct sk_buff *skb, const struct nft_expr *expr)
nf_jiffies64_to_msecs(priv->timeout),
NFTA_DYNSET_PAD))
goto nla_put_failure;
- if (priv->set->num_exprs == 0) {
+ if (priv->set->num_exprs == 0 || priv->override_exprs) {
if (priv->num_exprs == 1) {
if (nft_expr_dump(skb, NFTA_DYNSET_EXPR,
priv->expr_array[0]))
diff --git a/net/netfilter/nft_exthdr.c b/net/netfilter/nft_exthdr.c
index f3fefe70abff..4e9b0c1c5d30 100644
--- a/net/netfilter/nft_exthdr.c
+++ b/net/netfilter/nft_exthdr.c
@@ -672,7 +672,8 @@ static int nft_exthdr_dump_common(struct sk_buff *skb, const struct nft_exthdr *
return -1;
}
-static int nft_exthdr_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_exthdr_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_exthdr *priv = nft_expr_priv(expr);
@@ -682,7 +683,8 @@ static int nft_exthdr_dump(struct sk_buff *skb, const struct nft_expr *expr)
return nft_exthdr_dump_common(skb, priv);
}
-static int nft_exthdr_dump_set(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_exthdr_dump_set(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_exthdr *priv = nft_expr_priv(expr);
@@ -692,7 +694,8 @@ static int nft_exthdr_dump_set(struct sk_buff *skb, const struct nft_expr *expr)
return nft_exthdr_dump_common(skb, priv);
}
-static int nft_exthdr_dump_strip(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_exthdr_dump_strip(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_exthdr *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_fib.c b/net/netfilter/nft_fib.c
index 60913a97a47f..3acecd7b59bc 100644
--- a/net/netfilter/nft_fib.c
+++ b/net/netfilter/nft_fib.c
@@ -30,6 +30,15 @@ int nft_fib_validate(const struct nft_ctx *ctx, const struct nft_expr *expr)
const struct nft_fib *priv = nft_expr_priv(expr);
unsigned int hooks;
+ switch (ctx->family) {
+ case NFPROTO_IPV4:
+ case NFPROTO_IPV6:
+ case NFPROTO_INET:
+ break;
+ default:
+ return -EOPNOTSUPP;
+ }
+
switch (priv->result) {
case NFT_FIB_RESULT_OIF:
case NFT_FIB_RESULT_OIFNAME:
@@ -121,7 +130,7 @@ int nft_fib_init(const struct nft_ctx *ctx, const struct nft_expr *expr,
}
EXPORT_SYMBOL_GPL(nft_fib_init);
-int nft_fib_dump(struct sk_buff *skb, const struct nft_expr *expr)
+int nft_fib_dump(struct sk_buff *skb, const struct nft_expr *expr, bool reset)
{
const struct nft_fib *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_fib_netdev.c b/net/netfilter/nft_fib_netdev.c
index 9121ec64e918..ad62a06040ab 100644
--- a/net/netfilter/nft_fib_netdev.c
+++ b/net/netfilter/nft_fib_netdev.c
@@ -50,6 +50,33 @@ static void nft_fib_netdev_eval(const struct nft_expr *expr,
regs->verdict.code = NFT_BREAK;
}
+static int nft_fib_netdev_validate(const struct nft_ctx *ctx,
+ const struct nft_expr *expr)
+{
+ const struct nft_fib *priv = nft_expr_priv(expr);
+ unsigned int hooks;
+
+ switch (priv->result) {
+ case NFT_FIB_RESULT_OIF:
+ case NFT_FIB_RESULT_OIFNAME:
+ hooks = (1 << NF_NETDEV_INGRESS);
+ break;
+ case NFT_FIB_RESULT_ADDRTYPE:
+ if (priv->flags & NFTA_FIB_F_IIF)
+ hooks = (1 << NF_NETDEV_INGRESS);
+ else if (priv->flags & NFTA_FIB_F_OIF)
+ hooks = (1 << NF_NETDEV_EGRESS);
+ else
+ hooks = (1 << NF_NETDEV_INGRESS) |
+ (1 << NF_NETDEV_EGRESS);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return nft_chain_validate_hooks(ctx->chain, hooks);
+}
+
static struct nft_expr_type nft_fib_netdev_type;
static const struct nft_expr_ops nft_fib_netdev_ops = {
.type = &nft_fib_netdev_type,
@@ -57,7 +84,7 @@ static const struct nft_expr_ops nft_fib_netdev_ops = {
.eval = nft_fib_netdev_eval,
.init = nft_fib_init,
.dump = nft_fib_dump,
- .validate = nft_fib_validate,
+ .validate = nft_fib_netdev_validate,
.reduce = nft_fib_reduce,
};
diff --git a/net/netfilter/nft_flow_offload.c b/net/netfilter/nft_flow_offload.c
index 6bdafe91bc78..449d4b4725d2 100644
--- a/net/netfilter/nft_flow_offload.c
+++ b/net/netfilter/nft_flow_offload.c
@@ -149,10 +149,11 @@ static void nft_dev_path_info(const struct net_device_path_stack *stack,
info->num_encaps++;
break;
case DEV_PATH_BR_VLAN_UNTAG:
- if (WARN_ON_ONCE(info->num_encaps-- == 0)) {
+ if (info->num_encaps == 0) {
info->indev = NULL;
break;
}
+ info->num_encaps--;
break;
case DEV_PATH_BR_VLAN_KEEP:
break;
@@ -374,6 +375,7 @@ static void nft_flow_offload_eval(const struct nft_expr *expr,
if (tcph)
flow_offload_ct_tcp(ct);
+ __set_bit(NF_FLOW_HW_BIDIRECTIONAL, &flow->flags);
ret = flow_offload_add(flowtable, flow);
if (ret < 0)
goto err_flow_add;
@@ -416,6 +418,7 @@ static int nft_flow_offload_init(const struct nft_ctx *ctx,
struct nft_flow_offload *priv = nft_expr_priv(expr);
u8 genmask = nft_genmask_next(ctx->net);
struct nft_flowtable *flowtable;
+ int err;
if (!tb[NFTA_FLOW_TABLE_NAME])
return -EINVAL;
@@ -430,7 +433,11 @@ static int nft_flow_offload_init(const struct nft_ctx *ctx,
priv->flowtable = flowtable;
- return nf_ct_netns_get(ctx->net, ctx->family);
+ err = nf_ct_netns_get(ctx->net, ctx->family);
+ if (err < 0)
+ nft_use_dec(&flowtable->use);
+
+ return err;
}
static void nft_flow_offload_deactivate(const struct nft_ctx *ctx,
@@ -456,7 +463,8 @@ static void nft_flow_offload_destroy(const struct nft_ctx *ctx,
nf_ct_netns_put(ctx->net, ctx->family);
}
-static int nft_flow_offload_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_flow_offload_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_flow_offload *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_fwd_netdev.c b/net/netfilter/nft_fwd_netdev.c
index d1afebeef354..3b2dc5c3a138 100644
--- a/net/netfilter/nft_fwd_netdev.c
+++ b/net/netfilter/nft_fwd_netdev.c
@@ -56,7 +56,8 @@ static int nft_fwd_netdev_init(const struct nft_ctx *ctx,
sizeof(int));
}
-static int nft_fwd_netdev_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_fwd_netdev_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_fwd_netdev *priv = nft_expr_priv(expr);
@@ -196,7 +197,8 @@ static int nft_fwd_neigh_init(const struct nft_ctx *ctx,
addr_len);
}
-static int nft_fwd_neigh_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_fwd_neigh_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_fwd_neigh *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_hash.c b/net/netfilter/nft_hash.c
index e15d0edc761c..9833fa429e5e 100644
--- a/net/netfilter/nft_hash.c
+++ b/net/netfilter/nft_hash.c
@@ -139,7 +139,7 @@ static int nft_symhash_init(const struct nft_ctx *ctx,
}
static int nft_jhash_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_jhash *priv = nft_expr_priv(expr);
@@ -176,7 +176,7 @@ static bool nft_jhash_reduce(struct nft_regs_track *track,
}
static int nft_symhash_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_symhash *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_immediate.c b/net/netfilter/nft_immediate.c
index 44d564241847..02ee5fb69871 100644
--- a/net/netfilter/nft_immediate.c
+++ b/net/netfilter/nft_immediate.c
@@ -228,7 +228,8 @@ static void nft_immediate_destroy(const struct nft_ctx *ctx,
}
}
-static int nft_immediate_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_immediate_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_immediate_expr *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_last.c b/net/netfilter/nft_last.c
index 2ff387b649cf..de1b6066bfa8 100644
--- a/net/netfilter/nft_last.c
+++ b/net/netfilter/nft_last.c
@@ -65,7 +65,8 @@ static void nft_last_eval(const struct nft_expr *expr,
WRITE_ONCE(last->set, 1);
}
-static int nft_last_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_last_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_last_priv *priv = nft_expr_priv(expr);
struct nft_last *last = priv->last;
diff --git a/net/netfilter/nft_limit.c b/net/netfilter/nft_limit.c
index a263596da99a..21d26b79b460 100644
--- a/net/netfilter/nft_limit.c
+++ b/net/netfilter/nft_limit.c
@@ -207,7 +207,8 @@ static int nft_limit_pkts_init(const struct nft_ctx *ctx,
return 0;
}
-static int nft_limit_pkts_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_limit_pkts_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_limit_priv_pkts *priv = nft_expr_priv(expr);
@@ -266,7 +267,7 @@ static int nft_limit_bytes_init(const struct nft_ctx *ctx,
}
static int nft_limit_bytes_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_limit_priv *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_log.c b/net/netfilter/nft_log.c
index b510b01a06ab..bf01cf8a8907 100644
--- a/net/netfilter/nft_log.c
+++ b/net/netfilter/nft_log.c
@@ -185,7 +185,8 @@ static void nft_log_destroy(const struct nft_ctx *ctx,
nf_logger_put(ctx->family, li->type);
}
-static int nft_log_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_log_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_log *priv = nft_expr_priv(expr);
const struct nf_loginfo *li = &priv->loginfo;
diff --git a/net/netfilter/nft_lookup.c b/net/netfilter/nft_lookup.c
index fb4c129c63d3..344d6d110b8d 100644
--- a/net/netfilter/nft_lookup.c
+++ b/net/netfilter/nft_lookup.c
@@ -64,14 +64,16 @@ void nft_lookup_eval(const struct nft_expr *expr,
const struct net *net = nft_net(pkt);
bool found;
- found = nft_set_do_lookup(net, set, ®s->data[priv->sreg], &ext) ^
- priv->invert;
+ found = nft_set_do_lookup(net, set, ®s->data[priv->sreg], &ext);
if (!found) {
ext = nft_set_catchall_lookup(net, set);
- if (!ext) {
- regs->verdict.code = NFT_BREAK;
- return;
- }
+ found = !!ext;
+ }
+
+ found ^= priv->invert;
+ if (!found) {
+ regs->verdict.code = NFT_BREAK;
+ return;
}
if (ext) {
@@ -179,7 +181,8 @@ static void nft_lookup_destroy(const struct nft_ctx *ctx,
nf_tables_destroy_set(ctx, priv->set);
}
-static int nft_lookup_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_lookup_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_lookup *priv = nft_expr_priv(expr);
u32 flags = priv->invert ? NFT_LOOKUP_F_INV : 0;
diff --git a/net/netfilter/nft_masq.c b/net/netfilter/nft_masq.c
index b38f455a6c99..c9e42af8e81c 100644
--- a/net/netfilter/nft_masq.c
+++ b/net/netfilter/nft_masq.c
@@ -72,7 +72,8 @@ static int nft_masq_init(const struct nft_ctx *ctx,
return nf_ct_netns_get(ctx->net, ctx->family);
}
-static int nft_masq_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_masq_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_masq *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_meta.c b/net/netfilter/nft_meta.c
index 671923f4d9eb..0421751c97e0 100644
--- a/net/netfilter/nft_meta.c
+++ b/net/netfilter/nft_meta.c
@@ -667,7 +667,7 @@ int nft_meta_set_init(const struct nft_ctx *ctx,
EXPORT_SYMBOL_GPL(nft_meta_set_init);
int nft_meta_get_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_meta *priv = nft_expr_priv(expr);
@@ -682,7 +682,8 @@ int nft_meta_get_dump(struct sk_buff *skb,
}
EXPORT_SYMBOL_GPL(nft_meta_get_dump);
-int nft_meta_set_dump(struct sk_buff *skb, const struct nft_expr *expr)
+int nft_meta_set_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_meta *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_nat.c b/net/netfilter/nft_nat.c
index e1e2e46dbea7..3b644e6f5a68 100644
--- a/net/netfilter/nft_nat.c
+++ b/net/netfilter/nft_nat.c
@@ -259,7 +259,8 @@ static int nft_nat_init(const struct nft_ctx *ctx, const struct nft_expr *expr,
return nf_ct_netns_get(ctx->net, family);
}
-static int nft_nat_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_nat_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_nat *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_numgen.c b/net/netfilter/nft_numgen.c
index 5b4dde41ec36..bd058babfc82 100644
--- a/net/netfilter/nft_numgen.c
+++ b/net/netfilter/nft_numgen.c
@@ -112,7 +112,8 @@ static int nft_ng_dump(struct sk_buff *skb, enum nft_registers dreg,
return -1;
}
-static int nft_ng_inc_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_ng_inc_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_ng_inc *priv = nft_expr_priv(expr);
@@ -168,7 +169,8 @@ static int nft_ng_random_init(const struct nft_ctx *ctx,
NULL, NFT_DATA_VALUE, sizeof(u32));
}
-static int nft_ng_random_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_ng_random_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_ng_random *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_objref.c b/net/netfilter/nft_objref.c
index 35d8042464b3..b7a30683b6e5 100644
--- a/net/netfilter/nft_objref.c
+++ b/net/netfilter/nft_objref.c
@@ -78,7 +78,8 @@ static int nft_objref_init(const struct nft_ctx *ctx,
return 0;
}
-static int nft_objref_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_objref_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_object *obj = nft_objref_priv(expr);
@@ -188,7 +189,8 @@ static int nft_objref_map_init(const struct nft_ctx *ctx,
return 0;
}
-static int nft_objref_map_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_objref_map_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_objref_map *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_osf.c b/net/netfilter/nft_osf.c
index c81842649105..96096819dbfa 100644
--- a/net/netfilter/nft_osf.c
+++ b/net/netfilter/nft_osf.c
@@ -97,7 +97,8 @@ static int nft_osf_init(const struct nft_ctx *ctx,
return 0;
}
-static int nft_osf_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_osf_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_osf *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_payload.c b/net/netfilter/nft_payload.c
index 0669b91526c2..64ade8be817c 100644
--- a/net/netfilter/nft_payload.c
+++ b/net/netfilter/nft_payload.c
@@ -198,7 +198,8 @@ static int nft_payload_init(const struct nft_ctx *ctx,
priv->len);
}
-static int nft_payload_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_payload_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_payload *priv = nft_expr_priv(expr);
@@ -851,7 +852,8 @@ static int nft_payload_set_init(const struct nft_ctx *ctx,
priv->len);
}
-static int nft_payload_set_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_payload_set_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_payload_set *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_queue.c b/net/netfilter/nft_queue.c
index 5fb9e89d929d..344fe311878f 100644
--- a/net/netfilter/nft_queue.c
+++ b/net/netfilter/nft_queue.c
@@ -151,7 +151,8 @@ static int nft_queue_sreg_init(const struct nft_ctx *ctx,
return 0;
}
-static int nft_queue_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_queue_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_queue *priv = nft_expr_priv(expr);
@@ -167,7 +168,8 @@ static int nft_queue_dump(struct sk_buff *skb, const struct nft_expr *expr)
}
static int
-nft_queue_sreg_dump(struct sk_buff *skb, const struct nft_expr *expr)
+nft_queue_sreg_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_queue *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_quota.c b/net/netfilter/nft_quota.c
index 60e6d0c5f04e..4bbb63d9b947 100644
--- a/net/netfilter/nft_quota.c
+++ b/net/netfilter/nft_quota.c
@@ -223,7 +223,8 @@ static int nft_quota_init(const struct nft_ctx *ctx,
return nft_quota_do_init(tb, priv);
}
-static int nft_quota_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_quota_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_quota *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_range.c b/net/netfilter/nft_range.c
index 229f274f1823..d75772986d05 100644
--- a/net/netfilter/nft_range.c
+++ b/net/netfilter/nft_range.c
@@ -111,7 +111,8 @@ static int nft_range_init(const struct nft_ctx *ctx, const struct nft_expr *expr
return err;
}
-static int nft_range_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_range_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_range_expr *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_redir.c b/net/netfilter/nft_redir.c
index ffde20e5cb5e..ddbb6a79262a 100644
--- a/net/netfilter/nft_redir.c
+++ b/net/netfilter/nft_redir.c
@@ -76,7 +76,8 @@ static int nft_redir_init(const struct nft_ctx *ctx,
return nf_ct_netns_get(ctx->net, ctx->family);
}
-static int nft_redir_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_redir_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_redir *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_reject.c b/net/netfilter/nft_reject.c
index 3ef8c41d6876..9dab7828e1b8 100644
--- a/net/netfilter/nft_reject.c
+++ b/net/netfilter/nft_reject.c
@@ -68,7 +68,8 @@ int nft_reject_init(const struct nft_ctx *ctx,
}
EXPORT_SYMBOL_GPL(nft_reject_init);
-int nft_reject_dump(struct sk_buff *skb, const struct nft_expr *expr)
+int nft_reject_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
const struct nft_reject *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_rt.c b/net/netfilter/nft_rt.c
index 8c94bc82f635..0e95d0d3c557 100644
--- a/net/netfilter/nft_rt.c
+++ b/net/netfilter/nft_rt.c
@@ -146,7 +146,7 @@ static int nft_rt_get_init(const struct nft_ctx *ctx,
}
static int nft_rt_get_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_rt *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_set_bitmap.c b/net/netfilter/nft_set_bitmap.c
index cbf7f7825f1b..06b77100bef7 100644
--- a/net/netfilter/nft_set_bitmap.c
+++ b/net/netfilter/nft_set_bitmap.c
@@ -123,7 +123,7 @@ static void *nft_bitmap_get(const struct net *net, const struct nft_set *set,
static int nft_bitmap_insert(const struct net *net, const struct nft_set *set,
const struct nft_set_elem *elem,
- struct nft_set_ext **ext)
+ void **elem_priv)
{
struct nft_bitmap *priv = nft_set_priv(set);
struct nft_bitmap_elem *new = elem->priv, *be;
@@ -132,7 +132,7 @@ static int nft_bitmap_insert(const struct net *net, const struct nft_set *set,
be = nft_bitmap_elem_find(set, new, genmask);
if (be) {
- *ext = &be->ext;
+ *elem_priv = be;
return -EEXIST;
}
diff --git a/net/netfilter/nft_set_hash.c b/net/netfilter/nft_set_hash.c
index 5a74ee4b7dfb..60c0d72fb98b 100644
--- a/net/netfilter/nft_set_hash.c
+++ b/net/netfilter/nft_set_hash.c
@@ -168,7 +168,7 @@ static bool nft_rhash_update(struct nft_set *set, const u32 *key,
static int nft_rhash_insert(const struct net *net, const struct nft_set *set,
const struct nft_set_elem *elem,
- struct nft_set_ext **ext)
+ void **elem_priv)
{
struct nft_rhash *priv = nft_set_priv(set);
struct nft_rhash_elem *he = elem->priv;
@@ -185,7 +185,7 @@ static int nft_rhash_insert(const struct net *net, const struct nft_set *set,
if (IS_ERR(prev))
return PTR_ERR(prev);
if (prev) {
- *ext = &prev->ext;
+ *elem_priv = prev;
return -EEXIST;
}
return 0;
@@ -588,7 +588,7 @@ static u32 nft_jhash(const struct nft_set *set, const struct nft_hash *priv,
static int nft_hash_insert(const struct net *net, const struct nft_set *set,
const struct nft_set_elem *elem,
- struct nft_set_ext **ext)
+ void **elem_priv)
{
struct nft_hash_elem *this = elem->priv, *he;
struct nft_hash *priv = nft_set_priv(set);
@@ -600,7 +600,7 @@ static int nft_hash_insert(const struct net *net, const struct nft_set *set,
if (!memcmp(nft_set_ext_key(&this->ext),
nft_set_ext_key(&he->ext), set->klen) &&
nft_set_elem_active(&he->ext, genmask)) {
- *ext = &he->ext;
+ *elem_priv = he;
return -EEXIST;
}
}
diff --git a/net/netfilter/nft_set_pipapo.c b/net/netfilter/nft_set_pipapo.c
index 4b37c6684bff..13cb3059744e 100644
--- a/net/netfilter/nft_set_pipapo.c
+++ b/net/netfilter/nft_set_pipapo.c
@@ -1260,13 +1260,13 @@ static struct nft_pipapo_match *pipapo_maybe_clone(const struct nft_set *set)
* @net: Network namespace
* @set: nftables API set representation
* @elem: nftables API element representation containing key data
- * @ext2: Filled with pointer to &struct nft_set_ext in inserted element
+ * @elem_priv: Filled with pointer to opaque set element
*
* Return: 0 on success, error pointer on failure.
*/
static int nft_pipapo_insert(const struct net *net, const struct nft_set *set,
const struct nft_set_elem *elem,
- struct nft_set_ext **ext2)
+ void **elem_priv)
{
const struct nft_set_ext *ext = nft_set_elem_ext(set, elem->priv);
union nft_pipapo_map_bucket rulemap[NFT_PIPAPO_MAX_FIELDS];
@@ -1301,7 +1301,7 @@ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set,
if (!memcmp(start, dup_key->data, sizeof(*dup_key->data)) &&
!memcmp(end, dup_end->data, sizeof(*dup_end->data))) {
- *ext2 = &dup->ext;
+ *elem_priv = dup;
return -EEXIST;
}
@@ -1317,7 +1317,7 @@ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set,
if (PTR_ERR(dup) != -ENOENT) {
if (IS_ERR(dup))
return PTR_ERR(dup);
- *ext2 = &dup->ext;
+ *elem_priv = dup;
return -ENOTEMPTY;
}
@@ -1379,7 +1379,7 @@ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set,
put_cpu_ptr(m->scratch);
}
- *ext2 = &e->ext;
+ *elem_priv = e;
pipapo_map(m, rulemap, e);
@@ -1466,7 +1466,7 @@ static struct nft_pipapo_match *pipapo_clone(struct nft_pipapo_match *old)
goto out_mt;
dst->mt = kvmalloc_array(src->rules, sizeof(*src->mt),
- GFP_KERNEL);
+ GFP_KERNEL_ACCOUNT);
if (!dst->mt)
goto out_mt;
@@ -2200,13 +2200,20 @@ static void nft_pipapo_walk(const struct nft_ctx *ctx, struct nft_set *set,
const struct nft_pipapo_match *m;
switch (iter->type) {
- case NFT_ITER_UPDATE:
+ case NFT_ITER_UPDATE_CLONE:
m = pipapo_maybe_clone(set);
if (!m) {
iter->err = -ENOMEM;
return;
}
-
+ nft_pipapo_do_walk(ctx, set, m, iter);
+ break;
+ case NFT_ITER_UPDATE:
+ if (priv->clone)
+ m = priv->clone;
+ else
+ m = rcu_dereference_protected(priv->match,
+ nft_pipapo_transaction_mutex_held(set));
nft_pipapo_do_walk(ctx, set, m, iter);
break;
case NFT_ITER_READ:
@@ -2387,6 +2394,9 @@ static void nft_pipapo_destroy(const struct nft_ctx *ctx,
if (m) {
rcu_barrier();
+ if (!priv->clone)
+ nft_set_pipapo_match_destroy(ctx, set, m);
+
for_each_possible_cpu(cpu)
pipapo_free_scratch(m, cpu);
free_percpu(m->scratch);
diff --git a/net/netfilter/nft_set_rbtree.c b/net/netfilter/nft_set_rbtree.c
index 433094c4200f..e82e5cf107ee 100644
--- a/net/netfilter/nft_set_rbtree.c
+++ b/net/netfilter/nft_set_rbtree.c
@@ -319,7 +319,7 @@ static struct nft_rbtree_elem *nft_rbtree_prev_active(struct nft_rbtree_elem *rb
static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set,
struct nft_rbtree_elem *new,
- struct nft_set_ext **ext)
+ void **elem_priv)
{
struct nft_rbtree_elem *rbe, *rbe_le = NULL, *rbe_ge = NULL, *rbe_prev;
struct rb_node *node, *next, *parent, **p, *first = NULL;
@@ -437,7 +437,7 @@ static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set,
*/
if (rbe_ge && !nft_rbtree_cmp(set, new, rbe_ge) &&
nft_rbtree_interval_start(rbe_ge) == nft_rbtree_interval_start(new)) {
- *ext = &rbe_ge->ext;
+ *elem_priv = rbe_ge;
return -EEXIST;
}
@@ -446,7 +446,7 @@ static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set,
*/
if (rbe_le && !nft_rbtree_cmp(set, new, rbe_le) &&
nft_rbtree_interval_end(rbe_le) == nft_rbtree_interval_end(new)) {
- *ext = &rbe_le->ext;
+ *elem_priv = rbe_le;
return -EEXIST;
}
@@ -506,7 +506,7 @@ static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set,
static int nft_rbtree_insert(const struct net *net, const struct nft_set *set,
const struct nft_set_elem *elem,
- struct nft_set_ext **ext)
+ void **elem_priv)
{
struct nft_rbtree *priv = nft_set_priv(set);
struct nft_rbtree_elem *rbe = elem->priv;
@@ -520,7 +520,7 @@ static int nft_rbtree_insert(const struct net *net, const struct nft_set *set,
write_lock_bh(&priv->lock);
write_seqcount_begin(&priv->count);
- err = __nft_rbtree_insert(net, set, rbe, ext);
+ err = __nft_rbtree_insert(net, set, rbe, elem_priv);
write_seqcount_end(&priv->count);
write_unlock_bh(&priv->lock);
} while (err == -EAGAIN);
diff --git a/net/netfilter/nft_socket.c b/net/netfilter/nft_socket.c
index f56953f1022e..6d6bba8d7fb7 100644
--- a/net/netfilter/nft_socket.c
+++ b/net/netfilter/nft_socket.c
@@ -235,7 +235,7 @@ static int nft_socket_init(const struct nft_ctx *ctx,
}
static int nft_socket_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_socket *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_synproxy.c b/net/netfilter/nft_synproxy.c
index 3c674137eb81..08e9e1416b87 100644
--- a/net/netfilter/nft_synproxy.c
+++ b/net/netfilter/nft_synproxy.c
@@ -274,7 +274,8 @@ static void nft_synproxy_destroy(const struct nft_ctx *ctx,
nft_synproxy_do_destroy(ctx);
}
-static int nft_synproxy_dump(struct sk_buff *skb, const struct nft_expr *expr)
+static int nft_synproxy_dump(struct sk_buff *skb,
+ const struct nft_expr *expr, bool reset)
{
struct nft_synproxy *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_tproxy.c b/net/netfilter/nft_tproxy.c
index b848539f72a8..430c6ef0fa85 100644
--- a/net/netfilter/nft_tproxy.c
+++ b/net/netfilter/nft_tproxy.c
@@ -294,7 +294,7 @@ static void nft_tproxy_destroy(const struct nft_ctx *ctx,
}
static int nft_tproxy_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_tproxy *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_tunnel.c b/net/netfilter/nft_tunnel.c
index 1dda2096949d..92e56b656c18 100644
--- a/net/netfilter/nft_tunnel.c
+++ b/net/netfilter/nft_tunnel.c
@@ -108,7 +108,7 @@ static int nft_tunnel_get_init(const struct nft_ctx *ctx,
}
static int nft_tunnel_get_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_tunnel *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/nft_xfrm.c b/net/netfilter/nft_xfrm.c
index 34a37699f90f..82be7823d9c8 100644
--- a/net/netfilter/nft_xfrm.c
+++ b/net/netfilter/nft_xfrm.c
@@ -212,7 +212,7 @@ static void nft_xfrm_get_eval(const struct nft_expr *expr,
}
static int nft_xfrm_get_dump(struct sk_buff *skb,
- const struct nft_expr *expr)
+ const struct nft_expr *expr, bool reset)
{
const struct nft_xfrm *priv = nft_expr_priv(expr);
diff --git a/net/netfilter/xt_CT.c b/net/netfilter/xt_CT.c
index 498f5871c84a..90a295cf5ec6 100644
--- a/net/netfilter/xt_CT.c
+++ b/net/netfilter/xt_CT.c
@@ -284,7 +284,7 @@ static void xt_ct_tg_destroy(const struct xt_tgdtor_param *par,
struct nf_conn_help *help;
if (ct) {
- if (info->helper[0] || info->timeout[0])
+ if (info->helper[0])
nf_queue_nf_hook_drop(par->net);
help = nfct_help(ct);
diff --git a/net/netfilter/xt_IDLETIMER.c b/net/netfilter/xt_IDLETIMER.c
index 9733f49847a6..aa93c812317e 100644
--- a/net/netfilter/xt_IDLETIMER.c
+++ b/net/netfilter/xt_IDLETIMER.c
@@ -415,21 +415,23 @@ static void idletimer_tg_destroy(const struct xt_tgdtor_param *par)
mutex_lock(&list_mutex);
- if (--info->timer->refcnt == 0) {
- pr_debug("deleting timer %s\n", info->label);
-
- list_del(&info->timer->entry);
- del_timer_sync(&info->timer->timer);
- cancel_work_sync(&info->timer->work);
- sysfs_remove_file(idletimer_tg_kobj, &info->timer->attr.attr);
- kfree(info->timer->attr.attr.name);
- kfree(info->timer);
- } else {
+ if (--info->timer->refcnt > 0) {
pr_debug("decreased refcnt of timer %s to %u\n",
info->label, info->timer->refcnt);
+ mutex_unlock(&list_mutex);
+ return;
}
+ pr_debug("deleting timer %s\n", info->label);
+
+ list_del(&info->timer->entry);
mutex_unlock(&list_mutex);
+
+ del_timer_sync(&info->timer->timer);
+ cancel_work_sync(&info->timer->work);
+ sysfs_remove_file(idletimer_tg_kobj, &info->timer->attr.attr);
+ kfree(info->timer->attr.attr.name);
+ kfree(info->timer);
}
static void idletimer_tg_destroy_v1(const struct xt_tgdtor_param *par)
@@ -440,25 +442,27 @@ static void idletimer_tg_destroy_v1(const struct xt_tgdtor_param *par)
mutex_lock(&list_mutex);
- if (--info->timer->refcnt == 0) {
- pr_debug("deleting timer %s\n", info->label);
-
- list_del(&info->timer->entry);
- if (info->timer->timer_type & XT_IDLETIMER_ALARM) {
- alarm_cancel(&info->timer->alarm);
- } else {
- del_timer_sync(&info->timer->timer);
- }
- cancel_work_sync(&info->timer->work);
- sysfs_remove_file(idletimer_tg_kobj, &info->timer->attr.attr);
- kfree(info->timer->attr.attr.name);
- kfree(info->timer);
- } else {
+ if (--info->timer->refcnt > 0) {
pr_debug("decreased refcnt of timer %s to %u\n",
info->label, info->timer->refcnt);
+ mutex_unlock(&list_mutex);
+ return;
}
+ pr_debug("deleting timer %s\n", info->label);
+
+ list_del(&info->timer->entry);
mutex_unlock(&list_mutex);
+
+ if (info->timer->timer_type & XT_IDLETIMER_ALARM) {
+ alarm_cancel(&info->timer->alarm);
+ } else {
+ del_timer_sync(&info->timer->timer);
+ }
+ cancel_work_sync(&info->timer->work);
+ sysfs_remove_file(idletimer_tg_kobj, &info->timer->attr.attr);
+ kfree(info->timer->attr.attr.name);
+ kfree(info->timer);
}
diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c
index f5d4eba785d0..81f4494ed67e 100644
--- a/net/netlink/af_netlink.c
+++ b/net/netlink/af_netlink.c
@@ -463,7 +463,7 @@ netlink_unlock_table(void)
struct netlink_compare_arg
{
- possible_net_t pnet;
+ u64 net_cookie;
u32 portid;
};
@@ -478,14 +478,14 @@ static inline int netlink_compare(struct rhashtable_compare_arg *arg,
const struct netlink_sock *nlk = ptr;
return nlk->portid != x->portid ||
- !net_eq(sock_net(&nlk->sk), read_pnet(&x->pnet));
+ sock_net(&nlk->sk)->net_cookie != x->net_cookie;
}
static void netlink_compare_arg_init(struct netlink_compare_arg *arg,
struct net *net, u32 portid)
{
memset(arg, 0, sizeof(*arg));
- write_pnet(&arg->pnet, net);
+ arg->net_cookie = net->net_cookie;
arg->portid = portid;
}
diff --git a/net/netlink/genetlink.c b/net/netlink/genetlink.c
index 3a3b29ff63b3..4ededd793767 100644
--- a/net/netlink/genetlink.c
+++ b/net/netlink/genetlink.c
@@ -1159,6 +1159,7 @@ static int genl_ctrl_event(int event, const struct genl_family *family,
struct ctrl_dump_policy_ctx {
struct netlink_policy_dump_state *state;
const struct genl_family *rt;
+ struct module *owner;
unsigned int opidx;
u32 op;
u16 fam_id;
@@ -1202,6 +1203,9 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
return -ENOENT;
ctx->rt = rt;
+ ctx->owner = rt->module;
+ if (!try_module_get(ctx->owner))
+ return -ENOENT;
if (tb[CTRL_ATTR_OP]) {
ctx->single_op = true;
@@ -1210,14 +1214,19 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
err = genl_get_cmd(ctx->op, rt, &op);
if (err) {
NL_SET_BAD_ATTR(cb->extack, tb[CTRL_ATTR_OP]);
- return err;
+ goto err_put_owner;
}
- if (!op.policy)
- return -ENODATA;
+ if (!op.policy) {
+ err = -ENODATA;
+ goto err_put_owner;
+ }
- return netlink_policy_dump_add_policy(&ctx->state, op.policy,
- op.maxattr);
+ err = netlink_policy_dump_add_policy(&ctx->state, op.policy,
+ op.maxattr);
+ if (err)
+ goto err_free_state;
+ return 0;
}
for (i = 0; i < genl_get_cmd_cnt(rt); i++) {
@@ -1232,12 +1241,16 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb)
}
}
- if (!ctx->state)
- return -ENODATA;
+ if (!ctx->state) {
+ err = -ENODATA;
+ goto err_put_owner;
+ }
return 0;
err_free_state:
netlink_policy_dump_free(ctx->state);
+err_put_owner:
+ module_put(ctx->owner);
return err;
}
@@ -1372,6 +1385,7 @@ static int ctrl_dumppolicy_done(struct netlink_callback *cb)
struct ctrl_dump_policy_ctx *ctx = (void *)cb->ctx;
netlink_policy_dump_free(ctx->state);
+ module_put(ctx->owner);
return 0;
}
diff --git a/net/nfc/nci/core.c b/net/nfc/nci/core.c
index bc5f7068e6db..bfbe405a474e 100644
--- a/net/nfc/nci/core.c
+++ b/net/nfc/nci/core.c
@@ -566,6 +566,10 @@ static int nci_close_device(struct nci_dev *ndev)
flush_workqueue(ndev->cmd_wq);
del_timer_sync(&ndev->cmd_timer);
del_timer_sync(&ndev->data_timer);
+ if (test_bit(NCI_DATA_EXCHANGE, &ndev->flags))
+ nci_data_exchange_complete(ndev, NULL,
+ ndev->cur_conn_id,
+ -ENODEV);
mutex_unlock(&ndev->req_lock);
return 0;
}
@@ -596,6 +600,11 @@ static int nci_close_device(struct nci_dev *ndev)
flush_workqueue(ndev->cmd_wq);
del_timer_sync(&ndev->cmd_timer);
+ del_timer_sync(&ndev->data_timer);
+
+ if (test_bit(NCI_DATA_EXCHANGE, &ndev->flags))
+ nci_data_exchange_complete(ndev, NULL, ndev->cur_conn_id,
+ -ENODEV);
/* Clear flags except NCI_UNREG */
ndev->flags &= BIT(NCI_UNREG);
diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c
index 679409953ac5..00a3e0af9565 100644
--- a/net/openvswitch/actions.c
+++ b/net/openvswitch/actions.c
@@ -380,7 +380,12 @@ static void set_ip_addr(struct sk_buff *skb, struct iphdr *nh,
static void update_ipv6_checksum(struct sk_buff *skb, u8 l4_proto,
__be32 addr[4], const __be32 new_addr[4])
{
- int transport_len = skb->len - skb_transport_offset(skb);
+ int transport_len;
+
+ if (l4_proto == NEXTHDR_FRAGMENT)
+ return;
+
+ transport_len = skb->len - skb_transport_offset(skb);
if (l4_proto == NEXTHDR_TCP) {
if (likely(transport_len >= sizeof(struct tcphdr)))
diff --git a/net/openvswitch/conntrack.c b/net/openvswitch/conntrack.c
index a7bb9e360ed3..9407fdcbb52a 100644
--- a/net/openvswitch/conntrack.c
+++ b/net/openvswitch/conntrack.c
@@ -435,16 +435,14 @@ static int ovs_ct_set_labels(struct nf_conn *ct, struct sw_flow_key *key,
}
/* 'skb' should already be pulled to nh_ofs. */
-static int ovs_ct_helper(struct sk_buff *skb, u16 proto)
+static int nf_ct_helper(struct sk_buff *skb, struct nf_conn *ct,
+ enum ip_conntrack_info ctinfo, u16 proto)
{
const struct nf_conntrack_helper *helper;
const struct nf_conn_help *help;
- enum ip_conntrack_info ctinfo;
unsigned int protoff;
- struct nf_conn *ct;
int err;
- ct = nf_ct_get(skb, &ctinfo);
if (!ct || ctinfo == IP_CT_RELATED_REPLY)
return NF_ACCEPT;
@@ -799,7 +797,7 @@ static void ovs_nat_update_key(struct sw_flow_key *key,
/* Modelled after nf_nat_ipv[46]_fn().
* range is only used for new, uninitialized NAT state.
- * Returns either NF_ACCEPT or NF_DROP.
+ * Returns NF_ACCEPT, NF_DROP or NF_STOLEN.
*/
static int ovs_ct_nat_execute(struct sk_buff *skb, struct nf_conn *ct,
enum ip_conntrack_info ctinfo,
@@ -874,6 +872,9 @@ static int ovs_ct_nat_execute(struct sk_buff *skb, struct nf_conn *ct,
err = nf_nat_packet(ct, ctinfo, hooknum, skb);
push:
+ if ((err & NF_VERDICT_MASK) == NF_STOLEN)
+ return err;
+
skb_push_rcsum(skb, nh_off);
/* Update the flow key if NAT successful. */
@@ -951,6 +952,14 @@ static int ovs_ct_nat(struct net *net, struct sw_flow_key *key,
}
#endif
+/* Translate netfilter verdicts without evaluating the argument twice. */
+#define verdict_to_errno(verdict) ({ \
+ unsigned int __verdict = (verdict) & NF_VERDICT_MASK; \
+ \
+ __verdict == NF_ACCEPT ? 0 : \
+ __verdict == NF_STOLEN ? -EINPROGRESS : -EINVAL; \
+})
+
/* Pass 'skb' through conntrack in 'net', using zone configured in 'info', if
* not done already. Update key with new CT state after passing the packet
* through conntrack.
@@ -1001,7 +1010,7 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
err = nf_conntrack_in(skb, &state);
if (err != NF_ACCEPT)
- return -ENOENT;
+ return verdict_to_errno(err);
/* Clear CT state NAT flags to mark that we have not yet done
* NAT after the nf_conntrack_in() call. We can actually clear
@@ -1015,8 +1024,6 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
ct = nf_ct_get(skb, &ctinfo);
if (ct) {
- bool add_helper = false;
-
/* Packets starting a new connection must be NATted before the
* helper, so that the helper knows about the NAT. We enforce
* this by delaying both NAT and helper calls for unconfirmed
@@ -1028,9 +1035,12 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
* the key->ct_state.
*/
if (info->nat && !(key->ct_state & OVS_CS_F_NAT_MASK) &&
- (nf_ct_is_confirmed(ct) || info->commit) &&
- ovs_ct_nat(net, key, info, skb, ct, ctinfo) != NF_ACCEPT) {
- return -EINVAL;
+ (nf_ct_is_confirmed(ct) || info->commit)) {
+ int err = ovs_ct_nat(net, key, info, skb, ct, ctinfo);
+
+ err = verdict_to_errno(err);
+ if (err)
+ return err;
}
/* Userspace may decide to perform a ct lookup without a helper
@@ -1045,7 +1055,6 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
GFP_ATOMIC);
if (err)
return err;
- add_helper = true;
/* helper installed, add seqadj if NAT is required */
if (info->nat && !nfct_seqadj(ct)) {
@@ -1054,16 +1063,19 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key,
}
}
- /* Call the helper only if:
- * - nf_conntrack_in() was executed above ("!cached") or a
- * helper was just attached ("add_helper") for a confirmed
- * connection, or
- * - When committing an unconfirmed connection.
+ /* Call the helper only if nf_conntrack_in() was executed
+ * above ("!cached").
+ *
+ * For unconfirmed connections it will be called later during
+ * commit as we need to have all the other extensions allocated
+ * before the call.
*/
- if ((nf_ct_is_confirmed(ct) ? !cached || add_helper :
- info->commit) &&
- ovs_ct_helper(skb, info->family) != NF_ACCEPT) {
- return -EINVAL;
+ if (nf_ct_is_confirmed(ct) && !cached) {
+ int err = nf_ct_helper(skb, ct, ctinfo, info->family);
+
+ err = verdict_to_errno(err);
+ if (err)
+ return err;
}
if (nf_ct_protonum(ct) == IPPROTO_TCP &&
@@ -1281,6 +1293,14 @@ static int ovs_ct_commit(struct net *net, struct sw_flow_key *key,
return err;
nf_conn_act_ct_ext_add(skb, ct, ctinfo);
+
+ /* Call the helpers now. We couldn't do this before as
+ * all the extensions must be allocated before the call.
+ */
+ err = nf_ct_helper(skb, ct, ctinfo, info->family);
+ err = verdict_to_errno(err);
+ if (err)
+ return err;
} else if (IS_ENABLED(CONFIG_NF_CONNTRACK_LABELS) &&
labels_nonzero(&info->labels.mask)) {
err = ovs_ct_set_labels(ct, key, &info->labels.value,
@@ -1291,10 +1311,9 @@ static int ovs_ct_commit(struct net *net, struct sw_flow_key *key,
/* This will take care of sending queued events even if the connection
* is already confirmed.
*/
- if (nf_conntrack_confirm(skb) != NF_ACCEPT)
- return -EINVAL;
+ err = nf_conntrack_confirm(skb);
- return 0;
+ return verdict_to_errno(err);
}
/* Trim the skb to the length specified by the IP/IPv6 header,
@@ -1357,6 +1376,10 @@ int ovs_ct_execute(struct net *net, struct sk_buff *skb,
else
err = ovs_ct_lookup(net, key, info, skb);
+ /* conntrack core returned NF_STOLEN */
+ if (err == -EINPROGRESS)
+ return err;
+
skb_push_rcsum(skb, nh_ofs);
if (err)
kfree_skb(skb);
diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c
index bb68c3ce9519..2f777d9d2434 100644
--- a/net/openvswitch/datapath.c
+++ b/net/openvswitch/datapath.c
@@ -1476,6 +1476,12 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb)
return -ENODEV;
}
+ /*
+ * Not needed for safety. Stops every spin_unlock_bh() in
+ * ovs_flow_stats_get() from running the softirq backlog.
+ */
+ local_bh_disable();
+
ti = rcu_dereference(dp->table.ti);
for (;;) {
struct sw_flow *flow;
@@ -1496,6 +1502,8 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb)
cb->args[0] = bucket;
cb->args[1] = obj;
}
+
+ local_bh_enable();
rcu_read_unlock();
return skb->len;
}
diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c
index 0e77868d53f8..73663db442cc 100644
--- a/net/packet/af_packet.c
+++ b/net/packet/af_packet.c
@@ -87,6 +87,7 @@
#include <linux/errqueue.h>
#include <linux/net_tstamp.h>
#include <linux/percpu.h>
+#include <linux/workqueue.h>
#ifdef CONFIG_INET
#include <net/inet_common.h>
#endif
@@ -1396,6 +1397,8 @@ static void packet_sock_destruct(struct sock *sk)
WARN_ON(atomic_read(&sk->sk_rmem_alloc));
WARN_ON(refcount_read(&sk->sk_wmem_alloc));
+ packet_free_pending(pkt_sk(sk));
+
if (!sock_flag(sk, SOCK_DEAD)) {
pr_err("Attempt to release alive packet socket: %p\n", sk);
return;
@@ -1964,7 +1967,8 @@ static int packet_rcv_spkt(struct sk_buff *skb, struct net_device *dev,
spkt = &PACKET_SKB_CB(skb)->sa.pkt;
- skb_push(skb, skb->data - skb_mac_header(skb));
+ if (dev_has_header(dev))
+ skb_push(skb, skb->data - skb_mac_header(skb));
/*
* The SOCK_PACKET socket receives _all_ frames.
@@ -2597,11 +2601,11 @@ static void tpacket_destruct_skb(struct sk_buff *skb)
__u32 ts;
ph = skb_zcopy_get_nouarg(skb);
- packet_dec_pending(&po->tx_ring);
ts = __packet_set_timestamp(po, ph, skb);
__packet_set_status(po, ph, TP_STATUS_AVAILABLE | ts);
+ packet_dec_pending(&po->tx_ring);
complete(&po->skb_completion);
}
@@ -2912,6 +2916,9 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg)
vnet_hdr.hdr_len);
has_vnet_hdr = true;
}
+ /* Prevent the byte count from wrapping into a negative errno */
+ if (unlikely(tp_len > INT_MAX - len_sum))
+ break;
copylen = max_t(int, copylen, hard_header_len);
skb = sock_alloc_send_skb(&po->sk,
hlen + tlen + sizeof(struct sockaddr_ll) +
@@ -3262,7 +3269,6 @@ static int packet_release(struct socket *sock)
/* Purge queues */
skb_queue_purge(&sk->sk_receive_queue);
- packet_free_pending(po);
sk_refcnt_debug_release(sk);
sock_put(sk);
@@ -4398,11 +4404,26 @@ static const struct vm_operations_struct packet_mmap_ops = {
.close = packet_mm_close,
};
+struct packet_pg_vec {
+ struct packet_pg_vec_free *deferred;
+ unsigned int order;
+ unsigned int len;
+ struct pgv pg_vec[];
+};
+
+struct packet_pg_vec_free {
+ struct delayed_work work;
+ struct sock *sk;
+ struct packet_pg_vec *vec;
+};
+
static void free_pg_vec(struct pgv *pg_vec, unsigned int order,
unsigned int len)
{
+ struct packet_pg_vec *vec;
int i;
+ vec = container_of_const(pg_vec, struct packet_pg_vec, pg_vec[0]);
for (i = 0; i < len; i++) {
if (likely(pg_vec[i].buffer)) {
if (is_vmalloc_addr(pg_vec[i].buffer))
@@ -4413,7 +4434,46 @@ static void free_pg_vec(struct pgv *pg_vec, unsigned int order,
pg_vec[i].buffer = NULL;
}
}
- kfree(pg_vec);
+ kfree(vec->deferred);
+ kfree(vec);
+}
+
+static void packet_free_pg_vec_work(struct work_struct *work)
+{
+ struct packet_pg_vec_free *deferred;
+ struct packet_pg_vec *vec;
+ struct sock *sk;
+
+ deferred = container_of_const(to_delayed_work(work),
+ struct packet_pg_vec_free, work);
+ vec = deferred->vec;
+ sk = deferred->sk;
+ if (sk_wmem_alloc_get(sk)) {
+ queue_delayed_work(system_long_wq, &deferred->work, 1);
+ return;
+ }
+
+ free_pg_vec(vec->pg_vec, vec->order, vec->len);
+ sock_put(sk);
+}
+
+static void packet_free_tx_ring(struct sock *sk, struct pgv *pg_vec,
+ unsigned int order, unsigned int len)
+{
+ struct packet_pg_vec_free *deferred;
+ struct packet_pg_vec *vec;
+
+ vec = container_of_const(pg_vec, struct packet_pg_vec, pg_vec[0]);
+ deferred = vec->deferred;
+ if (!deferred || !sk_wmem_alloc_get(sk)) {
+ free_pg_vec(pg_vec, order, len);
+ return;
+ }
+
+ /* A detached ring's pending count can miss late skb destructors. */
+ deferred->sk = sk;
+ sock_hold(sk);
+ queue_delayed_work(system_long_wq, &deferred->work, 0);
}
static char *alloc_one_pg_vec_page(unsigned long order)
@@ -4441,20 +4501,36 @@ static char *alloc_one_pg_vec_page(unsigned long order)
return NULL;
}
-static struct pgv *alloc_pg_vec(struct tpacket_req *req, int order)
+static struct pgv *alloc_pg_vec(struct tpacket_req *req, int order, bool tx_ring)
{
unsigned int block_nr = req->tp_block_nr;
+ struct packet_pg_vec *vec;
struct pgv *pg_vec;
int i;
- pg_vec = kcalloc(block_nr, sizeof(struct pgv), GFP_KERNEL | __GFP_NOWARN);
- if (unlikely(!pg_vec))
- goto out;
+ vec = kzalloc(struct_size(vec, pg_vec, block_nr),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (unlikely(!vec))
+ return NULL;
+ vec->order = order;
+ vec->len = block_nr;
+ pg_vec = vec->pg_vec;
for (i = 0; i < block_nr; i++) {
pg_vec[i].buffer = alloc_one_pg_vec_page(order);
if (unlikely(!pg_vec[i].buffer))
goto out_free_pgvec;
+
+ if (tx_ring && !vec->deferred &&
+ is_vmalloc_addr(pg_vec[i].buffer)) {
+ vec->deferred = kzalloc(sizeof(*vec->deferred),
+ GFP_KERNEL | __GFP_NOWARN);
+ if (!vec->deferred)
+ goto out_free_pgvec;
+ vec->deferred->vec = vec;
+ INIT_DELAYED_WORK(&vec->deferred->work,
+ packet_free_pg_vec_work);
+ }
}
out:
@@ -4537,7 +4613,7 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
err = -ENOMEM;
order = get_order(req->tp_block_size);
- pg_vec = alloc_pg_vec(req, order);
+ pg_vec = alloc_pg_vec(req, order, tx_ring);
if (unlikely(!pg_vec))
goto out;
switch (po->tp_version) {
@@ -4589,6 +4665,9 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
err = -EBUSY;
mutex_lock(&po->pg_vec_lock);
if (closing || atomic_long_read(&po->mapped) == 0) {
+ if (tx_ring && !closing && packet_read_pending(rb))
+ goto out_unlock;
+
err = 0;
spin_lock_bh(&rb_queue->lock);
swap(rb->pg_vec, pg_vec);
@@ -4610,6 +4689,7 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
pr_err("packet_mmap: vma is busy: %ld\n",
atomic_long_read(&po->mapped));
}
+out_unlock:
mutex_unlock(&po->pg_vec_lock);
spin_lock(&po->bind_lock);
@@ -4631,7 +4711,10 @@ static int packet_set_ring(struct sock *sk, union tpacket_req_u *req_u,
out_free_pg_vec:
if (pg_vec) {
bitmap_free(rx_owner_map);
- free_pg_vec(pg_vec, order, req->tp_block_nr);
+ if (tx_ring && closing)
+ packet_free_tx_ring(sk, pg_vec, order, req->tp_block_nr);
+ else
+ free_pg_vec(pg_vec, order, req->tp_block_nr);
}
out:
return err;
diff --git a/net/phonet/socket.c b/net/phonet/socket.c
index 71e2caf6ab85..c15eef431650 100644
--- a/net/phonet/socket.c
+++ b/net/phonet/socket.c
@@ -208,9 +208,15 @@ static int pn_socket_autobind(struct socket *sock)
sa.spn_family = AF_PHONET;
err = pn_socket_bind(sock, (struct sockaddr *)&sa,
sizeof(struct sockaddr_pn));
- if (err != -EINVAL)
+ /*
+ * pn_socket_bind() also returns -EINVAL when sk_state != TCP_CLOSE
+ * without a prior bind, so -EINVAL alone is not sufficient to infer
+ * that the socket was already bound. Only treat it as "already
+ * bound" when the port is non-zero; otherwise propagate the error
+ * instead of crashing the kernel.
+ */
+ if (err != -EINVAL || unlikely(!pn_port(pn_sk(sock->sk)->sobject)))
return err;
- BUG_ON(!pn_port(pn_sk(sock->sk)->sobject));
return 0; /* socket was already bound */
}
diff --git a/net/qrtr/mhi.c b/net/qrtr/mhi.c
index 9ced13c0627a..aa9cd496b549 100644
--- a/net/qrtr/mhi.c
+++ b/net/qrtr/mhi.c
@@ -24,13 +24,25 @@ static void qcom_mhi_qrtr_dl_callback(struct mhi_device *mhi_dev,
struct qrtr_mhi_dev *qdev = dev_get_drvdata(&mhi_dev->dev);
int rc;
- if (!qdev || mhi_res->transaction_status)
+ if (!qdev || (mhi_res->transaction_status && mhi_res->transaction_status != -ENOTCONN))
return;
+ /* Channel got reset. So just free the buffer */
+ if (mhi_res->transaction_status == -ENOTCONN) {
+ devm_kfree(&mhi_dev->dev, mhi_res->buf_addr);
+ return;
+ }
+
rc = qrtr_endpoint_post(&qdev->ep, mhi_res->buf_addr,
mhi_res->bytes_xferd);
if (rc == -EINVAL)
dev_err(qdev->dev, "invalid ipcrouter packet\n");
+
+ /* Done with the buffer, now recycle it for future use */
+ rc = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, mhi_res->buf_addr,
+ mhi_dev->mhi_cntrl->buffer_len, MHI_EOT);
+ if (rc)
+ dev_err(&mhi_dev->dev, "Failed to recycle the buffer: %d\n", rc);
}
/* From QRTR to MHI */
@@ -72,6 +84,29 @@ static int qcom_mhi_qrtr_send(struct qrtr_endpoint *ep, struct sk_buff *skb)
return rc;
}
+static int qcom_mhi_qrtr_queue_dl_buffers(struct mhi_device *mhi_dev)
+{
+ u32 free_desc;
+ void *buf;
+ int ret;
+
+ free_desc = mhi_get_free_desc_count(mhi_dev, DMA_FROM_DEVICE);
+ while (free_desc--) {
+ buf = devm_kmalloc(&mhi_dev->dev, mhi_dev->mhi_cntrl->buffer_len, GFP_KERNEL);
+ if (!buf)
+ return -ENOMEM;
+
+ ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, buf, mhi_dev->mhi_cntrl->buffer_len,
+ MHI_EOT);
+ if (ret) {
+ dev_err(&mhi_dev->dev, "Failed to queue buffer: %d\n", ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
static int qcom_mhi_qrtr_probe(struct mhi_device *mhi_dev,
const struct mhi_device_id *id)
{
@@ -87,20 +122,30 @@ static int qcom_mhi_qrtr_probe(struct mhi_device *mhi_dev,
qdev->ep.xmit = qcom_mhi_qrtr_send;
dev_set_drvdata(&mhi_dev->dev, qdev);
- rc = qrtr_endpoint_register(&qdev->ep, QRTR_EP_NID_AUTO);
- if (rc)
- return rc;
/* start channels */
- rc = mhi_prepare_for_transfer_autoqueue(mhi_dev);
- if (rc) {
- qrtr_endpoint_unregister(&qdev->ep);
+ rc = mhi_prepare_for_transfer(mhi_dev);
+ if (rc)
return rc;
- }
+
+ rc = qrtr_endpoint_register(&qdev->ep, QRTR_EP_NID_AUTO);
+ if (rc)
+ goto err_unprepare;
+
+ rc = qcom_mhi_qrtr_queue_dl_buffers(mhi_dev);
+ if (rc)
+ goto err_unregister;
dev_dbg(qdev->dev, "Qualcomm MHI QRTR driver probed\n");
return 0;
+
+err_unregister:
+ qrtr_endpoint_unregister(&qdev->ep);
+err_unprepare:
+ mhi_unprepare_from_transfer(mhi_dev);
+
+ return rc;
}
static void qcom_mhi_qrtr_remove(struct mhi_device *mhi_dev)
diff --git a/net/rds/rdma_transport.c b/net/rds/rdma_transport.c
index b15cf316b23a..7be38be1228e 100644
--- a/net/rds/rdma_transport.c
+++ b/net/rds/rdma_transport.c
@@ -52,7 +52,7 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id,
{
/* this can be null in the listening path */
struct rds_connection *conn = cm_id->context;
- struct rds_transport *trans;
+ struct rds_transport *trans = &rds_ib_transport;
int ret = 0;
int *err;
u8 len;
@@ -60,9 +60,6 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id,
rdsdebug("conn %p id %p handling event %u (%s)\n", conn, cm_id,
event->event, rdma_event_msg(event->event));
- if (cm_id->device->node_type == RDMA_NODE_IB_CA)
- trans = &rds_ib_transport;
-
/* Prevent shutdown from tearing down the connection
* while we're executing. */
if (conn) {
@@ -82,16 +79,47 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id,
switch (event->event) {
case RDMA_CM_EVENT_CONNECT_REQUEST:
+ /* Only the IB transport is supported, but RDS listens on
+ * every RDMA device: reject a request that arrived on any
+ * other kind. A non-zero return has rdma_cm destroy the
+ * request's id, which is what we want here and only here.
+ */
+ if (cm_id->device->node_type != RDMA_NODE_IB_CA) {
+ ret = 1;
+ break;
+ }
ret = trans->cm_handle_connect(cm_id, event, isv6);
break;
case RDMA_CM_EVENT_ADDR_RESOLVED:
if (conn) {
+ /* The address resolved to a device RDS has no
+ * transport for. Do not go on to resolve a route:
+ * an iWARP route leaves cm_id->route.path_rec
+ * unset, which the ROUTE_RESOLVED case below
+ * dereferences. Drop the connection instead; the
+ * id is still ic->i_cm_id, so return 0 and let the
+ * shutdown destroy it.
+ */
+ if (cm_id->device->node_type != RDMA_NODE_IB_CA) {
+ rds_conn_drop(conn);
+ break;
+ }
rdma_set_service_type(cm_id, conn->c_tos);
rdma_set_min_rnr_timer(cm_id, IB_RNR_TIMER_000_32);
- /* XXX do we need to clean up if this fails? */
ret = rdma_resolve_route(cm_id,
RDS_RDMA_RESOLVE_TIMEOUT_MS);
+ if (ret) {
+ /* A non-zero return has the rdma_cm destroy
+ * the id, but it is still ic->i_cm_id, which
+ * the connection's shutdown would then
+ * disconnect and destroy again. Drop the
+ * connection and keep the id for that
+ * shutdown.
+ */
+ rds_conn_drop(conn);
+ ret = 0;
+ }
}
break;
diff --git a/net/rds/tcp.c b/net/rds/tcp.c
index 397953c3048d..0967540f90f6 100644
--- a/net/rds/tcp.c
+++ b/net/rds/tcp.c
@@ -53,8 +53,10 @@ static atomic_t rds_tcp_unloading = ATOMIC_INIT(0);
static struct kmem_cache *rds_tcp_conn_slab;
-static int rds_tcp_skbuf_handler(struct ctl_table *ctl, int write,
- void *buffer, size_t *lenp, loff_t *fpos);
+static int rds_tcp_sndbuf_handler(struct ctl_table *ctl, int write,
+ void *buffer, size_t *lenp, loff_t *fpos);
+static int rds_tcp_rcvbuf_handler(struct ctl_table *ctl, int write,
+ void *buffer, size_t *lenp, loff_t *fpos);
static int rds_tcp_min_sndbuf = SOCK_MIN_SNDBUF;
static int rds_tcp_min_rcvbuf = SOCK_MIN_RCVBUF;
@@ -66,7 +68,7 @@ static struct ctl_table rds_tcp_sysctl_table[] = {
/* data is per-net pointer */
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = rds_tcp_skbuf_handler,
+ .proc_handler = rds_tcp_sndbuf_handler,
.extra1 = &rds_tcp_min_sndbuf,
},
#define RDS_TCP_RCVBUF 1
@@ -75,7 +77,7 @@ static struct ctl_table rds_tcp_sysctl_table[] = {
/* data is per-net pointer */
.maxlen = sizeof(int),
.mode = 0644,
- .proc_handler = rds_tcp_skbuf_handler,
+ .proc_handler = rds_tcp_rcvbuf_handler,
.extra1 = &rds_tcp_min_rcvbuf,
},
{ }
@@ -231,16 +233,19 @@ void rds_tcp_set_callbacks(struct socket *sock, struct rds_conn_path *cp)
rdsdebug("setting sock %p callbacks to tc %p\n", sock, tc);
write_lock_bh(&sock->sk->sk_callback_lock);
- /* done under the callback_lock to serialize with write_space */
+ /* done under the callback_lock to serialize with write_space.
+ * Set t_sock inside rds_tcp_tc_list_lock so readers walking
+ * rds_tcp_tc_list under the same lock cannot observe an
+ * entry whose t_sock is NULL.
+ */
spin_lock(&rds_tcp_tc_list_lock);
+ tc->t_sock = sock;
list_add_tail(&tc->t_list_item, &rds_tcp_tc_list);
spin_unlock(&rds_tcp_tc_list_lock);
/* accepted sockets need our listen data ready undone */
if (sock->sk->sk_data_ready == rds_tcp_listen_data_ready)
sock->sk->sk_data_ready = sock->sk->sk_user_data;
-
- tc->t_sock = sock;
tc->t_cpath = cp;
tc->t_orig_data_ready = sock->sk->sk_data_ready;
tc->t_orig_write_space = sock->sk->sk_write_space;
@@ -564,19 +569,25 @@ bool rds_tcp_tune(struct socket *sock)
struct rds_tcp_net *rtn;
tcp_sock_set_nodelay(sock->sk);
- lock_sock(sk);
/* TCP timer functions might access net namespace even after
* a process which created this net namespace terminated.
*/
if (!sk->sk_net_refcnt) {
- if (!maybe_get_net(net)) {
- release_sock(sk);
+ if (!maybe_get_net(net))
return false;
- }
+ /*
+ * The sk_net_refcnt upgrade must be done before lock_sock()
+ * because netns_tracker_alloc() does a GFP_KERNEL allocation,
+ * which can trigger fs_reclaim and create a circular lock
+ * dependency with the socket lock. The fields it modifies
+ * (sk_net_refcnt, ns_tracker) are not accessed by any
+ * concurrent code path at this point.
+ */
sk->sk_net_refcnt = 1;
netns_tracker_alloc(net, &sk->ns_tracker, GFP_KERNEL);
sock_inuse_add(net, 1);
}
+ lock_sock(sk);
rtn = net_generic(net, rds_tcp_netid);
if (rtn->sndbuf_size > 0) {
sk->sk_sndbuf = rtn->sndbuf_size;
@@ -700,13 +711,13 @@ static void __net_exit rds_tcp_exit_net(struct net *net)
{
struct rds_tcp_net *rtn = net_generic(net, rds_tcp_netid);
- rds_tcp_kill_sock(net);
-
if (rtn->rds_tcp_sysctl)
unregister_net_sysctl_table(rtn->rds_tcp_sysctl);
if (net != &init_net)
kfree(rtn->ctl_table);
+
+ rds_tcp_kill_sock(net);
}
static struct pernet_operations rds_tcp_net_ops = {
@@ -749,10 +760,10 @@ static void rds_tcp_sysctl_reset(struct net *net)
spin_unlock_irq(&rds_tcp_conn_lock);
}
-static int rds_tcp_skbuf_handler(struct ctl_table *ctl, int write,
+static int rds_tcp_skbuf_handler(struct rds_tcp_net *rtn,
+ struct ctl_table *ctl, int write,
void *buffer, size_t *lenp, loff_t *fpos)
{
- struct net *net = current->nsproxy->net_ns;
int err;
err = proc_dointvec_minmax(ctl, write, buffer, lenp, fpos);
@@ -761,11 +772,34 @@ static int rds_tcp_skbuf_handler(struct ctl_table *ctl, int write,
*(int *)(ctl->extra1));
return err;
}
- if (write)
+
+ if (write && rtn->rds_tcp_listen_sock && rtn->rds_tcp_listen_sock->sk) {
+ struct net *net = sock_net(rtn->rds_tcp_listen_sock->sk);
+
rds_tcp_sysctl_reset(net);
+ }
+
return 0;
}
+static int rds_tcp_sndbuf_handler(struct ctl_table *ctl, int write,
+ void *buffer, size_t *lenp, loff_t *fpos)
+{
+ struct rds_tcp_net *rtn = container_of(ctl->data, struct rds_tcp_net,
+ sndbuf_size);
+
+ return rds_tcp_skbuf_handler(rtn, ctl, write, buffer, lenp, fpos);
+}
+
+static int rds_tcp_rcvbuf_handler(struct ctl_table *ctl, int write,
+ void *buffer, size_t *lenp, loff_t *fpos)
+{
+ struct rds_tcp_net *rtn = container_of(ctl->data, struct rds_tcp_net,
+ rcvbuf_size);
+
+ return rds_tcp_skbuf_handler(rtn, ctl, write, buffer, lenp, fpos);
+}
+
static void rds_tcp_exit(void)
{
rds_tcp_set_unloading();
diff --git a/net/rxrpc/ar-internal.h b/net/rxrpc/ar-internal.h
index 8499ceb7719c..fbec55c7e6dc 100644
--- a/net/rxrpc/ar-internal.h
+++ b/net/rxrpc/ar-internal.h
@@ -303,7 +303,7 @@ struct rxrpc_peer {
struct hlist_head error_targets; /* targets for net error distribution */
struct rb_root service_conns; /* Service connections */
struct list_head keepalive_link; /* Link in net->peer_keepalive[] */
- time64_t last_tx_at; /* Last time packet sent here */
+ unsigned int last_tx_at; /* Last time packet sent here (time64_t LSW) */
seqlock_t service_conn_lock;
spinlock_t lock; /* access lock */
unsigned int if_mtu; /* interface MTU for this peer */
@@ -986,6 +986,13 @@ void rxrpc_encap_err_rcv(struct sock *sk, struct sk_buff *skb, unsigned int udp_
void rxrpc_error_report(struct sock *);
void rxrpc_peer_keepalive_worker(struct work_struct *);
+/* Update the last transmission time on a peer for keepalive purposes. */
+static inline void rxrpc_peer_mark_tx(struct rxrpc_peer *peer)
+{
+ /* To avoid tearing on 32-bit systems, we only keep the LSW. */
+ WRITE_ONCE(peer->last_tx_at, ktime_get_seconds());
+}
+
/*
* peer_object.c
*/
diff --git a/net/rxrpc/call_accept.c b/net/rxrpc/call_accept.c
index 4a1e5d7a0726..8001c330b2be 100644
--- a/net/rxrpc/call_accept.c
+++ b/net/rxrpc/call_accept.c
@@ -151,6 +151,11 @@ static int rxrpc_service_prealloc_one(struct rxrpc_sock *rx,
id_in_use:
write_unlock(&rx->call_lock);
+ call->abort_code = RX_CALL_DEAD;
+ call->error = -EBADSLT;
+ call->completion = RXRPC_CALL_LOCAL_ERROR;
+ call->state = RXRPC_CALL_COMPLETE;
+ set_bit(RXRPC_CALL_RELEASED, &call->flags);
rxrpc_cleanup_call(call);
_leave(" = -EBADSLT");
return -EBADSLT;
diff --git a/net/rxrpc/conn_event.c b/net/rxrpc/conn_event.c
index 293922df2a89..ef9303dcf7bd 100644
--- a/net/rxrpc/conn_event.c
+++ b/net/rxrpc/conn_event.c
@@ -142,7 +142,7 @@ static void rxrpc_conn_retransmit_call(struct rxrpc_connection *conn,
}
ret = kernel_sendmsg(conn->params.local->socket, &msg, iov, ioc, len);
- conn->params.peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(conn->params.peer);
if (ret < 0)
trace_rxrpc_tx_fail(chan->call_debug_id, serial, ret,
rxrpc_tx_point_call_final_resend);
@@ -262,7 +262,7 @@ static int rxrpc_abort_connection(struct rxrpc_connection *conn,
trace_rxrpc_tx_packet(conn->debug_id, &whdr, rxrpc_tx_point_conn_abort);
- conn->params.peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(conn->params.peer);
_leave(" = 0");
return 0;
@@ -285,6 +285,34 @@ static void rxrpc_call_is_secure(struct rxrpc_call *call)
}
}
+static int rxrpc_verify_response(struct rxrpc_connection *conn,
+ struct sk_buff *skb,
+ u32 *_abort_code)
+{
+ int ret;
+
+ if (skb_cloned(skb)) {
+ /* Copy the packet if shared so that we can do in-place
+ * decryption.
+ */
+ struct sk_buff *nskb = skb_copy(skb, GFP_NOFS);
+
+ if (nskb) {
+ rxrpc_new_skb(nskb, rxrpc_skb_unshared);
+ ret = conn->security->verify_response(conn, nskb, _abort_code);
+ rxrpc_free_skb(nskb, rxrpc_skb_freed);
+ } else {
+ /* OOM - Drop the packet. */
+ rxrpc_see_skb(skb, rxrpc_skb_unshared_nomem);
+ ret = -ENOMEM;
+ }
+ } else {
+ ret = conn->security->verify_response(conn, skb, _abort_code);
+ }
+
+ return ret;
+}
+
/*
* connection-level Rx packet processor
*/
@@ -345,7 +373,7 @@ static int rxrpc_process_event(struct rxrpc_connection *conn,
}
spin_unlock_bh(&conn->state_lock);
- ret = conn->security->verify_response(conn, skb, _abort_code);
+ ret = rxrpc_verify_response(conn, skb, _abort_code);
if (ret < 0)
return ret;
diff --git a/net/rxrpc/output.c b/net/rxrpc/output.c
index 08c117bc083e..6fb0c67ef8bb 100644
--- a/net/rxrpc/output.c
+++ b/net/rxrpc/output.c
@@ -87,7 +87,7 @@ static size_t rxrpc_fill_out_ack(struct rxrpc_connection *conn,
return 0;
/* Barrier against rxrpc_input_data(). */
- serial = call->ackr_serial;
+ serial = reason == RXRPC_ACK_DELAY ? 0 : call->ackr_serial;
hard_ack = READ_ONCE(call->rx_hard_ack);
top = smp_load_acquire(&call->rx_top);
*_hard_ack = hard_ack;
@@ -254,7 +254,7 @@ int rxrpc_send_ack_packet(struct rxrpc_call *call, bool ping,
rtt_slot = rxrpc_begin_rtt_probe(call, serial, rxrpc_rtt_tx_ping);
ret = kernel_sendmsg(conn->params.local->socket, &msg, iov, 2, len);
- conn->params.peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(conn->params.peer);
if (ret < 0)
trace_rxrpc_tx_fail(call->debug_id, serial, ret,
rxrpc_tx_point_call_ack);
@@ -333,7 +333,7 @@ int rxrpc_send_abort_packet(struct rxrpc_call *call)
ret = kernel_sendmsg(conn->params.local->socket,
&msg, iov, 1, sizeof(pkt));
- conn->params.peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(conn->params.peer);
if (ret < 0)
trace_rxrpc_tx_fail(call->debug_id, serial, ret,
rxrpc_tx_point_call_abort);
@@ -449,7 +449,7 @@ int rxrpc_send_data_packet(struct rxrpc_call *call, struct sk_buff *skb,
* message and update the peer record
*/
ret = kernel_sendmsg(conn->params.local->socket, &msg, iov, 2, len);
- conn->params.peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(conn->params.peer);
up_read(&conn->params.local->defrag_sem);
if (ret < 0) {
@@ -525,7 +525,7 @@ int rxrpc_send_data_packet(struct rxrpc_call *call, struct sk_buff *skb,
IP_PMTUDISC_DONT);
ret = kernel_sendmsg(conn->params.local->socket, &msg,
iov, 2, len);
- conn->params.peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(conn->params.peer);
ip_sock_set_mtu_discover(conn->params.local->socket->sk,
IP_PMTUDISC_DO);
@@ -674,6 +674,6 @@ void rxrpc_send_keepalive(struct rxrpc_peer *peer)
trace_rxrpc_tx_packet(peer->debug_id, &whdr,
rxrpc_tx_point_version_keepalive);
- peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(peer);
_leave("");
}
diff --git a/net/rxrpc/peer_event.c b/net/rxrpc/peer_event.c
index 32561e9567fe..77005c97dd90 100644
--- a/net/rxrpc/peer_event.c
+++ b/net/rxrpc/peer_event.c
@@ -515,6 +515,21 @@ static void rxrpc_distribute_error(struct rxrpc_peer *peer, int error,
}
}
+/*
+ * Reconstruct the last transmission time. The difference calculated should be
+ * valid provided no more than ~68 years elapsed since the last transmission.
+ */
+static time64_t rxrpc_peer_get_tx_mark(const struct rxrpc_peer *peer, time64_t base)
+{
+ s32 last_tx_at = READ_ONCE(peer->last_tx_at);
+ s32 base_lsw = base;
+ s32 diff = last_tx_at - base_lsw;
+
+ diff = clamp(diff, -RXRPC_KEEPALIVE_TIME, RXRPC_KEEPALIVE_TIME);
+
+ return diff + base;
+}
+
/*
* Perform keep-alive pings.
*/
@@ -541,7 +556,7 @@ static void rxrpc_peer_keepalive_dispatch(struct rxrpc_net *rxnet,
if (__rxrpc_use_local(peer->local)) {
spin_unlock_bh(&rxnet->peer_hash_lock);
- keepalive_at = peer->last_tx_at + RXRPC_KEEPALIVE_TIME;
+ keepalive_at = rxrpc_peer_get_tx_mark(peer, base) + RXRPC_KEEPALIVE_TIME;
slot = keepalive_at - base;
_debug("%02x peer %u t=%d {%pISp}",
cursor, peer->debug_id, slot, &peer->srx.transport);
diff --git a/net/rxrpc/proc.c b/net/rxrpc/proc.c
index 47d36554ad31..ee07785860a0 100644
--- a/net/rxrpc/proc.c
+++ b/net/rxrpc/proc.c
@@ -234,13 +234,13 @@ static int rxrpc_peer_seq_show(struct seq_file *seq, void *v)
now = ktime_get_seconds();
seq_printf(seq,
"UDP %-47.47s %-47.47s %3u"
- " %3u %5u %6llus %8u %8u\n",
+ " %3u %5u %6ds %8u %8u\n",
lbuff,
rbuff,
refcount_read(&peer->ref),
peer->cong_cwnd,
peer->mtu,
- now - peer->last_tx_at,
+ (s32)now - (s32)READ_ONCE(peer->last_tx_at),
peer->srtt_us >> 3,
jiffies_to_usecs(peer->rto_j));
diff --git a/net/rxrpc/recvmsg.c b/net/rxrpc/recvmsg.c
index 99c679f40e7b..870fab693653 100644
--- a/net/rxrpc/recvmsg.c
+++ b/net/rxrpc/recvmsg.c
@@ -546,6 +546,15 @@ int rxrpc_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
write_lock_bh(&rx->recvmsg_lock);
l = rx->recvmsg_q.next;
call = list_entry(l, struct rxrpc_call, recvmsg_link);
+
+ if (test_bit(RXRPC_CALL_RELEASED, &call->flags)) {
+ list_del_init(&call->recvmsg_link);
+ write_unlock_bh(&rx->recvmsg_lock);
+ release_sock(&rx->sk);
+ rxrpc_put_call(call, rxrpc_call_put);
+ goto try_again;
+ }
+
if (!(flags & MSG_PEEK))
list_del_init(&call->recvmsg_link);
else
@@ -568,8 +577,11 @@ int rxrpc_recvmsg(struct socket *sock, struct msghdr *msg, size_t len,
release_sock(&rx->sk);
- if (test_bit(RXRPC_CALL_RELEASED, &call->flags))
- BUG();
+ if (test_bit(RXRPC_CALL_RELEASED, &call->flags)) {
+ mutex_unlock(&call->user_mutex);
+ rxrpc_put_call(call, rxrpc_call_put);
+ goto try_again;
+ }
if (test_bit(RXRPC_CALL_HAS_USERID, &call->flags)) {
if (flags & MSG_CMSG_COMPAT) {
diff --git a/net/rxrpc/rxkad.c b/net/rxrpc/rxkad.c
index 84a61fa18bc0..262b0df1e447 100644
--- a/net/rxrpc/rxkad.c
+++ b/net/rxrpc/rxkad.c
@@ -780,7 +780,7 @@ static int rxkad_issue_challenge(struct rxrpc_connection *conn)
return -EAGAIN;
}
- conn->params.peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(conn->params.peer);
trace_rxrpc_tx_packet(conn->debug_id, &whdr,
rxrpc_tx_point_rxkad_challenge);
_leave(" = 0");
@@ -838,7 +838,7 @@ static int rxkad_send_response(struct rxrpc_connection *conn,
return -EAGAIN;
}
- conn->params.peer->last_tx_at = ktime_get_seconds();
+ rxrpc_peer_mark_tx(conn->params.peer);
_leave(" = 0");
return 0;
}
@@ -1311,6 +1311,7 @@ static int rxkad_verify_response(struct rxrpc_connection *conn,
kfree(ticket);
kfree(response);
+ key_put(server_key);
_leave(" = 0");
return 0;
diff --git a/net/sched/act_api.c b/net/sched/act_api.c
index 630f285f8db0..0ab2c4877599 100644
--- a/net/sched/act_api.c
+++ b/net/sched/act_api.c
@@ -40,11 +40,9 @@ int tcf_dev_queue_xmit(struct sk_buff *skb, int (*xmit)(struct sk_buff *skb))
}
EXPORT_SYMBOL_GPL(tcf_dev_queue_xmit);
-static void tcf_action_goto_chain_exec(const struct tc_action *a,
+static void tcf_action_goto_chain_exec(const struct tcf_chain *chain,
struct tcf_result *res)
{
- const struct tcf_chain *chain = rcu_dereference_bh(a->goto_chain);
-
res->goto_tp = rcu_dereference_bh(chain->filter_chain);
}
@@ -147,10 +145,15 @@ static void offload_action_hw_count_dec(struct tc_action *act,
static unsigned int tcf_offload_act_num_actions_single(struct tc_action *act)
{
- if (is_tcf_pedit(act))
- return tcf_pedit_nkeys(act);
- else
- return 1;
+ unsigned int count;
+
+ if (is_tcf_pedit(act)) {
+ spin_lock_bh(&act->tcfa_lock);
+ count = tcf_pedit_nkeys_locked(act);
+ spin_unlock_bh(&act->tcfa_lock);
+ return count;
+ }
+ return 1;
}
static bool tc_act_skip_hw(u32 flags)
@@ -375,9 +378,10 @@ static int __tcf_action_put(struct tc_action *p, bool bind)
{
struct tcf_idrinfo *idrinfo = p->idrinfo;
- if (refcount_dec_and_mutex_lock(&p->tcfa_refcnt, &idrinfo->lock)) {
- if (bind)
- atomic_dec(&p->tcfa_bindcnt);
+ mutex_lock(&idrinfo->lock);
+ if (bind)
+ atomic_dec(&p->tcfa_bindcnt);
+ if (refcount_dec_and_test(&p->tcfa_refcnt)) {
idr_remove(&idrinfo->action_idr, p->tcfa_index);
mutex_unlock(&idrinfo->lock);
@@ -385,9 +389,7 @@ static int __tcf_action_put(struct tc_action *p, bool bind)
return 1;
}
- if (bind)
- atomic_dec(&p->tcfa_bindcnt);
-
+ mutex_unlock(&idrinfo->lock);
return 0;
}
@@ -1125,11 +1127,14 @@ int tcf_action_exec(struct sk_buff *skb, struct tc_action **actions,
return TC_ACT_OK;
}
} else if (TC_ACT_EXT_CMP(ret, TC_ACT_GOTO_CHAIN)) {
- if (unlikely(!rcu_access_pointer(a->goto_chain))) {
- net_warn_ratelimited("can't go to NULL chain!\n");
+ struct tcf_chain *chain = rcu_dereference_bh(a->goto_chain);
+
+ if (unlikely(!chain)) {
+ tcf_set_drop_reason(skb,
+ SKB_DROP_REASON_TC_CHAIN_NOTFOUND);
return TC_ACT_SHOT;
}
- tcf_action_goto_chain_exec(a, res);
+ tcf_action_goto_chain_exec(chain, res);
}
if (ret != TC_ACT_PIPE)
@@ -1142,19 +1147,14 @@ EXPORT_SYMBOL(tcf_action_exec);
int tcf_action_destroy(struct tc_action *actions[], int bind)
{
- const struct tc_action_ops *ops;
struct tc_action *a;
int ret = 0, i;
for (i = 0; i < TCA_ACT_MAX_PRIO && actions[i]; i++) {
a = actions[i];
actions[i] = NULL;
- ops = a->ops;
- ret = __tcf_idr_release(a, bind, true);
- if (ret == ACT_P_DELETED)
- module_put(ops->owner);
- else if (ret < 0)
- return ret;
+ /* Drop our reference even if the action is still bound to a filter. */
+ ret = tcf_idr_release(a, bind);
}
return ret;
}
diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c
index a25203a49270..2e340d8b6286 100644
--- a/net/sched/act_ife.c
+++ b/net/sched/act_ife.c
@@ -28,6 +28,7 @@
#include <uapi/linux/tc_act/tc_ife.h>
#include <net/tc_act/tc_ife.h>
#include <linux/etherdevice.h>
+#include <linux/if_arp.h>
#include <net/ife.h>
static int max_metacnt = IFE_META_MAX + 1;
@@ -722,7 +723,7 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a,
tcf_lastuse_update(&ife->tcf_tm);
if (skb_at_tc_ingress(skb))
- skb_push(skb, skb->dev->hard_header_len);
+ skb_push(skb, ETH_HLEN);
tlv_data = ife_decode(skb, &metalen);
if (unlikely(!tlv_data)) {
@@ -794,7 +795,7 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a,
where ORIGDATA = original ethernet header ...
*/
u16 metalen = ife_get_sz(skb, p);
- int hdrm = metalen + skb->dev->hard_header_len + IFE_METAHDRLEN;
+ int hdrm = metalen + ETH_HLEN + IFE_METAHDRLEN;
unsigned int skboff = 0;
int new_len = skb->len + hdrm;
bool exceed_mtu = false;
@@ -825,7 +826,7 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a,
}
if (skb_at_tc_ingress(skb))
- skb_push(skb, skb->dev->hard_header_len);
+ skb_push(skb, ETH_HLEN);
ife_meta = ife_encode(skb, metalen);
if (!ife_meta)
@@ -855,11 +856,27 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a,
oethh->h_proto = htons(p->eth_type);
if (skb_at_tc_ingress(skb))
- skb_pull(skb, skb->dev->hard_header_len);
+ skb_pull(skb, ETH_HLEN);
return action;
}
+/* IFE encapsulates the original Ethernet header and, on decode, expects to
+ * find one, so it can only ever work on skbs that carry one. Loopback carries
+ * Ethernet header as well, so it qualifies here.
+ * At ingress, also verify that the L2 header about to be pushed back really
+ * is an Ethernet header because the skb could've been redirected with mirred
+ * from a non-Ethernet device.
+ */
+static bool tcf_ife_is_eth_skb(const struct sk_buff *skb)
+{
+ if (skb->dev->type != ARPHRD_ETHER &&
+ skb->dev->type != ARPHRD_LOOPBACK)
+ return false;
+
+ return !skb_at_tc_ingress(skb) || skb->mac_len == ETH_HLEN;
+}
+
static int tcf_ife_act(struct sk_buff *skb, const struct tc_action *a,
struct tcf_result *res)
{
@@ -867,6 +884,13 @@ static int tcf_ife_act(struct sk_buff *skb, const struct tc_action *a,
struct tcf_ife_params *p;
int ret;
+ if (unlikely(!tcf_ife_is_eth_skb(skb))) {
+ bstats_update(this_cpu_ptr(ife->common.cpu_bstats), skb);
+ tcf_lastuse_update(&ife->tcf_tm);
+ qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats));
+ return TC_ACT_SHOT;
+ }
+
p = rcu_dereference_bh(ife->params);
if (p->flags & IFE_ENCODE) {
ret = tcf_ife_encode(skb, a, res, p);
diff --git a/net/sched/act_mirred.c b/net/sched/act_mirred.c
index bbc34987bd09..896bffd50aa8 100644
--- a/net/sched/act_mirred.c
+++ b/net/sched/act_mirred.c
@@ -205,18 +205,14 @@ static int tcf_mirred_init(struct net *net, struct nlattr *nla,
return err;
}
-static bool is_mirred_nested(void)
-{
- return unlikely(__this_cpu_read(mirred_nest_level) > 1);
-}
-
-static int tcf_mirred_forward(bool want_ingress, struct sk_buff *skb)
+static int
+tcf_mirred_forward(bool at_ingress, bool want_ingress, struct sk_buff *skb)
{
int err;
if (!want_ingress)
err = tcf_dev_queue_xmit(skb, dev_queue_xmit);
- else if (is_mirred_nested())
+ else if (!at_ingress)
err = netif_rx(skb);
else
err = netif_receive_skb(skb);
@@ -312,7 +308,7 @@ static int tcf_mirred_act(struct sk_buff *skb, const struct tc_action *a,
/* let's the caller reinsert the packet, if possible */
if (use_reinsert) {
- err = tcf_mirred_forward(want_ingress, skb);
+ err = tcf_mirred_forward(at_ingress, want_ingress, skb);
if (err)
tcf_action_inc_overlimit_qstats(&m->common);
__this_cpu_dec(mirred_nest_level);
@@ -320,7 +316,7 @@ static int tcf_mirred_act(struct sk_buff *skb, const struct tc_action *a,
}
}
- err = tcf_mirred_forward(want_ingress, skb2);
+ err = tcf_mirred_forward(at_ingress, want_ingress, skb2);
if (err)
tcf_action_inc_overlimit_qstats(&m->common);
__this_cpu_dec(mirred_nest_level);
diff --git a/net/sched/act_pedit.c b/net/sched/act_pedit.c
index 1d06873da8f1..e1be146b0463 100644
--- a/net/sched/act_pedit.c
+++ b/net/sched/act_pedit.c
@@ -556,9 +556,18 @@ static int tcf_pedit_offload_act_setup(struct tc_action *act, void *entry_data,
{
if (bind) {
struct flow_action_entry *entry = entry_data;
+ int nkeys = tcf_pedit_nkeys_locked(act);
int k;
- for (k = 0; k < tcf_pedit_nkeys(act); k++) {
+ /* If the required keys exceed the remaining capacity return
+ * -ENOSPC to abort the offload and fallback to software.
+ */
+ if (nkeys > *index_inc) {
+ NL_SET_ERR_MSG_MOD(extack, "Not enough space to offload all pedit keys");
+ return -ENOSPC;
+ }
+
+ for (k = 0; k < nkeys; k++) {
switch (tcf_pedit_cmd(act, k)) {
case TCA_PEDIT_KEY_EX_CMD_SET:
entry->id = FLOW_ACTION_MANGLE;
diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c
index 341128a00420..603b7caebad2 100644
--- a/net/sched/cls_api.c
+++ b/net/sched/cls_api.c
@@ -285,6 +285,7 @@ static struct tcf_proto *tcf_proto_create(const char *kind, u32 protocol,
tp->protocol = protocol;
tp->prio = prio;
tp->chain = chain;
+ tp->usesw = !tp->ops->reoffload;
spin_lock_init(&tp->lock);
refcount_set(&tp->refcnt, 1);
@@ -305,12 +306,55 @@ static void tcf_proto_get(struct tcf_proto *tp)
refcount_inc(&tp->refcnt);
}
+static void tcf_proto_count_usesw(struct tcf_proto *tp, bool add)
+{
+#ifdef CONFIG_NET_CLS_ACT
+ struct tcf_block *block = tp->chain->block;
+ bool counted = false;
+
+ if (!add) {
+ if (tp->usesw && tp->counted) {
+ if (!atomic_dec_return(&block->useswcnt))
+ static_branch_dec(&tcf_sw_enabled_key);
+ tp->counted = false;
+ }
+ return;
+ }
+
+ spin_lock(&tp->lock);
+ if (tp->usesw && !tp->counted) {
+ counted = true;
+ tp->counted = true;
+ }
+ spin_unlock(&tp->lock);
+
+ if (counted && atomic_inc_return(&block->useswcnt) == 1)
+ static_branch_inc(&tcf_sw_enabled_key);
+#endif
+}
+
static void tcf_chain_put(struct tcf_chain *chain);
static void tcf_proto_destroy(struct tcf_proto *tp, bool rtnl_held,
bool sig_destroy, struct netlink_ext_ack *extack)
{
- tp->ops->destroy(tp, rtnl_held, extack);
+ /* A locked classifier's destroy callback (e.g. u32_destroy) uses
+ * rtnl_dereference() and mutates shared structures (e.g. the
+ * tc_u_common hash list) that are only safe under rtnl_lock. When an
+ * unlocked classifier's request (e.g. flower on ingress) loses the
+ * tcf_chain_tp_insert_unique() race and ends up dropping the last
+ * reference on a locked classifier's proto, destroy() would run
+ * without rtnl held. Take it here in that case.
+ */
+ bool not_lockless = !rtnl_held &&
+ !(tp->ops->flags & TCF_PROTO_OPS_DOIT_UNLOCKED);
+
+ if (not_lockless)
+ rtnl_lock();
+ tp->ops->destroy(tp, rtnl_held || not_lockless, extack);
+ if (not_lockless)
+ rtnl_unlock();
+ tcf_proto_count_usesw(tp, false);
if (sig_destroy)
tcf_proto_signal_destroyed(tp->chain, tp);
tcf_chain_put(tp->chain);
@@ -1431,6 +1475,9 @@ tcf_block_playback_offloads(struct tcf_block *block, flow_setup_cb_t *cb,
chain_prev = chain,
chain = __tcf_get_next_chain(block, chain),
tcf_chain_put(chain_prev)) {
+ if (chain->tmplt_ops && add)
+ chain->tmplt_ops->tmplt_reoffload(chain, true, cb,
+ cb_priv);
for (tp = __tcf_get_next_proto(chain, NULL); tp;
tp_prev = tp,
tp = __tcf_get_next_proto(chain, tp),
@@ -1446,6 +1493,9 @@ tcf_block_playback_offloads(struct tcf_block *block, flow_setup_cb_t *cb,
goto err_playback_remove;
}
}
+ if (chain->tmplt_ops && !add)
+ chain->tmplt_ops->tmplt_reoffload(chain, false, cb,
+ cb_priv);
}
return 0;
@@ -2205,13 +2255,27 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n,
tfilter_notify(net, skb, n, tp, block, q, parent, fh,
RTM_NEWTFILTER, false, rtnl_held, extack);
tfilter_put(tp, fh);
+ tcf_proto_count_usesw(tp, true);
/* q pointer is NULL for shared blocks */
if (q)
q->flags &= ~TCQ_F_CAN_BYPASS;
}
errout:
- if (err && tp_state == TP_CREATED)
+ if (err && !IS_ERR_OR_NULL(tp) &&
+ (tp_state == TP_CREATED || tp->ops->delete_empty))
+ /*
+ * The request is dropping its reference to tp. If it was
+ * the last user (the idr is empty now), reclaim the proto.
+ * A tp this request created is reclaimed unconditionally:
+ * it is the only owner, so marking it for deletion is
+ * safe. Otherwise only classifiers with a delete_empty
+ * callback are reclaimed -- the callback admits an empty
+ * proto only, so a live shared proto is never unlinked.
+ * Classifiers without one (deleting is set
+ * unconditionally) are rtnl-serialized, so the raced
+ * window this guard closes cannot arise for them.
+ */
tcf_chain_tp_delete_empty(chain, tp, rtnl_held, NULL);
errout_tp:
if (chain) {
@@ -2848,7 +2912,8 @@ static int tc_chain_tmplt_add(struct tcf_chain *chain, struct net *net,
ops = tcf_proto_lookup_ops(name, true, extack);
if (IS_ERR(ops))
return PTR_ERR(ops);
- if (!ops->tmplt_create || !ops->tmplt_destroy || !ops->tmplt_dump) {
+ if (!ops->tmplt_create || !ops->tmplt_destroy || !ops->tmplt_dump ||
+ !ops->tmplt_reoffload) {
NL_SET_ERR_MSG(extack, "Chain templates are not supported with specified classifier");
module_put(ops->owner);
return -EOPNOTSUPP;
@@ -3611,12 +3676,21 @@ int tc_setup_action(struct flow_action *flow_action,
entry = &flow_action->entries[j];
spin_lock_bh(&act->tcfa_lock);
+
+ /* Abort the offload if we have exhausted the allocated capacity */
+ if (j >= flow_action->num_entries) {
+ NL_SET_ERR_MSG_MOD(extack, "Flow action buffer overflow");
+ err = -ENOSPC;
+ goto err_out_locked;
+ }
+
err = tcf_act_get_cookie(entry, act);
if (err)
goto err_out_locked;
- index = 0;
- err = tc_setup_offload_act(act, entry, &index, extack);
+ index = flow_action->num_entries - j;
+ err = tc_setup_offload_act(act, entry, &index,
+ extack);
if (err)
goto err_out_locked;
@@ -3662,10 +3736,13 @@ unsigned int tcf_exts_num_actions(struct tcf_exts *exts)
int i;
tcf_exts_for_each_action(i, act, exts) {
- if (is_tcf_pedit(act))
- num_acts += tcf_pedit_nkeys(act);
- else
+ if (is_tcf_pedit(act)) {
+ spin_lock_bh(&act->tcfa_lock);
+ num_acts += tcf_pedit_nkeys_locked(act);
+ spin_unlock_bh(&act->tcfa_lock);
+ } else {
num_acts++;
+ }
}
return num_acts;
}
diff --git a/net/sched/cls_basic.c b/net/sched/cls_basic.c
index d229ce99e554..bf500510f60f 100644
--- a/net/sched/cls_basic.c
+++ b/net/sched/cls_basic.c
@@ -191,7 +191,7 @@ static int basic_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
@@ -210,9 +210,11 @@ static int basic_change(struct net *net, struct sk_buff *in_skb,
if (err)
goto errout;
fnew->handle = handle;
- fnew->pf = alloc_percpu(struct tc_basic_pcnt);
+ fnew->pf = alloc_percpu_gfp(struct tc_basic_pcnt, GFP_KERNEL_ACCOUNT);
if (!fnew->pf) {
err = -ENOMEM;
+ if (!fold)
+ idr_remove(&head->handle_idr, fnew->handle);
goto errout;
}
diff --git a/net/sched/cls_bpf.c b/net/sched/cls_bpf.c
index 00159978c450..3e5316c05d2c 100644
--- a/net/sched/cls_bpf.c
+++ b/net/sched/cls_bpf.c
@@ -350,7 +350,7 @@ static int cls_bpf_prog_from_ops(struct nlattr **tb, struct cls_bpf_prog *prog)
if (bpf_size != nla_len(tb[TCA_BPF_OPS]))
return -EINVAL;
- bpf_ops = kmemdup(nla_data(tb[TCA_BPF_OPS]), bpf_size, GFP_KERNEL);
+ bpf_ops = kmemdup(nla_data(tb[TCA_BPF_OPS]), bpf_size, GFP_KERNEL_ACCOUNT);
if (bpf_ops == NULL)
return -ENOMEM;
@@ -387,7 +387,7 @@ static int cls_bpf_prog_from_efd(struct nlattr **tb, struct cls_bpf_prog *prog,
return PTR_ERR(fp);
if (tb[TCA_BPF_NAME]) {
- name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL);
+ name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL_ACCOUNT);
if (!name) {
bpf_prog_put(fp);
return -ENOMEM;
@@ -427,7 +427,7 @@ static int cls_bpf_change(struct net *net, struct sk_buff *in_skb,
if (ret < 0)
return ret;
- prog = kzalloc(sizeof(*prog), GFP_KERNEL);
+ prog = kzalloc(sizeof(*prog), GFP_KERNEL_ACCOUNT);
if (!prog)
return -ENOBUFS;
@@ -507,6 +507,8 @@ static int cls_bpf_change(struct net *net, struct sk_buff *in_skb,
if (!tc_in_hw(prog->gen_flags))
prog->gen_flags |= TCA_CLS_FLAGS_NOT_IN_HW;
+ tcf_proto_update_usesw(tp, prog->gen_flags);
+
if (oldprog) {
idr_replace(&head->handle_idr, prog, handle);
list_replace_rcu(&oldprog->link, &prog->link);
diff --git a/net/sched/cls_cgroup.c b/net/sched/cls_cgroup.c
index ed00001b528a..300806f80b86 100644
--- a/net/sched/cls_cgroup.c
+++ b/net/sched/cls_cgroup.c
@@ -93,7 +93,7 @@ static int cls_cgroup_change(struct net *net, struct sk_buff *in_skb,
if (head && handle != head->handle)
return -ENOENT;
- new = kzalloc(sizeof(*head), GFP_KERNEL);
+ new = kzalloc(sizeof(*head), GFP_KERNEL_ACCOUNT);
if (!new)
return -ENOBUFS;
diff --git a/net/sched/cls_flow.c b/net/sched/cls_flow.c
index 315d689adf65..50a707f2fe8e 100644
--- a/net/sched/cls_flow.c
+++ b/net/sched/cls_flow.c
@@ -436,7 +436,7 @@ static int flow_change(struct net *net, struct sk_buff *in_skb,
return -EOPNOTSUPP;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c
index a40a9e84c75f..c6eb857bce89 100644
--- a/net/sched/cls_flower.c
+++ b/net/sched/cls_flower.c
@@ -1548,6 +1548,11 @@ static int fl_set_enc_opt(struct nlattr **tb, struct fl_flow_key *key,
return -EINVAL;
}
nla_opt_msk = nla_next(nla_opt_msk, &msk_depth);
+
+ if (msk_depth && !nla_ok(nla_opt_msk, msk_depth)) {
+ NL_SET_ERR_MSG(extack, "A mask attribute is invalid");
+ return -EINVAL;
+ }
}
return 0;
@@ -2003,7 +2008,7 @@ static struct fl_flow_mask *fl_create_new_mask(struct cls_fl_head *head,
struct fl_flow_mask *newmask;
int err;
- newmask = kzalloc(sizeof(*newmask), GFP_KERNEL);
+ newmask = kzalloc(sizeof(*newmask), GFP_KERNEL_ACCOUNT);
if (!newmask)
return ERR_PTR(-ENOMEM);
@@ -2196,7 +2201,7 @@ static int fl_change(struct net *net, struct sk_buff *in_skb,
goto errout_tb;
}
- fnew = kzalloc(sizeof(*fnew), GFP_KERNEL);
+ fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT);
if (!fnew) {
err = -ENOBUFS;
goto errout_tb;
@@ -2240,6 +2245,8 @@ static int fl_change(struct net *net, struct sk_buff *in_skb,
if (!tc_in_hw(fnew->flags))
fnew->flags |= TCA_CLS_FLAGS_NOT_IN_HW;
+ tcf_proto_update_usesw(tp, fnew->flags);
+
spin_lock(&tp->lock);
/* tp was deleted concurrently. -EAGAIN will cause caller to lookup
@@ -2596,6 +2603,28 @@ static void fl_tmplt_destroy(void *tmplt_priv)
kfree(tmplt);
}
+static void fl_tmplt_reoffload(struct tcf_chain *chain, bool add,
+ flow_setup_cb_t *cb, void *cb_priv)
+{
+ struct fl_flow_tmplt *tmplt = chain->tmplt_priv;
+ struct flow_cls_offload cls_flower = {};
+
+ cls_flower.rule = flow_rule_alloc(0);
+ if (!cls_flower.rule)
+ return;
+
+ cls_flower.common.chain_index = chain->index;
+ cls_flower.command = add ? FLOW_CLS_TMPLT_CREATE :
+ FLOW_CLS_TMPLT_DESTROY;
+ cls_flower.cookie = (unsigned long) tmplt;
+ cls_flower.rule->match.dissector = &tmplt->dissector;
+ cls_flower.rule->match.mask = &tmplt->mask;
+ cls_flower.rule->match.key = &tmplt->dummy_key;
+
+ cb(TC_SETUP_CLSFLOWER, &cls_flower, cb_priv);
+ kfree(cls_flower.rule);
+}
+
static int fl_dump_key_val(struct sk_buff *skb,
void *val, int val_type,
void *mask, int mask_type, int len)
@@ -3452,6 +3481,7 @@ static struct tcf_proto_ops cls_fl_ops __read_mostly = {
.bind_class = fl_bind_class,
.tmplt_create = fl_tmplt_create,
.tmplt_destroy = fl_tmplt_destroy,
+ .tmplt_reoffload = fl_tmplt_reoffload,
.tmplt_dump = fl_tmplt_dump,
.owner = THIS_MODULE,
.flags = TCF_PROTO_OPS_DOIT_UNLOCKED,
diff --git a/net/sched/cls_fw.c b/net/sched/cls_fw.c
index 979662292dce..fe6097b45707 100644
--- a/net/sched/cls_fw.c
+++ b/net/sched/cls_fw.c
@@ -274,7 +274,7 @@ static int fw_change(struct net *net, struct sk_buff *in_skb,
if (f->id != handle && handle)
return -EINVAL;
- fnew = kzalloc(sizeof(struct fw_filter), GFP_KERNEL);
+ fnew = kzalloc(sizeof(struct fw_filter), GFP_KERNEL_ACCOUNT);
if (!fnew)
return -ENOBUFS;
@@ -328,7 +328,7 @@ static int fw_change(struct net *net, struct sk_buff *in_skb,
rcu_assign_pointer(tp->root, head);
}
- f = kzalloc(sizeof(struct fw_filter), GFP_KERNEL);
+ f = kzalloc(sizeof(struct fw_filter), GFP_KERNEL_ACCOUNT);
if (f == NULL)
return -ENOBUFS;
diff --git a/net/sched/cls_matchall.c b/net/sched/cls_matchall.c
index 43f8df584741..4e54e7babda1 100644
--- a/net/sched/cls_matchall.c
+++ b/net/sched/cls_matchall.c
@@ -187,7 +187,7 @@ static int mall_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
}
- new = kzalloc(sizeof(*new), GFP_KERNEL);
+ new = kzalloc(sizeof(*new), GFP_KERNEL_ACCOUNT);
if (!new)
return -ENOBUFS;
@@ -199,7 +199,7 @@ static int mall_change(struct net *net, struct sk_buff *in_skb,
handle = 1;
new->handle = handle;
new->flags = userflags;
- new->pf = alloc_percpu(struct tc_matchall_pcnt);
+ new->pf = alloc_percpu_gfp(struct tc_matchall_pcnt, GFP_KERNEL_ACCOUNT);
if (!new->pf) {
err = -ENOMEM;
goto err_alloc_percpu;
@@ -226,6 +226,8 @@ static int mall_change(struct net *net, struct sk_buff *in_skb,
if (!tc_in_hw(new->flags))
new->flags |= TCA_CLS_FLAGS_NOT_IN_HW;
+ tcf_proto_update_usesw(tp, new->flags);
+
*arg = head;
rcu_assign_pointer(tp->root, new);
return 0;
diff --git a/net/sched/cls_route.c b/net/sched/cls_route.c
index c7fe52f59521..3f63a37bc2bb 100644
--- a/net/sched/cls_route.c
+++ b/net/sched/cls_route.c
@@ -442,7 +442,7 @@ static int route4_set_parms(struct net *net, struct tcf_proto *tp,
h1 = to_hash(nhandle);
b = rtnl_dereference(head->table[h1]);
if (!b) {
- b = kzalloc(sizeof(struct route4_bucket), GFP_KERNEL);
+ b = kzalloc(sizeof(struct route4_bucket), GFP_KERNEL_ACCOUNT);
if (b == NULL)
return -ENOBUFS;
@@ -513,7 +513,7 @@ static int route4_change(struct net *net, struct sk_buff *in_skb,
return -EINVAL;
err = -ENOBUFS;
- f = kzalloc(sizeof(struct route4_filter), GFP_KERNEL);
+ f = kzalloc(sizeof(struct route4_filter), GFP_KERNEL_ACCOUNT);
if (!f)
goto errout;
diff --git a/net/sched/cls_u32.c b/net/sched/cls_u32.c
index 276ee95f6501..26874ff2db3e 100644
--- a/net/sched/cls_u32.c
+++ b/net/sched/cls_u32.c
@@ -839,7 +839,8 @@ static struct tc_u_knode *u32_init_knode(struct net *net, struct tcf_proto *tp,
struct tc_u32_sel *s = &n->sel;
struct tc_u_knode *new;
- new = kzalloc(struct_size(new, sel.keys, s->nkeys), GFP_KERNEL);
+ new = kzalloc(struct_size(new, sel.keys, s->nkeys),
+ GFP_KERNEL_ACCOUNT);
if (!new)
return NULL;
@@ -964,6 +965,8 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
if (!tc_in_hw(new->flags))
new->flags |= TCA_CLS_FLAGS_NOT_IN_HW;
+ tcf_proto_update_usesw(tp, new->flags);
+
u32_replace_knode(tp, tp_c, new);
tcf_unbind_filter(tp, &n->res);
tcf_exts_get_net(&n->exts);
@@ -1128,15 +1131,23 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
goto erridr;
}
- n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL);
+ if (s->offshift >= 16) {
+ NL_SET_ERR_MSG_MOD(extack,
+ "offshift must be less than 16");
+ err = -EINVAL;
+ goto erridr;
+ }
+
+ n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL_ACCOUNT);
if (n == NULL) {
err = -ENOBUFS;
goto erridr;
}
#ifdef CONFIG_CLS_U32_PERF
- n->pf = __alloc_percpu(struct_size(n->pf, kcnts, s->nkeys),
- __alignof__(struct tc_u32_pcnt));
+ n->pf = __alloc_percpu_gfp(struct_size(n->pf, kcnts, s->nkeys),
+ __alignof__(struct tc_u32_pcnt),
+ GFP_KERNEL_ACCOUNT);
if (!n->pf) {
err = -ENOBUFS;
goto errfree;
@@ -1158,7 +1169,7 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
goto errout;
#ifdef CONFIG_CLS_U32_MARK
- n->pcpu_success = alloc_percpu(u32);
+ n->pcpu_success = alloc_percpu_gfp(u32, GFP_KERNEL_ACCOUNT);
if (!n->pcpu_success) {
err = -ENOMEM;
goto errout;
@@ -1189,6 +1200,8 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,
if (!tc_in_hw(n->flags))
n->flags |= TCA_CLS_FLAGS_NOT_IN_HW;
+ tcf_proto_update_usesw(tp, n->flags);
+
ins = &ht->ht[TC_U32_HASH(handle)];
for (pins = rtnl_dereference(*ins); pins;
ins = &pins->next, pins = rtnl_dereference(*ins))
diff --git a/net/sched/em_text.c b/net/sched/em_text.c
index 32aae8a9deda..d612ad4e5c12 100644
--- a/net/sched/em_text.c
+++ b/net/sched/em_text.c
@@ -58,6 +58,9 @@ static int em_text_change(struct net *net, void *data, int len,
if (len < sizeof(*conf) || len < (sizeof(*conf) + conf->pattern_len))
return -EINVAL;
+ if (!memchr(conf->algo, '\0', sizeof(conf->algo)))
+ return -EINVAL;
+
if (conf->from_layer > conf->to_layer)
return -EINVAL;
diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c
index a911ed58d078..8fa5325920eb 100644
--- a/net/sched/sch_cake.c
+++ b/net/sched/sch_cake.c
@@ -1720,7 +1720,7 @@ static u32 cake_classify(struct Qdisc *sch, struct cake_tin_data **t,
goto hash;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
diff --git a/net/sched/sch_codel.c b/net/sched/sch_codel.c
index c05be1eeea56..655eee9e629b 100644
--- a/net/sched/sch_codel.c
+++ b/net/sched/sch_codel.c
@@ -168,9 +168,9 @@ static const struct nla_policy codel_policy[TCA_CODEL_MAX + 1] = {
static int codel_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct codel_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_CODEL_MAX + 1];
- unsigned int qlen, dropped = 0;
int err;
err = nla_parse_nested_deprecated(tb, TCA_CODEL_MAX, opt,
@@ -204,15 +204,17 @@ static int codel_change(struct Qdisc *sch, struct nlattr *opt,
if (tb[TCA_CODEL_ECN])
q->params.ecn = !!nla_get_u32(tb[TCA_CODEL_ECN]);
- qlen = sch->q.qlen;
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, true);
- dropped += qdisc_pkt_len(skb);
- qdisc_qstats_backlog_dec(sch, skb);
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_qdisc_drop(skb, sch);
}
- qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen, dropped);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
@@ -228,7 +230,7 @@ static int codel_init(struct Qdisc *sch, struct nlattr *opt,
codel_params_init(&q->params);
codel_vars_init(&q->vars);
codel_stats_init(&q->stats);
- q->params.mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 256, 1 << 20);
+ q->params.mtu = min_t(u32, psched_mtu(qdisc_dev(sch)), 1 << 20);
if (opt) {
int err = codel_change(sch, opt, extack);
diff --git a/net/sched/sch_drr.c b/net/sched/sch_drr.c
index 63ec888d6b49..62c02ebdb4ec 100644
--- a/net/sched/sch_drr.c
+++ b/net/sched/sch_drr.c
@@ -312,7 +312,7 @@ static struct drr_class *drr_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
fl = rcu_dereference_bh(q->filter_list);
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_ets.c b/net/sched/sch_ets.c
index 2f1e8a5d5b37..f65dae86c618 100644
--- a/net/sched/sch_ets.c
+++ b/net/sched/sch_ets.c
@@ -389,7 +389,7 @@ static struct ets_class *ets_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
if (TC_H_MAJ(skb->priority) != sch->handle) {
fl = rcu_dereference_bh(q->filter_list);
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_fifo.c b/net/sched/sch_fifo.c
index af5f2ab69b8d..91c9bd34a4a9 100644
--- a/net/sched/sch_fifo.c
+++ b/net/sched/sch_fifo.c
@@ -19,7 +19,8 @@
static int bfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
- if (likely(sch->qstats.backlog + qdisc_pkt_len(skb) <= sch->limit))
+ if (likely((u64)sch->qstats.backlog + qdisc_pkt_len(skb) <=
+ READ_ONCE(sch->limit)))
return qdisc_enqueue_tail(skb, sch);
return qdisc_drop(skb, sch, to_free);
@@ -28,7 +29,7 @@ static int bfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch,
static int pfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch,
struct sk_buff **to_free)
{
- if (likely(sch->q.qlen < sch->limit))
+ if (likely(sch->q.qlen < READ_ONCE(sch->limit)))
return qdisc_enqueue_tail(skb, sch);
return qdisc_drop(skb, sch, to_free);
@@ -42,7 +43,7 @@ static int pfifo_tail_enqueue(struct sk_buff *skb, struct Qdisc *sch,
if (unlikely(READ_ONCE(sch->limit) == 0))
return qdisc_drop(skb, sch, to_free);
- if (likely(sch->q.qlen < sch->limit))
+ if (likely(sch->q.qlen < READ_ONCE(sch->limit)))
return qdisc_enqueue_tail(skb, sch);
prev_backlog = sch->qstats.backlog;
@@ -108,14 +109,14 @@ static int __fifo_init(struct Qdisc *sch, struct nlattr *opt,
if (is_bfifo)
limit *= psched_mtu(qdisc_dev(sch));
- sch->limit = limit;
+ WRITE_ONCE(sch->limit, limit);
} else {
struct tc_fifo_qopt *ctl = nla_data(opt);
if (nla_len(opt) < sizeof(*ctl))
return -EINVAL;
- sch->limit = ctl->limit;
+ WRITE_ONCE(sch->limit, ctl->limit);
}
if (is_bfifo)
@@ -157,7 +158,7 @@ static void fifo_destroy(struct Qdisc *sch)
static int __fifo_dump(struct Qdisc *sch, struct sk_buff *skb)
{
- struct tc_fifo_qopt opt = { .limit = sch->limit };
+ struct tc_fifo_qopt opt = { .limit = READ_ONCE(sch->limit) };
if (nla_put(skb, TCA_OPTIONS, sizeof(opt), &opt))
goto nla_put_failure;
diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c
index 91f5ef6be0f2..ee8629d11647 100644
--- a/net/sched/sch_fq.c
+++ b/net/sched/sch_fq.c
@@ -806,11 +806,11 @@ static const struct nla_policy fq_policy[TCA_FQ_MAX + 1] = {
static int fq_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct fq_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_FQ_MAX + 1];
- int err, drop_count = 0;
- unsigned drop_len = 0;
u32 fq_log;
+ int err;
err = nla_parse_nested_deprecated(tb, TCA_FQ_MAX, opt, fq_policy,
NULL);
@@ -900,16 +900,18 @@ static int fq_change(struct Qdisc *sch, struct nlattr *opt,
err = fq_resize(sch, fq_log);
sch_tree_lock(sch);
}
+
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, false);
if (!skb)
break;
- drop_len += qdisc_pkt_len(skb);
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_kfree_skbs(skb, skb);
- drop_count++;
}
- qdisc_tree_reduce_backlog(sch, drop_count, drop_len);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return err;
@@ -928,12 +930,14 @@ static int fq_init(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
struct fq_sched_data *q = qdisc_priv(sch);
+ u32 mtu;
int err;
sch->limit = 10000;
q->flow_plimit = 100;
- q->quantum = 2 * psched_mtu(qdisc_dev(sch));
- q->initial_quantum = 10 * psched_mtu(qdisc_dev(sch));
+ mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 1, 1 << 20);
+ q->quantum = min_t(u32, 2 * mtu, 1 << 20);
+ q->initial_quantum = min_t(u32, 10 * mtu, 1 << 20);
q->flow_refill_delay = msecs_to_jiffies(40);
q->flow_max_rate = ~0UL;
q->time_next_delayed_flow = ~0ULL;
diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c
index ac2f643dd842..8a4f4c496370 100644
--- a/net/sched/sch_fq_codel.c
+++ b/net/sched/sch_fq_codel.c
@@ -91,7 +91,7 @@ static unsigned int fq_codel_classify(struct sk_buff *skb, struct Qdisc *sch,
return fq_codel_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
@@ -402,6 +402,7 @@ static const struct nla_policy fq_codel_policy[TCA_FQ_CODEL_MAX + 1] = {
static int fq_codel_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct fq_codel_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_FQ_CODEL_MAX + 1];
u32 quantum = 0;
@@ -470,13 +471,14 @@ static int fq_codel_change(struct Qdisc *sch, struct nlattr *opt,
q->memory_usage > q->memory_limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, false);
- q->cstats.drop_len += qdisc_pkt_len(skb);
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_kfree_skbs(skb, skb);
- q->cstats.drop_count++;
}
- qdisc_tree_reduce_backlog(sch, q->cstats.drop_count, q->cstats.drop_len);
- q->cstats.drop_count = 0;
- q->cstats.drop_len = 0;
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
@@ -503,8 +505,8 @@ static int fq_codel_init(struct Qdisc *sch, struct nlattr *opt,
q->flows_cnt = 1024;
q->memory_limit = 32 << 20; /* 32 MBytes */
q->drop_batch_size = 64;
- mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 256, FQ_CODEL_QUANTUM_MAX);
- q->quantum = mtu;
+ mtu = min_t(u32, psched_mtu(qdisc_dev(sch)), FQ_CODEL_QUANTUM_MAX);
+ q->quantum = clamp_t(u32, mtu, 256, FQ_CODEL_QUANTUM_MAX);
INIT_LIST_HEAD(&q->new_flows);
INIT_LIST_HEAD(&q->old_flows);
codel_params_init(&q->cparams);
diff --git a/net/sched/sch_fq_pie.c b/net/sched/sch_fq_pie.c
index 75d6e7716ae8..1bf86b65dacf 100644
--- a/net/sched/sch_fq_pie.c
+++ b/net/sched/sch_fq_pie.c
@@ -95,7 +95,7 @@ static unsigned int fq_pie_classify(struct sk_buff *skb, struct Qdisc *sch,
return fq_pie_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, filter, &res, false);
+ result = tcf_classify_qdisc(skb, filter, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
@@ -284,10 +284,9 @@ static struct sk_buff *fq_pie_qdisc_dequeue(struct Qdisc *sch)
static int fq_pie_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct fq_pie_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_FQ_PIE_MAX + 1];
- unsigned int len_dropped = 0;
- unsigned int num_dropped = 0;
int err;
err = nla_parse_nested(tb, TCA_FQ_PIE_MAX, opt, fq_pie_policy, extack);
@@ -366,11 +365,14 @@ static int fq_pie_change(struct Qdisc *sch, struct nlattr *opt,
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, false);
- len_dropped += qdisc_pkt_len(skb);
- num_dropped += 1;
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_kfree_skbs(skb, skb);
}
- qdisc_tree_reduce_backlog(sch, num_dropped, len_dropped);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c
index f58ec57e31f4..3d5f86be5a95 100644
--- a/net/sched/sch_generic.c
+++ b/net/sched/sch_generic.c
@@ -541,28 +541,43 @@ static void dev_watchdog(struct timer_list *t)
dev->netdev_ops->ndo_tx_timeout(dev, i);
netif_unfreeze_queues(dev);
}
- if (!mod_timer(&dev->watchdog_timer,
- round_jiffies(oldest_start +
- dev->watchdog_timeo)))
- release = false;
+ spin_lock(&dev->watchdog_lock);
+ mod_timer(&dev->watchdog_timer,
+ round_jiffies(oldest_start +
+ dev->watchdog_timeo));
+ release = false;
+ spin_unlock(&dev->watchdog_lock);
}
}
spin_unlock(&dev->tx_global_lock);
- if (release)
+ spin_lock(&dev->watchdog_lock);
+ if (timer_pending(&dev->watchdog_timer))
+ release = false;
+ if (release && dev->watchdog_ref_held) {
netdev_put(dev, &dev->watchdog_dev_tracker);
+ dev->watchdog_ref_held = false;
+ }
+ spin_unlock(&dev->watchdog_lock);
}
void __netdev_watchdog_up(struct net_device *dev)
{
- if (dev->netdev_ops->ndo_tx_timeout) {
- if (dev->watchdog_timeo <= 0)
- dev->watchdog_timeo = 5*HZ;
- if (!mod_timer(&dev->watchdog_timer,
- round_jiffies(jiffies + dev->watchdog_timeo)))
+ if (!dev->netdev_ops->ndo_tx_timeout)
+ return;
+ if (dev->watchdog_timeo <= 0)
+ dev->watchdog_timeo = 5*HZ;
+
+ spin_lock_bh(&dev->watchdog_lock);
+ if (!mod_timer(&dev->watchdog_timer,
+ round_jiffies(jiffies + dev->watchdog_timeo))) {
+ if (!dev->watchdog_ref_held) {
netdev_hold(dev, &dev->watchdog_dev_tracker,
GFP_ATOMIC);
+ dev->watchdog_ref_held = true;
+ }
}
+ spin_unlock_bh(&dev->watchdog_lock);
}
EXPORT_SYMBOL_GPL(__netdev_watchdog_up);
@@ -574,8 +589,14 @@ static void dev_watchdog_up(struct net_device *dev)
static void dev_watchdog_down(struct net_device *dev)
{
netif_tx_lock_bh(dev);
- if (del_timer(&dev->watchdog_timer))
+
+ spin_lock(&dev->watchdog_lock);
+ if (del_timer(&dev->watchdog_timer)) {
netdev_put(dev, &dev->watchdog_dev_tracker);
+ dev->watchdog_ref_held = false;
+ }
+ spin_unlock(&dev->watchdog_lock);
+
netif_tx_unlock_bh(dev);
}
@@ -592,8 +613,6 @@ void netif_carrier_on(struct net_device *dev)
return;
atomic_inc(&dev->carrier_up_count);
linkwatch_fire_event(dev);
- if (netif_running(dev))
- __netdev_watchdog_up(dev);
}
}
EXPORT_SYMBOL(netif_carrier_on);
@@ -684,6 +703,7 @@ struct Qdisc noop_qdisc = {
.qlen = 0,
.lock = __SPIN_LOCK_UNLOCKED(noop_qdisc.skb_bad_txq.lock),
},
+ .owner = -1,
};
EXPORT_SYMBOL(noop_qdisc);
@@ -986,6 +1006,7 @@ struct Qdisc *qdisc_alloc(struct netdev_queue *dev_queue,
sch->enqueue = ops->enqueue;
sch->dequeue = ops->dequeue;
sch->dev_queue = dev_queue;
+ sch->owner = -1;
netdev_hold(dev, &sch->dev_tracker, GFP_KERNEL);
refcount_set(&sch->refcnt, 1);
diff --git a/net/sched/sch_gred.c b/net/sched/sch_gred.c
index fa7a1b69c0f3..4f44a268286f 100644
--- a/net/sched/sch_gred.c
+++ b/net/sched/sch_gred.c
@@ -179,7 +179,7 @@ static int gred_enqueue(struct sk_buff *skb, struct Qdisc *sch,
* if no default DP has been configured. This
* allows for DP flows to be left untouched.
*/
- if (likely(sch->qstats.backlog + qdisc_pkt_len(skb) <=
+ if (likely((u64)sch->qstats.backlog + qdisc_pkt_len(skb) <=
sch->limit))
return qdisc_enqueue_tail(skb, sch);
else
@@ -244,7 +244,7 @@ static int gred_enqueue(struct sk_buff *skb, struct Qdisc *sch,
break;
}
- if (gred_backlog(t, q, sch) + qdisc_pkt_len(skb) <= q->limit) {
+ if ((u64)gred_backlog(t, q, sch) + qdisc_pkt_len(skb) <= q->limit) {
q->backlog += qdisc_pkt_len(skb);
return qdisc_enqueue_tail(skb, sch);
}
diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c
index eb99bc1a76f6..9550176e3a48 100644
--- a/net/sched/sch_hfsc.c
+++ b/net/sched/sch_hfsc.c
@@ -1152,7 +1152,7 @@ hfsc_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
head = &q->root;
drift = HFSC_CLASSIFY_MAX_DRIFT;
tcf = rcu_dereference_bh(q->root.filter_list);
- while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) {
+ while (tcf && (result = tcf_classify_qdisc(skb, tcf, &res, false)) >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
case TC_ACT_QUEUED:
diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c
index e1e8527e4519..17027d1f4d11 100644
--- a/net/sched/sch_hhf.c
+++ b/net/sched/sch_hhf.c
@@ -536,9 +536,9 @@ static const struct nla_policy hhf_policy[TCA_HHF_MAX + 1] = {
static int hhf_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct hhf_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_HHF_MAX + 1];
- unsigned int qlen, prev_backlog;
int err;
u64 non_hh_quantum;
u32 new_quantum = q->quantum;
@@ -585,15 +585,17 @@ static int hhf_change(struct Qdisc *sch, struct nlattr *opt,
q->hhf_evict_timeout = usecs_to_jiffies(us);
}
- qlen = sch->q.qlen;
- prev_backlog = sch->qstats.backlog;
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, false);
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_kfree_skbs(skb, skb);
}
- qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen,
- prev_backlog - sch->qstats.backlog);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c
index 1fbaedc130e7..7515de8511ad 100644
--- a/net/sched/sch_htb.c
+++ b/net/sched/sch_htb.c
@@ -240,7 +240,7 @@ static struct htb_class *htb_classify(struct sk_buff *skb, struct Qdisc *sch,
}
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) {
+ while (tcf && (result = tcf_classify_qdisc(skb, tcf, &res, false)) >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
case TC_ACT_QUEUED:
diff --git a/net/sched/sch_multiq.c b/net/sched/sch_multiq.c
index 62cc1a21a603..3dff4b1d4aa9 100644
--- a/net/sched/sch_multiq.c
+++ b/net/sched/sch_multiq.c
@@ -36,7 +36,7 @@ multiq_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
int err;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_netem.c b/net/sched/sch_netem.c
index 2c47bd8dba64..5d6e928ab750 100644
--- a/net/sched/sch_netem.c
+++ b/net/sched/sch_netem.c
@@ -508,8 +508,9 @@ static int netem_enqueue(struct sk_buff *skb, struct Qdisc *sch,
goto finish_segs;
}
- skb->data[prandom_u32_max(skb_headlen(skb))] ^=
- 1<<prandom_u32_max(8);
+ if (skb_headlen(skb))
+ skb->data[prandom_u32_max(skb_headlen(skb))] ^=
+ 1 << prandom_u32_max(8);
}
if (unlikely(sch->q.qlen >= sch->limit)) {
diff --git a/net/sched/sch_pie.c b/net/sched/sch_pie.c
index 1ed6f43c93c4..5dffe25457b8 100644
--- a/net/sched/sch_pie.c
+++ b/net/sched/sch_pie.c
@@ -138,9 +138,9 @@ static const struct nla_policy pie_policy[TCA_PIE_MAX + 1] = {
static int pie_change(struct Qdisc *sch, struct nlattr *opt,
struct netlink_ext_ack *extack)
{
+ unsigned int dropped_pkts = 0, dropped_bytes = 0;
struct pie_sched_data *q = qdisc_priv(sch);
struct nlattr *tb[TCA_PIE_MAX + 1];
- unsigned int qlen, dropped = 0;
int err;
err = nla_parse_nested_deprecated(tb, TCA_PIE_MAX, opt, pie_policy,
@@ -188,15 +188,17 @@ static int pie_change(struct Qdisc *sch, struct nlattr *opt,
nla_get_u32(tb[TCA_PIE_DQ_RATE_ESTIMATOR]);
/* Drop excess packets if new limit is lower */
- qlen = sch->q.qlen;
while (sch->q.qlen > sch->limit) {
struct sk_buff *skb = qdisc_dequeue_internal(sch, true);
- dropped += qdisc_pkt_len(skb);
- qdisc_qstats_backlog_dec(sch, skb);
+ if (!skb)
+ break;
+
+ dropped_pkts++;
+ dropped_bytes += qdisc_pkt_len(skb);
rtnl_qdisc_drop(skb, sch);
}
- qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen, dropped);
+ qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes);
sch_tree_unlock(sch);
return 0;
diff --git a/net/sched/sch_plug.c b/net/sched/sch_plug.c
index 35f49edf63db..6638343ca671 100644
--- a/net/sched/sch_plug.c
+++ b/net/sched/sch_plug.c
@@ -89,7 +89,7 @@ static int plug_enqueue(struct sk_buff *skb, struct Qdisc *sch,
{
struct plug_sched_data *q = qdisc_priv(sch);
- if (likely(sch->qstats.backlog + skb->len <= q->limit)) {
+ if (likely((u64)sch->qstats.backlog + skb->len <= q->limit)) {
if (!q->unplug_indefinite)
q->pkts_current_epoch++;
return qdisc_enqueue_tail(skb, sch);
diff --git a/net/sched/sch_prio.c b/net/sched/sch_prio.c
index fdd9caa41e80..7ef77ead388e 100644
--- a/net/sched/sch_prio.c
+++ b/net/sched/sch_prio.c
@@ -39,7 +39,7 @@ prio_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr)
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
if (TC_H_MAJ(skb->priority) != sch->handle) {
fl = rcu_dereference_bh(q->filter_list);
- err = tcf_classify(skb, NULL, fl, &res, false);
+ err = tcf_classify_qdisc(skb, fl, &res, false);
#ifdef CONFIG_NET_CLS_ACT
switch (err) {
case TC_ACT_STOLEN:
diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c
index 51d962e5113b..62ac5eba222f 100644
--- a/net/sched/sch_qfq.c
+++ b/net/sched/sch_qfq.c
@@ -711,7 +711,7 @@ static struct qfq_class *qfq_classify(struct sk_buff *skb, struct Qdisc *sch,
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
fl = rcu_dereference_bh(q->filter_list);
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_sfb.c b/net/sched/sch_sfb.c
index 58bf4c803f7f..693f562bd864 100644
--- a/net/sched/sch_sfb.c
+++ b/net/sched/sch_sfb.c
@@ -260,7 +260,7 @@ static bool sfb_classify(struct sk_buff *skb, struct tcf_proto *fl,
struct tcf_result res;
int result;
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c
index 779e632ede19..31b5aeb416ce 100644
--- a/net/sched/sch_sfq.c
+++ b/net/sched/sch_sfq.c
@@ -171,7 +171,7 @@ static unsigned int sfq_classify(struct sk_buff *skb, struct Qdisc *sch,
return sfq_hash(q, skb) + 1;
*qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS;
- result = tcf_classify(skb, NULL, fl, &res, false);
+ result = tcf_classify_qdisc(skb, fl, &res, false);
if (result >= 0) {
#ifdef CONFIG_NET_CLS_ACT
switch (result) {
diff --git a/net/sctp/bind_addr.c b/net/sctp/bind_addr.c
index 6b95d3ba8fe1..0947b276d1e0 100644
--- a/net/sctp/bind_addr.c
+++ b/net/sctp/bind_addr.c
@@ -275,6 +275,16 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list,
param = (struct sctp_paramhdr *)raw_addr_list;
rawaddr = (union sctp_addr_param *)raw_addr_list;
+ if (addrs_len < sizeof(*param)) {
+ retval = -EINVAL;
+ goto out_err;
+ }
+ len = ntohs(param->length);
+ if (addrs_len < len) {
+ retval = -EINVAL;
+ goto out_err;
+ }
+
af = sctp_get_af_specific(param_type2af(param->type));
if (unlikely(!af) ||
!af->from_addr_param(&addr, rawaddr, htons(port), 0)) {
@@ -291,7 +301,6 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list,
goto out_err;
next:
- len = ntohs(param->length);
addrs_len -= len;
raw_addr_list += len;
}
diff --git a/net/sctp/inqueue.c b/net/sctp/inqueue.c
index 5f988b3a8814..d666cec6b194 100644
--- a/net/sctp/inqueue.c
+++ b/net/sctp/inqueue.c
@@ -212,8 +212,10 @@ struct sctp_chunk *sctp_inq_pop(struct sctp_inq *queue)
chunk->chunk_end = ((__u8 *)ch) + SCTP_PAD4(ntohs(ch->length));
skb_pull(chunk->skb, sizeof(*ch));
chunk->subh.v = NULL; /* Subheader is no longer valid. */
-
- if (chunk->chunk_end + sizeof(*ch) <= skb_tail_pointer(chunk->skb)) {
+ if (unlikely(ntohs(ch->length) < sizeof(*ch))) {
+ chunk->pdiscard = 1;
+ } else if (chunk->chunk_end + sizeof(*ch) <=
+ skb_tail_pointer(chunk->skb)) {
/* This is not a singleton */
chunk->singleton = 0;
} else if (chunk->chunk_end > skb_tail_pointer(chunk->skb)) {
diff --git a/net/sctp/ipv6.c b/net/sctp/ipv6.c
index 3b07b9f5d752..01fefac2f6e7 100644
--- a/net/sctp/ipv6.c
+++ b/net/sctp/ipv6.c
@@ -227,6 +227,7 @@ static int sctp_v6_xmit(struct sk_buff *skb, struct sctp_transport *t)
struct ipv6_pinfo *np = inet6_sk(sk);
__u8 tclass = np->tclass;
__be32 label;
+ int err;
pr_debug("%s: skb:%p, len:%d, src:%pI6 dst:%pI6\n", __func__, skb,
skb->len, &fl6->saddr, &fl6->daddr);
@@ -263,9 +264,12 @@ static int sctp_v6_xmit(struct sk_buff *skb, struct sctp_transport *t)
skb_set_inner_ipproto(skb, IPPROTO_SCTP);
label = ip6_make_flowlabel(sock_net(sk), skb, fl6->flowlabel, true, fl6);
- return udp_tunnel6_xmit_skb(dst, sk, skb, NULL, &fl6->saddr,
- &fl6->daddr, tclass, ip6_dst_hoplimit(dst),
- label, sctp_sk(sk)->udp_port, t->encap_port, false);
+ local_bh_disable();
+ err = udp_tunnel6_xmit_skb(dst, sk, skb, NULL, &fl6->saddr,
+ &fl6->daddr, tclass, ip6_dst_hoplimit(dst),
+ label, sctp_sk(sk)->udp_port, t->encap_port, false);
+ local_bh_enable();
+ return err;
}
/* Returns the dst cache entry for the given source and destination ip
diff --git a/net/sctp/protocol.c b/net/sctp/protocol.c
index 94878fc63f07..bf8ef2737778 100644
--- a/net/sctp/protocol.c
+++ b/net/sctp/protocol.c
@@ -685,8 +685,9 @@ static void sctp_free_addr_wq(struct net *net)
struct sctp_sockaddr_entry *addrw;
struct sctp_sockaddr_entry *temp;
+ timer_shutdown_sync(&net->sctp.addr_wq_timer);
+
spin_lock_bh(&net->sctp.addr_wq_lock);
- del_timer(&net->sctp.addr_wq_timer);
list_for_each_entry_safe(addrw, temp, &net->sctp.addr_waitq, list) {
list_del(&addrw->list);
kfree(addrw);
@@ -1078,9 +1079,11 @@ static inline int sctp_v4_xmit(struct sk_buff *skb, struct sctp_transport *t)
skb_reset_inner_mac_header(skb);
skb_reset_inner_transport_header(skb);
skb_set_inner_ipproto(skb, IPPROTO_SCTP);
+ local_bh_disable();
udp_tunnel_xmit_skb((struct rtable *)dst, sk, skb, fl4->saddr,
fl4->daddr, dscp, ip4_dst_hoplimit(dst), df,
sctp_sk(sk)->udp_port, t->encap_port, false, false);
+ local_bh_enable();
return 0;
}
diff --git a/net/sctp/sm_make_chunk.c b/net/sctp/sm_make_chunk.c
index 649384ff0e54..b5ade45d6c66 100644
--- a/net/sctp/sm_make_chunk.c
+++ b/net/sctp/sm_make_chunk.c
@@ -1750,8 +1750,9 @@ struct sctp_association *sctp_unpack_cookie(
struct sk_buff *skb = chunk->skb;
struct sctp_cookie *bear_cookie;
__u8 *digest = ep->digest;
+ struct sctp_chunkhdr *ch;
+ unsigned int len, chlen;
enum sctp_scope scope;
- unsigned int len;
ktime_t kt;
/* Header size is static data prior to the actual cookie, including
@@ -1779,6 +1780,17 @@ struct sctp_association *sctp_unpack_cookie(
cookie = chunk->subh.cookie_hdr;
bear_cookie = &cookie->c;
+ ch = (struct sctp_chunkhdr *)&bear_cookie->peer_init[0];
+ if (ch->type != SCTP_CID_INIT)
+ goto malformed;
+ chlen = ntohs(ch->length);
+ if (chlen < sizeof(struct sctp_init_chunk))
+ goto malformed;
+ if (chlen > len - fixed_size)
+ goto malformed;
+ if (bear_cookie->raw_addr_list_len > len - fixed_size - chlen)
+ goto malformed;
+
if (!sctp_sk(ep->base.sk)->hmac)
goto no_hmac;
@@ -2328,7 +2340,8 @@ int sctp_verify_init(struct net *net, const struct sctp_endpoint *ep,
* VIOLATION error. We build the ERROR chunk here and let the normal
* error handling code build and send the packet.
*/
- if (param.v != (void *)chunk->chunk_end)
+ if (param.v != (void *)peer_init +
+ SCTP_PAD4(ntohs(peer_init->chunk_hdr.length)))
return sctp_process_inv_paramlength(asoc, param.p, chunk, errp);
/* The only missing mandatory param possible today is
diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c
index 052a9f74354c..57d81cc01f72 100644
--- a/net/sctp/sm_statefuns.c
+++ b/net/sctp/sm_statefuns.c
@@ -707,11 +707,12 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net,
struct sctp_cmd_seq *commands)
{
struct sctp_ulpevent *ev, *ai_ev = NULL, *auth_ev = NULL;
+ struct sctp_chunk *err_chk_p = NULL;
struct sctp_association *new_asoc;
struct sctp_init_chunk *peer_init;
struct sctp_chunk *chunk = arg;
- struct sctp_chunk *err_chk_p;
struct sctp_chunk *repl;
+ enum sctp_cid cid;
struct sock *sk;
int error = 0;
@@ -785,6 +786,19 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net,
}
}
+ peer_init = &chunk->subh.cookie_hdr->c.peer_init[0];
+ cid = peer_init->chunk_hdr.type;
+ if (!sctp_sk(sk)->hmac &&
+ !sctp_verify_init(net, ep, asoc, cid, peer_init, chunk,
+ &err_chk_p)) {
+ sctp_association_free(new_asoc);
+ if (err_chk_p)
+ sctp_chunk_free(err_chk_p);
+ return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
+ }
+ if (err_chk_p)
+ sctp_chunk_free(err_chk_p);
+
if (security_sctp_assoc_request(new_asoc, chunk->head_skb ?: chunk->skb)) {
sctp_association_free(new_asoc);
return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands);
@@ -798,8 +812,6 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net,
/* This is a brand-new association, so these are not yet side
* effects--it is safe to run them here.
*/
- peer_init = &chunk->subh.cookie_hdr->c.peer_init[0];
-
if (!sctp_process_init(new_asoc, chunk,
&chunk->subh.cookie_hdr->c.peer_addr,
peer_init, GFP_ATOMIC))
@@ -2217,10 +2229,12 @@ enum sctp_disposition sctp_sf_do_5_2_4_dupcook(
void *arg,
struct sctp_cmd_seq *commands)
{
+ struct sctp_chunk *err_chk_p = NULL;
struct sctp_association *new_asoc;
+ struct sctp_init_chunk *peer_init;
struct sctp_chunk *chunk = arg;
enum sctp_disposition retval;
- struct sctp_chunk *err_chk_p;
+ enum sctp_cid cid;
int error = 0;
char action;
@@ -2289,6 +2303,20 @@ enum sctp_disposition sctp_sf_do_5_2_4_dupcook(
switch (action) {
case 'A': /* Association restart. */
case 'B': /* Collision case B. */
+ peer_init = &chunk->subh.cookie_hdr->c.peer_init[0];
+ cid = peer_init->chunk_hdr.type;
+ if (!sctp_sk(ep->base.sk)->hmac &&
+ !sctp_verify_init(net, ep, asoc, cid, peer_init, chunk,
+ &err_chk_p)) {
+ sctp_association_free(new_asoc);
+ if (err_chk_p)
+ sctp_chunk_free(err_chk_p);
+ return sctp_sf_pdiscard(net, ep, asoc, type, arg,
+ commands);
+ }
+ if (err_chk_p)
+ sctp_chunk_free(err_chk_p);
+ fallthrough;
case 'D': /* Collision case D. */
/* Update socket peer label if first association. */
if (security_sctp_assoc_request((struct sctp_association *)asoc,
diff --git a/net/sctp/socket.c b/net/sctp/socket.c
index 82170d333fac..28d5d2cf6091 100644
--- a/net/sctp/socket.c
+++ b/net/sctp/socket.c
@@ -1784,6 +1784,18 @@ static int sctp_sendmsg_check_sflags(struct sctp_association *asoc,
return 1;
}
+static int sctp_sendmsg_check_stream(struct sctp_association *asoc,
+ struct sctp_sndrcvinfo *sinfo)
+{
+ if (unlikely(sinfo->sinfo_stream >= asoc->stream.outcnt))
+ return -EINVAL;
+
+ if (unlikely(!SCTP_SO(&asoc->stream, sinfo->sinfo_stream)->ext))
+ return sctp_stream_init_ext(&asoc->stream, sinfo->sinfo_stream);
+
+ return 0;
+}
+
static int sctp_sendmsg_to_asoc(struct sctp_association *asoc,
struct msghdr *msg, size_t msg_len,
struct sctp_transport *transport,
@@ -1798,16 +1810,9 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc,
long timeo;
int err;
- if (sinfo->sinfo_stream >= asoc->stream.outcnt) {
- err = -EINVAL;
+ err = sctp_sendmsg_check_stream(asoc, sinfo);
+ if (err)
goto err;
- }
-
- if (unlikely(!SCTP_SO(&asoc->stream, sinfo->sinfo_stream)->ext)) {
- err = sctp_stream_init_ext(&asoc->stream, sinfo->sinfo_stream);
- if (err)
- goto err;
- }
if (sp->disable_fragments && msg_len > asoc->frag_point) {
err = -EMSGSIZE;
@@ -1828,10 +1833,9 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc,
err = sctp_wait_for_sndbuf(asoc, transport, &timeo, msg_len);
if (err)
goto err;
- if (unlikely(sinfo->sinfo_stream >= asoc->stream.outcnt)) {
- err = -EINVAL;
+ err = sctp_sendmsg_check_stream(asoc, sinfo);
+ if (err)
goto err;
- }
}
if (sctp_state(asoc, CLOSED)) {
@@ -1842,6 +1846,11 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc,
if (asoc->ep->intl_enable) {
timeo = sock_sndtimeo(sk, 0);
err = sctp_wait_for_connect(asoc, &timeo);
+ if (err || sk->sk_shutdown & RCV_SHUTDOWN) {
+ err = -ESRCH;
+ goto err;
+ }
+ err = sctp_sendmsg_check_stream(asoc, sinfo);
if (err) {
err = -ESRCH;
goto err;
@@ -7520,11 +7529,7 @@ static int sctp_getsockopt_pr_streamstatus(struct sock *sk, int len,
/* Not allocated yet, means all stats are 0 */
params.sprstat_abandoned_unsent = 0;
params.sprstat_abandoned_sent = 0;
- retval = 0;
- goto out;
- }
-
- if (policy == SCTP_PR_SCTP_ALL) {
+ } else if (policy == SCTP_PR_SCTP_ALL) {
params.sprstat_abandoned_unsent = 0;
params.sprstat_abandoned_sent = 0;
for (policy = 0; policy <= SCTP_PR_INDEX(MAX); policy++) {
diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c
index 89fa7295275c..4c158cde1f50 100644
--- a/net/smc/af_smc.c
+++ b/net/smc/af_smc.c
@@ -406,6 +406,7 @@ static struct sock *smc_sock_alloc(struct net *net, struct socket *sock,
sk_refcnt_debug_inc(sk);
mutex_init(&smc->clcsock_release_lock);
smc_init_saved_callbacks(smc);
+ smc_close_init(smc);
return sk;
}
@@ -1277,7 +1278,6 @@ static int smc_connect_rdma(struct smc_sock *smc,
goto connect_abort;
}
- smc_close_init(smc);
smc_rx_init(smc);
if (ini->first_contact_local) {
@@ -1397,7 +1397,6 @@ static int smc_connect_ism(struct smc_sock *smc,
}
smc_conn_save_peer_info(smc, aclc);
- smc_close_init(smc);
smc_rx_init(smc);
smc_tx_init(smc);
@@ -2430,7 +2429,6 @@ static void smc_listen_work(struct work_struct *work)
goto out_decl;
mutex_lock(&smc_server_lgr_pending);
- smc_close_init(new_smc);
smc_rx_init(new_smc);
smc_tx_init(new_smc);
@@ -2470,8 +2468,9 @@ static void smc_listen_work(struct work_struct *work)
mutex_unlock(&smc_server_lgr_pending);
}
smc_conn_save_peer_info(new_smc, cclc);
- smc_listen_out_connected(new_smc);
SMC_STAT_SERV_SUCC_INC(sock_net(newclcsock->sk), ini);
+ /* smc_listen_out() will release smcsk */
+ smc_listen_out_connected(new_smc);
goto out_free;
out_unlock:
@@ -2677,6 +2676,7 @@ static int smc_getname(struct socket *sock, struct sockaddr *addr,
int peer)
{
struct smc_sock *smc;
+ int rc = -EBADF;
if (peer && (sock->sk->sk_state != SMC_ACTIVE) &&
(sock->sk->sk_state != SMC_APPCLOSEWAIT1))
@@ -2684,7 +2684,11 @@ static int smc_getname(struct socket *sock, struct sockaddr *addr,
smc = smc_sk(sock->sk);
- return smc->clcsock->ops->getname(smc->clcsock, addr, peer);
+ mutex_lock(&smc->clcsock_release_lock);
+ if (smc->clcsock)
+ rc = smc->clcsock->ops->getname(smc->clcsock, addr, peer);
+ mutex_unlock(&smc->clcsock_release_lock);
+ return rc;
}
static int smc_sendmsg(struct socket *sock, struct msghdr *msg, size_t len)
@@ -3271,8 +3275,8 @@ static const struct proto_ops smc_sock_ops = {
.splice_read = smc_splice_read,
};
-static int __smc_create(struct net *net, struct socket *sock, int protocol,
- int kern, struct socket *clcsock)
+static int smc_create(struct net *net, struct socket *sock, int protocol,
+ int kern)
{
int family = (protocol == SMCPROTO_SMC6) ? PF_INET6 : PF_INET;
struct smc_sock *smc;
@@ -3302,92 +3306,33 @@ static int __smc_create(struct net *net, struct socket *sock, int protocol,
/* default behavior from limit_smc_hs in every net namespace */
smc->limit_smc_hs = net->smc.limit_smc_hs;
- rc = 0;
- if (!clcsock) {
- rc = sock_create_kern(net, family, SOCK_STREAM, IPPROTO_TCP,
- &smc->clcsock);
- if (rc) {
- sk_common_release(sk);
- goto out;
- }
- } else {
- smc->clcsock = clcsock;
+ rc = sock_create_kern(net, family, SOCK_STREAM, IPPROTO_TCP,
+ &smc->clcsock);
+ if (rc) {
+ sk_common_release(sk);
+ goto out;
}
+ /* smc_clcsock_release() does not wait smc->clcsock->sk's
+ * destruction; its sk_state might not be TCP_CLOSE after
+ * smc->sk is close()d, and TCP timers can be fired later,
+ * which need net ref.
+ */
+ sk = smc->clcsock->sk;
+ sk->sk_net_refcnt = 1;
+ get_net_track(net, &sk->ns_tracker, GFP_KERNEL);
+ sock_inuse_add(net, 1);
+
out:
return rc;
}
-static int smc_create(struct net *net, struct socket *sock, int protocol,
- int kern)
-{
- return __smc_create(net, sock, protocol, kern, NULL);
-}
-
static const struct net_proto_family smc_sock_family_ops = {
.family = PF_SMC,
.owner = THIS_MODULE,
.create = smc_create,
};
-static int smc_ulp_init(struct sock *sk)
-{
- struct socket *tcp = sk->sk_socket;
- struct net *net = sock_net(sk);
- struct socket *smcsock;
- int protocol, ret;
-
- /* only TCP can be replaced */
- if (tcp->type != SOCK_STREAM || sk->sk_protocol != IPPROTO_TCP ||
- (sk->sk_family != AF_INET && sk->sk_family != AF_INET6))
- return -ESOCKTNOSUPPORT;
- /* don't handle wq now */
- if (tcp->state != SS_UNCONNECTED || !tcp->file || tcp->wq.fasync_list)
- return -ENOTCONN;
-
- if (sk->sk_family == AF_INET)
- protocol = SMCPROTO_SMC;
- else
- protocol = SMCPROTO_SMC6;
-
- smcsock = sock_alloc();
- if (!smcsock)
- return -ENFILE;
-
- smcsock->type = SOCK_STREAM;
- __module_get(THIS_MODULE); /* tried in __tcp_ulp_find_autoload */
- ret = __smc_create(net, smcsock, protocol, 1, tcp);
- if (ret) {
- sock_release(smcsock); /* module_put() which ops won't be NULL */
- return ret;
- }
-
- /* replace tcp socket to smc */
- smcsock->file = tcp->file;
- smcsock->file->private_data = smcsock;
- smcsock->file->f_inode = SOCK_INODE(smcsock); /* replace inode when sock_close */
- smcsock->file->f_path.dentry->d_inode = SOCK_INODE(smcsock); /* dput() in __fput */
- tcp->file = NULL;
-
- return ret;
-}
-
-static void smc_ulp_clone(const struct request_sock *req, struct sock *newsk,
- const gfp_t priority)
-{
- struct inet_connection_sock *icsk = inet_csk(newsk);
-
- /* don't inherit ulp ops to child when listen */
- icsk->icsk_ulp_ops = NULL;
-}
-
-static struct tcp_ulp_ops smc_ulp_ops __read_mostly = {
- .name = "smc",
- .owner = THIS_MODULE,
- .init = smc_ulp_init,
- .clone = smc_ulp_clone,
-};
-
unsigned int smc_net_id;
static __net_init int smc_net_init(struct net *net)
@@ -3507,17 +3452,9 @@ static int __init smc_init(void)
goto out_sock;
}
- rc = tcp_register_ulp(&smc_ulp_ops);
- if (rc) {
- pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc);
- goto out_ib;
- }
-
static_branch_enable(&tcp_have_smc);
return 0;
-out_ib:
- smc_ib_unregister_client();
out_sock:
sock_unregister(PF_SMC);
out_proto6:
@@ -3547,7 +3484,6 @@ static int __init smc_init(void)
static void __exit smc_exit(void)
{
static_branch_disable(&tcp_have_smc);
- tcp_unregister_ulp(&smc_ulp_ops);
sock_unregister(PF_SMC);
smc_core_exit();
smc_ib_unregister_client();
@@ -3571,5 +3507,4 @@ MODULE_AUTHOR("Ursula Braun <ubraun@linux.vnet.ibm.com>");
MODULE_DESCRIPTION("smc socket address family");
MODULE_LICENSE("GPL");
MODULE_ALIAS_NETPROTO(PF_SMC);
-MODULE_ALIAS_TCP_ULP("smc");
MODULE_ALIAS_GENL_FAMILY(SMC_GENL_FAMILY_NAME);
diff --git a/net/smc/smc.h b/net/smc/smc.h
index f480b956c45e..8ad6ff655f91 100644
--- a/net/smc/smc.h
+++ b/net/smc/smc.h
@@ -233,9 +233,9 @@ struct smc_connection {
* 0 for SMC-R, 32 for SMC-D
*/
u64 peer_token; /* SMC-D token of peer */
- u8 killed : 1; /* abnormal termination */
- u8 freed : 1; /* normal termiation */
- u8 out_of_sync : 1; /* out of sync with peer */
+ u8 killed; /* abnormal termination */
+ u8 freed; /* normal termiation */
+ u8 out_of_sync; /* out of sync with peer */
};
struct smc_sock { /* smc sock container */
diff --git a/net/strparser/strparser.c b/net/strparser/strparser.c
index 2a805c964210..be422d077ec7 100644
--- a/net/strparser/strparser.c
+++ b/net/strparser/strparser.c
@@ -518,12 +518,16 @@ void strp_unpause(struct strparser *strp)
EXPORT_SYMBOL_GPL(strp_unpause);
/* strp must already be stopped so that strp_recv will no longer be called.
- * Note that strp_done is not called with the lower socket held.
+ * Note that strp_done must not be called with the lower socket held.
*/
void strp_done(struct strparser *strp)
{
WARN_ON(!strp->stopped);
+ lock_sock(strp->sk);
+ /* sync with pending strp_recv */
+ release_sock(strp->sk);
+
cancel_delayed_work_sync(&strp->msg_timer_work);
cancel_work_sync(&strp->work);
diff --git a/net/sunrpc/auth_gss/gss_krb5_mech.c b/net/sunrpc/auth_gss/gss_krb5_mech.c
index 1c092b05c2bb..8fc6f25468ac 100644
--- a/net/sunrpc/auth_gss/gss_krb5_mech.c
+++ b/net/sunrpc/auth_gss/gss_krb5_mech.c
@@ -471,6 +471,7 @@ gss_import_v2_context(const void *p, const void *end, struct krb5_ctx *ctx,
u64 seq_send64;
int keylen;
u32 time32;
+ int ret;
p = simple_get_bytes(p, end, &ctx->flags, sizeof(ctx->flags));
if (IS_ERR(p))
@@ -528,14 +529,24 @@ gss_import_v2_context(const void *p, const void *end, struct krb5_ctx *ctx,
switch (ctx->enctype) {
case ENCTYPE_DES3_CBC_RAW:
- return context_derive_keys_des3(ctx, gfp_mask);
+ ret = context_derive_keys_des3(ctx, gfp_mask);
+ break;
case ENCTYPE_AES128_CTS_HMAC_SHA1_96:
case ENCTYPE_AES256_CTS_HMAC_SHA1_96:
- return context_derive_keys_new(ctx, gfp_mask);
+ ret = context_derive_keys_new(ctx, gfp_mask);
+ break;
default:
- return -EINVAL;
+ ret = -EINVAL;
+ }
+ if (ret) {
+ p = ERR_PTR(ret);
+ goto out_free;
}
+ return 0;
+
+out_free:
+ kfree(ctx->mech_used.data);
out_err:
return PTR_ERR(p);
}
diff --git a/net/sunrpc/auth_gss/gss_krb5_unseal.c b/net/sunrpc/auth_gss/gss_krb5_unseal.c
index ba04e3ec970a..b7adf2fc7cf5 100644
--- a/net/sunrpc/auth_gss/gss_krb5_unseal.c
+++ b/net/sunrpc/auth_gss/gss_krb5_unseal.c
@@ -159,6 +159,9 @@ gss_verify_mic_v2(struct krb5_ctx *ctx,
dprintk("RPC: %s\n", __func__);
+ if (read_token->len < GSS_KRB5_TOK_HDR_LEN + ctx->gk5e->cksumlength)
+ return GSS_S_DEFECTIVE_TOKEN;
+
memcpy(&be16_ptr, (char *) ptr, 2);
if (be16_to_cpu(be16_ptr) != KG2_TOK_MIC)
return GSS_S_DEFECTIVE_TOKEN;
diff --git a/net/sunrpc/auth_gss/gss_rpc_xdr.c b/net/sunrpc/auth_gss/gss_rpc_xdr.c
index ee91f4d641e6..967943ae61c6 100644
--- a/net/sunrpc/auth_gss/gss_rpc_xdr.c
+++ b/net/sunrpc/auth_gss/gss_rpc_xdr.c
@@ -222,7 +222,8 @@ static int gssx_dec_linux_creds(struct xdr_stream *xdr,
return 0;
out_free_groups:
- groups_free(creds->cr_group_info);
+ put_group_info(creds->cr_group_info);
+ creds->cr_group_info = NULL;
return err;
}
@@ -230,6 +231,7 @@ static int gssx_dec_option_array(struct xdr_stream *xdr,
struct gssx_option_array *oa)
{
struct svc_cred *creds;
+ bool creds_decoded = false;
u32 count, i;
__be32 *p;
int err;
@@ -242,12 +244,12 @@ static int gssx_dec_option_array(struct xdr_stream *xdr,
return 0;
/* we recognize only 1 currently: CREDS_VALUE */
- oa->count = 1;
-
oa->data = kmalloc(sizeof(struct gssx_option), GFP_KERNEL);
if (!oa->data)
return -ENOMEM;
+ oa->count = 1;
+
creds = kzalloc(sizeof(struct svc_cred), GFP_KERNEL);
if (!creds) {
err = -ENOMEM;
@@ -280,9 +282,14 @@ static int gssx_dec_option_array(struct xdr_stream *xdr,
if (length == sizeof(CREDS_VALUE) &&
memcmp(p, CREDS_VALUE, sizeof(CREDS_VALUE)) == 0) {
/* We have creds here. parse them */
+ if (creds_decoded) {
+ err = -EINVAL;
+ goto free_creds;
+ }
err = gssx_dec_linux_creds(xdr, creds);
if (err)
goto free_creds;
+ creds_decoded = true;
oa->data[0].value.len = 1; /* presence */
} else {
/* consume uninteresting buffer */
@@ -294,8 +301,10 @@ static int gssx_dec_option_array(struct xdr_stream *xdr,
return 0;
free_creds:
+ free_svc_cred(creds);
kfree(creds);
free_oa:
+ oa->count = 0;
kfree(oa->data);
oa->data = NULL;
return err;
diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/svcauth_gss.c
index 36549f6d5e01..5567c1f426a7 100644
--- a/net/sunrpc/auth_gss/svcauth_gss.c
+++ b/net/sunrpc/auth_gss/svcauth_gss.c
@@ -49,11 +49,36 @@
#include <linux/sunrpc/svcauth.h>
#include <linux/sunrpc/svcauth_gss.h>
#include <linux/sunrpc/cache.h>
+#include <linux/sunrpc/gss_krb5.h>
#include <trace/events/rpcgss.h>
#include "gss_rpc_upcall.h"
+/*
+ * Unfortunately there isn't a maximum checksum size exported via the
+ * GSS API. Manufacture one based on GSS mechanisms supported by this
+ * implementation.
+ */
+#define GSS_MAX_CKSUMSIZE (GSS_KRB5_TOK_HDR_LEN + GSS_KRB5_MAX_CKSUM_LEN)
+
+/*
+ * This value may be increased in the future to accommodate other
+ * usage of the scratch buffer.
+ */
+#define GSS_SCRATCH_SIZE GSS_MAX_CKSUMSIZE
+
+struct gss_svc_data {
+ /* decoded gss client cred: */
+ struct rpc_gss_wire_cred clcred;
+ /* save a pointer to the beginning of the encoded verifier,
+ * for use in encryption/checksumming in svcauth_gss_release: */
+ __be32 *verf_start;
+ struct rsc *rsci;
+
+ /* for temporary results */
+ u8 gsd_scratch[GSS_SCRATCH_SIZE];
+};
/* The rpcsec_init cache is used for mapping RPCSEC_GSS_{,CONT_}INIT requests
* into replies.
@@ -887,13 +912,11 @@ read_u32_from_xdr_buf(struct xdr_buf *buf, int base, u32 *obj)
static int
unwrap_integ_data(struct svc_rqst *rqstp, struct xdr_buf *buf, u32 seq, struct gss_ctx *ctx)
{
+ struct gss_svc_data *gsd = rqstp->rq_auth_data;
u32 integ_len, rseqno, maj_stat;
- int stat = -EINVAL;
struct xdr_netobj mic;
struct xdr_buf integ_buf;
- mic.data = NULL;
-
/* NFS READ normally uses splice to send data in-place. However
* the data in cache can change after the reply's MIC is computed
* but before the RPC reply is sent. To prevent the client from
@@ -917,11 +940,9 @@ unwrap_integ_data(struct svc_rqst *rqstp, struct xdr_buf *buf, u32 seq, struct g
/* copy out mic... */
if (read_u32_from_xdr_buf(buf, integ_len, &mic.len))
goto unwrap_failed;
- if (mic.len > RPC_MAX_AUTH_SIZE)
- goto unwrap_failed;
- mic.data = kmalloc(mic.len, GFP_KERNEL);
- if (!mic.data)
+ if (mic.len > sizeof(gsd->gsd_scratch))
goto unwrap_failed;
+ mic.data = gsd->gsd_scratch;
if (read_bytes_from_xdr_buf(buf, integ_len + 4, mic.data, mic.len))
goto unwrap_failed;
maj_stat = gss_verify_mic(ctx, &integ_buf, &mic);
@@ -932,20 +953,17 @@ unwrap_integ_data(struct svc_rqst *rqstp, struct xdr_buf *buf, u32 seq, struct g
goto bad_seqno;
/* trim off the mic and padding at the end before returning */
xdr_buf_trim(buf, round_up_to_quad(mic.len) + 4);
- stat = 0;
-out:
- kfree(mic.data);
- return stat;
+ return 0;
unwrap_failed:
trace_rpcgss_svc_unwrap_failed(rqstp);
- goto out;
+ return -EINVAL;
bad_seqno:
trace_rpcgss_svc_seqno_bad(rqstp, seq, rseqno);
- goto out;
+ return -EINVAL;
bad_mic:
trace_rpcgss_svc_mic(rqstp, maj_stat);
- goto out;
+ return -EINVAL;
}
static inline int
@@ -954,61 +972,52 @@ total_buf_len(struct xdr_buf *buf)
return buf->head[0].iov_len + buf->page_len + buf->tail[0].iov_len;
}
-static void
-fix_priv_head(struct xdr_buf *buf, int pad)
-{
- if (buf->page_len == 0) {
- /* We need to adjust head and buf->len in tandem in this
- * case to make svc_defer() work--it finds the original
- * buffer start using buf->len - buf->head[0].iov_len. */
- buf->head[0].iov_len -= pad;
- }
-}
-
-static int
-unwrap_priv_data(struct svc_rqst *rqstp, struct xdr_buf *buf, u32 seq, struct gss_ctx *ctx)
+/*
+ * RFC 2203, Section 5.3.2.3
+ *
+ * struct rpc_gss_priv_data {
+ * opaque databody_priv<>
+ * };
+ *
+ * struct rpc_gss_data_t {
+ * unsigned int seq_num;
+ * proc_req_arg_t arg;
+ * };
+ */
+static noinline_for_stack int
+svcauth_gss_unwrap_priv(struct svc_rqst *rqstp, u32 seq, struct gss_ctx *ctx)
{
- u32 priv_len, maj_stat;
- int pad, remaining_len, offset;
- u32 rseqno;
+ struct xdr_stream *xdr = &rqstp->rq_arg_stream;
+ u32 len, maj_stat, seq_num, offset;
+ struct xdr_buf *buf = xdr->buf;
+ unsigned int saved_len;
clear_bit(RQ_SPLICE_OK, &rqstp->rq_flags);
- priv_len = svc_getnl(&buf->head[0]);
+ if (xdr_stream_decode_u32(xdr, &len) < 0)
+ goto unwrap_failed;
if (rqstp->rq_deferred) {
/* Already decrypted last time through! The sequence number
* check at out_seq is unnecessary but harmless: */
goto out_seq;
}
- /* buf->len is the number of bytes from the original start of the
- * request to the end, where head[0].iov_len is just the bytes
- * not yet read from the head, so these two values are different: */
- remaining_len = total_buf_len(buf);
- if (priv_len > remaining_len)
+ if (len > xdr_stream_remaining(xdr))
goto unwrap_failed;
- pad = remaining_len - priv_len;
- buf->len -= pad;
- fix_priv_head(buf, pad);
-
- maj_stat = gss_unwrap(ctx, 0, priv_len, buf);
- pad = priv_len - buf->len;
- /* The upper layers assume the buffer is aligned on 4-byte boundaries.
- * In the krb5p case, at least, the data ends up offset, so we need to
- * move it around. */
- /* XXX: This is very inefficient. It would be better to either do
- * this while we encrypt, or maybe in the receive code, if we can peak
- * ahead and work out the service and mechanism there. */
- offset = xdr_pad_size(buf->head[0].iov_len);
- if (offset) {
- buf->buflen = RPCSVC_MAXPAYLOAD;
- xdr_shift_buf(buf, offset);
- fix_priv_head(buf, pad);
- }
+ if (len <= GSS_KRB5_TOK_HDR_LEN)
+ goto unwrap_failed;
+ offset = xdr_stream_pos(xdr);
+
+ saved_len = buf->len;
+ maj_stat = gss_unwrap(ctx, offset, offset + len, buf);
if (maj_stat != GSS_S_COMPLETE)
goto bad_unwrap;
+ xdr->nwords -= XDR_QUADLEN(saved_len - buf->len);
+
out_seq:
- rseqno = svc_getnl(&buf->head[0]);
- if (rseqno != seq)
+ /* gss_unwrap() decrypted the sequence number. */
+ if (xdr_stream_decode_u32(xdr, &seq_num) < 0)
+ goto unwrap_failed;
+ if (seq_num != seq)
goto bad_seqno;
return 0;
@@ -1016,22 +1025,13 @@ unwrap_priv_data(struct svc_rqst *rqstp, struct xdr_buf *buf, u32 seq, struct gs
trace_rpcgss_svc_unwrap_failed(rqstp);
return -EINVAL;
bad_seqno:
- trace_rpcgss_svc_seqno_bad(rqstp, seq, rseqno);
+ trace_rpcgss_svc_seqno_bad(rqstp, seq, seq_num);
return -EINVAL;
bad_unwrap:
trace_rpcgss_svc_unwrap(rqstp, maj_stat);
return -EINVAL;
}
-struct gss_svc_data {
- /* decoded gss client cred: */
- struct rpc_gss_wire_cred clcred;
- /* save a pointer to the beginning of the encoded verifier,
- * for use in encryption/checksumming in svcauth_gss_release: */
- __be32 *verf_start;
- struct rsc *rsci;
-};
-
static int
svcauth_gss_set_client(struct svc_rqst *rqstp)
{
@@ -1643,8 +1643,9 @@ svcauth_gss_accept(struct svc_rqst *rqstp)
/* placeholders for length and seq. number: */
svc_putnl(resv, 0);
svc_putnl(resv, 0);
- if (unwrap_priv_data(rqstp, &rqstp->rq_arg,
- gc->gc_seq, rsci->mechctx))
+ svcxdr_init_decode(rqstp);
+ if (svcauth_gss_unwrap_priv(rqstp, gc->gc_seq,
+ rsci->mechctx))
goto garbage_args;
rqstp->rq_auth_slack = RPC_MAX_AUTH_SIZE * 2;
break;
diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c
index a390a4e5592f..212e1a470596 100644
--- a/net/sunrpc/clnt.c
+++ b/net/sunrpc/clnt.c
@@ -101,7 +101,10 @@ static void rpc_unregister_client(struct rpc_clnt *clnt)
static void __rpc_clnt_remove_pipedir(struct rpc_clnt *clnt)
{
- rpc_remove_client_dir(clnt);
+ if (clnt->pipefs_sb) {
+ rpc_remove_client_dir(clnt);
+ clnt->pipefs_sb = NULL;
+ }
}
static void rpc_clnt_remove_pipedir(struct rpc_clnt *clnt)
@@ -186,23 +189,30 @@ static int __rpc_clnt_handle_event(struct rpc_clnt *clnt, unsigned long event,
struct super_block *sb)
{
struct dentry *dentry;
+ int err = 0;
switch (event) {
case RPC_PIPEFS_MOUNT:
+ clnt->pipefs_sb = sb;
dentry = rpc_setup_pipedir_sb(sb, clnt);
if (!dentry)
- return -ENOENT;
- if (IS_ERR(dentry))
- return PTR_ERR(dentry);
+ err = -ENOENT;
+ else if (IS_ERR(dentry))
+ err = PTR_ERR(dentry);
+ if (err)
+ clnt->pipefs_sb = NULL;
break;
case RPC_PIPEFS_UMOUNT:
- __rpc_clnt_remove_pipedir(clnt);
+ if (clnt->pipefs_sb == sb) {
+ __rpc_clnt_remove_pipedir(clnt);
+ clnt->pipefs_sb = NULL;
+ }
break;
default:
printk(KERN_ERR "%s: unknown event: %ld\n", __func__, event);
return -ENOTSUPP;
}
- return 0;
+ return err;
}
static int __rpc_pipefs_event(struct rpc_clnt *clnt, unsigned long event,
diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c
index 112dc001459d..ec977c5c4332 100644
--- a/net/sunrpc/svc.c
+++ b/net/sunrpc/svc.c
@@ -874,6 +874,15 @@ void svc_rqst_replace_page(struct svc_rqst *rqstp, struct page *page)
}
EXPORT_SYMBOL_GPL(svc_rqst_replace_page);
+static void svc_rqst_free_rcu(struct rcu_head *head)
+{
+ struct svc_rqst *rqstp = container_of(head, struct svc_rqst, rq_rcu_head);
+
+ kfree(rqstp->rq_resp);
+ kfree(rqstp->rq_argp);
+ kfree(rqstp);
+}
+
/*
* Called from a server thread as it's exiting. Caller must hold the "service
* mutex" for the service.
@@ -884,10 +893,8 @@ svc_rqst_free(struct svc_rqst *rqstp)
svc_release_buffer(rqstp);
if (rqstp->rq_scratch_page)
put_page(rqstp->rq_scratch_page);
- kfree(rqstp->rq_resp);
- kfree(rqstp->rq_argp);
kfree(rqstp->rq_auth_data);
- kfree_rcu(rqstp, rq_rcu_head);
+ call_rcu(&rqstp->rq_rcu_head, svc_rqst_free_rcu);
}
EXPORT_SYMBOL_GPL(svc_rqst_free);
diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c
index 23b4c728de59..69d40d0980cf 100644
--- a/net/sunrpc/svcsock.c
+++ b/net/sunrpc/svcsock.c
@@ -1457,6 +1457,9 @@ static struct svc_xprt *svc_create_socket(struct svc_serv *serv,
newlen = error;
if (protocol == IPPROTO_TCP) {
+ sock->sk->sk_net_refcnt = 1;
+ get_net_track(net, &sock->sk->ns_tracker, GFP_KERNEL);
+ sock_inuse_add(net, 1);
if ((error = kernel_listen(sock, 64)) < 0)
goto bummer;
}
diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c
index 612b1a5572eb..7c4e966a6aad 100644
--- a/net/sunrpc/xdr.c
+++ b/net/sunrpc/xdr.c
@@ -863,13 +863,6 @@ static unsigned int xdr_shrink_pagelen(struct xdr_buf *buf, unsigned int len)
return shift;
}
-void
-xdr_shift_buf(struct xdr_buf *buf, size_t len)
-{
- xdr_shrink_bufhead(buf, buf->head->iov_len - len);
-}
-EXPORT_SYMBOL_GPL(xdr_shift_buf);
-
/**
* xdr_stream_pos - Return the current offset from the start of the xdr_stream
* @xdr: pointer to struct xdr_stream
diff --git a/net/sunrpc/xprtrdma/svc_rdma_pcl.c b/net/sunrpc/xprtrdma/svc_rdma_pcl.c
index b63cfeaa2923..a1c61311714d 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_pcl.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_pcl.c
@@ -213,7 +213,6 @@ bool pcl_alloc_write(struct svc_rdma_recv_ctxt *rctxt,
chunk = pcl_alloc_chunk(segcount, 0);
if (!chunk)
return false;
- list_add_tail(&chunk->ch_list, &pcl->cl_chunks);
for (j = 0; j < segcount; j++) {
segment = &chunk->ch_segments[j];
@@ -225,6 +224,7 @@ bool pcl_alloc_write(struct svc_rdma_recv_ctxt *rctxt,
chunk->ch_length += segment->rs_length;
chunk->ch_segcount++;
}
+ list_add_tail(&chunk->ch_list, &pcl->cl_chunks);
}
return true;
}
diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
index 186c9c12432b..06c64aeed5fd 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c
@@ -415,11 +415,14 @@ static void svc_rdma_build_arg_xdr(struct svc_rqst *rqstp,
* to the first byte past the Read list. rc_read_pcl and
* rc_call_pcl cl_count fields are set to the number of
* Read segments in the list.
- * %false: Read list is corrupt. @rctxt's xdr_stream is left in an
- * unknown state.
+ * %false: Read list is corrupt or exceeds the page budget. @rctxt's
+ * xdr_stream is left in an unknown state.
*/
static bool xdr_count_read_segments(struct svc_rdma_recv_ctxt *rctxt, __be32 *p)
{
+ unsigned int maxlen = rctxt->rc_maxpages << PAGE_SHIFT;
+ unsigned int total_len = 0;
+
rctxt->rc_call_pcl.cl_count = 0;
rctxt->rc_read_pcl.cl_count = 0;
while (xdr_item_is_present(p)) {
@@ -433,6 +436,11 @@ static bool xdr_count_read_segments(struct svc_rdma_recv_ctxt *rctxt, __be32 *p)
xdr_decode_read_segment(p, &position, &handle,
&length, &offset);
+ if (length > maxlen)
+ return false;
+ total_len += length;
+ if (PAGE_ALIGN(total_len) > maxlen)
+ return false;
if (position) {
if (position & 3)
return false;
@@ -483,10 +491,13 @@ static bool xdr_check_write_chunk(struct svc_rdma_recv_ctxt *rctxt)
return false;
/* Before trusting the segcount value enough to use it in
- * a computation, perform a simple range check. This is an
- * arbitrary but sensible limit (ie, not architectural).
+ * a computation, perform a simple range check. A zero
+ * segcount describes no remote buffer and is rejected so
+ * downstream consumers never see a degenerate ch_segcount==0
+ * chunk. The upper bound is an arbitrary but sensible limit
+ * (ie, not architectural).
*/
- if (unlikely(segcount > RPCSVC_MAXPAGES))
+ if (segcount == 0 || unlikely(segcount > RPCSVC_MAXPAGES))
return false;
p = xdr_inline_decode(&rctxt->rc_stream,
@@ -830,6 +841,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp)
ctxt->rc_recv_sge.addr, ctxt->rc_byte_len,
DMA_FROM_DEVICE);
svc_rdma_build_arg_xdr(rqstp, ctxt);
+ ctxt->rc_maxpages = rqstp->rq_page_end - rqstp->rq_pages;
ret = svc_rdma_xdr_decode_req(&rqstp->rq_arg, ctxt);
if (ret < 0)
diff --git a/net/sunrpc/xprtrdma/svc_rdma_rw.c b/net/sunrpc/xprtrdma/svc_rdma_rw.c
index 1bc9dfa1e5bb..15140246f03e 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_rw.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_rw.c
@@ -786,7 +786,7 @@ static int svc_rdma_build_read_chunk(struct svc_rdma_read_info *info,
/**
* svc_rdma_copy_inline_range - Copy part of the inline content into pages
* @info: context for RDMA Reads
- * @offset: offset into the Receive buffer of region to copy
+ * @offset: offset into the inline content of region to copy
* @remaining: length of region to copy
*
* Take a page at a time from rqstp->rq_pages and copy the inline
@@ -803,14 +803,21 @@ static int svc_rdma_copy_inline_range(struct svc_rdma_read_info *info,
unsigned int remaining)
{
struct svc_rdma_recv_ctxt *head = info->ri_readctxt;
- unsigned char *dst, *src = head->rc_recv_buf;
struct svc_rqst *rqstp = info->ri_rqst;
+ unsigned char *dst, *src = rqstp->rq_arg.head[0].iov_base;
+ unsigned int inline_len = rqstp->rq_arg.head[0].iov_len;
unsigned int page_no, numpages;
+ if (offset > inline_len || remaining > inline_len - offset)
+ return -EINVAL;
+
numpages = PAGE_ALIGN(info->ri_pageoff + remaining) >> PAGE_SHIFT;
for (page_no = 0; page_no < numpages; page_no++) {
unsigned int page_len;
+ if (&rqstp->rq_pages[info->ri_pageno] >= rqstp->rq_page_end)
+ return -EINVAL;
+
page_len = min_t(unsigned int, remaining,
PAGE_SIZE - info->ri_pageoff);
@@ -853,9 +860,10 @@ static noinline int svc_rdma_read_multiple_chunks(struct svc_rdma_read_info *inf
const struct svc_rdma_pcl *pcl = &head->rc_read_pcl;
struct xdr_buf *buf = &info->ri_rqst->rq_arg;
struct svc_rdma_chunk *chunk, *next;
- unsigned int start, length;
+ unsigned int inline_len, start, length;
int ret;
+ inline_len = buf->head[0].iov_len;
start = 0;
chunk = pcl_first_chunk(pcl);
length = chunk->ch_position;
@@ -873,6 +881,8 @@ static noinline int svc_rdma_read_multiple_chunks(struct svc_rdma_read_info *inf
break;
start += length;
+ if (info->ri_totalbytes > next->ch_position)
+ return -EINVAL;
length = next->ch_position - info->ri_totalbytes;
ret = svc_rdma_copy_inline_range(info, start, length);
if (ret < 0)
@@ -880,7 +890,9 @@ static noinline int svc_rdma_read_multiple_chunks(struct svc_rdma_read_info *inf
}
start += length;
- length = head->rc_byte_len - start;
+ if (start > inline_len)
+ return -EINVAL;
+ length = inline_len - start;
ret = svc_rdma_copy_inline_range(info, start, length);
if (ret < 0)
return ret;
@@ -921,6 +933,9 @@ static int svc_rdma_read_data_item(struct svc_rdma_read_info *info)
int ret;
chunk = pcl_first_chunk(&head->rc_read_pcl);
+ if (chunk->ch_position > buf->head[0].iov_len)
+ return -EINVAL;
+
ret = svc_rdma_build_read_chunk(info, chunk);
if (ret < 0)
goto out;
@@ -1021,14 +1036,17 @@ static int svc_rdma_read_call_chunk(struct svc_rdma_read_info *info)
pcl_first_chunk(&head->rc_call_pcl);
const struct svc_rdma_pcl *pcl = &head->rc_read_pcl;
struct svc_rdma_chunk *chunk, *next;
- unsigned int start, length;
+ unsigned int call_len, start, length;
int ret;
if (pcl_is_empty(pcl))
return svc_rdma_build_read_chunk(info, call_chunk);
+ call_len = call_chunk->ch_length;
start = 0;
chunk = pcl_first_chunk(pcl);
+ if (chunk->ch_position > call_len)
+ return -EINVAL;
length = chunk->ch_position;
ret = svc_rdma_read_chunk_range(info, call_chunk, start, length);
if (ret < 0)
@@ -1044,7 +1062,11 @@ static int svc_rdma_read_call_chunk(struct svc_rdma_read_info *info)
break;
start += length;
+ if (info->ri_totalbytes > next->ch_position)
+ return -EINVAL;
length = next->ch_position - info->ri_totalbytes;
+ if (length > call_len - start)
+ return -EINVAL;
ret = svc_rdma_read_chunk_range(info, call_chunk,
start, length);
if (ret < 0)
@@ -1052,7 +1074,9 @@ static int svc_rdma_read_call_chunk(struct svc_rdma_read_info *info)
}
start += length;
- length = call_chunk->ch_length - start;
+ if (start > call_len)
+ return -EINVAL;
+ length = call_len - start;
return svc_rdma_read_chunk_range(info, call_chunk, start, length);
}
diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
index 22a871e6fe4d..fd6ed6901c7c 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c
@@ -645,20 +645,21 @@ static int svc_rdma_xb_count_sges(const struct xdr_buf *xdr,
}
/**
- * svc_rdma_pull_up_needed - Determine whether to use pull-up
+ * svc_rdma_check_pull_up - Determine whether to use pull-up
* @rdma: controlling transport
* @sctxt: send_ctxt for the Send WR
* @rctxt: Write and Reply chunks provided by client
* @xdr: xdr_buf containing RPC message to transmit
*
* Returns:
- * %true if pull-up must be used
- * %false otherwise
+ * %1 if pull-up must be used
+ * %0 if pull-up is not needed
+ * %-E2BIG if the reply is too large to be pulled up
*/
-static bool svc_rdma_pull_up_needed(const struct svcxprt_rdma *rdma,
- const struct svc_rdma_send_ctxt *sctxt,
- const struct svc_rdma_recv_ctxt *rctxt,
- const struct xdr_buf *xdr)
+static int svc_rdma_check_pull_up(const struct svcxprt_rdma *rdma,
+ const struct svc_rdma_send_ctxt *sctxt,
+ const struct svc_rdma_recv_ctxt *rctxt,
+ const struct xdr_buf *xdr)
{
/* Resources needed for the transport header */
struct svc_rdma_pullup_data args = {
@@ -670,11 +671,22 @@ static bool svc_rdma_pull_up_needed(const struct svcxprt_rdma *rdma,
ret = pcl_process_nonpayloads(&rctxt->rc_write_pcl, xdr,
svc_rdma_xb_count_sges, &args);
if (ret < 0)
- return false;
+ return 0;
if (args.pd_length < RPCRDMA_PULLUP_THRESH)
- return true;
- return args.pd_num_sges >= rdma->sc_max_send_sges;
+ return 1;
+ if (args.pd_num_sges < rdma->sc_max_send_sges)
+ return 0;
+
+ /*
+ * The reply has too many SGEs to Send inline, so it has to be
+ * linearized into sc_xprt_buf. That buffer holds only
+ * sc_max_req_size bytes, so a larger reply cannot be pulled up.
+ * RFC 8166 Section 4.5.3 requires responding with ERR_CHUNK.
+ */
+ if (args.pd_length > rdma->sc_max_req_size)
+ return -E2BIG;
+ return 1;
}
/**
@@ -730,7 +742,7 @@ static int svc_rdma_xb_linearize(const struct xdr_buf *xdr,
* Assemble the elements of @xdr into the transport header buffer.
*
* Assumptions:
- * pull_up_needed has determined that @xdr will fit in the buffer.
+ * check_pull_up has determined that @xdr will fit in the buffer.
*
* Returns:
* %0 if pull-up was successful
@@ -764,6 +776,7 @@ static int svc_rdma_pull_up_reply_msg(const struct svcxprt_rdma *rdma,
*
* Returns:
* %0 if DMA mapping was successful.
+ * %-E2BIG if the reply is too large to be pulled up
* %-EMSGSIZE if a buffer manipulation problem occurred
* %-EIO if DMA mapping failed
*
@@ -778,6 +791,7 @@ int svc_rdma_map_reply_msg(struct svcxprt_rdma *rdma,
.md_rdma = rdma,
.md_ctxt = sctxt,
};
+ int ret;
/* Set up the (persistently-mapped) transport header SGE. */
sctxt->sc_send_wr.num_sge = 1;
@@ -792,7 +806,10 @@ int svc_rdma_map_reply_msg(struct svcxprt_rdma *rdma,
/* For pull-up, svc_rdma_send() will sync the transport header.
* No additional DMA mapping is necessary.
*/
- if (svc_rdma_pull_up_needed(rdma, sctxt, rctxt, xdr))
+ ret = svc_rdma_check_pull_up(rdma, sctxt, rctxt, xdr);
+ if (ret < 0)
+ return ret;
+ if (ret)
return svc_rdma_pull_up_reply_msg(rdma, sctxt, rctxt, xdr);
return pcl_process_nonpayloads(&rctxt->rc_write_pcl, xdr,
@@ -955,7 +972,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
ret = svc_rdma_send_reply_chunk(rdma, rctxt, &rqstp->rq_res);
if (ret < 0)
- goto reply_chunk;
+ goto send_err;
rc_size = ret;
*p++ = *rdma_argp;
@@ -975,7 +992,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
ret = svc_rdma_send_reply_msg(rdma, sctxt, rctxt, rqstp);
if (ret < 0)
- goto put_ctxt;
+ goto send_err;
/* Prevent svc_xprt_release() from releasing the page backing
* rq_res.head[0].iov_base. It's no longer being accessed by
@@ -983,7 +1000,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp)
rqstp->rq_respages++;
return 0;
-reply_chunk:
+send_err:
if (ret != -E2BIG && ret != -EINVAL)
goto put_ctxt;
diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c
index 45b0fef0b5e2..c84e80ae374e 100644
--- a/net/sunrpc/xprtrdma/svc_rdma_transport.c
+++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c
@@ -64,7 +64,9 @@
#define RPCDBG_FACILITY RPCDBG_SVCXPRT
static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,
- struct net *net);
+ struct net *net, int node);
+static int svc_rdma_listen_handler(struct rdma_cm_id *cma_id,
+ struct rdma_cm_event *event);
static struct svc_xprt *svc_rdma_create(struct svc_serv *serv,
struct net *net,
struct sockaddr *sa, int salen,
@@ -124,15 +126,50 @@ static void qp_event_handler(struct ib_event *event, void *context)
}
}
+static struct rdma_cm_id *
+svc_rdma_create_listen_id(struct net *net, struct sockaddr *sap,
+ void *context)
+{
+ struct rdma_cm_id *listen_id;
+ int ret;
+
+ listen_id = rdma_create_id(net, svc_rdma_listen_handler, context,
+ RDMA_PS_TCP, IB_QPT_RC);
+ if (IS_ERR(listen_id))
+ return listen_id;
+
+ /* Allow both IPv4 and IPv6 sockets to bind a single port
+ * at the same time.
+ */
+#if IS_ENABLED(CONFIG_IPV6)
+ ret = rdma_set_afonly(listen_id, 1);
+ if (ret)
+ goto out_destroy;
+#endif
+ ret = rdma_bind_addr(listen_id, sap);
+ if (ret)
+ goto out_destroy;
+
+ ret = rdma_listen(listen_id, RPCRDMA_LISTEN_BACKLOG);
+ if (ret)
+ goto out_destroy;
+
+ return listen_id;
+
+out_destroy:
+ rdma_destroy_id(listen_id);
+ return ERR_PTR(ret);
+}
+
static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv,
- struct net *net)
+ struct net *net, int node)
{
- struct svcxprt_rdma *cma_xprt = kzalloc(sizeof *cma_xprt, GFP_KERNEL);
+ struct svcxprt_rdma *cma_xprt;
- if (!cma_xprt) {
- dprintk("svcrdma: failed to create new transport\n");
+ cma_xprt = kzalloc_node(sizeof(*cma_xprt), GFP_KERNEL, node);
+ if (!cma_xprt)
return NULL;
- }
+
svc_xprt_init(net, &svc_rdma_class, &cma_xprt->sc_xprt, serv);
INIT_LIST_HEAD(&cma_xprt->sc_accept_q);
INIT_LIST_HEAD(&cma_xprt->sc_rq_dto_q);
@@ -184,22 +221,26 @@ svc_rdma_parse_connect_private(struct svcxprt_rdma *newxprt,
* structure for the listening endpoint.
*
* This function creates a new xprt for the new connection and enqueues it on
- * the accept queue for the listent xprt. When the listen thread is kicked, it
+ * the accept queue for the listen xprt. When the listen thread is kicked, it
* will call the recvfrom method on the listen xprt which will accept the new
* connection.
+ *
+ * Return values:
+ * %0: Do not destroy @new_cma_id
+ * %1: Destroy @new_cma_id (allocation failure)
*/
-static void handle_connect_req(struct rdma_cm_id *new_cma_id,
- struct rdma_conn_param *param)
+static int handle_connect_req(struct rdma_cm_id *new_cma_id,
+ struct rdma_conn_param *param)
{
struct svcxprt_rdma *listen_xprt = new_cma_id->context;
struct svcxprt_rdma *newxprt;
struct sockaddr *sa;
- /* Create a new transport */
newxprt = svc_rdma_create_xprt(listen_xprt->sc_xprt.xpt_server,
- listen_xprt->sc_xprt.xpt_net);
+ listen_xprt->sc_xprt.xpt_net,
+ ibdev_to_node(new_cma_id->device));
if (!newxprt)
- return;
+ return 1;
newxprt->sc_cm_id = new_cma_id;
new_cma_id->context = newxprt;
svc_rdma_parse_connect_private(newxprt, param);
@@ -233,6 +274,7 @@ static void handle_connect_req(struct rdma_cm_id *new_cma_id,
set_bit(XPT_CONN, &listen_xprt->sc_xprt.xpt_flags);
svc_xprt_enqueue(&listen_xprt->sc_xprt);
+ return 0;
}
/**
@@ -242,17 +284,30 @@ static void handle_connect_req(struct rdma_cm_id *new_cma_id,
*
* Return values:
* %0: Do not destroy @cma_id
- * %1: Destroy @cma_id (never returned here)
+ * %1: Destroy @cma_id
*
* NB: There is never a DEVICE_REMOVAL event for INADDR_ANY listeners.
*/
static int svc_rdma_listen_handler(struct rdma_cm_id *cma_id,
struct rdma_cm_event *event)
{
+ struct sockaddr *sap = (struct sockaddr *)&cma_id->route.addr.src_addr;
+ struct svcxprt_rdma *cma_xprt = cma_id->context;
+ struct svc_xprt *cma_rdma = &cma_xprt->sc_xprt;
+ struct rdma_cm_id *listen_id;
+
switch (event->event) {
case RDMA_CM_EVENT_CONNECT_REQUEST:
- handle_connect_req(cma_id, &event->param.conn);
- break;
+ return handle_connect_req(cma_id, &event->param.conn);
+ case RDMA_CM_EVENT_ADDR_CHANGE:
+ listen_id = svc_rdma_create_listen_id(cma_rdma->xpt_net,
+ sap, cma_xprt);
+ if (IS_ERR(listen_id)) {
+ pr_err("Listener dead, address change failed for device %s\n",
+ cma_id->device->name);
+ } else
+ cma_xprt->sc_cm_id = listen_id;
+ return 1;
default:
break;
}
@@ -302,40 +357,28 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv,
{
struct rdma_cm_id *listen_id;
struct svcxprt_rdma *cma_xprt;
- int ret;
if (sa->sa_family != AF_INET && sa->sa_family != AF_INET6)
return ERR_PTR(-EAFNOSUPPORT);
- cma_xprt = svc_rdma_create_xprt(serv, net);
+ cma_xprt = svc_rdma_create_xprt(serv, net, NUMA_NO_NODE);
if (!cma_xprt)
return ERR_PTR(-ENOMEM);
set_bit(XPT_LISTENER, &cma_xprt->sc_xprt.xpt_flags);
strcpy(cma_xprt->sc_xprt.xpt_remotebuf, "listener");
- listen_id = rdma_create_id(net, svc_rdma_listen_handler, cma_xprt,
- RDMA_PS_TCP, IB_QPT_RC);
+ listen_id = svc_rdma_create_listen_id(net, sa, cma_xprt);
if (IS_ERR(listen_id)) {
- ret = PTR_ERR(listen_id);
- goto err0;
+ /* _svc_xprt_create() acquired one module reference and
+ * puts it on xpo_create failure. svc_xprt_free() puts
+ * a second one when the kref drops to zero. Take a
+ * compensating reference so both puts are balanced.
+ */
+ __module_get(cma_xprt->sc_xprt.xpt_class->xcl_owner);
+ svc_xprt_put(&cma_xprt->sc_xprt);
+ return ERR_CAST(listen_id);
}
-
- /* Allow both IPv4 and IPv6 sockets to bind a single port
- * at the same time.
- */
-#if IS_ENABLED(CONFIG_IPV6)
- ret = rdma_set_afonly(listen_id, 1);
- if (ret)
- goto err1;
-#endif
- ret = rdma_bind_addr(listen_id, sa);
- if (ret)
- goto err1;
cma_xprt->sc_cm_id = listen_id;
- ret = rdma_listen(listen_id, RPCRDMA_LISTEN_BACKLOG);
- if (ret)
- goto err1;
-
/*
* We need to use the address from the cm_id in case the
* caller specified 0 for the port number.
@@ -344,12 +387,6 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv,
svc_xprt_set_local(&cma_xprt->sc_xprt, sa, salen);
return &cma_xprt->sc_xprt;
-
- err1:
- rdma_destroy_id(listen_id);
- err0:
- kfree(cma_xprt);
- return ERR_PTR(ret);
}
/*
diff --git a/net/sunrpc/xprtsock.c b/net/sunrpc/xprtsock.c
index b9dc8e197dde..e812396e869b 100644
--- a/net/sunrpc/xprtsock.c
+++ b/net/sunrpc/xprtsock.c
@@ -1115,9 +1115,9 @@ static void xs_save_old_callbacks(struct sock_xprt *transport, struct sock *sk)
static void xs_restore_old_callbacks(struct sock_xprt *transport, struct sock *sk)
{
- sk->sk_data_ready = transport->old_data_ready;
+ WRITE_ONCE(sk->sk_data_ready, transport->old_data_ready);
sk->sk_state_change = transport->old_state_change;
- sk->sk_write_space = transport->old_write_space;
+ WRITE_ONCE(sk->sk_write_space, transport->old_write_space);
sk->sk_error_report = transport->old_error_report;
}
@@ -1563,6 +1563,7 @@ static void xs_udp_do_set_buffer_size(struct rpc_xprt *xprt)
{
struct sock_xprt *transport = container_of(xprt, struct sock_xprt, xprt);
struct sock *sk = transport->inet;
+ void (*write_space)(struct sock *sock);
if (transport->rcvsize) {
sk->sk_userlocks |= SOCK_RCVBUF_LOCK;
@@ -1571,7 +1572,8 @@ static void xs_udp_do_set_buffer_size(struct rpc_xprt *xprt)
if (transport->sndsize) {
sk->sk_userlocks |= SOCK_SNDBUF_LOCK;
sk->sk_sndbuf = transport->sndsize * xprt->max_reqs * 2;
- sk->sk_write_space(sk);
+ write_space = READ_ONCE(sk->sk_write_space);
+ write_space(sk);
}
}
@@ -1855,6 +1857,12 @@ static struct socket *xs_create_sock(struct rpc_xprt *xprt,
goto out;
}
+ if (protocol == IPPROTO_TCP) {
+ sock->sk->sk_net_refcnt = 1;
+ get_net_track(xprt->xprt_net, &sock->sk->ns_tracker, GFP_KERNEL);
+ sock_inuse_add(xprt->xprt_net, 1);
+ }
+
filp = sock_alloc_file(sock, O_NONBLOCK, NULL);
if (IS_ERR(filp))
return ERR_CAST(filp);
@@ -1879,10 +1887,11 @@ static int xs_local_finish_connecting(struct rpc_xprt *xprt,
xs_save_old_callbacks(transport, sk);
sk->sk_user_data = xprt;
- sk->sk_data_ready = xs_data_ready;
- sk->sk_write_space = xs_udp_write_space;
+ WRITE_ONCE(sk->sk_data_ready, xs_data_ready);
+ WRITE_ONCE(sk->sk_write_space, xs_udp_write_space);
sk->sk_state_change = xs_local_state_change;
sk->sk_error_report = xs_error_report;
+ sk->sk_use_task_frag = false;
xprt_clear_connected(xprt);
@@ -2081,8 +2090,9 @@ static void xs_udp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
xs_save_old_callbacks(transport, sk);
sk->sk_user_data = xprt;
- sk->sk_data_ready = xs_data_ready;
- sk->sk_write_space = xs_udp_write_space;
+ WRITE_ONCE(sk->sk_data_ready, xs_data_ready);
+ WRITE_ONCE(sk->sk_write_space, xs_udp_write_space);
+ sk->sk_use_task_frag = false;
xprt_set_connected(xprt);
@@ -2247,10 +2257,11 @@ static int xs_tcp_finish_connecting(struct rpc_xprt *xprt, struct socket *sock)
xs_save_old_callbacks(transport, sk);
sk->sk_user_data = xprt;
- sk->sk_data_ready = xs_data_ready;
+ WRITE_ONCE(sk->sk_data_ready, xs_data_ready);
sk->sk_state_change = xs_tcp_state_change;
- sk->sk_write_space = xs_tcp_write_space;
+ WRITE_ONCE(sk->sk_write_space, xs_tcp_write_space);
sk->sk_error_report = xs_error_report;
+ sk->sk_use_task_frag = false;
/* socket options */
sock_reset_flag(sk, SOCK_LINGER);
diff --git a/net/tipc/core.c b/net/tipc/core.c
index 1ddecea1df6e..315975c3be81 100644
--- a/net/tipc/core.c
+++ b/net/tipc/core.c
@@ -45,6 +45,7 @@
#include "crypto.h"
#include <linux/module.h>
+#include <linux/wait_bit.h>
/* configurable TIPC parameters */
unsigned int tipc_net_id __read_mostly;
@@ -118,8 +119,7 @@ static void __net_exit tipc_exit_net(struct net *net)
#ifdef CONFIG_TIPC_CRYPTO
tipc_crypto_stop(&tipc_net(net)->crypto_tx);
#endif
- while (atomic_read(&tn->wq_count))
- cond_resched();
+ wait_var_event(&tn->wq_count, atomic_read(&tn->wq_count) == 0);
}
static void __net_exit tipc_pernet_pre_exit(struct net *net)
diff --git a/net/tipc/crypto.c b/net/tipc/crypto.c
index d86b3a8f7078..dc2239b2cf85 100644
--- a/net/tipc/crypto.c
+++ b/net/tipc/crypto.c
@@ -144,7 +144,7 @@ struct tipc_tfm {
* @rcu: struct rcu_head
* @key: the aead key
* @gen: the key's generation
- * @seqno: the key seqno (cluster scope)
+ * @seqno: the per-key TX nonce counter
* @refcnt: the key reference counter
*/
struct tipc_aead {
@@ -190,7 +190,6 @@ struct tipc_crypto_stats {
* @rekeying_intv: rekeying interval (in minutes)
* @stats: the crypto statistics
* @name: the crypto name
- * @sndnxt: the per-peer sndnxt (TX)
* @timer1: general timer 1 (jiffies)
* @timer2: general timer 2 (jiffies)
* @working: the crypto is working or not
@@ -219,7 +218,6 @@ struct tipc_crypto {
struct tipc_crypto_stats __percpu *stats;
char name[48];
- atomic64_t sndnxt ____cacheline_aligned;
unsigned long timer1;
unsigned long timer2;
union {
@@ -1060,14 +1058,11 @@ static int tipc_ehdr_build(struct net *net, struct tipc_aead *aead,
WARN_ON(skb_headroom(skb) < ehsz);
ehdr = (struct tipc_ehdr *)skb_push(skb, ehsz);
- /* Obtain a seqno first:
- * Use the key seqno (= cluster wise) if dest is unknown or we're in
- * cluster key mode, otherwise it's better for a per-peer seqno!
+ /*
+ * Keep the nonce unique for the lifetime of the TX key,
+ * including key state changes and peer reconnection.
*/
- if (!__rx || aead->mode == CLUSTER_KEY)
- seqno = atomic64_inc_return(&aead->seqno);
- else
- seqno = atomic64_inc_return(&__rx->sndnxt);
+ seqno = atomic64_inc_return(&aead->seqno);
/* Revoke the key if seqno is wrapped around */
if (unlikely(!seqno))
@@ -1246,7 +1241,6 @@ void tipc_crypto_key_flush(struct tipc_crypto *c)
tipc_crypto_key_set_state(c, 0, 0, 0);
for (k = KEY_MIN; k <= KEY_MAX; k++)
tipc_crypto_key_detach(c->aead[k], &c->lock);
- atomic64_set(&c->sndnxt, 0);
spin_unlock_bh(&c->lock);
}
@@ -1393,8 +1387,6 @@ static struct tipc_aead *tipc_crypto_key_pick_tx(struct tipc_crypto *tx,
* It also considers if peer has no key, then we need to make own master key
* (if any) taking over i.e. starting grace period and also trigger key
* distributing process.
- *
- * The "per-peer" sndnxt is also reset when the peer key has switched.
*/
static void tipc_crypto_key_synch(struct tipc_crypto *rx, struct sk_buff *skb)
{
@@ -1445,7 +1437,6 @@ static void tipc_crypto_key_synch(struct tipc_crypto *rx, struct sk_buff *skb)
if (cur)
tipc_aead_users_dec(tx->aead[cur], 0);
- atomic64_set(&rx->sndnxt, 0);
/* Mark the point TX key users changed */
tx->timer1 = jiffies;
@@ -1510,7 +1501,6 @@ int tipc_crypto_start(struct tipc_crypto **crypto, struct net *net,
tipc_crypto_key_set_state(c, 0, 0, 0);
atomic_set(&c->key_distr, 0);
atomic_set(&c->peer_rx_active, 0);
- atomic64_set(&c->sndnxt, 0);
c->timer1 = jiffies;
c->timer2 = jiffies;
c->rekeying_intv = TIPC_REKEYING_INTV_DEF;
diff --git a/net/tipc/node.c b/net/tipc/node.c
index 256904240987..e27e70c6366c 100644
--- a/net/tipc/node.c
+++ b/net/tipc/node.c
@@ -1335,7 +1335,9 @@ static void tipc_node_reset_links(struct tipc_node *n)
pr_warn("Resetting all links to %x\n", n->addr);
+ tipc_node_write_lock(n);
trace_tipc_node_reset_links(n, true, " ");
+ tipc_node_write_unlock_fast(n);
for (i = 0; i < MAX_BEARERS; i++) {
tipc_node_link_down(n, i, false);
}
@@ -2421,8 +2423,11 @@ static struct tipc_node *tipc_node_find_by_name(struct net *net,
}
}
tipc_node_read_unlock(n);
- if (found_node)
+ if (found_node) {
+ if (!kref_get_unless_zero(&found_node->kref))
+ found_node = NULL;
break;
+ }
}
rcu_read_unlock();
@@ -2507,6 +2512,7 @@ int tipc_nl_node_set_link(struct sk_buff *skb, struct genl_info *info)
tipc_node_read_unlock(node);
tipc_bearer_xmit(net, bearer_id, &xmitq, &node->links[bearer_id].maddr,
NULL);
+ tipc_node_put(node);
return res;
}
@@ -2558,12 +2564,14 @@ int tipc_nl_node_get_link(struct sk_buff *skb, struct genl_info *info)
link = node->links[bearer_id].link;
if (!link) {
tipc_node_read_unlock(node);
+ tipc_node_put(node);
err = -EINVAL;
goto err_free;
}
err = __tipc_nl_add_link(net, &msg, link, 0);
tipc_node_read_unlock(node);
+ tipc_node_put(node);
if (err)
goto err_free;
}
@@ -2634,11 +2642,13 @@ int tipc_nl_node_reset_link_stats(struct sk_buff *skb, struct genl_info *info)
if (!link) {
spin_unlock_bh(&le->lock);
tipc_node_read_unlock(node);
+ tipc_node_put(node);
return -EINVAL;
}
tipc_link_reset_stats(link);
spin_unlock_bh(&le->lock);
tipc_node_read_unlock(node);
+ tipc_node_put(node);
return 0;
}
diff --git a/net/tipc/topsrv.c b/net/tipc/topsrv.c
index 0445c91db009..435fa14468e4 100644
--- a/net/tipc/topsrv.c
+++ b/net/tipc/topsrv.c
@@ -54,7 +54,7 @@
/**
* struct tipc_topsrv - TIPC server structure
* @conn_idr: identifier set of connection
- * @idr_lock: protect the connection identifier set
+ * @idr_lock: protect the connection identifier set and listener
* @idr_in_use: amount of allocated identifier entry
* @net: network namspace instance
* @awork: accept work item
@@ -217,6 +217,10 @@ static struct tipc_conn *tipc_conn_lookup(struct tipc_topsrv *s, int conid)
struct tipc_conn *con;
spin_lock_bh(&s->idr_lock);
+ if (!s->listener) {
+ spin_unlock_bh(&s->idr_lock);
+ return NULL;
+ }
con = idr_find(&s->conn_idr, conid);
if (!connected(con) || !kref_get_unless_zero(&con->kref))
con = NULL;
@@ -300,10 +304,20 @@ static void tipc_conn_send_to_sock(struct tipc_conn *con)
static void tipc_conn_send_work(struct work_struct *work)
{
struct tipc_conn *con = container_of(work, struct tipc_conn, swork);
+ struct tipc_topsrv *srv;
+
+ srv = con->server;
+ spin_lock_bh(&srv->idr_lock);
+ if (!srv->listener) {
+ spin_unlock_bh(&srv->idr_lock);
+ goto out;
+ }
+ spin_unlock_bh(&srv->idr_lock);
if (connected(con))
tipc_conn_send_to_sock(con);
+out:
conn_put(con);
}
@@ -333,8 +347,14 @@ void tipc_topsrv_queue_evt(struct net *net, int conid,
list_add_tail(&e->list, &con->outqueue);
spin_unlock_bh(&con->outqueue_lock);
- if (queue_work(srv->send_wq, &con->swork))
- return;
+ spin_lock_bh(&srv->idr_lock);
+ if (srv->listener) {
+ if (queue_work(srv->send_wq, &con->swork)) {
+ spin_unlock_bh(&srv->idr_lock);
+ return;
+ }
+ }
+ spin_unlock_bh(&srv->idr_lock);
err:
conn_put(con);
}
@@ -345,14 +365,20 @@ void tipc_topsrv_queue_evt(struct net *net, int conid,
*/
static void tipc_conn_write_space(struct sock *sk)
{
+ struct tipc_topsrv *srv;
struct tipc_conn *con;
read_lock_bh(&sk->sk_callback_lock);
con = sk->sk_user_data;
if (connected(con)) {
- conn_get(con);
- if (!queue_work(con->server->send_wq, &con->swork))
- conn_put(con);
+ srv = con->server;
+ spin_lock_bh(&srv->idr_lock);
+ if (srv->listener) {
+ conn_get(con);
+ if (!queue_work(srv->send_wq, &con->swork))
+ conn_put(con);
+ }
+ spin_unlock_bh(&srv->idr_lock);
}
read_unlock_bh(&sk->sk_callback_lock);
}
@@ -417,8 +443,17 @@ static int tipc_conn_rcv_from_sock(struct tipc_conn *con)
static void tipc_conn_recv_work(struct work_struct *work)
{
struct tipc_conn *con = container_of(work, struct tipc_conn, rwork);
+ struct tipc_topsrv *srv;
int count = 0;
+ srv = con->server;
+ spin_lock_bh(&srv->idr_lock);
+ if (!srv->listener) {
+ spin_unlock_bh(&srv->idr_lock);
+ goto out;
+ }
+ spin_unlock_bh(&srv->idr_lock);
+
while (connected(con)) {
if (tipc_conn_rcv_from_sock(con))
break;
@@ -429,6 +464,7 @@ static void tipc_conn_recv_work(struct work_struct *work)
count = 0;
}
}
+out:
conn_put(con);
}
@@ -437,14 +473,20 @@ static void tipc_conn_recv_work(struct work_struct *work)
*/
static void tipc_conn_data_ready(struct sock *sk)
{
+ struct tipc_topsrv *srv;
struct tipc_conn *con;
read_lock_bh(&sk->sk_callback_lock);
con = sk->sk_user_data;
if (connected(con)) {
- conn_get(con);
- if (!queue_work(con->server->rcv_wq, &con->rwork))
- conn_put(con);
+ srv = con->server;
+ spin_lock_bh(&srv->idr_lock);
+ if (srv->listener) {
+ conn_get(con);
+ if (!queue_work(srv->rcv_wq, &con->rwork))
+ conn_put(con);
+ }
+ spin_unlock_bh(&srv->idr_lock);
}
read_unlock_bh(&sk->sk_callback_lock);
}
@@ -498,8 +540,12 @@ static void tipc_topsrv_listener_data_ready(struct sock *sk)
read_lock_bh(&sk->sk_callback_lock);
srv = sk->sk_user_data;
- if (srv)
- queue_work(srv->rcv_wq, &srv->awork);
+ if (srv) {
+ spin_lock_bh(&srv->idr_lock);
+ if (srv->listener)
+ queue_work(srv->rcv_wq, &srv->awork);
+ spin_unlock_bh(&srv->idr_lock);
+ }
read_unlock_bh(&sk->sk_callback_lock);
}
@@ -695,23 +741,27 @@ static void tipc_topsrv_stop(struct net *net)
struct tipc_conn *con;
int id;
+ spin_lock_bh(&srv->idr_lock);
+ srv->listener = NULL;
+ spin_unlock_bh(&srv->idr_lock);
+ tipc_topsrv_work_stop(srv);
+
spin_lock_bh(&srv->idr_lock);
for (id = 0; srv->idr_in_use; id++) {
con = idr_find(&srv->conn_idr, id);
if (con) {
- conn_get(con);
spin_unlock_bh(&srv->idr_lock);
tipc_conn_close(con);
- conn_put(con);
spin_lock_bh(&srv->idr_lock);
+ continue;
}
+ spin_unlock_bh(&srv->idr_lock);
+ spin_lock_bh(&srv->idr_lock);
}
__module_get(lsock->ops->owner);
__module_get(lsock->sk->sk_prot_creator->owner);
- srv->listener = NULL;
spin_unlock_bh(&srv->idr_lock);
- tipc_topsrv_work_stop(srv);
sock_release(lsock);
idr_destroy(&srv->conn_idr);
kfree(srv);
diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c
index e1305d159834..479597a7267c 100644
--- a/net/tipc/udp_media.c
+++ b/net/tipc/udp_media.c
@@ -40,6 +40,7 @@
#include <linux/igmp.h>
#include <linux/kernel.h>
#include <linux/workqueue.h>
+#include <linux/wait_bit.h>
#include <linux/list.h>
#include <net/sock.h>
#include <net/ip.h>
@@ -94,6 +95,8 @@ struct udp_replicast {
* @ifindex: local address scope
* @work: used to schedule deferred work on a bearer
* @rcast: associated udp_replicast container
+ * @rcast_lock: serialize updates to @rcast.list against concurrent updaters
+ * @disabled: bearer is being torn down; reject further @rcast.list additions
*/
struct udp_bearer {
struct tipc_bearer __rcu *bearer;
@@ -101,6 +104,8 @@ struct udp_bearer {
u32 ifindex;
struct work_struct work;
struct udp_replicast rcast;
+ spinlock_t rcast_lock;
+ bool disabled;
};
static int tipc_udp_is_mcast_addr(struct udp_media_addr *addr)
@@ -279,26 +284,6 @@ static int tipc_udp_send_msg(struct net *net, struct sk_buff *skb,
return err;
}
-static bool tipc_udp_is_known_peer(struct tipc_bearer *b,
- struct udp_media_addr *addr)
-{
- struct udp_replicast *rcast, *tmp;
- struct udp_bearer *ub;
-
- ub = rcu_dereference_rtnl(b->media_ptr);
- if (!ub) {
- pr_err_ratelimited("UDP bearer instance not found\n");
- return false;
- }
-
- list_for_each_entry_safe(rcast, tmp, &ub->rcast.list, list) {
- if (!memcmp(&rcast->addr, addr, sizeof(struct udp_media_addr)))
- return true;
- }
-
- return false;
-}
-
static int tipc_udp_rcast_add(struct tipc_bearer *b,
struct udp_media_addr *addr)
{
@@ -309,16 +294,34 @@ static int tipc_udp_rcast_add(struct tipc_bearer *b,
if (!ub)
return -ENODEV;
+ spin_lock_bh(&ub->rcast_lock);
+ if (ub->disabled) {
+ spin_unlock_bh(&ub->rcast_lock);
+ return 0;
+ }
+ list_for_each_entry(rcast, &ub->rcast.list, list) {
+ if (!memcmp(&rcast->addr, addr, sizeof(*addr))) {
+ spin_unlock_bh(&ub->rcast_lock);
+ return 0;
+ }
+ }
+
rcast = kmalloc(sizeof(*rcast), GFP_ATOMIC);
- if (!rcast)
+ if (!rcast) {
+ spin_unlock_bh(&ub->rcast_lock);
return -ENOMEM;
+ }
if (dst_cache_init(&rcast->dst_cache, GFP_ATOMIC)) {
+ spin_unlock_bh(&ub->rcast_lock);
kfree(rcast);
return -ENOMEM;
}
memcpy(&rcast->addr, addr, sizeof(struct udp_media_addr));
+ list_add_rcu(&rcast->list, &ub->rcast.list);
+ b->bcast_addr.broadcast = TIPC_REPLICAST_SUPPORT;
+ spin_unlock_bh(&ub->rcast_lock);
if (ntohs(addr->proto) == ETH_P_IP)
pr_info("New replicast peer: %pI4\n", &rcast->addr.ipv4);
@@ -326,8 +329,6 @@ static int tipc_udp_rcast_add(struct tipc_bearer *b,
else if (ntohs(addr->proto) == ETH_P_IPV6)
pr_info("New replicast peer: %pI6\n", &rcast->addr.ipv6);
#endif
- b->bcast_addr.broadcast = TIPC_REPLICAST_SUPPORT;
- list_add_rcu(&rcast->list, &ub->rcast.list);
return 0;
}
@@ -362,9 +363,6 @@ static int tipc_udp_rcast_disc(struct tipc_bearer *b, struct sk_buff *skb)
return 0;
}
- if (likely(tipc_udp_is_known_peer(b, &src)))
- return 0;
-
return tipc_udp_rcast_add(b, &src);
}
@@ -646,9 +644,6 @@ int tipc_udp_nl_bearer_add(struct tipc_bearer *b, struct nlattr *attr)
return -EINVAL;
}
- if (tipc_udp_is_known_peer(b, &addr))
- return 0;
-
return tipc_udp_rcast_add(b, &addr);
}
@@ -680,6 +675,7 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b,
return -ENOMEM;
INIT_LIST_HEAD(&ub->rcast.list);
+ spin_lock_init(&ub->rcast_lock);
if (!attrs[TIPC_NLA_BEARER_UDP_OPTS])
goto err;
@@ -806,6 +802,14 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b,
return err;
}
+static void rcast_free_rcu(struct rcu_head *rcu)
+{
+ struct udp_replicast *rcast = container_of(rcu, struct udp_replicast, rcu);
+
+ dst_cache_destroy(&rcast->dst_cache);
+ kfree(rcast);
+}
+
/* cleanup_bearer - break the socket/bearer association */
static void cleanup_bearer(struct work_struct *work)
{
@@ -813,20 +817,23 @@ static void cleanup_bearer(struct work_struct *work)
struct udp_replicast *rcast, *tmp;
struct tipc_net *tn;
+ spin_lock_bh(&ub->rcast_lock);
list_for_each_entry_safe(rcast, tmp, &ub->rcast.list, list) {
- dst_cache_destroy(&rcast->dst_cache);
list_del_rcu(&rcast->list);
- kfree_rcu(rcast, rcu);
+ call_rcu(&rcast->rcu, rcast_free_rcu);
}
+ ub->disabled = true;
+ spin_unlock_bh(&ub->rcast_lock);
tn = tipc_net(sock_net(ub->ubsock->sk));
- dst_cache_destroy(&ub->rcast.dst_cache);
udp_tunnel_sock_release(ub->ubsock);
- /* Note: could use a call_rcu() to avoid another synchronize_net() */
synchronize_net();
- atomic_dec(&tn->wq_count);
+
+ dst_cache_destroy(&ub->rcast.dst_cache);
+ if (atomic_dec_and_test(&tn->wq_count))
+ wake_up_var(&tn->wq_count);
kfree(ub);
}
diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c
index 753c45ecc356..74b61318a60a 100644
--- a/net/tls/tls_device.c
+++ b/net/tls/tls_device.c
@@ -239,7 +239,7 @@ static void tls_device_resync_tx(struct sock *sk, struct tls_context *tls_ctx,
int err = 0;
u8 *rcd_sn;
- skb = tcp_write_queue_tail(sk);
+ skb = tcp_write_queue_tail(sk) ?: tcp_rtx_queue_tail(sk);
if (skb)
TCP_SKB_CB(skb)->eor = 1;
@@ -1181,7 +1181,7 @@ int tls_set_device_offload(struct sock *sk, struct tls_context *ctx)
* SKBs where only part of the payload needs to be encrypted.
* So mark the last skb in the write queue as end of record.
*/
- skb = tcp_write_queue_tail(sk);
+ skb = tcp_write_queue_tail(sk) ?: tcp_rtx_queue_tail(sk);
if (skb)
TCP_SKB_CB(skb)->eor = 1;
diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c
index 68b8a59684e3..2aeeeca57da6 100644
--- a/net/tls/tls_sw.c
+++ b/net/tls/tls_sw.c
@@ -1807,6 +1807,8 @@ tls_decrypt_sw(struct sock *sk, struct tls_context *tls_ctx,
/* If opportunistic TLS 1.3 ZC failed retry without ZC */
if (unlikely(darg->zc && prot->version == TLS_1_3_VERSION &&
darg->tail != TLS_RECORD_TYPE_DATA)) {
+ iov_iter_revert(&msg->msg_iter, strp_msg(darg->skb)->full_len -
+ prot->overhead_size);
darg->zc = false;
if (!darg->tail)
TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXNOPADVIOL);
diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c
index 92685d6da9ad..5675f224ea7d 100644
--- a/net/vmw_vsock/af_vsock.c
+++ b/net/vmw_vsock/af_vsock.c
@@ -331,6 +331,38 @@ struct sock *vsock_find_connected_socket(struct sockaddr_vm *src,
}
EXPORT_SYMBOL_GPL(vsock_find_connected_socket);
+/**
+ * vsock_check_source - validate a packet source against a socket peer
+ * @vsk: socket receiving the packet
+ * @transport: transport receiving the packet
+ * @src: source address from the packet
+ *
+ * Return: true if the packet arrived on the socket's assigned transport and
+ * its source matches the stored peer. Loopback packets are generated
+ * internally and always use the local CID as their source, including
+ * connections using a valid CID alias.
+ *
+ * The caller must hold the socket lock and must not call this for listening
+ * sockets, which accept packets from any source and have no assigned
+ * transport.
+ */
+bool vsock_check_source(const struct vsock_sock *vsk,
+ const struct vsock_transport *transport,
+ const struct sockaddr_vm *src)
+{
+ if (vsk->transport != transport)
+ return false;
+
+ if (src->svm_port != vsk->remote_addr.svm_port)
+ return false;
+
+ if (src->svm_cid == vsk->remote_addr.svm_cid)
+ return true;
+
+ return transport->get_local_cid() == VMADDR_CID_LOCAL;
+}
+EXPORT_SYMBOL_GPL(vsock_check_source);
+
void vsock_remove_sock(struct vsock_sock *vsk)
{
/* Transport reassignment must not remove the binding. */
@@ -1502,23 +1534,22 @@ static int vsock_connect(struct socket *sock, struct sockaddr *addr,
timeout = schedule_timeout(timeout);
lock_sock(sk);
- /* Connection established. Whatever happens to socket once we
- * release it, that's not connect()'s concern. No need to go
+ /* Connection was established. Whatever happens to socket once
+ * we release it, that's not connect()'s concern. No need to go
* into signal and timeout handling. Call it a day.
*
* Note that allowing to "reset" an already established socket
* here is racy and insecure.
*/
- if (sk->sk_state == TCP_ESTABLISHED)
- break;
+ if (sk->sk_state == TCP_ESTABLISHED ||
+ sk->sk_state == TCP_CLOSING) {
+ err = 0;
+ goto out_wait;
+ }
/* If connection was _not_ established and a signal/timeout came
* to be, we want the socket's state reset. User space may want
- * to retry.
- *
- * sk_state != TCP_ESTABLISHED implies that socket is not on
- * vsock_connected_table. We keep the binding and the transport
- * assigned.
+ * to retry, so we keep the binding and the transport assigned.
*/
if (signal_pending(current) || timeout == 0) {
err = timeout == 0 ? -ETIMEDOUT : sock_intr_errno(timeout);
@@ -1586,8 +1617,7 @@ static int vsock_accept(struct socket *sock, struct socket *newsock, int flags,
timeout = sock_rcvtimeo(listener, flags & O_NONBLOCK);
prepare_to_wait(sk_sleep(listener), &wait, TASK_INTERRUPTIBLE);
- while ((connected = vsock_dequeue_accept(listener)) == NULL &&
- listener->sk_err == 0) {
+ while ((connected = vsock_dequeue_accept(listener)) == NULL) {
release_sock(listener);
timeout = schedule_timeout(timeout);
finish_wait(sk_sleep(listener), &wait);
@@ -1605,9 +1635,6 @@ static int vsock_accept(struct socket *sock, struct socket *newsock, int flags,
}
finish_wait(sk_sleep(listener), &wait);
- if (listener->sk_err)
- err = -listener->sk_err;
-
if (connected) {
sk_acceptq_removed(listener);
diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c
index 6ea00b863035..81bb95875f20 100644
--- a/net/vmw_vsock/virtio_transport_common.c
+++ b/net/vmw_vsock/virtio_transport_common.c
@@ -1372,7 +1372,8 @@ void virtio_transport_recv_pkt(struct virtio_transport *t,
* lock_sock (note: listener sockets are not assigned to any transport)
*/
if (sock_flag(sk, SOCK_DONE) ||
- (sk->sk_state != TCP_LISTEN && vsk->transport != &t->transport)) {
+ (sk->sk_state != TCP_LISTEN &&
+ !vsock_check_source(vsk, &t->transport, &src))) {
(void)virtio_transport_reset_no_sock(t, skb);
release_sock(sk);
sock_put(sk);
diff --git a/net/wireless/core.c b/net/wireless/core.c
index 6f83ea71b760..3179eb8899d6 100644
--- a/net/wireless/core.c
+++ b/net/wireless/core.c
@@ -232,6 +232,9 @@ void cfg80211_stop_p2p_device(struct cfg80211_registered_device *rdev,
if (!wdev_running(wdev))
return;
+ wdev_lock(wdev);
+ cfg80211_pmsr_wdev_down(wdev);
+ wdev_unlock(wdev);
rdev_stop_p2p_device(rdev, wdev);
wdev->is_running = false;
@@ -257,6 +260,10 @@ void cfg80211_stop_nan(struct cfg80211_registered_device *rdev,
if (!wdev_running(wdev))
return;
+ wdev_lock(wdev);
+ cfg80211_pmsr_wdev_down(wdev);
+ wdev_unlock(wdev);
+
rdev_stop_nan(rdev, wdev);
wdev->is_running = false;
@@ -686,6 +693,24 @@ static int wiphy_verify_combinations(struct wiphy *wiphy)
return 0;
}
+static bool wiphy_cipher_suites_valid(const struct wiphy *wiphy)
+{
+ int i, j;
+
+ if (wiphy->n_cipher_suites && !wiphy->cipher_suites)
+ return false;
+
+ for (i = 0; i < wiphy->n_cipher_suites; i++) {
+ for (j = 0; j < i; j++) {
+ if (wiphy->cipher_suites[i] ==
+ wiphy->cipher_suites[j])
+ return false;
+ }
+ }
+
+ return true;
+}
+
int wiphy_register(struct wiphy *wiphy)
{
struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
@@ -817,6 +842,9 @@ int wiphy_register(struct wiphy *wiphy)
if (res)
return res;
+ if (!wiphy_cipher_suites_valid(wiphy))
+ return -EINVAL;
+
/* sanity check supported bands/channels */
for (band = 0; band < NUM_NL80211_BANDS; band++) {
u16 types = 0;
@@ -1201,7 +1229,7 @@ static void _cfg80211_unregister_wdev(struct wireless_dev *wdev,
ASSERT_RTNL();
lockdep_assert_held(&rdev->wiphy.mtx);
- flush_work(&wdev->pmsr_free_wk);
+ wiphy_work_flush(wdev->wiphy, &wdev->pmsr_free_wk);
nl80211_notify_iface(rdev, wdev, NL80211_CMD_DEL_INTERFACE);
@@ -1216,6 +1244,7 @@ static void _cfg80211_unregister_wdev(struct wireless_dev *wdev,
list_del_rcu(&wdev->list);
synchronize_net();
rdev->devlist_generation++;
+ wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk);
cfg80211_mlme_purge_registrations(wdev);
@@ -1382,7 +1411,7 @@ void cfg80211_init_wdev(struct wireless_dev *wdev)
INIT_LIST_HEAD(&wdev->mgmt_registrations);
INIT_LIST_HEAD(&wdev->pmsr_list);
spin_lock_init(&wdev->pmsr_lock);
- INIT_WORK(&wdev->pmsr_free_wk, cfg80211_pmsr_free_wk);
+ wiphy_work_init(&wdev->pmsr_free_wk, cfg80211_pmsr_free_wk);
#ifdef CONFIG_CFG80211_WEXT
wdev->wext.default_key = -1;
@@ -1404,7 +1433,7 @@ void cfg80211_init_wdev(struct wireless_dev *wdev)
wdev->iftype == NL80211_IFTYPE_ADHOC) && !wdev->use_4addr)
wdev->netdev->priv_flags |= IFF_DONT_BRIDGE;
- INIT_WORK(&wdev->disconnect_wk, cfg80211_autodisconnect_wk);
+ wiphy_work_init(&wdev->disconnect_wk, cfg80211_autodisconnect_wk);
}
void cfg80211_register_wdev(struct cfg80211_registered_device *rdev,
@@ -1512,9 +1541,9 @@ static int cfg80211_netdev_notifier_call(struct notifier_block *nb,
wiphy_lock(&rdev->wiphy);
cfg80211_leave(rdev, wdev);
cfg80211_remove_links(wdev);
- wiphy_unlock(&rdev->wiphy);
/* since we just did cfg80211_leave() nothing to do there */
- cancel_work_sync(&wdev->disconnect_wk);
+ wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk);
+ wiphy_unlock(&rdev->wiphy);
break;
case NETDEV_DOWN:
wiphy_lock(&rdev->wiphy);
diff --git a/net/wireless/core.h b/net/wireless/core.h
index 987c41b12856..733eb2e587c1 100644
--- a/net/wireless/core.h
+++ b/net/wireless/core.h
@@ -427,7 +427,7 @@ void __cfg80211_port_authorized(struct wireless_dev *wdev, const u8 *bssid,
const u8 *td_bitmap, u8 td_bitmap_len);
int cfg80211_mgd_wext_connect(struct cfg80211_registered_device *rdev,
struct wireless_dev *wdev);
-void cfg80211_autodisconnect_wk(struct work_struct *work);
+void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work);
/* SME implementation */
void cfg80211_conn_work(struct work_struct *work);
@@ -575,7 +575,7 @@ cfg80211_bss_update(struct cfg80211_registered_device *rdev,
void cfg80211_release_pmsr(struct wireless_dev *wdev, u32 portid);
void cfg80211_pmsr_wdev_down(struct wireless_dev *wdev);
-void cfg80211_pmsr_free_wk(struct work_struct *work);
+void cfg80211_pmsr_free_wk(struct wiphy *wiphy, struct wiphy_work *work);
void cfg80211_remove_link(struct wireless_dev *wdev, unsigned int link_id);
void cfg80211_remove_links(struct wireless_dev *wdev);
diff --git a/net/wireless/lib80211_crypt_tkip.c b/net/wireless/lib80211_crypt_tkip.c
index 1b4d6c87a5c5..823274728711 100644
--- a/net/wireless/lib80211_crypt_tkip.c
+++ b/net/wireless/lib80211_crypt_tkip.c
@@ -38,6 +38,7 @@ MODULE_DESCRIPTION("lib80211 crypt: TKIP");
MODULE_LICENSE("GPL");
#define TKIP_HDR_LEN 8
+#define MICHAEL_MIC_LEN 8
struct lib80211_tkip_data {
#define TKIP_KEY_LEN 32
@@ -585,16 +586,18 @@ static int lib80211_michael_mic_verify(struct sk_buff *skb, int keyidx,
int hdr_len, void *priv)
{
struct lib80211_tkip_data *tkey = priv;
- u8 mic[8];
+ u8 mic[MICHAEL_MIC_LEN];
- if (!tkey->key_set)
+ if (!tkey->key_set || skb->len < hdr_len + MICHAEL_MIC_LEN)
return -1;
michael_mic_hdr(skb, tkey->rx_hdr);
if (michael_mic(tkey->rx_tfm_michael, &tkey->key[24], tkey->rx_hdr,
- skb->data + hdr_len, skb->len - 8 - hdr_len, mic))
+ skb->data + hdr_len,
+ skb->len - MICHAEL_MIC_LEN - hdr_len, mic))
return -1;
- if (memcmp(mic, skb->data + skb->len - 8, 8) != 0) {
+ if (memcmp(mic, skb->data + skb->len - MICHAEL_MIC_LEN,
+ MICHAEL_MIC_LEN) != 0) {
struct ieee80211_hdr *hdr;
hdr = (struct ieee80211_hdr *)skb->data;
printk(KERN_DEBUG "%s: Michael MIC verification failed for "
@@ -612,7 +615,7 @@ static int lib80211_michael_mic_verify(struct sk_buff *skb, int keyidx,
tkey->rx_iv32 = tkey->rx_iv32_new;
tkey->rx_iv16 = tkey->rx_iv16_new;
- skb_trim(skb, skb->len - 8);
+ skb_trim(skb, skb->len - MICHAEL_MIC_LEN);
return 0;
}
diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c
index d749fa91f399..fab77741663f 100644
--- a/net/wireless/nl80211.c
+++ b/net/wireless/nl80211.c
@@ -15968,15 +15968,15 @@ static int nl80211_color_change(struct sk_buff *skb, struct genl_info *info)
params.count = nla_get_u8(info->attrs[NL80211_ATTR_COLOR_CHANGE_COUNT]);
params.color = nla_get_u8(info->attrs[NL80211_ATTR_COLOR_CHANGE_COLOR]);
- err = nl80211_parse_beacon(rdev, info->attrs, ¶ms.beacon_next,
- info->extack);
- if (err)
- return err;
-
tb = kcalloc(NL80211_ATTR_MAX + 1, sizeof(*tb), GFP_KERNEL);
if (!tb)
return -ENOMEM;
+ err = nl80211_parse_beacon(rdev, info->attrs, ¶ms.beacon_next,
+ info->extack);
+ if (err)
+ goto out;
+
err = nla_parse_nested(tb, NL80211_ATTR_MAX,
info->attrs[NL80211_ATTR_COLOR_CHANGE_ELEMS],
nl80211_policy, info->extack);
@@ -19798,7 +19798,8 @@ static int nl80211_netlink_notify(struct notifier_block * nb,
wdev->nl_owner_dead = true;
schedule_work(&rdev->destroy_work);
} else if (wdev->conn_owner_nlportid == notify->portid) {
- schedule_work(&wdev->disconnect_wk);
+ wiphy_work_queue(wdev->wiphy,
+ &wdev->disconnect_wk);
}
cfg80211_release_pmsr(wdev, notify->portid);
diff --git a/net/wireless/pmsr.c b/net/wireless/pmsr.c
index e3987ba2d40a..c84e218930f7 100644
--- a/net/wireless/pmsr.c
+++ b/net/wireless/pmsr.c
@@ -635,7 +635,7 @@ static void cfg80211_pmsr_process_abort(struct wireless_dev *wdev)
}
}
-void cfg80211_pmsr_free_wk(struct work_struct *work)
+void cfg80211_pmsr_free_wk(struct wiphy *wiphy, struct wiphy_work *work)
{
struct wireless_dev *wdev = container_of(work, struct wireless_dev,
pmsr_free_wk);
@@ -657,7 +657,7 @@ void cfg80211_pmsr_wdev_down(struct wireless_dev *wdev)
}
spin_unlock_bh(&wdev->pmsr_lock);
- cancel_work_sync(&wdev->pmsr_free_wk);
+ wiphy_work_cancel(wdev->wiphy, &wdev->pmsr_free_wk);
if (found)
cfg80211_pmsr_process_abort(wdev);
@@ -672,7 +672,7 @@ void cfg80211_release_pmsr(struct wireless_dev *wdev, u32 portid)
list_for_each_entry(req, &wdev->pmsr_list, list) {
if (req->nl_portid == portid) {
req->nl_portid = 0;
- schedule_work(&wdev->pmsr_free_wk);
+ wiphy_work_queue(wdev->wiphy, &wdev->pmsr_free_wk);
}
}
spin_unlock_bh(&wdev->pmsr_lock);
diff --git a/net/wireless/scan.c b/net/wireless/scan.c
index ecf902121938..8998a48d0be5 100644
--- a/net/wireless/scan.c
+++ b/net/wireless/scan.c
@@ -995,6 +995,7 @@ int cfg80211_scan(struct cfg80211_registered_device *rdev)
struct cfg80211_scan_request *request;
struct cfg80211_scan_request *rdev_req = rdev->scan_req;
u32 n_channels = 0, idx, i;
+ int err;
if (!(rdev->wiphy.flags & WIPHY_FLAG_SPLIT_SCAN_6GHZ))
return rdev_scan(rdev, rdev_req);
@@ -1021,8 +1022,14 @@ int cfg80211_scan(struct cfg80211_registered_device *rdev)
}
rdev_req->scan_6ghz = false;
+ err = rdev_scan(rdev, request);
+ if (err) {
+ kfree(request);
+ return err;
+ }
+
rdev->int_scan_req = request;
- return rdev_scan(rdev, request);
+ return 0;
}
void ___cfg80211_scan_done(struct cfg80211_registered_device *rdev,
@@ -1784,20 +1791,22 @@ cfg80211_update_known_bss(struct cfg80211_registered_device *rdev,
return false;
}
- old = rcu_access_pointer(known->pub.beacon_ies);
+ if (!new->pub.hidden_beacon_bss) {
+ old = rcu_access_pointer(known->pub.beacon_ies);
- rcu_assign_pointer(known->pub.beacon_ies, new->pub.beacon_ies);
+ rcu_assign_pointer(known->pub.beacon_ies, new->pub.beacon_ies);
- /* Override IEs if they were from a beacon before */
- if (old == rcu_access_pointer(known->pub.ies))
- rcu_assign_pointer(known->pub.ies, new->pub.beacon_ies);
+ /* Override IEs if they were from a beacon before */
+ if (old == rcu_access_pointer(known->pub.ies))
+ rcu_assign_pointer(known->pub.ies, new->pub.beacon_ies);
- cfg80211_update_hidden_bsses(known,
- rcu_access_pointer(new->pub.beacon_ies),
- old);
+ cfg80211_update_hidden_bsses(known,
+ rcu_access_pointer(new->pub.beacon_ies),
+ old);
- if (old)
- kfree_rcu((struct cfg80211_bss_ies *)old, rcu_head);
+ if (old)
+ kfree_rcu((struct cfg80211_bss_ies *)old, rcu_head);
+ }
}
known->pub.beacon_interval = new->pub.beacon_interval;
@@ -2695,10 +2704,9 @@ void cfg80211_update_assoc_bss_entry(struct wireless_dev *wdev,
if (new) {
/* to save time, update IEs for transmitting bss only */
- if (cfg80211_update_known_bss(rdev, cbss, new, false)) {
- new->pub.proberesp_ies = NULL;
- new->pub.beacon_ies = NULL;
- }
+ cfg80211_update_known_bss(rdev, cbss, new, false);
+ new->pub.proberesp_ies = NULL;
+ new->pub.beacon_ies = NULL;
list_for_each_entry_safe(nontrans_bss, tmp,
&new->pub.nontrans_list,
diff --git a/net/wireless/sme.c b/net/wireless/sme.c
index 5adf3912f585..09d872835310 100644
--- a/net/wireless/sme.c
+++ b/net/wireless/sme.c
@@ -1557,13 +1557,12 @@ int cfg80211_disconnect(struct cfg80211_registered_device *rdev,
* Used to clean up after the connection / connection attempt owner socket
* disconnects
*/
-void cfg80211_autodisconnect_wk(struct work_struct *work)
+void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work)
{
struct wireless_dev *wdev =
container_of(work, struct wireless_dev, disconnect_wk);
- struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy);
+ struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy);
- wiphy_lock(wdev->wiphy);
wdev_lock(wdev);
if (wdev->conn_owner_nlportid) {
@@ -1602,5 +1601,4 @@ void cfg80211_autodisconnect_wk(struct work_struct *work)
}
wdev_unlock(wdev);
- wiphy_unlock(wdev->wiphy);
}
diff --git a/net/xfrm/espintcp.c b/net/xfrm/espintcp.c
index b98fa2887c86..a35a170c3920 100644
--- a/net/xfrm/espintcp.c
+++ b/net/xfrm/espintcp.c
@@ -491,6 +491,7 @@ static int espintcp_init_sk(struct sock *sk)
/* avoid using task_frag */
sk->sk_allocation = GFP_ATOMIC;
+ sk->sk_use_task_frag = false;
return 0;
diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c
index 313c0eb3381f..b16c4657fe0f 100644
--- a/net/xfrm/xfrm_device.c
+++ b/net/xfrm/xfrm_device.c
@@ -215,6 +215,14 @@ struct sk_buff *validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t featur
pskb = skb2;
}
+ /* skb_gso_segment() set skb->prev to the last segment, but async
+ * crypto may have stolen it above without updating ->prev. Repoint
+ * it at the last retained segment so validate_xmit_skb_list() does
+ * not chain onto a segment now owned by the crypto engine.
+ */
+ if (skb)
+ skb->prev = pskb;
+
return skb;
}
EXPORT_SYMBOL_GPL(validate_xmit_xfrm);
@@ -424,6 +432,13 @@ static int xfrm_dev_down(struct net_device *dev)
return NOTIFY_DONE;
}
+static int xfrm_dev_unregister(struct net_device *dev)
+{
+ xfrm_dev_state_flush(dev_net(dev), dev, true);
+
+ return NOTIFY_DONE;
+}
+
static int xfrm_dev_event(struct notifier_block *this, unsigned long event, void *ptr)
{
struct net_device *dev = netdev_notifier_info_to_dev(ptr);
@@ -436,8 +451,10 @@ static int xfrm_dev_event(struct notifier_block *this, unsigned long event, void
return xfrm_api_check(dev);
case NETDEV_DOWN:
- case NETDEV_UNREGISTER:
return xfrm_dev_down(dev);
+
+ case NETDEV_UNREGISTER:
+ return xfrm_dev_unregister(dev);
}
return NOTIFY_DONE;
}
diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c
index de2e8b16b217..b39bb4662d4b 100644
--- a/net/xfrm/xfrm_input.c
+++ b/net/xfrm/xfrm_input.c
@@ -634,11 +634,14 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
else
nexthdr = x->type->input(x, skb);
- if (nexthdr == -EINPROGRESS)
+ if (nexthdr == -EINPROGRESS) {
+ if (async)
+ dev_put(skb->dev);
return 0;
-resume:
- dev_put(skb->dev);
+ }
+ dev_put(skb->dev);
+resume:
spin_lock(&x->lock);
if (nexthdr < 0) {
if (nexthdr == -EBADMSG) {
@@ -704,6 +707,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
sp->olen = 0;
if (skb_valid_dst(skb))
skb_dst_drop(skb);
+ if (async)
+ dev_put(skb->dev);
gro_cells_receive(&gro_cells, skb);
return 0;
} else {
@@ -723,6 +728,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
sp->olen = 0;
if (skb_valid_dst(skb))
skb_dst_drop(skb);
+ if (async)
+ dev_put(skb->dev);
gro_cells_receive(&gro_cells, skb);
return err;
}
@@ -733,6 +740,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type)
drop_unlock:
spin_unlock(&x->lock);
drop:
+ if (async)
+ dev_put(skb->dev);
xfrm_rcv_cb(skb, family, x && x->type ? x->type->proto : nexthdr, -1);
kfree_skb(skb);
return 0;
diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c
index 2a47f5b5776e..7ababb73d08e 100644
--- a/net/xfrm/xfrm_policy.c
+++ b/net/xfrm/xfrm_policy.c
@@ -1381,7 +1381,10 @@ static void xfrm_hash_rebuild(struct work_struct *work)
void xfrm_policy_hash_rebuild(struct net *net)
{
- schedule_work(&net->xfrm.policy_hthresh.work);
+ write_seqlock(&net->xfrm.policy_hthresh.lock);
+ if (!net->xfrm.policy_hthresh.work_disabled)
+ schedule_work(&net->xfrm.policy_hthresh.work);
+ write_sequnlock(&net->xfrm.policy_hthresh.lock);
}
EXPORT_SYMBOL(xfrm_policy_hash_rebuild);
@@ -4094,6 +4097,7 @@ static int __net_init xfrm_policy_init(struct net *net)
net->xfrm.policy_hthresh.rbits6 = 128;
seqlock_init(&net->xfrm.policy_hthresh.lock);
+ net->xfrm.policy_hthresh.work_disabled = false;
INIT_LIST_HEAD(&net->xfrm.policy_all);
INIT_LIST_HEAD(&net->xfrm.inexact_bins);
@@ -4119,12 +4123,27 @@ static void xfrm_policy_fini(struct net *net)
unsigned int sz;
int dir;
+ /* Prevent new policy hash rebuilds before draining the work item.
+ *
+ * The upstream fix uses disable_work_sync(), which is unavailable
+ * in v6.6. Protecting work_disabled and schedule_work() with the
+ * same seqlock closes the check-to-queue race, while the subsequent
+ * cancel_work_sync() drains work that was queued or running before
+ * teardown disabled it.
+ */
+ write_seqlock(&net->xfrm.policy_hthresh.lock);
+ net->xfrm.policy_hthresh.work_disabled = true;
+ write_sequnlock(&net->xfrm.policy_hthresh.lock);
+ cancel_work_sync(&net->xfrm.policy_hthresh.work);
+
flush_work(&net->xfrm.policy_hash_work);
#ifdef CONFIG_XFRM_SUB_POLICY
xfrm_policy_flush(net, XFRM_POLICY_TYPE_SUB, false);
#endif
xfrm_policy_flush(net, XFRM_POLICY_TYPE_MAIN, false);
+ synchronize_rcu();
+
WARN_ON(!list_empty(&net->xfrm.policy_all));
for (dir = 0; dir < XFRM_POLICY_MAX; dir++) {
diff --git a/net/xfrm/xfrm_state.c b/net/xfrm/xfrm_state.c
index 4b0835cb58f5..6b4fa60de6d8 100644
--- a/net/xfrm/xfrm_state.c
+++ b/net/xfrm/xfrm_state.c
@@ -34,6 +34,8 @@
#define xfrm_state_deref_prot(table, net) \
rcu_dereference_protected((table), lockdep_is_held(&(net)->xfrm.xfrm_state_lock))
+#define xfrm_state_deref_check(table, net) \
+ rcu_dereference_check((table), lockdep_is_held(&(net)->xfrm.xfrm_state_lock))
static void xfrm_state_gc_task(struct work_struct *work);
@@ -61,6 +63,8 @@ static inline unsigned int xfrm_dst_hash(struct net *net,
u32 reqid,
unsigned short family)
{
+ lockdep_assert_held(&net->xfrm.xfrm_state_lock);
+
return __xfrm_dst_hash(daddr, saddr, reqid, family, net->xfrm.state_hmask);
}
@@ -69,6 +73,8 @@ static inline unsigned int xfrm_src_hash(struct net *net,
const xfrm_address_t *saddr,
unsigned short family)
{
+ lockdep_assert_held(&net->xfrm.xfrm_state_lock);
+
return __xfrm_src_hash(daddr, saddr, family, net->xfrm.state_hmask);
}
@@ -76,11 +82,15 @@ static inline unsigned int
xfrm_spi_hash(struct net *net, const xfrm_address_t *daddr,
__be32 spi, u8 proto, unsigned short family)
{
+ lockdep_assert_held(&net->xfrm.xfrm_state_lock);
+
return __xfrm_spi_hash(daddr, spi, proto, family, net->xfrm.state_hmask);
}
static unsigned int xfrm_seq_hash(struct net *net, u32 seq)
{
+ lockdep_assert_held(&net->xfrm.xfrm_state_lock);
+
return __xfrm_seq_hash(seq, net->xfrm.state_hmask);
}
@@ -944,15 +954,37 @@ xfrm_init_tempstate(struct xfrm_state *x, const struct flowi *fl,
x->props.family = tmpl->encap_family;
}
-static struct xfrm_state *__xfrm_state_lookup(struct net *net, u32 mark,
+struct xfrm_hash_state_ptrs {
+ const struct hlist_head *bydst;
+ const struct hlist_head *bysrc;
+ const struct hlist_head *byspi;
+ unsigned int hmask;
+};
+
+static void xfrm_hash_ptrs_get(const struct net *net, struct xfrm_hash_state_ptrs *ptrs)
+{
+ unsigned int sequence;
+
+ do {
+ sequence = read_seqcount_begin(&net->xfrm.xfrm_state_hash_generation);
+
+ ptrs->bydst = xfrm_state_deref_check(net->xfrm.state_bydst, net);
+ ptrs->bysrc = xfrm_state_deref_check(net->xfrm.state_bysrc, net);
+ ptrs->byspi = xfrm_state_deref_check(net->xfrm.state_byspi, net);
+ ptrs->hmask = net->xfrm.state_hmask;
+ } while (read_seqcount_retry(&net->xfrm.xfrm_state_hash_generation, sequence));
+}
+
+static struct xfrm_state *__xfrm_state_lookup(const struct xfrm_hash_state_ptrs *state_ptrs,
+ u32 mark,
const xfrm_address_t *daddr,
__be32 spi, u8 proto,
unsigned short family)
{
- unsigned int h = xfrm_spi_hash(net, daddr, spi, proto, family);
+ unsigned int h = __xfrm_spi_hash(daddr, spi, proto, family, state_ptrs->hmask);
struct xfrm_state *x;
- hlist_for_each_entry_rcu(x, net->xfrm.state_byspi + h, byspi) {
+ hlist_for_each_entry_rcu(x, state_ptrs->byspi + h, byspi) {
if (x->props.family != family ||
x->id.spi != spi ||
x->id.proto != proto ||
@@ -969,15 +1001,16 @@ static struct xfrm_state *__xfrm_state_lookup(struct net *net, u32 mark,
return NULL;
}
-static struct xfrm_state *__xfrm_state_lookup_byaddr(struct net *net, u32 mark,
+static struct xfrm_state *__xfrm_state_lookup_byaddr(const struct xfrm_hash_state_ptrs *state_ptrs,
+ u32 mark,
const xfrm_address_t *daddr,
const xfrm_address_t *saddr,
u8 proto, unsigned short family)
{
- unsigned int h = xfrm_src_hash(net, daddr, saddr, family);
+ unsigned int h = __xfrm_src_hash(daddr, saddr, family, state_ptrs->hmask);
struct xfrm_state *x;
- hlist_for_each_entry_rcu(x, net->xfrm.state_bysrc + h, bysrc) {
+ hlist_for_each_entry_rcu(x, state_ptrs->bysrc + h, bysrc) {
if (x->props.family != family ||
x->id.proto != proto ||
!xfrm_addr_equal(&x->id.daddr, daddr, family) ||
@@ -997,14 +1030,17 @@ static struct xfrm_state *__xfrm_state_lookup_byaddr(struct net *net, u32 mark,
static inline struct xfrm_state *
__xfrm_state_locate(struct xfrm_state *x, int use_spi, int family)
{
+ struct xfrm_hash_state_ptrs state_ptrs;
struct net *net = xs_net(x);
u32 mark = x->mark.v & x->mark.m;
+ xfrm_hash_ptrs_get(net, &state_ptrs);
+
if (use_spi)
- return __xfrm_state_lookup(net, mark, &x->id.daddr,
+ return __xfrm_state_lookup(&state_ptrs, mark, &x->id.daddr,
x->id.spi, x->id.proto, family);
else
- return __xfrm_state_lookup_byaddr(net, mark,
+ return __xfrm_state_lookup_byaddr(&state_ptrs, mark,
&x->id.daddr,
&x->props.saddr,
x->id.proto, family);
@@ -1067,6 +1103,7 @@ xfrm_state_find(const xfrm_address_t *daddr, const xfrm_address_t *saddr,
unsigned short family, u32 if_id)
{
static xfrm_address_t saddr_wildcard = { };
+ struct xfrm_hash_state_ptrs state_ptrs;
struct net *net = xp_net(pol);
unsigned int h, h_wildcard;
struct xfrm_state *x, *x0, *to_put;
@@ -1083,8 +1120,10 @@ xfrm_state_find(const xfrm_address_t *daddr, const xfrm_address_t *saddr,
sequence = read_seqcount_begin(&net->xfrm.xfrm_state_hash_generation);
rcu_read_lock();
- h = xfrm_dst_hash(net, daddr, saddr, tmpl->reqid, encap_family);
- hlist_for_each_entry_rcu(x, net->xfrm.state_bydst + h, bydst) {
+ xfrm_hash_ptrs_get(net, &state_ptrs);
+
+ h = __xfrm_dst_hash(daddr, saddr, tmpl->reqid, encap_family, state_ptrs.hmask);
+ hlist_for_each_entry_rcu(x, state_ptrs.bydst + h, bydst) {
if (x->props.family == encap_family &&
x->props.reqid == tmpl->reqid &&
(mark & x->mark.m) == x->mark.v &&
@@ -1100,8 +1139,9 @@ xfrm_state_find(const xfrm_address_t *daddr, const xfrm_address_t *saddr,
if (best || acquire_in_progress)
goto found;
- h_wildcard = xfrm_dst_hash(net, daddr, &saddr_wildcard, tmpl->reqid, encap_family);
- hlist_for_each_entry_rcu(x, net->xfrm.state_bydst + h_wildcard, bydst) {
+ h_wildcard = __xfrm_dst_hash(daddr, &saddr_wildcard, tmpl->reqid,
+ encap_family, state_ptrs.hmask);
+ hlist_for_each_entry_rcu(x, state_ptrs.bydst + h_wildcard, bydst) {
if (x->props.family == encap_family &&
x->props.reqid == tmpl->reqid &&
(mark & x->mark.m) == x->mark.v &&
@@ -1119,7 +1159,7 @@ xfrm_state_find(const xfrm_address_t *daddr, const xfrm_address_t *saddr,
x = best;
if (!x && !error && !acquire_in_progress) {
if (tmpl->id.spi &&
- (x0 = __xfrm_state_lookup(net, mark, daddr, tmpl->id.spi,
+ (x0 = __xfrm_state_lookup(&state_ptrs, mark, daddr, tmpl->id.spi,
tmpl->id.proto, encap_family)) != NULL) {
to_put = x0;
error = -EEXIST;
@@ -1264,6 +1304,26 @@ struct xfrm_state *xfrm_state_lookup_byspi(struct net *net, __be32 spi,
}
EXPORT_SYMBOL(xfrm_state_lookup_byspi);
+static struct xfrm_state *xfrm_state_lookup_spi_proto(struct net *net, __be32 spi, u8 proto)
+{
+ struct xfrm_state *x;
+ unsigned int i;
+
+ rcu_read_lock();
+ for (i = 0; i <= net->xfrm.state_hmask; i++) {
+ hlist_for_each_entry_rcu(x, &net->xfrm.state_byspi[i], byspi) {
+ if (x->id.spi == spi && x->id.proto == proto) {
+ if (!xfrm_state_hold_rcu(x))
+ continue;
+ rcu_read_unlock();
+ return x;
+ }
+ }
+ }
+ rcu_read_unlock();
+ return NULL;
+}
+
static void __xfrm_state_insert(struct xfrm_state *x)
{
struct net *net = xs_net(x);
@@ -1807,10 +1867,13 @@ struct xfrm_state *
xfrm_state_lookup(struct net *net, u32 mark, const xfrm_address_t *daddr, __be32 spi,
u8 proto, unsigned short family)
{
+ struct xfrm_hash_state_ptrs state_ptrs;
struct xfrm_state *x;
rcu_read_lock();
- x = __xfrm_state_lookup(net, mark, daddr, spi, proto, family);
+ xfrm_hash_ptrs_get(net, &state_ptrs);
+
+ x = __xfrm_state_lookup(&state_ptrs, mark, daddr, spi, proto, family);
rcu_read_unlock();
return x;
}
@@ -1821,10 +1884,14 @@ xfrm_state_lookup_byaddr(struct net *net, u32 mark,
const xfrm_address_t *daddr, const xfrm_address_t *saddr,
u8 proto, unsigned short family)
{
+ struct xfrm_hash_state_ptrs state_ptrs;
struct xfrm_state *x;
spin_lock_bh(&net->xfrm.xfrm_state_lock);
- x = __xfrm_state_lookup_byaddr(net, mark, daddr, saddr, proto, family);
+
+ xfrm_hash_ptrs_get(net, &state_ptrs);
+
+ x = __xfrm_state_lookup_byaddr(&state_ptrs, mark, daddr, saddr, proto, family);
spin_unlock_bh(&net->xfrm.xfrm_state_lock);
return x;
}
@@ -2044,10 +2111,8 @@ int xfrm_alloc_spi(struct xfrm_state *x, u32 low, u32 high)
unsigned int h;
struct xfrm_state *x0;
int err = -ENOENT;
- __be32 minspi = htonl(low);
- __be32 maxspi = htonl(high);
+ u32 range = high - low + 1;
__be32 newspi = 0;
- u32 mark = x->mark.v & x->mark.m;
spin_lock_bh(&x->lock);
if (x->km.state == XFRM_STATE_DEAD)
@@ -2059,33 +2124,30 @@ int xfrm_alloc_spi(struct xfrm_state *x, u32 low, u32 high)
err = -ENOENT;
- if (minspi == maxspi) {
- x0 = xfrm_state_lookup(net, mark, &x->id.daddr, minspi, x->id.proto, x->props.family);
- if (x0) {
- xfrm_state_put(x0);
+ for (h = 0; h < range; h++) {
+ u32 spi = (low == high) ? low : low + prandom_u32_max(high - low + 1);
+ newspi = htonl(spi);
+
+ spin_lock_bh(&net->xfrm.xfrm_state_lock);
+ x0 = xfrm_state_lookup_spi_proto(net, newspi, x->id.proto);
+ if (!x0) {
+ x->id.spi = newspi;
+ h = xfrm_spi_hash(net, &x->id.daddr, newspi, x->id.proto, x->props.family);
+ hlist_add_head_rcu(&x->byspi, net->xfrm.state_byspi + h);
+ spin_unlock_bh(&net->xfrm.xfrm_state_lock);
+ err = 0;
goto unlock;
}
- newspi = minspi;
- } else {
- u32 spi = 0;
- for (h = 0; h < high-low+1; h++) {
- spi = low + prandom_u32_max(high - low + 1);
- x0 = xfrm_state_lookup(net, mark, &x->id.daddr, htonl(spi), x->id.proto, x->props.family);
- if (x0 == NULL) {
- newspi = htonl(spi);
- break;
- }
- xfrm_state_put(x0);
- }
- }
- if (newspi) {
- spin_lock_bh(&net->xfrm.xfrm_state_lock);
- x->id.spi = newspi;
- h = xfrm_spi_hash(net, &x->id.daddr, x->id.spi, x->id.proto, x->props.family);
- hlist_add_head_rcu(&x->byspi, net->xfrm.state_byspi + h);
+ xfrm_state_put(x0);
spin_unlock_bh(&net->xfrm.xfrm_state_lock);
- err = 0;
+ if (signal_pending(current)) {
+ err = -ERESTARTSYS;
+ goto unlock;
+ }
+
+ if (low == high)
+ break;
}
unlock:
diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c
index 6fb79def144c..9041a9885854 100644
--- a/net/xfrm/xfrm_user.c
+++ b/net/xfrm/xfrm_user.c
@@ -2366,7 +2366,8 @@ static inline unsigned int xfrm_aevent_msgsize(struct xfrm_state *x)
+ nla_total_size_64bit(sizeof(struct xfrm_lifetime_cur))
+ nla_total_size(sizeof(struct xfrm_mark))
+ nla_total_size(4) /* XFRM_AE_RTHR */
- + nla_total_size(4); /* XFRM_AE_ETHR */
+ + nla_total_size(4) /* XFRM_AE_ETHR */
+ + nla_total_size(sizeof(x->if_id)); /* XFRMA_IF_ID */
}
static int build_aevent(struct sk_buff *skb, struct xfrm_state *x, const struct km_event *c)
@@ -2467,7 +2468,12 @@ static int xfrm_get_ae(struct sk_buff *skb, struct nlmsghdr *nlh,
c.portid = nlh->nlmsg_pid;
err = build_aevent(r_skb, x, &c);
- BUG_ON(err < 0);
+ if (err < 0) {
+ spin_unlock_bh(&x->lock);
+ xfrm_state_put(x);
+ kfree_skb(r_skb);
+ return err;
+ }
err = nlmsg_unicast(xfrm_net_nlsk(net, skb), r_skb, NETLINK_CB(skb).portid);
spin_unlock_bh(&x->lock);
diff --git a/samples/ftrace/ftrace-direct-multi-modify.c b/samples/ftrace/ftrace-direct-multi-modify.c
index 6f23b1413745..6859bc931ad0 100644
--- a/samples/ftrace/ftrace-direct-multi-modify.c
+++ b/samples/ftrace/ftrace-direct-multi-modify.c
@@ -140,9 +140,16 @@ static int __init ftrace_direct_multi_init(void)
ret = register_ftrace_direct_multi(&direct, my_tramp);
- if (!ret)
- simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn");
- return ret;
+ if (ret)
+ return ret;
+ simple_tsk = kthread_run(simple_thread, NULL, "event-sample-fn");
+ if (IS_ERR(simple_tsk)) {
+ unregister_ftrace_direct_multi(&direct, my_tramp);
+ ftrace_free_filter(&direct);
+ return PTR_ERR(simple_tsk);
+ }
+
+ return 0;
}
static void __exit ftrace_direct_multi_exit(void)
diff --git a/samples/pidfd/Makefile b/samples/pidfd/Makefile
index 9754e2d81f70..d82409f7061f 100644
--- a/samples/pidfd/Makefile
+++ b/samples/pidfd/Makefile
@@ -1,4 +1,4 @@
# SPDX-License-Identifier: GPL-2.0
-usertprogs-always-y += pidfd-metadata
+userprogs-always-y += pidfd-metadata
userccflags += -I usr/include
diff --git a/security/apparmor/domain.c b/security/apparmor/domain.c
index a6308d103f76..e844e0e48943 100644
--- a/security/apparmor/domain.c
+++ b/security/apparmor/domain.c
@@ -1006,8 +1006,8 @@ static struct aa_label *build_change_hat(struct aa_profile *profile,
if (!hat) {
error = -ENOENT;
if (COMPLAIN_MODE(profile)) {
- hat = aa_new_null_profile(profile, true, name,
- GFP_KERNEL);
+ hat = __aa_new_null_profile(profile, true, name,
+ GFP_KERNEL);
if (!hat) {
info = "failed null profile create";
error = -ENOMEM;
@@ -1043,6 +1043,7 @@ static struct aa_label *change_hat(struct aa_label *label, const char *hats[],
bool sibling = false;
const char *name, *info = NULL;
int i, error;
+ bool needput = false;
AA_BUG(!label);
AA_BUG(!hats);
@@ -1055,7 +1056,6 @@ static struct aa_label *change_hat(struct aa_label *label, const char *hats[],
* the profiles and label, we can rely on the namespaces being live
* and avoid incrementing their refcounts while grabbing the lock.
*/
- label = aa_get_label(label);
ns = labels_ns(label);
retry:
@@ -1063,15 +1063,19 @@ static struct aa_label *change_hat(struct aa_label *label, const char *hats[],
if (label_is_stale(label)) {
new = aa_get_newest_label(label);
new_ns = labels_ns(new);
+
+ if (needput)
+ /* aa_put_label() is safe to call when under lock */
+ aa_put_label(label);
+ label = new;
+ needput = true;
+ /* check if replaced with label in parent ns, and lock there */
if (new_ns != ns) {
- aa_put_label(new);
mutex_unlock(&ns->lock);
ns = new_ns;
- label = new;
+ /* retry will bottom out at the root of the tree */
goto retry;
}
- aa_put_label(label);
- label = new;
}
if (PROFILE_IS_HAT(labels_profile(label)))
@@ -1082,7 +1086,8 @@ static struct aa_label *change_hat(struct aa_label *label, const char *hats[],
name = hats[i];
label_for_each_in_ns(it, labels_ns(label), label, profile) {
if (sibling && PROFILE_IS_HAT(profile)) {
- root = aa_get_profile(profile->parent);
+ root = aa_get_profile(rcu_dereference_protected(profile->parent,
+ mutex_is_locked(&ns->lock)));
} else if (!sibling && !PROFILE_IS_HAT(profile)) {
root = aa_get_profile(profile);
} else { /* conflicting change type */
@@ -1142,6 +1147,8 @@ static struct aa_label *change_hat(struct aa_label *label, const char *hats[],
}
}
mutex_unlock(&ns->lock);
+ if (needput)
+ aa_put_label(label);
return ERR_PTR(error);
build:
@@ -1154,6 +1161,8 @@ static struct aa_label *change_hat(struct aa_label *label, const char *hats[],
goto fail;
} /* else if (IS_ERR) build_change_hat has logged error so return new */
mutex_unlock(&ns->lock);
+ if (needput)
+ aa_put_label(label);
return new;
}
@@ -1463,7 +1472,7 @@ int aa_change_profile(const char *fqname, int flags)
}
/* full transition will be built in exec path */
- error = aa_set_current_onexec(target, stack);
+ aa_set_current_onexec(target, stack);
}
goto audit;
diff --git a/security/apparmor/include/cred.h b/security/apparmor/include/cred.h
index 0b9ae4804ef7..ea3819827753 100644
--- a/security/apparmor/include/cred.h
+++ b/security/apparmor/include/cred.h
@@ -161,13 +161,9 @@ static inline struct aa_label *begin_current_label_crit_section(void)
{
struct aa_label *label = aa_current_raw_label();
- might_sleep();
-
if (label_is_stale(label)) {
label = aa_get_newest_label(label);
- if (aa_replace_current_label(label) == 0)
- /* task cred will keep the reference */
- aa_put_label(label);
+ aa_schedule_stale_label_replacement();
}
return label;
diff --git a/security/apparmor/include/label.h b/security/apparmor/include/label.h
index 9887bfbab0f3..43565faae2ba 100644
--- a/security/apparmor/include/label.h
+++ b/security/apparmor/include/label.h
@@ -22,7 +22,7 @@ struct aa_ns;
#define LOCAL_VEC_ENTRIES 8
#define DEFINE_VEC(T, V) \
- struct aa_ ## T *(_ ## V ## _localtmp)[LOCAL_VEC_ENTRIES]; \
+ struct aa_ ## T *(_ ## V ## _localtmp)[LOCAL_VEC_ENTRIES + 1]; \
struct aa_ ## T **(V)
#define vec_setup(T, V, N, GFP) \
@@ -30,10 +30,10 @@ struct aa_ns;
if ((N) <= LOCAL_VEC_ENTRIES) { \
typeof(N) i; \
(V) = (_ ## V ## _localtmp); \
- for (i = 0; i < (N); i++) \
+ for (i = 0; i <= (N); i++) \
(V)[i] = NULL; \
} else \
- (V) = kzalloc(sizeof(struct aa_ ## T *) * (N), (GFP)); \
+ (V) = kcalloc((N) + 1, sizeof(struct aa_ ## T *), (GFP)); \
(V) ? 0 : -ENOMEM; \
})
diff --git a/security/apparmor/include/match.h b/security/apparmor/include/match.h
index 611ae908469b..72562aaab9ae 100644
--- a/security/apparmor/include/match.h
+++ b/security/apparmor/include/match.h
@@ -105,16 +105,18 @@ struct aa_dfa {
extern struct aa_dfa *nulldfa;
extern struct aa_dfa *stacksplitdfa;
-#define byte_to_byte(X) (X)
-
#define UNPACK_ARRAY(TABLE, BLOB, LEN, TTYPE, BTYPE, NTOHX) \
do { \
typeof(LEN) __i; \
TTYPE *__t = (TTYPE *) TABLE; \
BTYPE *__b = (BTYPE *) BLOB; \
- for (__i = 0; __i < LEN; __i++) { \
- __t[__i] = NTOHX(__b[__i]); \
- } \
+ BUILD_BUG_ON(sizeof(TTYPE) != sizeof(BTYPE)); \
+ if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN)) \
+ memcpy(__t, __b, (LEN) * sizeof(BTYPE)); \
+ else /* copy & convert from big-endian */ \
+ for (__i = 0; __i < LEN; __i++) { \
+ __t[__i] = NTOHX(&__b[__i]); \
+ } \
} while (0)
static inline size_t table_size(size_t len, size_t el_size)
diff --git a/security/apparmor/include/policy.h b/security/apparmor/include/policy.h
index 44471bb05762..ae8a3626ac75 100644
--- a/security/apparmor/include/policy.h
+++ b/security/apparmor/include/policy.h
@@ -181,6 +181,8 @@ void aa_add_profile(struct aa_policy *common, struct aa_profile *profile);
void aa_free_proxy_kref(struct kref *kref);
struct aa_profile *aa_alloc_profile(const char *name, struct aa_proxy *proxy,
gfp_t gfp);
+struct aa_profile *__aa_new_null_profile(struct aa_profile *parent, bool hat,
+ const char *base, gfp_t gfp);
struct aa_profile *aa_new_null_profile(struct aa_profile *parent, bool hat,
const char *base, gfp_t gfp);
void aa_free_profile(struct aa_profile *profile);
diff --git a/security/apparmor/include/task.h b/security/apparmor/include/task.h
index 13437d62c70f..3c203b622b43 100644
--- a/security/apparmor/include/task.h
+++ b/security/apparmor/include/task.h
@@ -21,16 +21,23 @@ static inline struct aa_task_ctx *task_ctx(struct task_struct *task)
* @onexec: profile to transition to on next exec (MAY BE NULL)
* @previous: profile the task may return to (MAY BE NULL)
* @token: magic value the task must know for returning to @previous_profile
+ * @label_replacement_tw: for aa_schedule_stale_label_replacement()
+ * @label_replacement_pending: is @label_replacement_tw pending?
+ *
+ * When changing this, check if aa_dup_task_ctx() needs to be updated.
*/
struct aa_task_ctx {
struct aa_label *nnp;
struct aa_label *onexec;
struct aa_label *previous;
u64 token;
+ struct callback_head label_replacement_tw;
+ bool label_replacement_pending;
};
int aa_replace_current_label(struct aa_label *label);
-int aa_set_current_onexec(struct aa_label *label, bool stack);
+void aa_schedule_stale_label_replacement(void);
+void aa_set_current_onexec(struct aa_label *label, bool stack);
int aa_set_current_hat(struct aa_label *label, u64 token);
int aa_restore_previous_label(u64 cookie);
struct aa_label *aa_get_task_label(struct task_struct *task);
@@ -56,10 +63,10 @@ static inline void aa_free_task_ctx(struct aa_task_ctx *ctx)
static inline void aa_dup_task_ctx(struct aa_task_ctx *new,
const struct aa_task_ctx *old)
{
- *new = *old;
- aa_get_label(new->nnp);
- aa_get_label(new->previous);
- aa_get_label(new->onexec);
+ new->nnp = aa_get_label(old->nnp);
+ new->onexec = aa_get_label(old->onexec);
+ new->previous = aa_get_label(old->previous);
+ new->token = old->token;
}
/**
diff --git a/security/apparmor/match.c b/security/apparmor/match.c
index 8972d1b57b7a..c284614dc028 100644
--- a/security/apparmor/match.c
+++ b/security/apparmor/match.c
@@ -8,6 +8,7 @@
* Copyright 2009-2012 Canonical Ltd.
*/
+#include <asm/unaligned.h>
#include <linux/errno.h>
#include <linux/kernel.h>
#include <linux/mm.h>
@@ -86,11 +87,11 @@ static struct table_header *unpack_table(char *blob, size_t bsize)
/* loaded td_id's start at 1, subtract 1 now to avoid doing
* it every time we use td_id as an index
*/
- th.td_id = be16_to_cpu(*(__be16 *) (blob)) - 1;
+ th.td_id = get_unaligned_be16(blob) - 1;
if (th.td_id > YYTD_ID_MAX)
goto out;
- th.td_flags = be16_to_cpu(*(__be16 *) (blob + 2));
- th.td_lolen = be32_to_cpu(*(__be32 *) (blob + 8));
+ th.td_flags = get_unaligned_be16(blob + 2);
+ th.td_lolen = get_unaligned_be32(blob + 8);
blob += sizeof(struct table_header);
if (!(th.td_flags == YYTD_DATA16 || th.td_flags == YYTD_DATA32 ||
@@ -110,14 +111,13 @@ static struct table_header *unpack_table(char *blob, size_t bsize)
table->td_flags = th.td_flags;
table->td_lolen = th.td_lolen;
if (th.td_flags == YYTD_DATA8)
- UNPACK_ARRAY(table->td_data, blob, th.td_lolen,
- u8, u8, byte_to_byte);
+ memcpy(table->td_data, blob, th.td_lolen);
else if (th.td_flags == YYTD_DATA16)
UNPACK_ARRAY(table->td_data, blob, th.td_lolen,
- u16, __be16, be16_to_cpu);
+ u16, __be16, get_unaligned_be16);
else if (th.td_flags == YYTD_DATA32)
UNPACK_ARRAY(table->td_data, blob, th.td_lolen,
- u32, __be32, be32_to_cpu);
+ u32, __be32, get_unaligned_be32);
else
goto fail;
/* if table was vmalloced make sure the page tables are synced
@@ -337,14 +337,14 @@ struct aa_dfa *aa_dfa_unpack(void *blob, size_t size, int flags)
if (size < sizeof(struct table_set_header))
goto fail;
- if (ntohl(*(__be32 *) data) != YYTH_MAGIC)
+ if (get_unaligned_be32(data) != YYTH_MAGIC)
goto fail;
- hsize = ntohl(*(__be32 *) (data + 4));
+ hsize = get_unaligned_be32(data + 4);
if (size < hsize)
goto fail;
- dfa->flags = ntohs(*(__be16 *) (data + 12));
+ dfa->flags = get_unaligned_be16(data + 12);
if (dfa->flags & ~(YYTH_FLAGS))
goto fail;
@@ -353,7 +353,7 @@ struct aa_dfa *aa_dfa_unpack(void *blob, size_t size, int flags)
* if (dfa->flags & YYTH_FLAGS_OOB_TRANS) {
* if (hsize < 16 + 4)
* goto fail;
- * dfa->max_oob = ntol(*(__be32 *) (data + 16));
+ * dfa->max_oob = get_unaligned_be32(data + 16);
* if (dfa->max <= MAX_OOB_SUPPORTED) {
* pr_err("AppArmor DFA OOB greater than supported\n");
* goto fail;
diff --git a/security/apparmor/policy.c b/security/apparmor/policy.c
index 257ca96a3d06..13d3041200cb 100644
--- a/security/apparmor/policy.c
+++ b/security/apparmor/policy.c
@@ -507,7 +507,7 @@ struct aa_profile *aa_fqlookupn_profile(struct aa_label *base,
}
/**
- * aa_new_null_profile - create or find a null-X learning profile
+ * __aa_new_null_profile - create or find a null-X learning profile
* @parent: profile that caused this profile to be created (NOT NULL)
* @hat: true if the null- learning profile is a hat
* @base: name to base the null profile off of
@@ -524,14 +524,15 @@ struct aa_profile *aa_fqlookupn_profile(struct aa_label *base,
*
* Returns: new refcounted profile else NULL on failure
*/
-struct aa_profile *aa_new_null_profile(struct aa_profile *parent, bool hat,
- const char *base, gfp_t gfp)
+struct aa_profile *__aa_new_null_profile(struct aa_profile *parent, bool hat,
+ const char *base, gfp_t gfp)
{
struct aa_profile *p, *profile;
const char *bname;
char *name = NULL;
AA_BUG(!parent);
+ AA_BUG(!mutex_is_locked(&parent->ns->lock));
if (base) {
name = kmalloc(strlen(parent->base.hname) + 8 + strlen(base),
@@ -572,7 +573,6 @@ struct aa_profile *aa_new_null_profile(struct aa_profile *parent, bool hat,
profile->file.dfa = aa_get_dfa(nulldfa);
profile->policy.dfa = aa_get_dfa(nulldfa);
- mutex_lock_nested(&profile->ns->lock, profile->ns->level);
p = __find_child(&parent->base.profiles, bname);
if (p) {
aa_free_profile(profile);
@@ -580,7 +580,6 @@ struct aa_profile *aa_new_null_profile(struct aa_profile *parent, bool hat,
} else {
__add_profile(&parent->base.profiles, profile);
}
- mutex_unlock(&profile->ns->lock);
/* refcount released by caller */
out:
@@ -594,6 +593,18 @@ struct aa_profile *aa_new_null_profile(struct aa_profile *parent, bool hat,
return NULL;
}
+struct aa_profile *aa_new_null_profile(struct aa_profile *parent, bool hat,
+ const char *base, gfp_t gfp)
+{
+ struct aa_profile *profile;
+
+ mutex_lock_nested(&parent->ns->lock, parent->ns->level);
+ profile = __aa_new_null_profile(parent, hat, base, gfp);
+ mutex_unlock(&parent->ns->lock);
+
+ return profile;
+}
+
/**
* replacement_allowed - test to see if replacement is allowed
* @profile: profile to test if it can be replaced (MAYBE NULL)
diff --git a/security/apparmor/task.c b/security/apparmor/task.c
index 503dc0877fb1..1d60b1521ed8 100644
--- a/security/apparmor/task.c
+++ b/security/apparmor/task.c
@@ -14,6 +14,7 @@
#include <linux/gfp.h>
#include <linux/ptrace.h>
+#include <linux/task_work.h>
#include "include/audit.h"
#include "include/cred.h"
@@ -88,14 +89,39 @@ int aa_replace_current_label(struct aa_label *label)
return 0;
}
+static void aa_replace_stale_label_tw_func(struct callback_head *tw)
+{
+ struct aa_task_ctx *ctx = task_ctx(current);
+ struct aa_label *label;
+
+ ctx->label_replacement_pending = false;
+ label = aa_current_raw_label();
+ if (!label_is_stale(label))
+ return;
+ label = aa_get_newest_label(label);
+ aa_replace_current_label(label);
+ aa_put_label(label);
+}
+
+/* replace the current task's stale label on syscall return */
+void aa_schedule_stale_label_replacement(void)
+{
+ struct aa_task_ctx *ctx = task_ctx(current);
+
+ if (ctx->label_replacement_pending)
+ return;
+ init_task_work(&ctx->label_replacement_tw, aa_replace_stale_label_tw_func);
+ if (task_work_add(current, &ctx->label_replacement_tw, TWA_RESUME) == 0)
+ ctx->label_replacement_pending = true;
+}
+
/**
* aa_set_current_onexec - set the tasks change_profile to happen onexec
* @label: system label to set at exec (MAYBE NULL to clear value)
* @stack: whether stacking should be done
- * Returns: 0 or error on failure
*/
-int aa_set_current_onexec(struct aa_label *label, bool stack)
+void aa_set_current_onexec(struct aa_label *label, bool stack)
{
struct aa_task_ctx *ctx = task_ctx(current);
@@ -103,8 +129,6 @@ int aa_set_current_onexec(struct aa_label *label, bool stack)
aa_put_label(ctx->onexec);
ctx->onexec = label;
ctx->token = stack;
-
- return 0;
}
/**
diff --git a/security/integrity/ima/ima_appraise.c b/security/integrity/ima/ima_appraise.c
index f9d0bd4d3613..c76f5030fb55 100644
--- a/security/integrity/ima/ima_appraise.c
+++ b/security/integrity/ima/ima_appraise.c
@@ -676,6 +676,15 @@ static int ima_protect_xattr(struct dentry *dentry, const char *xattr_name,
return 0;
}
+/*
+ * ima_reset_appraise_flags - reset ima_iint_cache flags
+ *
+ * @digsig: whether to clear/set IMA_DIGSIG flag, tristate values
+ * 0: clear IMA_DIGSIG
+ * 1: set IMA_DIGSIG
+ * -1: don't change IMA_DIGSIG
+ *
+ */
static void ima_reset_appraise_flags(struct inode *inode, int digsig)
{
struct integrity_iint_cache *iint;
@@ -688,9 +697,9 @@ static void ima_reset_appraise_flags(struct inode *inode, int digsig)
return;
iint->measured_pcrs = 0;
set_bit(IMA_CHANGE_XATTR, &iint->atomic_flags);
- if (digsig)
+ if (digsig == 1)
set_bit(IMA_DIGSIG, &iint->atomic_flags);
- else
+ else if (digsig == 0)
clear_bit(IMA_DIGSIG, &iint->atomic_flags);
}
@@ -777,6 +786,8 @@ int ima_inode_setxattr(struct dentry *dentry, const char *xattr_name,
digsig = (xvalue->type == EVM_IMA_XATTR_DIGSIG);
} else if (!strcmp(xattr_name, XATTR_NAME_EVM) && xattr_value_len > 0) {
digsig = (xvalue->type == EVM_XATTR_PORTABLE_DIGSIG);
+ } else {
+ digsig = -1;
}
if (result == 1 || evm_revalidate_status(xattr_name)) {
ima_reset_appraise_flags(d_backing_inode(dentry), digsig);
@@ -788,11 +799,13 @@ int ima_inode_setxattr(struct dentry *dentry, const char *xattr_name,
int ima_inode_removexattr(struct dentry *dentry, const char *xattr_name)
{
- int result;
+ int result, digsig = -1;
result = ima_protect_xattr(dentry, xattr_name, NULL, 0);
if (result == 1 || evm_revalidate_status(xattr_name)) {
- ima_reset_appraise_flags(d_backing_inode(dentry), 0);
+ if (!strcmp(xattr_name, XATTR_NAME_IMA))
+ digsig = 0;
+ ima_reset_appraise_flags(d_backing_inode(dentry), digsig);
if (result == 1)
result = 0;
}
diff --git a/security/integrity/ima/ima_kexec.c b/security/integrity/ima/ima_kexec.c
index 419dc405c831..9a63c9a327d4 100644
--- a/security/integrity/ima/ima_kexec.c
+++ b/security/integrity/ima/ima_kexec.c
@@ -12,6 +12,8 @@
#include <linux/kexec.h>
#include <linux/of.h>
#include <linux/ima.h>
+#include <linux/mm.h>
+#include <linux/overflow.h>
#include "ima.h"
#ifdef CONFIG_IMA_KEXEC
@@ -164,3 +166,36 @@ void __init ima_load_kexec_buffer(void)
pr_debug("Error restoring the measurement list: %d\n", rc);
}
}
+
+/*
+ * ima_validate_range - verify a physical buffer lies in addressable RAM
+ * @phys: physical start address of the buffer from previous kernel
+ * @size: size of the buffer
+ *
+ * On success return 0. On failure returns -EINVAL so callers can skip
+ * restoring.
+ */
+int ima_validate_range(phys_addr_t phys, size_t size)
+{
+ unsigned long start_pfn, end_pfn;
+ phys_addr_t end_phys;
+
+ if (check_add_overflow(phys, (phys_addr_t)size - 1, &end_phys))
+ return -EINVAL;
+
+ start_pfn = PHYS_PFN(phys);
+ end_pfn = PHYS_PFN(end_phys);
+
+#ifdef CONFIG_X86
+ if (!pfn_range_is_mapped(start_pfn, end_pfn))
+#else
+ if (!page_is_ram(start_pfn) || !page_is_ram(end_pfn))
+#endif
+ {
+ pr_warn("IMA: previous kernel measurement buffer %pa (size 0x%zx) lies outside available memory\n",
+ &phys, size);
+ return -EINVAL;
+ }
+
+ return 0;
+}
diff --git a/security/keys/key.c b/security/keys/key.c
index f2a84d86eab4..e671876619fc 100644
--- a/security/keys/key.c
+++ b/security/keys/key.c
@@ -586,11 +586,18 @@ int key_reject_and_link(struct key *key,
ret = -EBUSY;
if (keyring) {
- if (keyring->restrict_link)
- return -EPERM;
-
link_ret = __key_link_lock(keyring, &key->index_key);
if (link_ret == 0) {
+ /*
+ * Check the restriction under the keyring semaphore.
+ * keyring_restrict() installs it while holding the
+ * semaphore, so testing it beforehand races with a
+ * concurrent restriction install.
+ */
+ if (keyring->restrict_link) {
+ __key_link_end(keyring, &key->index_key, edit);
+ return -EPERM;
+ }
link_ret = __key_link_begin(keyring, &key->index_key, &edit);
if (link_ret < 0)
__key_link_end(keyring, &key->index_key, edit);
@@ -845,9 +852,6 @@ key_ref_t key_create_or_update(key_ref_t keyring_ref,
key_check(keyring);
- if (!(flags & KEY_ALLOC_BYPASS_RESTRICTION))
- restrict_link = keyring->restrict_link;
-
key_ref = ERR_PTR(-ENOTDIR);
if (keyring->type != &key_type_keyring)
goto error_put_type;
@@ -885,6 +889,9 @@ key_ref_t key_create_or_update(key_ref_t keyring_ref,
goto error_link_end;
}
+ if (!(flags & KEY_ALLOC_BYPASS_RESTRICTION))
+ restrict_link = keyring->restrict_link;
+
if (restrict_link && restrict_link->check) {
ret = restrict_link->check(keyring, index_key.type,
&prep.payload, restrict_link->key);
diff --git a/security/keys/keyctl.c b/security/keys/keyctl.c
index 127c17191785..e9fa1b94d0d0 100644
--- a/security/keys/keyctl.c
+++ b/security/keys/keyctl.c
@@ -677,12 +677,14 @@ long keyctl_describe_key(key_serial_t keyid,
/* calculate how much information we're going to return */
ret = -ENOMEM;
+ down_read(&key->sem);
infobuf = kasprintf(GFP_KERNEL,
"%s;%d;%d;%08x;",
key->type->name,
from_kuid_munged(current_user_ns(), key->uid),
from_kgid_munged(current_user_ns(), key->gid),
key->perm);
+ up_read(&key->sem);
if (!infobuf)
goto error2;
infolen = strlen(infobuf);
@@ -806,8 +808,12 @@ static long __keyctl_read_key(struct key *key, char *buffer, size_t buflen)
down_read(&key->sem);
ret = key_validate(key);
- if (ret == 0)
- ret = key->type->read(key, buffer, buflen);
+ if (ret == 0) {
+ if (key->type->read)
+ ret = key->type->read(key, buffer, buflen);
+ else
+ ret = -EOPNOTSUPP;
+ }
up_read(&key->sem);
return ret;
}
@@ -861,11 +867,6 @@ long keyctl_read_key(key_serial_t keyid, char __user *buffer, size_t buflen)
/* the key is probably readable - now try to read it */
can_read_key:
- if (!key->type->read) {
- ret = -EOPNOTSUPP;
- goto key_put_out;
- }
-
if (!buffer || !buflen) {
/* Get the key length from the read method */
ret = __keyctl_read_key(key, NULL, 0);
diff --git a/security/keys/keyring.c b/security/keys/keyring.c
index fd95a0eb7a46..74ed1372d22c 100644
--- a/security/keys/keyring.c
+++ b/security/keys/keyring.c
@@ -14,6 +14,9 @@
#include <linux/err.h>
#include <linux/user_namespace.h>
#include <linux/nsproxy.h>
+#include <linux/once.h>
+#include <linux/random.h>
+#include <linux/siphash.h>
#include <keys/keyring-type.h>
#include <keys/user-type.h>
#include <linux/assoc_array_priv.h>
@@ -147,49 +150,46 @@ static int keyring_instantiate(struct key *keyring,
}
/*
- * Multiply 64-bits by 32-bits to 96-bits and fold back to 64-bit. Ideally we'd
- * fold the carry back too, but that requires inline asm.
+ * Give key types and domains address-independent identities. Zero means that
+ * an ID has not yet been allocated. As with inode sequence numbers used by
+ * futexes, this relies on the 64-bit counter not wrapping in a machine's
+ * lifetime.
*/
-static u64 mult_64x32_and_fold(u64 x, u32 y)
+static u64 key_index_id(atomic64_t *id)
{
- u64 hi = (u64)(u32)(x >> 32) * y;
- u64 lo = (u64)(u32)(x) * y;
- return lo + ((u64)(u32)hi << 32) + (u32)(hi >> 32);
+ static atomic64_t next_id;
+ u64 old, new;
+
+ old = atomic64_read(id);
+ if (likely(old))
+ return old;
+
+ do {
+ new = atomic64_inc_return(&next_id);
+ } while (WARN_ON_ONCE(!new));
+
+ old = 0;
+ if (!atomic64_try_cmpxchg_relaxed(id, &old, new))
+ return old;
+ return new;
}
/*
- * Hash a key type and description.
+ * Hash a key type, domain and description. KEYCTL_READ exposes the tree's
+ * traversal order, so no part of the index may contain kernel addresses.
*/
static void hash_key_type_and_desc(struct keyring_index_key *index_key)
{
const unsigned level_shift = ASSOC_ARRAY_LEVEL_STEP;
const unsigned long fan_mask = ASSOC_ARRAY_FAN_MASK;
- const char *description = index_key->description;
- unsigned long hash, type;
- u32 piece;
+ static siphash_key_t hash_key __read_mostly;
+ unsigned long hash;
u64 acc;
- int n, desc_len = index_key->desc_len;
-
- type = (unsigned long)index_key->type;
- acc = mult_64x32_and_fold(type, desc_len + 13);
- acc = mult_64x32_and_fold(acc, 9207);
- piece = (unsigned long)index_key->domain_tag;
- acc = mult_64x32_and_fold(acc, piece);
- acc = mult_64x32_and_fold(acc, 9207);
-
- for (;;) {
- n = desc_len;
- if (n <= 0)
- break;
- if (n > 4)
- n = 4;
- piece = 0;
- memcpy(&piece, description, n);
- description += n;
- desc_len -= n;
- acc = mult_64x32_and_fold(acc, piece);
- acc = mult_64x32_and_fold(acc, 9207);
- }
+
+ get_random_once(&hash_key, sizeof(hash_key));
+ acc = siphash(index_key->description, index_key->desc_len, &hash_key);
+ acc = siphash_3u64(index_key->type_id, index_key->domain_id, acc,
+ &hash_key);
/* Fold the hash down to 32 bits if need be. */
hash = acc;
@@ -209,7 +209,7 @@ static void hash_key_type_and_desc(struct keyring_index_key *index_key)
/*
* Finalise an index key to include a part of the description actually in the
- * index key, to set the domain tag and to calculate the hash.
+ * index key, to set the domain tag and index IDs, and to calculate the hash.
*/
void key_set_index_key(struct keyring_index_key *index_key)
{
@@ -225,6 +225,14 @@ void key_set_index_key(struct keyring_index_key *index_key)
index_key->domain_tag = &default_domain_tag;
}
+ /* Preserve the IDs when copying an existing key's index. In
+ * particular, its type may already have been unregistered.
+ */
+ if (!index_key->type_id)
+ index_key->type_id = key_index_id(&index_key->type->index_id);
+ if (!index_key->domain_id)
+ index_key->domain_id = key_index_id(&index_key->domain_tag->index_id);
+
hash_key_type_and_desc(index_key);
}
@@ -263,15 +271,19 @@ void key_remove_domain(struct key_tag *domain_tag)
/*
* Build the next index key chunk.
*
+ * The hash and inline description are followed by the 64-bit type and domain
+ * IDs, low word first, and then by the rest of the description.
* We return it one word-sized chunk at a time.
*/
static unsigned long keyring_get_key_chunk(const void *data, int level)
{
+ const unsigned int id_chunks = sizeof(u64) / sizeof(unsigned long);
const struct keyring_index_key *index_key = data;
+ int desc_len = index_key->desc_len;
unsigned long chunk = 0;
- const u8 *d;
- int desc_len = index_key->desc_len, n = sizeof(chunk);
+ int n = sizeof(chunk);
unsigned int offset;
+ const u8 *d;
level /= ASSOC_ARRAY_KEY_CHUNK_SIZE;
switch (level) {
@@ -279,27 +291,30 @@ static unsigned long keyring_get_key_chunk(const void *data, int level)
return index_key->hash;
case 1:
return index_key->x;
- case 2:
- return (unsigned long)index_key->type;
- case 3:
- return (unsigned long)index_key->domain_tag;
- default:
- level -= 4;
- offset = sizeof(index_key->desc) + level * sizeof(long);
- if (desc_len <= offset)
- return 0;
-
- d = index_key->description + offset;
- desc_len -= offset;
- if (desc_len > n)
- desc_len = n;
- d += desc_len;
- do {
- chunk <<= 8;
- chunk |= *--d;
- } while (--desc_len > 0);
- return chunk;
}
+
+ level -= 2;
+ if (level < id_chunks)
+ return index_key->type_id >> (level * ASSOC_ARRAY_KEY_CHUNK_SIZE);
+ level -= id_chunks;
+ if (level < id_chunks)
+ return index_key->domain_id >> (level * ASSOC_ARRAY_KEY_CHUNK_SIZE);
+ level -= id_chunks;
+
+ offset = sizeof(index_key->desc) + level * sizeof(long);
+ if (desc_len <= offset)
+ return 0;
+
+ d = index_key->description + offset;
+ desc_len -= offset;
+ if (desc_len > n)
+ desc_len = n;
+ d += desc_len;
+ do {
+ chunk <<= 8;
+ chunk |= *--d;
+ } while (--desc_len > 0);
+ return chunk;
}
static unsigned long keyring_get_object_key_chunk(const void *object, int level)
@@ -331,6 +346,7 @@ static int keyring_diff_objects(const void *object, const void *data)
const struct keyring_index_key *b = data;
unsigned long seg_a, seg_b;
int level, i;
+ u64 diff;
level = 0;
seg_a = a->hash;
@@ -339,28 +355,22 @@ static int keyring_diff_objects(const void *object, const void *data)
goto differ;
level += ASSOC_ARRAY_KEY_CHUNK_SIZE / 8;
- /* The number of bits contributed by the hash is controlled by a
- * constant in the assoc_array headers. Everything else thereafter we
- * can deal with as being machine word-size dependent.
- */
+ /* The inline description contributes one machine-word-sized chunk. */
seg_a = a->x;
seg_b = b->x;
if ((seg_a ^ seg_b) != 0)
goto differ;
level += sizeof(unsigned long);
- /* The next bit may not work on big endian */
- seg_a = (unsigned long)a->type;
- seg_b = (unsigned long)b->type;
- if ((seg_a ^ seg_b) != 0)
- goto differ;
- level += sizeof(unsigned long);
+ diff = a->type_id ^ b->type_id;
+ if (diff)
+ return level * 8 + __ffs64(diff);
+ level += sizeof(u64);
- seg_a = (unsigned long)a->domain_tag;
- seg_b = (unsigned long)b->domain_tag;
- if ((seg_a ^ seg_b) != 0)
- goto differ;
- level += sizeof(unsigned long);
+ diff = a->domain_id ^ b->domain_id;
+ if (diff)
+ return level * 8 + __ffs64(diff);
+ level += sizeof(u64);
i = sizeof(a->desc);
if (a->desc_len <= i)
@@ -1158,7 +1168,7 @@ struct key *find_keyring_by_name(const char *name, bool uid_keyring)
* grants Search permission and that hasn't been revoked
*/
list_for_each_entry(keyring, &ns->keyring_name_list, name_link) {
- if (!kuid_has_mapping(ns, keyring->user->uid))
+ if (!kuid_has_mapping(ns, keyring->uid))
continue;
if (test_bit(KEY_FLAG_REVOKED, &keyring->flags))
diff --git a/security/keys/persistent.c b/security/keys/persistent.c
index 97af230aa4b2..bdab99539326 100644
--- a/security/keys/persistent.c
+++ b/security/keys/persistent.c
@@ -33,25 +33,31 @@ static int key_create_persistent_register(struct user_namespace *ns)
}
/*
- * Create the persistent keyring for the specified user.
+ * Get or create the persistent keyring for the specified user.
*
* Called with the namespace's sem locked for writing.
+ *
+ * Returns a boolean indicating whether the keyring already existed,
+ * or a negative error. On success, *persistent_ref holds a reference
+ * to the keyring.
*/
-static key_ref_t key_create_persistent(struct user_namespace *ns, kuid_t uid,
- struct keyring_index_key *index_key)
+static int key_get_or_create_persistent(struct user_namespace *ns, kuid_t uid,
+ struct keyring_index_key *index_key,
+ key_ref_t *persistent_ref)
{
struct key *persistent;
- key_ref_t reg_ref, persistent_ref;
+ key_ref_t reg_ref;
if (!ns->persistent_keyring_register) {
- long err = key_create_persistent_register(ns);
+ int err = key_create_persistent_register(ns);
+
if (err < 0)
- return ERR_PTR(err);
+ return err;
} else {
reg_ref = make_key_ref(ns->persistent_keyring_register, true);
- persistent_ref = find_key_to_update(reg_ref, index_key);
- if (persistent_ref)
- return persistent_ref;
+ *persistent_ref = find_key_to_update(reg_ref, index_key);
+ if (*persistent_ref)
+ return 1;
}
persistent = keyring_alloc(index_key->description,
@@ -61,9 +67,10 @@ static key_ref_t key_create_persistent(struct user_namespace *ns, kuid_t uid,
KEY_ALLOC_NOT_IN_QUOTA, NULL,
ns->persistent_keyring_register);
if (IS_ERR(persistent))
- return ERR_CAST(persistent);
+ return PTR_ERR(persistent);
- return make_key_ref(persistent, true);
+ *persistent_ref = make_key_ref(persistent, true);
+ return 0;
}
/*
@@ -78,6 +85,7 @@ static long key_get_persistent(struct user_namespace *ns, kuid_t uid,
key_ref_t reg_ref, persistent_ref;
char buf[32];
long ret;
+ bool created = false;
/* Look in the register if it exists */
memset(&index_key, 0, sizeof(index_key));
@@ -100,23 +108,24 @@ static long key_get_persistent(struct user_namespace *ns, kuid_t uid,
* also need to create the register.
*/
down_write(&ns->keyring_sem);
- persistent_ref = key_create_persistent(ns, uid, &index_key);
+ ret = key_get_or_create_persistent(ns, uid, &index_key,
+ &persistent_ref);
up_write(&ns->keyring_sem);
- if (!IS_ERR(persistent_ref))
- goto found;
-
- return PTR_ERR(persistent_ref);
+ if (ret < 0)
+ return ret;
+ created = ret == 0;
found:
+ persistent = key_ref_to_ptr(persistent_ref);
ret = key_task_permission(persistent_ref, current_cred(), KEY_NEED_LINK);
- if (ret == 0) {
- persistent = key_ref_to_ptr(persistent_ref);
+ if (ret == 0)
ret = key_link(key_ref_to_ptr(dest_ref), persistent);
- if (ret == 0) {
- key_set_timeout(persistent, persistent_keyring_expiry);
- ret = persistent->serial;
- }
- }
+
+ if (ret == 0 || created)
+ key_set_timeout(persistent, persistent_keyring_expiry);
+
+ if (ret == 0)
+ ret = persistent->serial;
key_ref_put(persistent_ref);
return ret;
diff --git a/security/keys/process_keys.c b/security/keys/process_keys.c
index b5d5333ab330..4b2b8a75eb1f 100644
--- a/security/keys/process_keys.c
+++ b/security/keys/process_keys.c
@@ -556,9 +556,8 @@ key_ref_t search_process_keyrings_rcu(struct keyring_search_context *ctx)
) {
const struct cred *cred = ctx->cred;
- if (key_validate(cred->request_key_auth) == 0) {
- rka = ctx->cred->request_key_auth->payload.data[0];
-
+ rka = dereference_key_rcu(cred->request_key_auth);
+ if (rka && key_validate(cred->request_key_auth) == 0) {
//// was search_process_keyrings() [ie. recursive]
ctx->cred = rka->cred;
key_ref = search_cred_keyrings_rcu(ctx);
diff --git a/security/keys/trusted-keys/trusted_tpm1.c b/security/keys/trusted-keys/trusted_tpm1.c
index 4863ee08b7b1..06a2b45ab6d8 100644
--- a/security/keys/trusted-keys/trusted_tpm1.c
+++ b/security/keys/trusted-keys/trusted_tpm1.c
@@ -959,8 +959,10 @@ static int trusted_tpm_unseal(struct trusted_key_payload *p, char *datablob)
ret = tpm2_unseal_trusted(chip, p, options);
else
ret = key_unseal(p, options);
- if (ret < 0)
+ if (ret < 0) {
pr_info("key_unseal failed (%d)\n", ret);
+ goto out;
+ }
if (options->pcrlock) {
ret = pcrlock(options->pcrlock);
@@ -1058,10 +1060,10 @@ static int __init trusted_tpm_init(void)
static void trusted_tpm_exit(void)
{
if (chip) {
+ unregister_key_type(&key_type_trusted);
put_device(&chip->dev);
kfree(digests);
trusted_shash_release();
- unregister_key_type(&key_type_trusted);
}
}
diff --git a/security/keys/trusted-keys/trusted_tpm2.c b/security/keys/trusted-keys/trusted_tpm2.c
index bbd6ffb67cd6..ebd83cba18a7 100644
--- a/security/keys/trusted-keys/trusted_tpm2.c
+++ b/security/keys/trusted-keys/trusted_tpm2.c
@@ -124,7 +124,7 @@ static int tpm2_key_decode(struct trusted_key_payload *payload,
if (ctx.priv_len + ctx.pub_len > MAX_BLOB_SIZE)
return -EINVAL;
- blob = kmalloc(ctx.priv_len + ctx.pub_len + 4, GFP_KERNEL);
+ blob = kmalloc(ctx.priv_len + ctx.pub_len, GFP_KERNEL);
if (!blob)
return -ENOMEM;
diff --git a/security/landlock/fs.c b/security/landlock/fs.c
index 8f20a146648c..4e03da721e63 100644
--- a/security/landlock/fs.c
+++ b/security/landlock/fs.c
@@ -911,17 +911,19 @@ static int current_check_refer_path(struct dentry *const old_dentry,
/*
* old_dentry may be the root of the common mount point and
* !IS_ROOT(old_dentry) at the same time (e.g. with open_tree() and
- * OPEN_TREE_CLONE). We do not need to call dget(old_parent) because
- * we keep a reference to old_dentry.
+ * OPEN_TREE_CLONE). Pin the dentry used as old_parent in either case.
+ * Otherwise, dget_parent() safely fetches and pins the current parent
+ * against a concurrent rename(2).
*/
- old_parent = (old_dentry == mnt_dir.dentry) ? old_dentry :
- old_dentry->d_parent;
+ old_parent = (old_dentry == mnt_dir.dentry) ? dget(old_dentry) :
+ dget_parent(old_dentry);
/* new_dir->dentry is equal to new_dentry->d_parent */
allow_parent1 = collect_domain_accesses(dom, mnt_dir.dentry, old_parent,
&layer_masks_parent1);
allow_parent2 = collect_domain_accesses(
dom, mnt_dir.dentry, new_dir->dentry, &layer_masks_parent2);
+ dput(old_parent);
if (allow_parent1 && allow_parent2)
return 0;
diff --git a/security/selinux/avc.c b/security/selinux/avc.c
index dcc8e1df4d7c..e26f9b01b730 100644
--- a/security/selinux/avc.c
+++ b/security/selinux/avc.c
@@ -589,7 +589,7 @@ static int avc_latest_notif_update(struct selinux_avc *avc,
}
} else {
if (seqno > avc->avc_cache.latest_notif)
- avc->avc_cache.latest_notif = seqno;
+ WRITE_ONCE(avc->avc_cache.latest_notif, seqno);
}
spin_unlock_irqrestore(¬if_lock, flag);
@@ -1202,7 +1202,8 @@ int avc_has_perm(struct selinux_state *state, u32 ssid, u32 tsid, u16 tclass,
u32 avc_policy_seqno(struct selinux_state *state)
{
- return state->avc->avc_cache.latest_notif;
+ /* Writers hold notif_lock; readers only sample the sequence number. */
+ return READ_ONCE(state->avc->avc_cache.latest_notif);
}
void avc_disable(void)
diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c
index 146ec9ef3aa5..299d7721ca9f 100644
--- a/security/selinux/hooks.c
+++ b/security/selinux/hooks.c
@@ -646,8 +646,12 @@ static int selinux_set_mnt_opts(struct super_block *sb,
* will be used for both mounts)
*/
if ((sbsec->flags & SE_SBINITIALIZED) && (sb->s_type->fs_flags & FS_BINARY_MOUNTDATA)
- && !opts)
+ && !opts) {
+ if ((kern_flags & SECURITY_LSM_NATIVE_LABELS) &&
+ sbsec->behavior == SECURITY_FS_USE_NATIVE)
+ *set_kern_flags |= SECURITY_LSM_NATIVE_LABELS;
goto out;
+ }
root_isec = backing_inode_security_novalidate(root);
@@ -691,6 +695,9 @@ static int selinux_set_mnt_opts(struct super_block *sb,
/* previously mounted with options, but not on this attempt? */
if ((sbsec->flags & SE_MNTMASK) && !opts)
goto out_double_mount;
+ if ((kern_flags & SECURITY_LSM_NATIVE_LABELS) &&
+ sbsec->behavior == SECURITY_FS_USE_NATIVE)
+ *set_kern_flags |= SECURITY_LSM_NATIVE_LABELS;
rc = 0;
goto out;
}
diff --git a/security/selinux/selinuxfs.c b/security/selinux/selinuxfs.c
index f74fdabaa541..5fbb24cce1b5 100644
--- a/security/selinux/selinuxfs.c
+++ b/security/selinux/selinuxfs.c
@@ -74,7 +74,6 @@ struct selinux_fs_info {
int *bool_pending_values;
struct dentry *class_dir;
unsigned long last_class_ino;
- bool policy_opened;
struct dentry *policycap_dir;
unsigned long last_ino;
struct selinux_state *state;
@@ -401,41 +400,29 @@ static int sel_open_policy(struct inode *inode, struct file *filp)
struct policy_load_memory *plm = NULL;
int rc;
- BUG_ON(filp->private_data);
-
- mutex_lock(&fsi->state->policy_mutex);
-
rc = avc_has_perm(&selinux_state,
current_sid(), SECINITSID_SECURITY,
SECCLASS_SECURITY, SECURITY__READ_POLICY, NULL);
if (rc)
- goto err;
-
- rc = -EBUSY;
- if (fsi->policy_opened)
- goto err;
+ return rc;
- rc = -ENOMEM;
plm = kzalloc(sizeof(*plm), GFP_KERNEL);
if (!plm)
- goto err;
+ return -ENOMEM;
+ mutex_lock(&fsi->state->policy_mutex);
rc = security_read_policy(state, &plm->data, &plm->len);
if (rc)
goto err;
-
if ((size_t)i_size_read(inode) != plm->len) {
inode_lock(inode);
i_size_write(inode, plm->len);
inode_unlock(inode);
}
-
- fsi->policy_opened = 1;
+ mutex_unlock(&fsi->state->policy_mutex);
filp->private_data = plm;
- mutex_unlock(&fsi->state->policy_mutex);
-
return 0;
err:
mutex_unlock(&fsi->state->policy_mutex);
@@ -448,13 +435,8 @@ static int sel_open_policy(struct inode *inode, struct file *filp)
static int sel_release_policy(struct inode *inode, struct file *filp)
{
- struct selinux_fs_info *fsi = inode->i_sb->s_fs_info;
struct policy_load_memory *plm = filp->private_data;
- BUG_ON(!plm);
-
- fsi->policy_opened = 0;
-
vfree(plm->data);
kfree(plm);
@@ -2211,7 +2193,6 @@ static struct file_system_type sel_fs_type = {
.kill_sb = sel_kill_sb,
};
-static struct vfsmount *selinuxfs_mount __ro_after_init;
struct path selinux_null __ro_after_init;
static int __init init_sel_fs(void)
@@ -2233,18 +2214,21 @@ static int __init init_sel_fs(void)
return err;
}
- selinux_null.mnt = selinuxfs_mount = kern_mount(&sel_fs_type);
- if (IS_ERR(selinuxfs_mount)) {
+ selinux_null.mnt = kern_mount(&sel_fs_type);
+ if (IS_ERR(selinux_null.mnt)) {
pr_err("selinuxfs: could not mount!\n");
- err = PTR_ERR(selinuxfs_mount);
- selinuxfs_mount = NULL;
+ err = PTR_ERR(selinux_null.mnt);
+ selinux_null.mnt = NULL;
+ return err;
}
+
selinux_null.dentry = d_hash_and_lookup(selinux_null.mnt->mnt_root,
&null_name);
if (IS_ERR(selinux_null.dentry)) {
pr_err("selinuxfs: could not lookup null!\n");
err = PTR_ERR(selinux_null.dentry);
selinux_null.dentry = NULL;
+ return err;
}
return err;
@@ -2257,7 +2241,7 @@ void exit_sel_fs(void)
{
sysfs_remove_mount_point(fs_kobj, "selinux");
dput(selinux_null.dentry);
- kern_unmount(selinuxfs_mount);
+ kern_unmount(selinux_null.mnt);
unregister_filesystem(&sel_fs_type);
}
#endif
diff --git a/security/selinux/ss/policydb.c b/security/selinux/ss/policydb.c
index 42bcd861bbfb..e6aa1069fd41 100644
--- a/security/selinux/ss/policydb.c
+++ b/security/selinux/ss/policydb.c
@@ -1126,6 +1126,10 @@ static int perm_read(struct policydb *p, struct symtab *s, void *fp)
len = le32_to_cpu(buf[0]);
perdatum->value = le32_to_cpu(buf[1]);
+ rc = -EINVAL;
+ /* indexes an nprim-sized array in security_get_permissions() */
+ if (perdatum->value > s->nprim)
+ goto bad;
rc = str_read(&key, GFP_KERNEL, fp, len);
if (rc)
diff --git a/sound/core/oss/mixer_oss.c b/sound/core/oss/mixer_oss.c
index abc99ae33328..ecb4af670959 100644
--- a/sound/core/oss/mixer_oss.c
+++ b/sound/core/oss/mixer_oss.c
@@ -540,6 +540,10 @@ static void snd_mixer_oss_get_volume1_vol(struct snd_mixer_oss_file *fmixer,
if (numid == ID_UNKNOWN)
return;
down_read(&card->controls_rwsem);
+ if (card->shutdown) {
+ up_read(&card->controls_rwsem);
+ return;
+ }
kctl = snd_ctl_find_numid_locked(card, numid);
if (!kctl) {
up_read(&card->controls_rwsem);
@@ -579,6 +583,10 @@ static void snd_mixer_oss_get_volume1_sw(struct snd_mixer_oss_file *fmixer,
if (numid == ID_UNKNOWN)
return;
down_read(&card->controls_rwsem);
+ if (card->shutdown) {
+ up_read(&card->controls_rwsem);
+ return;
+ }
kctl = snd_ctl_find_numid_locked(card, numid);
if (!kctl) {
up_read(&card->controls_rwsem);
@@ -645,6 +653,10 @@ static void snd_mixer_oss_put_volume1_vol(struct snd_mixer_oss_file *fmixer,
if (numid == ID_UNKNOWN)
return;
down_read(&card->controls_rwsem);
+ if (card->shutdown) {
+ up_read(&card->controls_rwsem);
+ return;
+ }
kctl = snd_ctl_find_numid_locked(card, numid);
if (!kctl) {
up_read(&card->controls_rwsem);
@@ -688,6 +700,10 @@ static void snd_mixer_oss_put_volume1_sw(struct snd_mixer_oss_file *fmixer,
if (numid == ID_UNKNOWN)
return;
down_read(&card->controls_rwsem);
+ if (card->shutdown) {
+ up_read(&card->controls_rwsem);
+ return;
+ }
kctl = snd_ctl_find_numid_locked(card, numid);
if (!kctl) {
up_read(&card->controls_rwsem);
@@ -833,6 +849,10 @@ static int snd_mixer_oss_get_recsrc2(struct snd_mixer_oss_file *fmixer, unsigned
goto __free_only;
}
down_read(&card->controls_rwsem);
+ if (card->shutdown) {
+ err = -ENODEV;
+ goto __unlock;
+ }
kctl = snd_mixer_oss_test_id(mixer, "Capture Source", 0);
if (! kctl) {
err = -ENOENT;
@@ -886,6 +906,10 @@ static int snd_mixer_oss_put_recsrc2(struct snd_mixer_oss_file *fmixer, unsigned
goto __free_only;
}
down_read(&card->controls_rwsem);
+ if (card->shutdown) {
+ err = -ENODEV;
+ goto __unlock;
+ }
kctl = snd_mixer_oss_test_id(mixer, "Capture Source", 0);
if (! kctl) {
err = -ENOENT;
@@ -937,6 +961,10 @@ static int snd_mixer_oss_build_test(struct snd_mixer_oss *mixer, struct slot *sl
int err;
down_read(&card->controls_rwsem);
+ if (card->shutdown) {
+ up_read(&card->controls_rwsem);
+ return -ENODEV;
+ }
kcontrol = snd_mixer_oss_test_id(mixer, name, index);
if (kcontrol == NULL) {
up_read(&card->controls_rwsem);
@@ -1069,6 +1097,10 @@ static int snd_mixer_oss_build_input(struct snd_mixer_oss *mixer,
if (snd_mixer_oss_build_test_all(mixer, ptr, &slot))
return 0;
down_read(&mixer->card->controls_rwsem);
+ if (mixer->card->shutdown) {
+ up_read(&mixer->card->controls_rwsem);
+ return -ENODEV;
+ }
kctl = NULL;
if (!ptr->index)
kctl = snd_mixer_oss_test_id(mixer, "Capture Source", 0);
diff --git a/sound/core/oss/pcm_oss.c b/sound/core/oss/pcm_oss.c
index 80dae370a0d5..d5cd728c30a0 100644
--- a/sound/core/oss/pcm_oss.c
+++ b/sound/core/oss/pcm_oss.c
@@ -2161,10 +2161,16 @@ static int snd_pcm_oss_get_trigger(struct snd_pcm_oss_file *pcm_oss_file)
psubstream = pcm_oss_file->streams[SNDRV_PCM_STREAM_PLAYBACK];
csubstream = pcm_oss_file->streams[SNDRV_PCM_STREAM_CAPTURE];
- if (psubstream && psubstream->runtime && psubstream->runtime->oss.trigger)
- result |= PCM_ENABLE_OUTPUT;
- if (csubstream && csubstream->runtime && csubstream->runtime->oss.trigger)
- result |= PCM_ENABLE_INPUT;
+ if (psubstream && psubstream->runtime) {
+ guard(mutex)(&psubstream->runtime->oss.params_lock);
+ if (psubstream->runtime->oss.trigger)
+ result |= PCM_ENABLE_OUTPUT;
+ }
+ if (csubstream && csubstream->runtime) {
+ guard(mutex)(&csubstream->runtime->oss.params_lock);
+ if (csubstream->runtime->oss.trigger)
+ result |= PCM_ENABLE_INPUT;
+ }
return result;
}
@@ -2839,6 +2845,17 @@ static int snd_pcm_oss_capture_ready(struct snd_pcm_substream *substream)
runtime->oss.period_frames;
}
+static bool need_input_retrigger(struct snd_pcm_runtime *runtime)
+{
+ bool ret;
+
+ guard(mutex)(&runtime->oss.params_lock);
+ ret = runtime->oss.trigger;
+ if (ret)
+ runtime->oss.trigger = 0;
+ return ret;
+}
+
static __poll_t snd_pcm_oss_poll(struct file *file, poll_table * wait)
{
struct snd_pcm_oss_file *pcm_oss_file;
@@ -2871,11 +2888,11 @@ static __poll_t snd_pcm_oss_poll(struct file *file, poll_table * wait)
snd_pcm_oss_capture_ready(csubstream))
mask |= EPOLLIN | EPOLLRDNORM;
snd_pcm_stream_unlock_irq(csubstream);
- if (ostate != SNDRV_PCM_STATE_RUNNING && runtime->oss.trigger) {
+ if (ostate != SNDRV_PCM_STATE_RUNNING &&
+ need_input_retrigger(runtime)) {
struct snd_pcm_oss_file ofile;
memset(&ofile, 0, sizeof(ofile));
ofile.streams[SNDRV_PCM_STREAM_CAPTURE] = pcm_oss_file->streams[SNDRV_PCM_STREAM_CAPTURE];
- runtime->oss.trigger = 0;
snd_pcm_oss_set_trigger(&ofile, PCM_ENABLE_INPUT);
}
}
diff --git a/sound/core/seq/oss/seq_oss_event.c b/sound/core/seq/oss/seq_oss_event.c
index 7b7c925dd3aa..4699c084d018 100644
--- a/sound/core/seq/oss/seq_oss_event.c
+++ b/sound/core/seq/oss/seq_oss_event.c
@@ -39,8 +39,10 @@ static int set_echo_event(struct seq_oss_devinfo *dp, union evrec *rec, struct s
*/
int
-snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd_seq_event *ev)
+snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q,
+ struct snd_seq_event *ev, snd_use_lock_t **lockp)
{
+ *lockp = NULL;
switch (q->s.code) {
case SEQ_EXTENDED:
return extended_event(dp, q, ev);
@@ -69,7 +71,7 @@ snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd
if (snd_seq_oss_midi_open(dp, q->s.dev, SNDRV_SEQ_OSS_FILE_WRITE))
break;
if (snd_seq_oss_midi_filemode(dp, q->s.dev) & SNDRV_SEQ_OSS_FILE_WRITE)
- return snd_seq_oss_midi_putc(dp, q->s.dev, q->s.parm1, ev);
+ return snd_seq_oss_midi_putc(dp, q->s.dev, q->s.parm1, ev, lockp);
break;
case SEQ_ECHO:
diff --git a/sound/core/seq/oss/seq_oss_event.h b/sound/core/seq/oss/seq_oss_event.h
index b4f723949a17..a4524e51d0e9 100644
--- a/sound/core/seq/oss/seq_oss_event.h
+++ b/sound/core/seq/oss/seq_oss_event.h
@@ -91,7 +91,8 @@ union evrec {
#define ev_is_long(ev) ((ev)->s.code >= 128)
#define ev_length(ev) ((ev)->s.code >= 128 ? LONG_EVENT_SIZE : SHORT_EVENT_SIZE)
-int snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd_seq_event *ev);
+int snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q,
+ struct snd_seq_event *ev, snd_use_lock_t **lockp);
int snd_seq_oss_process_timer_event(struct seq_oss_timer *rec, union evrec *q);
int snd_seq_oss_event_input(struct snd_seq_event *ev, int direct, void *private_data, int atomic, int hop);
diff --git a/sound/core/seq/oss/seq_oss_ioctl.c b/sound/core/seq/oss/seq_oss_ioctl.c
index ccf682689ec9..ce7a69d52b30 100644
--- a/sound/core/seq/oss/seq_oss_ioctl.c
+++ b/sound/core/seq/oss/seq_oss_ioctl.c
@@ -45,14 +45,17 @@ static int snd_seq_oss_oob_user(struct seq_oss_devinfo *dp, void __user *arg)
{
unsigned char ev[8];
struct snd_seq_event tmpev;
+ snd_use_lock_t *lock = NULL;
if (copy_from_user(ev, arg, 8))
return -EFAULT;
memset(&tmpev, 0, sizeof(tmpev));
snd_seq_oss_fill_addr(dp, &tmpev, dp->addr.client, dp->addr.port);
tmpev.time.tick = 0;
- if (! snd_seq_oss_process_event(dp, (union evrec *)ev, &tmpev)) {
+ if (!snd_seq_oss_process_event(dp, (union evrec *)ev, &tmpev, &lock)) {
snd_seq_oss_dispatch(dp, &tmpev, 0, 0);
+ if (lock)
+ snd_use_lock_free(lock);
}
return 0;
}
diff --git a/sound/core/seq/oss/seq_oss_midi.c b/sound/core/seq/oss/seq_oss_midi.c
index f2940b29595f..15ee97f3e907 100644
--- a/sound/core/seq/oss/seq_oss_midi.c
+++ b/sound/core/seq/oss/seq_oss_midi.c
@@ -639,7 +639,8 @@ send_midi_event(struct seq_oss_devinfo *dp, struct snd_seq_event *ev, struct seq
* non-zero : invalid - ignored
*/
int
-snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c, struct snd_seq_event *ev)
+snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c,
+ struct snd_seq_event *ev, snd_use_lock_t **lockp)
{
struct seq_oss_midi *mdev;
@@ -648,7 +649,8 @@ snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c, stru
return -ENODEV;
if (snd_midi_event_encode_byte(mdev->coder, c, ev)) {
snd_seq_oss_fill_addr(dp, ev, mdev->client, mdev->port);
- snd_use_lock_free(&mdev->use_lock);
+ /* the caller must release this later */
+ *lockp = &mdev->use_lock;
return 0;
}
snd_use_lock_free(&mdev->use_lock);
diff --git a/sound/core/seq/oss/seq_oss_midi.h b/sound/core/seq/oss/seq_oss_midi.h
index bcc1683773df..4819d4170bf6 100644
--- a/sound/core/seq/oss/seq_oss_midi.h
+++ b/sound/core/seq/oss/seq_oss_midi.h
@@ -26,7 +26,7 @@ void snd_seq_oss_midi_open_all(struct seq_oss_devinfo *dp, int file_mode);
int snd_seq_oss_midi_close(struct seq_oss_devinfo *dp, int dev);
void snd_seq_oss_midi_reset(struct seq_oss_devinfo *dp, int dev);
int snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c,
- struct snd_seq_event *ev);
+ struct snd_seq_event *ev, snd_use_lock_t **lockp);
int snd_seq_oss_midi_input(struct snd_seq_event *ev, int direct, void *private);
int snd_seq_oss_midi_filemode(struct seq_oss_devinfo *dp, int dev);
int snd_seq_oss_midi_make_info(struct seq_oss_devinfo *dp, int dev, struct midi_info *inf);
diff --git a/sound/core/seq/oss/seq_oss_readq.c b/sound/core/seq/oss/seq_oss_readq.c
index f0db5d3dcba4..b7cdb1c61f26 100644
--- a/sound/core/seq/oss/seq_oss_readq.c
+++ b/sound/core/seq/oss/seq_oss_readq.c
@@ -73,13 +73,17 @@ snd_seq_oss_readq_delete(struct seq_oss_readq *q)
void
snd_seq_oss_readq_clear(struct seq_oss_readq *q)
{
- if (q->qlen) {
- q->qlen = 0;
- q->head = q->tail = 0;
+ scoped_guard(spinlock_irqsave, &q->lock) {
+ if (q->qlen) {
+ q->qlen = 0;
+ q->head = 0;
+ q->tail = 0;
+ }
+ q->input_time = (unsigned long)-1;
}
+
/* if someone sleeping, wake'em up */
wake_up(&q->midi_sleep);
- q->input_time = (unsigned long)-1;
}
/*
@@ -136,28 +140,37 @@ int snd_seq_oss_readq_sysex(struct seq_oss_readq *q, int dev,
/*
* copy an event to input queue:
* return zero if enqueued
+ * caller must hold lock
*/
-int
-snd_seq_oss_readq_put_event(struct seq_oss_readq *q, union evrec *ev)
+static int snd_seq_oss_readq_put_event_locked(struct seq_oss_readq *q,
+ union evrec *ev)
{
- unsigned long flags;
-
- spin_lock_irqsave(&q->lock, flags);
- if (q->qlen >= q->maxlen - 1) {
- spin_unlock_irqrestore(&q->lock, flags);
+ if (q->qlen >= q->maxlen - 1)
return -ENOMEM;
- }
memcpy(&q->q[q->tail], ev, sizeof(*ev));
q->tail = (q->tail + 1) % q->maxlen;
q->qlen++;
- /* wake up sleeper */
- wake_up(&q->midi_sleep);
+ return 0;
+}
- spin_unlock_irqrestore(&q->lock, flags);
+/*
+ * copy an event to input queue:
+ * return zero if enqueued
+ */
+int
+snd_seq_oss_readq_put_event(struct seq_oss_readq *q, union evrec *ev)
+{
+ int rc;
- return 0;
+ scoped_guard(spinlock_irqsave, &q->lock) {
+ rc = snd_seq_oss_readq_put_event_locked(q, ev);
+ if (!rc)
+ wake_up(&q->midi_sleep);
+ }
+
+ return rc;
}
@@ -215,23 +228,31 @@ snd_seq_oss_readq_poll(struct seq_oss_readq *q, struct file *file, poll_table *w
int
snd_seq_oss_readq_put_timestamp(struct seq_oss_readq *q, unsigned long curt, int seq_mode)
{
- if (curt != q->input_time) {
- union evrec rec;
- memset(&rec, 0, sizeof(rec));
- switch (seq_mode) {
- case SNDRV_SEQ_OSS_MODE_SYNTH:
- rec.echo = (curt << 8) | SEQ_WAIT;
- snd_seq_oss_readq_put_event(q, &rec);
- break;
- case SNDRV_SEQ_OSS_MODE_MUSIC:
- rec.t.code = EV_TIMING;
- rec.t.cmd = TMR_WAIT_ABS;
- rec.t.time = curt;
- snd_seq_oss_readq_put_event(q, &rec);
- break;
+ int queued = 0;
+
+ scoped_guard(spinlock_irqsave, &q->lock) {
+ if (curt != q->input_time) {
+ union evrec rec;
+
+ memset(&rec, 0, sizeof(rec));
+ switch (seq_mode) {
+ case SNDRV_SEQ_OSS_MODE_SYNTH:
+ rec.echo = (curt << 8) | SEQ_WAIT;
+ queued = !snd_seq_oss_readq_put_event_locked(q, &rec);
+ break;
+ case SNDRV_SEQ_OSS_MODE_MUSIC:
+ rec.t.code = EV_TIMING;
+ rec.t.cmd = TMR_WAIT_ABS;
+ rec.t.time = curt;
+ queued = !snd_seq_oss_readq_put_event_locked(q, &rec);
+ break;
+ }
+ q->input_time = curt;
}
- q->input_time = curt;
}
+ if (queued)
+ wake_up(&q->midi_sleep);
+
return 0;
}
diff --git a/sound/core/seq/oss/seq_oss_rw.c b/sound/core/seq/oss/seq_oss_rw.c
index 45bb45859770..b7147ac78ee8 100644
--- a/sound/core/seq/oss/seq_oss_rw.c
+++ b/sound/core/seq/oss/seq_oss_rw.c
@@ -154,6 +154,7 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt)
{
int rc = 0;
struct snd_seq_event event;
+ snd_use_lock_t *lock = NULL;
/* if this is a timing event, process the current time */
if (snd_seq_oss_process_timer_event(dp->timer, rec))
@@ -165,7 +166,7 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt)
event.type = SNDRV_SEQ_EVENT_NOTEOFF;
snd_seq_oss_fill_addr(dp, &event, dp->addr.client, dp->addr.port);
- if (snd_seq_oss_process_event(dp, rec, &event))
+ if (snd_seq_oss_process_event(dp, rec, &event, &lock))
return 0; /* invalid event - no need to insert queue */
event.time.tick = snd_seq_oss_timer_cur_tick(dp->timer);
@@ -174,6 +175,8 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt)
else
rc = snd_seq_kernel_client_enqueue(dp->cseq, &event, opt,
!is_nonblock_mode(dp->file_mode));
+ if (lock)
+ snd_use_lock_free(lock);
return rc;
}
diff --git a/sound/core/seq/seq_clientmgr.c b/sound/core/seq/seq_clientmgr.c
index 37930d74e394..5763ad14f563 100644
--- a/sound/core/seq/seq_clientmgr.c
+++ b/sound/core/seq/seq_clientmgr.c
@@ -68,7 +68,7 @@ static int bounce_error_event(struct snd_seq_client *client,
int err, int atomic, int hop);
static int snd_seq_deliver_single_event(struct snd_seq_client *client,
struct snd_seq_event *event,
- int filter, int atomic, int hop);
+ int atomic, int hop);
/*
*/
@@ -439,7 +439,7 @@ static ssize_t snd_seq_read(struct file *file, char __user *buf, size_t count,
cell = NULL;
err = 0;
- snd_seq_fifo_lock(fifo);
+ guard(snd_seq_fifo)(fifo);
if (IS_ENABLED(CONFIG_SND_SEQ_UMP) && client->midi_version > 0)
aligned_size = sizeof(struct snd_seq_ump_event);
@@ -498,7 +498,6 @@ static ssize_t snd_seq_read(struct file *file, char __user *buf, size_t count,
if (err == -EAGAIN && result > 0)
err = 0;
}
- snd_seq_fifo_unlock(fifo);
return (err < 0) ? err : result;
}
@@ -516,10 +515,8 @@ static int check_port_perm(struct snd_seq_client_port *port, unsigned int flags)
/*
* check if the destination client is available, and return the pointer
- * if filter is non-zero, client filter bitmap is tested.
*/
-static struct snd_seq_client *get_event_dest_client(struct snd_seq_event *event,
- int filter)
+static struct snd_seq_client *get_event_dest_client(struct snd_seq_event *event)
{
struct snd_seq_client *dest;
@@ -534,8 +531,6 @@ static struct snd_seq_client *get_event_dest_client(struct snd_seq_event *event,
if ((dest->filter & SNDRV_SEQ_FILTER_USE_EVENT) &&
! test_bit(event->type, dest->event_filter))
goto __not_avail;
- if (filter && !(dest->filter & filter))
- goto __not_avail;
return dest; /* ok - accessible */
__not_avail:
@@ -559,7 +554,7 @@ static int bounce_error_event(struct snd_seq_client *client,
struct snd_seq_event *event,
int err, int atomic, int hop)
{
- struct snd_seq_event bounce_ev;
+ struct snd_seq_event bounce_ev, quoted;
int result;
if (client == NULL ||
@@ -569,17 +564,43 @@ static int bounce_error_event(struct snd_seq_client *client,
/* set up quoted error */
memset(&bounce_ev, 0, sizeof(bounce_ev));
- bounce_ev.type = SNDRV_SEQ_EVENT_KERNEL_ERROR;
- bounce_ev.flags = SNDRV_SEQ_EVENT_LENGTH_FIXED;
+
+ if (client->type == USER_CLIENT) {
+ /*
+ * For user clients, send SNDRV_SEQ_EVENT_BOUNCE with the
+ * original event embedded as variable-length data. This
+ * avoids exposing data.quote.event (a kernel pointer) to
+ * userspace. Sanitise the embedded copy too - a queued
+ * variable-length event carries the address of its own
+ * extension cell, and the payload goes out verbatim.
+ */
+ quoted = *event;
+ if (snd_seq_ev_is_variable("ed)) {
+ quoted.data.ext.len &= ~SNDRV_SEQ_EXT_MASK;
+ quoted.data.ext.ptr = NULL;
+ }
+ bounce_ev.type = SNDRV_SEQ_EVENT_BOUNCE;
+ bounce_ev.flags = SNDRV_SEQ_EVENT_LENGTH_VARIABLE;
+ bounce_ev.data.ext.len = sizeof(struct snd_seq_event);
+ bounce_ev.data.ext.ptr = (char *)"ed;
+ } else {
+ /*
+ * For kernel clients, quote the event pointer directly.
+ * Kernel consumers can safely dereference the pointer.
+ */
+ bounce_ev.type = SNDRV_SEQ_EVENT_KERNEL_ERROR;
+ bounce_ev.flags = SNDRV_SEQ_EVENT_LENGTH_FIXED;
+ bounce_ev.data.quote.origin = event->dest;
+ bounce_ev.data.quote.event = event;
+ bounce_ev.data.quote.value = -err; /* use positive value */
+ }
+
bounce_ev.queue = SNDRV_SEQ_QUEUE_DIRECT;
bounce_ev.source.client = SNDRV_SEQ_CLIENT_SYSTEM;
bounce_ev.source.port = SNDRV_SEQ_PORT_SYSTEM_ANNOUNCE;
bounce_ev.dest.client = client->number;
bounce_ev.dest.port = event->source.port;
- bounce_ev.data.quote.origin = event->dest;
- bounce_ev.data.quote.event = event;
- bounce_ev.data.quote.value = -err; /* use positive value */
- result = snd_seq_deliver_single_event(NULL, &bounce_ev, 0, atomic, hop + 1);
+ result = snd_seq_deliver_single_event(NULL, &bounce_ev, atomic, hop + 1);
if (result < 0) {
client->event_lost++;
return result;
@@ -625,7 +646,7 @@ static int update_timestamp_of_queue(struct snd_seq_event *event,
*/
static int snd_seq_deliver_single_event(struct snd_seq_client *client,
struct snd_seq_event *event,
- int filter, int atomic, int hop)
+ int atomic, int hop)
{
struct snd_seq_client *dest = NULL;
struct snd_seq_client_port *dest_port = NULL;
@@ -634,7 +655,7 @@ static int snd_seq_deliver_single_event(struct snd_seq_client *client,
direct = snd_seq_ev_is_direct(event);
- dest = get_event_dest_client(event, filter);
+ dest = get_event_dest_client(event);
if (dest == NULL)
goto __skip;
dest_port = snd_seq_port_use_ptr(dest, event->dest.port);
@@ -717,8 +738,7 @@ static int deliver_to_subscribers(struct snd_seq_client *client,
/* convert time according to flag with subscription */
update_timestamp_of_queue(event, subs->info.queue,
subs->info.flags & SNDRV_SEQ_PORT_SUBS_TIME_REAL);
- err = snd_seq_deliver_single_event(client, event,
- 0, atomic, hop);
+ err = snd_seq_deliver_single_event(client, event, atomic, hop);
if (err < 0) {
/* save first error that occurs and continue */
if (!result)
@@ -863,7 +883,7 @@ static int snd_seq_deliver_event(struct snd_seq_client *client, struct snd_seq_e
result = port_broadcast_event(client, event, atomic, hop);
#endif
else
- result = snd_seq_deliver_single_event(client, event, 0, atomic, hop);
+ result = snd_seq_deliver_single_event(client, event, atomic, hop);
return result;
}
diff --git a/sound/core/seq/seq_compat.c b/sound/core/seq/seq_compat.c
index c0ce6236dc7f..cdeb546edd21 100644
--- a/sound/core/seq/seq_compat.c
+++ b/sound/core/seq/seq_compat.c
@@ -44,7 +44,9 @@ static int snd_seq_call_port_info_ioctl(struct snd_seq_client *client, unsigned
goto error;
data->kernel = NULL;
- err = snd_seq_kernel_client_ctl(client->number, cmd, data);
+ scoped_guard(mutex, &client->ioctl_mutex) {
+ err = snd_seq_kernel_client_ctl(client->number, cmd, data);
+ }
if (err < 0)
goto error;
diff --git a/sound/core/seq/seq_fifo.c b/sound/core/seq/seq_fifo.c
index f8e02e98709a..c3dfe04c3883 100644
--- a/sound/core/seq/seq_fifo.c
+++ b/sound/core/seq/seq_fifo.c
@@ -88,12 +88,11 @@ void snd_seq_fifo_clear(struct snd_seq_fifo *f)
atomic_set(&f->overflow, 0);
snd_use_lock_sync(&f->use_lock);
- spin_lock_irq(&f->lock);
+ guard(spinlock_irq)(&f->lock);
/* drain the fifo */
while ((cell = fifo_cell_out(f)) != NULL) {
snd_seq_cell_free(cell);
}
- spin_unlock_irq(&f->lock);
}
@@ -102,38 +101,48 @@ int snd_seq_fifo_event_in(struct snd_seq_fifo *f,
struct snd_seq_event *event)
{
struct snd_seq_event_cell *cell;
- unsigned long flags;
+ struct snd_seq_pool *pool;
+ bool linked;
int err;
if (snd_BUG_ON(!f))
return -EINVAL;
- snd_use_lock_use(&f->use_lock);
- err = snd_seq_event_dup(f->pool, event, &cell, 1, NULL, NULL); /* always non-blocking */
+ guard(snd_seq_fifo)(f);
+retry:
+ pool = READ_ONCE(f->pool);
+ err = snd_seq_event_dup(pool, event, &cell, 1, NULL, NULL); /* always non-blocking */
if (err < 0) {
if ((err == -ENOMEM) || (err == -EAGAIN))
atomic_inc(&f->overflow);
- snd_use_lock_free(&f->use_lock);
return err;
}
/* append new cells to fifo */
- spin_lock_irqsave(&f->lock, flags);
- if (f->tail != NULL)
- f->tail->next = cell;
- f->tail = cell;
- if (f->head == NULL)
- f->head = cell;
- cell->next = NULL;
- f->cells++;
- spin_unlock_irqrestore(&f->lock, flags);
+ linked = false;
+ scoped_guard(spinlock_irqsave, &f->lock) {
+ if (cell->pool == f->pool) {
+ if (f->tail)
+ f->tail->next = cell;
+ f->tail = cell;
+ if (!f->head)
+ f->head = cell;
+ cell->next = NULL;
+ f->cells++;
+ linked = true;
+ }
+ }
+
+ if (!linked) {
+ /* Retry against the replacement pool after resize publishes it. */
+ snd_seq_cell_free(cell);
+ goto retry;
+ }
/* wakeup client */
if (waitqueue_active(&f->input_sleep))
wake_up(&f->input_sleep);
- snd_use_lock_free(&f->use_lock);
-
return 0; /* success */
}
@@ -199,16 +208,21 @@ int snd_seq_fifo_cell_out(struct snd_seq_fifo *f,
void snd_seq_fifo_cell_putback(struct snd_seq_fifo *f,
struct snd_seq_event_cell *cell)
{
- unsigned long flags;
+ bool linked = false;
if (cell) {
- spin_lock_irqsave(&f->lock, flags);
- cell->next = f->head;
- f->head = cell;
- if (!f->tail)
- f->tail = cell;
- f->cells++;
- spin_unlock_irqrestore(&f->lock, flags);
+ scoped_guard(spinlock_irqsave, &f->lock) {
+ if (cell->pool == f->pool) {
+ cell->next = f->head;
+ f->head = cell;
+ if (!f->tail)
+ f->tail = cell;
+ f->cells++;
+ linked = true;
+ }
+ }
+ if (!linked)
+ snd_seq_cell_free(cell);
}
}
@@ -239,17 +253,17 @@ int snd_seq_fifo_resize(struct snd_seq_fifo *f, int poolsize)
return -ENOMEM;
}
- spin_lock_irq(&f->lock);
- /* remember old pool */
- oldpool = f->pool;
- oldhead = f->head;
- /* exchange pools */
- f->pool = newpool;
- f->head = NULL;
- f->tail = NULL;
- f->cells = 0;
- /* NOTE: overflow flag is not cleared */
- spin_unlock_irq(&f->lock);
+ scoped_guard(spinlock_irq, &f->lock) {
+ /* remember old pool */
+ oldpool = f->pool;
+ oldhead = f->head;
+ /* exchange pools */
+ WRITE_ONCE(f->pool, newpool);
+ f->head = NULL;
+ f->tail = NULL;
+ f->cells = 0;
+ /* NOTE: overflow flag is not cleared */
+ }
/* close the old pool and wait until all users are gone */
snd_seq_pool_mark_closing(oldpool);
@@ -268,16 +282,10 @@ int snd_seq_fifo_resize(struct snd_seq_fifo *f, int poolsize)
/* get the number of unused cells safely */
int snd_seq_fifo_unused_cells(struct snd_seq_fifo *f)
{
- unsigned long flags;
- int cells;
-
if (!f)
return 0;
- snd_use_lock_use(&f->use_lock);
- spin_lock_irqsave(&f->lock, flags);
- cells = snd_seq_unused_cells(f->pool);
- spin_unlock_irqrestore(&f->lock, flags);
- snd_use_lock_free(&f->use_lock);
- return cells;
+ guard(snd_seq_fifo)(f);
+ guard(spinlock_irqsave)(&f->lock);
+ return snd_seq_unused_cells(f->pool);
}
diff --git a/sound/core/seq/seq_fifo.h b/sound/core/seq/seq_fifo.h
index b56a7b897c9c..4c9c49127746 100644
--- a/sound/core/seq/seq_fifo.h
+++ b/sound/core/seq/seq_fifo.h
@@ -37,6 +37,7 @@ int snd_seq_fifo_event_in(struct snd_seq_fifo *f, struct snd_seq_event *event);
/* lock fifo from release */
#define snd_seq_fifo_lock(fifo) snd_use_lock_use(&(fifo)->use_lock)
#define snd_seq_fifo_unlock(fifo) snd_use_lock_free(&(fifo)->use_lock)
+DEFINE_GUARD(snd_seq_fifo, struct snd_seq_fifo *, snd_seq_fifo_lock(_T), snd_seq_fifo_unlock(_T))
/* get a cell from fifo - fifo should be locked */
int snd_seq_fifo_cell_out(struct snd_seq_fifo *f, struct snd_seq_event_cell **cellp, int nonblock);
diff --git a/sound/drivers/mtpav.c b/sound/drivers/mtpav.c
index f212f233ea61..1fb6797fa45f 100644
--- a/sound/drivers/mtpav.c
+++ b/sound/drivers/mtpav.c
@@ -663,12 +663,8 @@ static int snd_mtpav_get_RAWMIDI(struct mtpav *mcard)
static void snd_mtpav_free(struct snd_card *card)
{
struct mtpav *crd = card->private_data;
- unsigned long flags;
- spin_lock_irqsave(&crd->spinlock, flags);
- if (crd->istimer > 0)
- snd_mtpav_remove_output_timer(crd);
- spin_unlock_irqrestore(&crd->spinlock, flags);
+ timer_shutdown_sync(&crd->timer);
}
/*
diff --git a/sound/pci/hda/cs35l41_hda.c b/sound/pci/hda/cs35l41_hda.c
index 63395d2bf45c..5f2bf06c0fbc 100644
--- a/sound/pci/hda/cs35l41_hda.c
+++ b/sound/pci/hda/cs35l41_hda.c
@@ -894,6 +894,44 @@ static int cs35l41_fw_type_ctl_info(struct snd_kcontrol *kcontrol, struct snd_ct
return snd_ctl_enum_info(uinfo, 1, ARRAY_SIZE(hda_cs_dsp_fw_ids), hda_cs_dsp_fw_ids);
}
+static void cs35l41_remove_controls(struct cs35l41_hda *cs35l41)
+{
+ if (!cs35l41->codec)
+ return;
+
+ if (cs35l41->fw_load_ctl) {
+ snd_ctl_remove(cs35l41->codec->card, cs35l41->fw_load_ctl);
+ cs35l41->fw_load_ctl = NULL;
+ }
+
+ if (cs35l41->fw_type_ctl) {
+ snd_ctl_remove(cs35l41->codec->card, cs35l41->fw_type_ctl);
+ cs35l41->fw_type_ctl = NULL;
+ }
+}
+
+static int cs35l41_add_control(struct cs35l41_hda *cs35l41,
+ struct snd_kcontrol_new *ctl,
+ struct snd_kcontrol **kctl)
+{
+ int ret;
+
+ *kctl = snd_ctl_new1(ctl, cs35l41);
+ if (!*kctl)
+ return -ENOMEM;
+
+ ret = snd_ctl_add(cs35l41->codec->card, *kctl);
+ if (ret) {
+ dev_err(cs35l41->dev, "Failed to add KControl %s = %d\n", ctl->name, ret);
+ *kctl = NULL;
+ return ret;
+ }
+
+ dev_dbg(cs35l41->dev, "Added Control %s\n", ctl->name);
+
+ return 0;
+}
+
static int cs35l41_create_controls(struct cs35l41_hda *cs35l41)
{
char fw_type_ctl_name[SNDRV_CTL_ELEM_ID_NAME_MAXLEN];
@@ -919,23 +957,19 @@ static int cs35l41_create_controls(struct cs35l41_hda *cs35l41)
scnprintf(fw_load_ctl_name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN, "%s DSP1 Firmware Load",
cs35l41->amp_name);
- ret = snd_ctl_add(cs35l41->codec->card, snd_ctl_new1(&fw_type_ctl, cs35l41));
- if (ret) {
- dev_err(cs35l41->dev, "Failed to add KControl %s = %d\n", fw_type_ctl.name, ret);
- return ret;
- }
-
- dev_dbg(cs35l41->dev, "Added Control %s\n", fw_type_ctl.name);
-
- ret = snd_ctl_add(cs35l41->codec->card, snd_ctl_new1(&fw_load_ctl, cs35l41));
- if (ret) {
- dev_err(cs35l41->dev, "Failed to add KControl %s = %d\n", fw_load_ctl.name, ret);
- return ret;
- }
+ ret = cs35l41_add_control(cs35l41, &fw_type_ctl, &cs35l41->fw_type_ctl);
+ if (ret)
+ goto err;
- dev_dbg(cs35l41->dev, "Added Control %s\n", fw_load_ctl.name);
+ ret = cs35l41_add_control(cs35l41, &fw_load_ctl, &cs35l41->fw_load_ctl);
+ if (ret)
+ goto err;
return 0;
+
+err:
+ cs35l41_remove_controls(cs35l41);
+ return ret;
}
static int cs35l41_hda_bind(struct device *dev, struct device *master, void *master_data)
@@ -990,8 +1024,13 @@ static void cs35l41_hda_unbind(struct device *dev, struct device *master, void *
struct cs35l41_hda *cs35l41 = dev_get_drvdata(dev);
struct hda_component *comps = master_data;
- if (comps[cs35l41->index].dev == dev)
+ if (comps[cs35l41->index].dev == dev) {
memset(&comps[cs35l41->index], 0, sizeof(*comps));
+
+ cs35l41_remove_controls(cs35l41);
+ cancel_work_sync(&cs35l41->fw_load_work);
+ cs35l41->codec = NULL;
+ }
}
static const struct component_ops cs35l41_hda_comp_ops = {
@@ -1529,6 +1568,9 @@ void cs35l41_hda_remove(struct device *dev)
{
struct cs35l41_hda *cs35l41 = dev_get_drvdata(dev);
+ component_del(cs35l41->dev, &cs35l41_hda_comp_ops);
+ cancel_work_sync(&cs35l41->fw_load_work);
+
pm_runtime_get_sync(cs35l41->dev);
pm_runtime_dont_use_autosuspend(cs35l41->dev);
pm_runtime_disable(cs35l41->dev);
@@ -1536,8 +1578,6 @@ void cs35l41_hda_remove(struct device *dev)
if (cs35l41->halo_initialized)
cs35l41_remove_dsp(cs35l41);
- component_del(cs35l41->dev, &cs35l41_hda_comp_ops);
-
pm_runtime_put_noidle(cs35l41->dev);
if (cs35l41_safe_reset(cs35l41->regmap, cs35l41->hw_cfg.bst_type))
diff --git a/sound/pci/hda/cs35l41_hda.h b/sound/pci/hda/cs35l41_hda.h
index bdb35f3be68a..f6b4358fa435 100644
--- a/sound/pci/hda/cs35l41_hda.h
+++ b/sound/pci/hda/cs35l41_hda.h
@@ -45,6 +45,8 @@ enum cs35l41_hda_gpio_function {
CS35l41_SYNC,
};
+struct snd_kcontrol;
+
struct cs35l41_hda {
struct device *dev;
struct regmap *regmap;
@@ -62,6 +64,8 @@ struct cs35l41_hda {
int speaker_id;
struct mutex fw_mutex;
struct work_struct fw_load_work;
+ struct snd_kcontrol *fw_type_ctl;
+ struct snd_kcontrol *fw_load_ctl;
struct regmap_irq_chip_data *irq_data;
bool firmware_running;
diff --git a/sound/soc/amd/acp/acp-mach-common.c b/sound/soc/amd/acp/acp-mach-common.c
index 4c69cb6e3400..093c3fa70829 100644
--- a/sound/soc/amd/acp/acp-mach-common.c
+++ b/sound/soc/amd/acp/acp-mach-common.c
@@ -54,10 +54,11 @@ static const struct snd_pcm_hw_constraint_list constraints_channels = {
.mask = 0,
};
-static int acp_clk_enable(struct acp_card_drvdata *drvdata)
+static int acp_clk_enable(struct acp_card_drvdata *drvdata,
+ unsigned int srate, unsigned int bclk_ratio)
{
- clk_set_rate(drvdata->wclk, 48000);
- clk_set_rate(drvdata->bclk, 48000 * 64);
+ clk_set_rate(drvdata->wclk, srate);
+ clk_set_rate(drvdata->bclk, srate * bclk_ratio);
return clk_prepare_enable(drvdata->wclk);
}
@@ -86,36 +87,13 @@ static int acp_card_rt5682_init(struct snd_soc_pcm_runtime *rtd)
if (drvdata->hs_codec_id != RT5682)
return -EINVAL;
- ret = snd_soc_dai_set_fmt(codec_dai, SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF
- | SND_SOC_DAIFMT_CBP_CFP);
- if (ret < 0) {
- dev_err(rtd->card->dev, "Failed to set dai fmt: %d\n", ret);
- return ret;
- }
-
- ret = snd_soc_dai_set_pll(codec_dai, RT5682_PLL2, RT5682_PLL2_S_MCLK,
- PCO_PLAT_CLK, RT5682_PLL_FREQ);
- if (ret < 0) {
- dev_err(rtd->dev, "Failed to set codec PLL: %d\n", ret);
- return ret;
- }
-
- ret = snd_soc_dai_set_sysclk(codec_dai, RT5682_SCLK_S_PLL2,
- RT5682_PLL_FREQ, SND_SOC_CLOCK_IN);
- if (ret < 0) {
- dev_err(rtd->dev, "Failed to set codec SYSCLK: %d\n", ret);
- return ret;
- }
+ drvdata->wclk = devm_clk_get(component->dev, "rt5682-dai-wclk");
+ if (IS_ERR(drvdata->wclk))
+ return PTR_ERR(drvdata->wclk);
- /* Set tdm/i2s1 master bclk ratio */
- ret = snd_soc_dai_set_bclk_ratio(codec_dai, 64);
- if (ret < 0) {
- dev_err(rtd->dev, "Failed to set rt5682 tdm bclk ratio: %d\n", ret);
- return ret;
- }
-
- drvdata->wclk = clk_get(component->dev, "rt5682-dai-wclk");
- drvdata->bclk = clk_get(component->dev, "rt5682-dai-bclk");
+ drvdata->bclk = devm_clk_get(component->dev, "rt5682-dai-bclk");
+ if (IS_ERR(drvdata->bclk))
+ return PTR_ERR(drvdata->bclk);
ret = snd_soc_card_jack_new(card, "Headset Jack",
SND_JACK_HEADSET | SND_JACK_LINEOUT |
@@ -167,13 +145,6 @@ static int acp_card_hs_startup(struct snd_pcm_substream *substream)
&constraints_channels);
snd_pcm_hw_constraint_list(runtime, 0, SNDRV_PCM_HW_PARAM_RATE,
&constraints_rates);
- if (!drvdata->soc_mclk) {
- ret = acp_clk_enable(drvdata);
- if (ret < 0) {
- dev_err(rtd->card->dev, "Failed to enable HS clk: %d\n", ret);
- return ret;
- }
- }
return ret;
}
@@ -188,54 +159,46 @@ static void acp_card_shutdown(struct snd_pcm_substream *substream)
clk_disable_unprepare(drvdata->wclk);
}
-static const struct snd_soc_ops acp_card_rt5682_ops = {
- .startup = acp_card_hs_startup,
- .shutdown = acp_card_shutdown,
-};
-
-/* Define RT5682S CODEC component*/
-SND_SOC_DAILINK_DEF(rt5682s,
- DAILINK_COMP_ARRAY(COMP_CODEC("i2c-RTL5682:00", "rt5682s-aif1")));
-
-static const struct snd_soc_dapm_route rt5682s_map[] = {
- { "Headphone Jack", NULL, "HPOL" },
- { "Headphone Jack", NULL, "HPOR" },
- { "IN1P", NULL, "Headset Mic" },
-};
-
-static int acp_card_rt5682s_init(struct snd_soc_pcm_runtime *rtd)
+static int acp_card_rt5682_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
struct snd_soc_card *card = rtd->card;
struct acp_card_drvdata *drvdata = card->drvdata;
struct snd_soc_dai *codec_dai = asoc_rtd_to_codec(rtd, 0);
- struct snd_soc_component *component = codec_dai->component;
- unsigned int fmt;
+ struct snd_soc_dai *cpu_dai = asoc_rtd_to_cpu(rtd, 0);
int ret;
+ unsigned int fmt, srate, ch, format;
- dev_info(rtd->dev, "codec dai name = %s\n", codec_dai->name);
-
- if (drvdata->hs_codec_id != RT5682S)
- return -EINVAL;
+ srate = params_rate(params);
+ ch = params_channels(params);
+ format = 8 * params_format(params);
if (drvdata->soc_mclk)
fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBC_CFC;
else
fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBP_CFP;
+ ret = snd_soc_dai_set_fmt(cpu_dai, fmt);
+ if (ret && ret != -ENOTSUPP) {
+ dev_err(rtd->dev, "Failed to set dai fmt: %d\n", ret);
+ return ret;
+ }
+
ret = snd_soc_dai_set_fmt(codec_dai, fmt);
if (ret < 0) {
dev_err(rtd->card->dev, "Failed to set dai fmt: %d\n", ret);
return ret;
}
- ret = snd_soc_dai_set_pll(codec_dai, RT5682S_PLL2, RT5682S_PLL_S_MCLK,
+ ret = snd_soc_dai_set_pll(codec_dai, RT5682_PLL2, RT5682_PLL2_S_MCLK,
PCO_PLAT_CLK, RT5682_PLL_FREQ);
if (ret < 0) {
dev_err(rtd->dev, "Failed to set codec PLL: %d\n", ret);
return ret;
}
- ret = snd_soc_dai_set_sysclk(codec_dai, RT5682S_SCLK_S_PLL2,
+ ret = snd_soc_dai_set_sysclk(codec_dai, RT5682_SCLK_S_PLL2,
RT5682_PLL_FREQ, SND_SOC_CLOCK_IN);
if (ret < 0) {
dev_err(rtd->dev, "Failed to set codec SYSCLK: %d\n", ret);
@@ -243,15 +206,60 @@ static int acp_card_rt5682s_init(struct snd_soc_pcm_runtime *rtd)
}
/* Set tdm/i2s1 master bclk ratio */
- ret = snd_soc_dai_set_bclk_ratio(codec_dai, 64);
+ ret = snd_soc_dai_set_bclk_ratio(codec_dai, ch * format);
if (ret < 0) {
dev_err(rtd->dev, "Failed to set rt5682 tdm bclk ratio: %d\n", ret);
return ret;
}
if (!drvdata->soc_mclk) {
- drvdata->wclk = clk_get(component->dev, "rt5682-dai-wclk");
- drvdata->bclk = clk_get(component->dev, "rt5682-dai-bclk");
+ ret = acp_clk_enable(drvdata, srate, ch * format);
+ if (ret < 0) {
+ dev_err(rtd->card->dev, "Failed to enable HS clk: %d\n", ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static const struct snd_soc_ops acp_card_rt5682_ops = {
+ .startup = acp_card_hs_startup,
+ .shutdown = acp_card_shutdown,
+ .hw_params = acp_card_rt5682_hw_params,
+};
+
+/* Define RT5682S CODEC component*/
+SND_SOC_DAILINK_DEF(rt5682s,
+ DAILINK_COMP_ARRAY(COMP_CODEC("i2c-RTL5682:00", "rt5682s-aif1")));
+
+static const struct snd_soc_dapm_route rt5682s_map[] = {
+ { "Headphone Jack", NULL, "HPOL" },
+ { "Headphone Jack", NULL, "HPOR" },
+ { "IN1P", NULL, "Headset Mic" },
+};
+
+static int acp_card_rt5682s_init(struct snd_soc_pcm_runtime *rtd)
+{
+ struct snd_soc_card *card = rtd->card;
+ struct acp_card_drvdata *drvdata = card->drvdata;
+ struct snd_soc_dai *codec_dai = asoc_rtd_to_codec(rtd, 0);
+ struct snd_soc_component *component = codec_dai->component;
+ int ret;
+
+ dev_info(rtd->dev, "codec dai name = %s\n", codec_dai->name);
+
+ if (drvdata->hs_codec_id != RT5682S)
+ return -EINVAL;
+
+ if (!drvdata->soc_mclk) {
+ drvdata->wclk = devm_clk_get(component->dev, "rt5682-dai-wclk");
+ if (IS_ERR(drvdata->wclk))
+ return PTR_ERR(drvdata->wclk);
+
+ drvdata->bclk = devm_clk_get(component->dev, "rt5682-dai-bclk");
+ if (IS_ERR(drvdata->bclk))
+ return PTR_ERR(drvdata->bclk);
}
ret = snd_soc_card_jack_new(card, "Headset Jack",
@@ -278,9 +286,68 @@ static int acp_card_rt5682s_init(struct snd_soc_pcm_runtime *rtd)
return snd_soc_dapm_add_routes(&rtd->card->dapm, rt5682s_map, ARRAY_SIZE(rt5682s_map));
}
+static int acp_card_rt5682s_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_card *card = rtd->card;
+ struct acp_card_drvdata *drvdata = card->drvdata;
+ struct snd_soc_dai *codec_dai = asoc_rtd_to_codec(rtd, 0);
+ struct snd_soc_dai *cpu_dai = asoc_rtd_to_cpu(rtd, 0);
+ int ret;
+ unsigned int fmt, srate, ch, format;
+
+ srate = params_rate(params);
+ ch = params_channels(params);
+ format = 8 * params_format(params);
+
+ if (drvdata->soc_mclk)
+ fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBC_CFC;
+ else
+ fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBP_CFP;
+
+ ret = snd_soc_dai_set_fmt(cpu_dai, fmt);
+ if (ret && ret != -ENOTSUPP) {
+ dev_err(rtd->dev, "Failed to set dai fmt: %d\n", ret);
+ return ret;
+ }
+
+ ret = snd_soc_dai_set_fmt(codec_dai, fmt);
+ if (ret < 0) {
+ dev_err(rtd->card->dev, "Failed to set dai fmt: %d\n", ret);
+ return ret;
+ }
+
+ ret = snd_soc_dai_set_pll(codec_dai, RT5682S_PLL2, RT5682S_PLL_S_MCLK,
+ PCO_PLAT_CLK, RT5682_PLL_FREQ);
+ if (ret < 0) {
+ dev_err(rtd->dev, "Failed to set codec PLL: %d\n", ret);
+ return ret;
+ }
+
+ ret = snd_soc_dai_set_sysclk(codec_dai, RT5682S_SCLK_S_PLL2,
+ RT5682_PLL_FREQ, SND_SOC_CLOCK_IN);
+ if (ret < 0) {
+ dev_err(rtd->dev, "Failed to set codec SYSCLK: %d\n", ret);
+ return ret;
+ }
+
+ /* Set tdm/i2s1 master bclk ratio */
+ ret = snd_soc_dai_set_bclk_ratio(codec_dai, ch * format);
+ if (ret < 0) {
+ dev_err(rtd->dev, "Failed to set rt5682 tdm bclk ratio: %d\n", ret);
+ return ret;
+ }
+
+ clk_set_rate(drvdata->wclk, srate);
+ clk_set_rate(drvdata->bclk, srate * ch * format);
+
+ return 0;
+}
+
static const struct snd_soc_ops acp_card_rt5682s_ops = {
.startup = acp_card_hs_startup,
- .shutdown = acp_card_shutdown,
+ .hw_params = acp_card_rt5682s_hw_params,
};
static const unsigned int dmic_channels[] = {
@@ -349,19 +416,34 @@ static int acp_card_rt1019_hw_params(struct snd_pcm_substream *substream,
struct snd_soc_card *card = rtd->card;
struct acp_card_drvdata *drvdata = card->drvdata;
struct snd_soc_dai *codec_dai;
- int srate, i, ret = 0;
+ struct snd_soc_dai *cpu_dai = asoc_rtd_to_cpu(rtd, 0);
+ int i, ret = 0;
+ unsigned int fmt, srate, ch, format;
srate = params_rate(params);
+ ch = params_channels(params);
+ format = 8 * params_format(params);
if (drvdata->amp_codec_id != RT1019)
return -EINVAL;
+ if (drvdata->soc_mclk)
+ fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBC_CFC;
+ else
+ fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBP_CFP;
+
+ ret = snd_soc_dai_set_fmt(cpu_dai, fmt);
+ if (ret && ret != -ENOTSUPP) {
+ dev_err(rtd->dev, "Failed to set dai fmt: %d\n", ret);
+ return ret;
+ }
+
for_each_rtd_codec_dais(rtd, i, codec_dai) {
if (strcmp(codec_dai->name, "rt1019-aif"))
continue;
ret = snd_soc_dai_set_pll(codec_dai, 0, RT1019_PLL_S_BCLK,
- 64 * srate, 256 * srate);
+ ch * format * srate, 256 * srate);
if (ret < 0)
return ret;
@@ -371,16 +453,20 @@ static int acp_card_rt1019_hw_params(struct snd_pcm_substream *substream,
return ret;
}
+ if (!drvdata->soc_mclk) {
+ ret = acp_clk_enable(drvdata, srate, ch * format);
+ if (ret < 0) {
+ dev_err(rtd->card->dev, "Failed to enable AMP clk: %d\n", ret);
+ return ret;
+ }
+ }
+
return 0;
}
static int acp_card_amp_startup(struct snd_pcm_substream *substream)
{
struct snd_pcm_runtime *runtime = substream->runtime;
- struct snd_soc_pcm_runtime *rtd = asoc_substream_to_rtd(substream);
- struct snd_soc_card *card = rtd->card;
- struct acp_card_drvdata *drvdata = card->drvdata;
- int ret = 0;
runtime->hw.channels_max = DUAL_CHANNEL;
snd_pcm_hw_constraint_list(runtime, 0, SNDRV_PCM_HW_PARAM_CHANNELS,
@@ -388,14 +474,7 @@ static int acp_card_amp_startup(struct snd_pcm_substream *substream)
snd_pcm_hw_constraint_list(runtime, 0, SNDRV_PCM_HW_PARAM_RATE,
&constraints_rates);
- if (!drvdata->soc_mclk) {
- ret = acp_clk_enable(drvdata);
- if (ret < 0) {
- dev_err(rtd->card->dev, "Failed to enable AMP clk: %d\n", ret);
- return ret;
- }
- }
- return ret;
+ return 0;
}
static const struct snd_soc_ops acp_card_rt1019_ops = {
@@ -424,9 +503,45 @@ static int acp_card_maxim_init(struct snd_soc_pcm_runtime *rtd)
ARRAY_SIZE(max98360a_map));
}
+static int acp_card_maxim_hw_params(struct snd_pcm_substream *substream,
+ struct snd_pcm_hw_params *params)
+{
+ struct snd_soc_pcm_runtime *rtd = substream->private_data;
+ struct snd_soc_card *card = rtd->card;
+ struct acp_card_drvdata *drvdata = card->drvdata;
+ struct snd_soc_dai *cpu_dai = asoc_rtd_to_cpu(rtd, 0);
+ unsigned int fmt, srate, ch, format;
+ int ret;
+
+ srate = params_rate(params);
+ ch = params_channels(params);
+ format = 8 * params_format(params);
+
+ if (drvdata->soc_mclk)
+ fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBC_CFC;
+ else
+ fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBP_CFP;
+
+ ret = snd_soc_dai_set_fmt(cpu_dai, fmt);
+ if (ret && ret != -ENOTSUPP) {
+ dev_err(rtd->dev, "Failed to set dai fmt: %d\n", ret);
+ return ret;
+ }
+
+ if (!drvdata->soc_mclk) {
+ ret = acp_clk_enable(drvdata, srate, ch * format);
+ if (ret < 0) {
+ dev_err(rtd->card->dev, "Failed to enable AMP clk: %d\n", ret);
+ return ret;
+ }
+ }
+ return 0;
+}
+
static const struct snd_soc_ops acp_card_maxim_ops = {
.startup = acp_card_amp_startup,
.shutdown = acp_card_shutdown,
+ .hw_params = acp_card_maxim_hw_params,
};
/* Declare nau8825 codec components */
@@ -444,7 +559,6 @@ static int acp_card_nau8825_init(struct snd_soc_pcm_runtime *rtd)
struct acp_card_drvdata *drvdata = card->drvdata;
struct snd_soc_dai *codec_dai = asoc_rtd_to_codec(rtd, 0);
struct snd_soc_component *component = codec_dai->component;
- unsigned int fmt;
int ret;
dev_info(rtd->dev, "codec dai name = %s\n", codec_dai->name);
@@ -452,16 +566,6 @@ static int acp_card_nau8825_init(struct snd_soc_pcm_runtime *rtd)
if (drvdata->hs_codec_id != NAU8825)
return -EINVAL;
- if (drvdata->soc_mclk)
- fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBC_CFC;
- else
- fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBP_CFP;
-
- ret = snd_soc_dai_set_fmt(codec_dai, fmt);
- if (ret < 0) {
- dev_err(rtd->card->dev, "Failed to set dai fmt: %d\n", ret);
- return ret;
- }
ret = snd_soc_card_jack_new(card, "Headset Jack",
SND_JACK_HEADSET | SND_JACK_LINEOUT |
SND_JACK_BTN_0 | SND_JACK_BTN_1 |
@@ -490,8 +594,12 @@ static int acp_nau8825_hw_params(struct snd_pcm_substream *substream,
struct snd_pcm_hw_params *params)
{
struct snd_soc_pcm_runtime *rtd = asoc_substream_to_rtd(substream);
+ struct snd_soc_card *card = rtd->card;
+ struct acp_card_drvdata *drvdata = card->drvdata;
struct snd_soc_dai *codec_dai = asoc_rtd_to_codec(rtd, 0);
+ struct snd_soc_dai *cpu_dai = asoc_rtd_to_cpu(rtd, 0);
int ret;
+ unsigned int fmt;
ret = snd_soc_dai_set_sysclk(codec_dai, NAU8825_CLK_FLL_FS,
(48000 * 256), SND_SOC_CLOCK_IN);
@@ -505,6 +613,23 @@ static int acp_nau8825_hw_params(struct snd_pcm_substream *substream,
return ret;
}
+ if (drvdata->soc_mclk)
+ fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBC_CFC;
+ else
+ fmt = SND_SOC_DAIFMT_I2S | SND_SOC_DAIFMT_NB_NF | SND_SOC_DAIFMT_CBP_CFP;
+
+ ret = snd_soc_dai_set_fmt(cpu_dai, fmt);
+ if (ret && ret != -ENOTSUPP) {
+ dev_err(rtd->dev, "Failed to set dai fmt: %d\n", ret);
+ return ret;
+ }
+
+ ret = snd_soc_dai_set_fmt(codec_dai, fmt);
+ if (ret < 0) {
+ dev_err(rtd->card->dev, "Failed to set dai fmt: %d\n", ret);
+ return ret;
+ }
+
return ret;
}
@@ -570,6 +695,50 @@ SND_SOC_DAILINK_DEF(sof_dmic,
SND_SOC_DAILINK_DEF(pdm_dmic,
DAILINK_COMP_ARRAY(COMP_CPU("acp-pdm-dmic")));
+static int acp_rtk_set_bias_level(struct snd_soc_card *card,
+ struct snd_soc_dapm_context *dapm,
+ enum snd_soc_bias_level level)
+{
+ struct snd_soc_component *component = dapm->component;
+ struct acp_card_drvdata *drvdata = card->drvdata;
+ int ret = 0;
+
+ if (!component)
+ return 0;
+
+ if (strncmp(component->name, "i2c-RTL5682", 11) &&
+ strncmp(component->name, "i2c-10EC1019", 12))
+ return 0;
+
+ /*
+ * For Realtek's codec and amplifier components,
+ * the lrck and bclk must be enabled brfore their all dapms be powered on,
+ * and must be disabled after their all dapms be powered down
+ * to avoid any pop.
+ */
+ switch (level) {
+ case SND_SOC_BIAS_STANDBY:
+ if (snd_soc_dapm_get_bias_level(dapm) == SND_SOC_BIAS_OFF) {
+
+ /* Increase bclk's enable_count */
+ ret = clk_prepare_enable(drvdata->bclk);
+ if (ret < 0)
+ dev_err(component->dev, "Failed to enable bclk %d\n", ret);
+ } else {
+ /*
+ * Decrease bclk's enable_count.
+ * While the enable_count is 0, the bclk would be closed.
+ */
+ clk_disable_unprepare(drvdata->bclk);
+ }
+ break;
+ default:
+ break;
+ }
+
+ return ret;
+}
+
int acp_sofdsp_dai_links_create(struct snd_soc_card *card)
{
struct snd_soc_dai_link *links;
@@ -730,6 +899,7 @@ int acp_sofdsp_dai_links_create(struct snd_soc_card *card)
card->dai_link = links;
card->num_links = num_links;
+ card->set_bias_level = acp_rtk_set_bias_level;
return 0;
}
@@ -907,6 +1077,7 @@ int acp_legacy_dai_links_create(struct snd_soc_card *card)
card->dai_link = links;
card->num_links = num_links;
+ card->set_bias_level = acp_rtk_set_bias_level;
return 0;
}
diff --git a/sound/soc/amd/yc/acp6x-mach.c b/sound/soc/amd/yc/acp6x-mach.c
index afbf1edbeeb5..64c217adfe1f 100644
--- a/sound/soc/amd/yc/acp6x-mach.c
+++ b/sound/soc/amd/yc/acp6x-mach.c
@@ -605,6 +605,13 @@ static const struct dmi_system_id yc_acp_quirk_table[] = {
DMI_MATCH(DMI_PRODUCT_NAME, "Thin A15 B7UC"),
}
},
+ {
+ .driver_data = &acp6x_card,
+ .matches = {
+ DMI_MATCH(DMI_BOARD_VENDOR, "MDC"),
+ DMI_MATCH(DMI_BOARD_NAME, "Herbag2_MDU"),
+ }
+ },
{
.driver_data = &acp6x_card,
.matches = {
diff --git a/sound/soc/codecs/adau1372.c b/sound/soc/codecs/adau1372.c
index 3e3b8955036d..f89555e3e525 100644
--- a/sound/soc/codecs/adau1372.c
+++ b/sound/soc/codecs/adau1372.c
@@ -798,7 +798,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable)
if (ret)
return ret;
if (adau1372->pd_gpio)
- gpiod_set_value(adau1372->pd_gpio, 0);
+ gpiod_set_value_cansleep(adau1372->pd_gpio, 0);
if (adau1372->switch_mode)
adau1372->switch_mode(adau1372->dev);
@@ -818,8 +818,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable)
ADAU1372_CLK_CTRL_PLL_EN,
0);
regcache_cache_only(adau1372->regmap, true);
- if (adau1372->pd_gpio)
- gpiod_set_value(adau1372->pd_gpio, 1);
+ gpiod_set_value_cansleep(adau1372->pd_gpio, 1);
clk_disable_unprepare(adau1372->mclk);
return ret;
}
@@ -836,7 +835,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable)
* map. No need to do any register writes to manually
* turn things off.
*/
- gpiod_set_value(adau1372->pd_gpio, 1);
+ gpiod_set_value_cansleep(adau1372->pd_gpio, 1);
regcache_mark_dirty(adau1372->regmap);
} else {
regmap_update_bits(adau1372->regmap, ADAU1372_REG_CLK_CTRL,
@@ -1001,6 +1000,17 @@ int adau1372_probe(struct device *dev, struct regmap *regmap,
regmap_write(regmap, ADAU1372_REG_OP_STAGE_MUTE, 0x0);
+ /*
+ * Set sane default values for the Output ASRC and DAC input muxes,
+ * since the power-on reset defaults are invalid "Reserved" states.
+ */
+ regmap_write(regmap, ADAU1372_REG_ASRCO_SOURCE_0_1,
+ 0x54); /* Decimator0, Decimator1 */
+ regmap_write(regmap, ADAU1372_REG_ASRCO_SOURCE_2_3,
+ 0x76); /* Decimator2, Decimator3 */
+ regmap_write(regmap, ADAU1372_REG_DAC_SOURCE,
+ 0xdc); /* Input ASRC0, Input ASRC1 */
+
regmap_write(regmap, 0x7, 0x01); /* CLOCK OUT */
return devm_snd_soc_register_component(dev, &adau1372_driver, &adau1372_dai_driver, 1);
diff --git a/sound/soc/codecs/cs42l52.c b/sound/soc/codecs/cs42l52.c
index 90bf535fc5a5..cf3ca3978336 100644
--- a/sound/soc/codecs/cs42l52.c
+++ b/sound/soc/codecs/cs42l52.c
@@ -273,7 +273,7 @@ static const char * const cs42l52_ng_type_text[] = {
};
static SOC_ENUM_SINGLE_DECL(ng_type_enum,
- CS42L52_NOISE_GATE_CTL, 6,
+ CS42L52_NOISE_GATE_CTL, 7,
cs42l52_ng_type_text);
static const char * const left_swap_text[] = {
diff --git a/sound/soc/codecs/cs42l56.c b/sound/soc/codecs/cs42l56.c
index 3b0e715549c9..206c45318ad4 100644
--- a/sound/soc/codecs/cs42l56.c
+++ b/sound/soc/codecs/cs42l56.c
@@ -449,7 +449,7 @@ static const struct snd_kcontrol_new cs42l56_snd_controls[] = {
0, 0x3f, 0, 0),
SOC_SINGLE_TLV("Limiter MAX", CS42L56_LIM_THRESHOLD_CTL,
5, 0x07, 1, alc_tlv),
- SOC_SINGLE_TLV("Limiter Cushion", CS42L56_ALC_THRESHOLD,
+ SOC_SINGLE_TLV("Limiter Cushion", CS42L56_LIM_THRESHOLD_CTL,
2, 0x07, 1, alc_tlv),
SOC_SINGLE("NG Switch", CS42L56_NOISE_GATE_CTL, 6, 1, 1),
diff --git a/sound/soc/codecs/hdac_hda.c b/sound/soc/codecs/hdac_hda.c
index 21a00c86a139..4dcb4eb04ec2 100644
--- a/sound/soc/codecs/hdac_hda.c
+++ b/sound/soc/codecs/hdac_hda.c
@@ -612,10 +612,8 @@ static int hdac_hda_dev_probe(struct hdac_device *hdev)
&hdac_hda_codec, hdac_hda_dais,
ARRAY_SIZE(hdac_hda_dais));
- if (ret < 0) {
+ if (ret < 0)
dev_err(&hdev->dev, "failed to register HDA codec %d\n", ret);
- return ret;
- }
snd_hdac_ext_bus_link_put(hdev->bus, hlink);
diff --git a/sound/soc/codecs/max98371.c b/sound/soc/codecs/max98371.c
index bac9d1bcf60a..ce6bfcbf3437 100644
--- a/sound/soc/codecs/max98371.c
+++ b/sound/soc/codecs/max98371.c
@@ -166,7 +166,7 @@ static const struct snd_kcontrol_new max98371_snd_controls[] = {
0, (1<<MAX98371_DHT_MAX_WIDTH)-1, 0,
max98371_dht_max_gain),
SOC_SINGLE_TLV("Speaker DHT Min Volume", MAX98371_DHT_GAIN,
- 0, (1<<MAX98371_DHT_GAIN_WIDTH)-1, 0,
+ 4, (1<<MAX98371_DHT_GAIN_WIDTH)-1, 0,
max98371_dht_min_gain),
SOC_SINGLE_TLV("Speaker DHT Rotation Volume", MAX98371_DHT_GAIN,
0, (1<<MAX98371_DHT_ROT_WIDTH)-1, 0,
diff --git a/sound/soc/codecs/nau8822.c b/sound/soc/codecs/nau8822.c
index cd5053cfd521..3670da488584 100644
--- a/sound/soc/codecs/nau8822.c
+++ b/sound/soc/codecs/nau8822.c
@@ -379,9 +379,9 @@ static const struct snd_kcontrol_new nau8822_snd_controls[] = {
NAU8822_REG_RIGHT_ADC_BOOST_CONTROL, 0, 7, 0, boost_tlv),
SOC_SINGLE("DAC 128x Oversampling Switch",
- NAU8822_REG_DAC_CONTROL, 5, 1, 0),
+ NAU8822_REG_DAC_CONTROL, 3, 1, 0),
SOC_SINGLE("ADC 128x Oversampling Switch",
- NAU8822_REG_ADC_CONTROL, 5, 1, 0),
+ NAU8822_REG_ADC_CONTROL, 3, 1, 0),
};
/* LMAIN and RMAIN Mixer */
diff --git a/sound/soc/codecs/pcm3168a.c b/sound/soc/codecs/pcm3168a.c
index e5ac1146588c..983ca03e7a75 100644
--- a/sound/soc/codecs/pcm3168a.c
+++ b/sound/soc/codecs/pcm3168a.c
@@ -104,7 +104,7 @@ static SOC_ENUM_SINGLE_DECL(pcm3168a_dac_zf_func, PCM3168A_DAC_ATT_DEMP_ZF,
static const char *const pcm3168a_pol[] = { "Active High", "Active Low" };
static SOC_ENUM_SINGLE_DECL(pcm3168a_dac_zf_pol, PCM3168A_DAC_ATT_DEMP_ZF,
- PCM3168A_DAC_ATSPDA_SHIFT, pcm3168a_pol);
+ PCM3168A_DAC_ZREV_SHIFT, pcm3168a_pol);
static const char *const pcm3168a_con[] = { "Differential", "Single-Ended" };
diff --git a/sound/soc/codecs/rt274.c b/sound/soc/codecs/rt274.c
index 4667bf7561b1..f664a6e13923 100644
--- a/sound/soc/codecs/rt274.c
+++ b/sound/soc/codecs/rt274.c
@@ -619,9 +619,11 @@ static int rt274_hw_params(struct snd_pcm_substream *substream,
unsigned int val = 0;
int d_len_code = 0, c_len_code = 0;
+ /* bit 14 Sample Rate Base, 0: 48kHz 1: 44.1kHz */
switch (params_rate(params)) {
- /* bit 14 0:48K 1:44.1K */
case 44100:
+ val |= BIT(14);
+ break;
case 48000:
break;
default:
diff --git a/sound/soc/codecs/rt286.c b/sound/soc/codecs/rt286.c
index ceb56647e369..42ed864c062f 100644
--- a/sound/soc/codecs/rt286.c
+++ b/sound/soc/codecs/rt286.c
@@ -951,7 +951,7 @@ static int rt286_probe(struct snd_soc_component *component)
if (rt286->i2c->irq)
schedule_delayed_work(&rt286->jack_detect_work,
- msecs_to_jiffies(50));
+ msecs_to_jiffies(1250));
return 0;
}
diff --git a/sound/soc/codecs/tas2562.c b/sound/soc/codecs/tas2562.c
index 07336352272b..bb9dfd10d2d7 100644
--- a/sound/soc/codecs/tas2562.c
+++ b/sound/soc/codecs/tas2562.c
@@ -474,10 +474,14 @@ static int tas2562_volume_control_put(struct snd_kcontrol *kcontrol,
{
struct snd_soc_component *component = snd_soc_kcontrol_component(kcontrol);
struct tas2562_data *tas2562 = snd_soc_component_get_drvdata(component);
- int ret;
+ int ret, index;
u32 reg_val;
- reg_val = float_vol_db_lookup[ucontrol->value.integer.value[0]/2];
+ index = ucontrol->value.integer.value[0] / 2;
+ if (index < 0 || index >= ARRAY_SIZE(float_vol_db_lookup))
+ return -EINVAL;
+
+ reg_val = float_vol_db_lookup[index];
/*
* The device applies the 32-bit coefficient to the playback path on
* the write to DVC_CFG4 (the LSB, book 0 page 2 reg 0x0F), so the
diff --git a/sound/soc/codecs/wm8903.c b/sound/soc/codecs/wm8903.c
index 41346e5ec5ad..aebe47e9dc0d 100644
--- a/sound/soc/codecs/wm8903.c
+++ b/sound/soc/codecs/wm8903.c
@@ -678,7 +678,7 @@ SOC_ENUM("DRC Decay Rate", drc_decay),
SOC_ENUM("DRC FF Delay", drc_ff_delay),
SOC_SINGLE("DRC Anticlip Switch", WM8903_DRC_0, 1, 1, 0),
SOC_SINGLE("DRC QR Switch", WM8903_DRC_0, 2, 1, 0),
-SOC_SINGLE_TLV("DRC QR Threshold Volume", WM8903_DRC_0, 6, 3, 0, drc_tlv_max),
+SOC_SINGLE_TLV("DRC QR Threshold Volume", WM8903_DRC_1, 6, 3, 0, drc_tlv_max),
SOC_ENUM("DRC QR Decay Rate", drc_qr_decay),
SOC_SINGLE("DRC Smoothing Switch", WM8903_DRC_0, 3, 1, 0),
SOC_SINGLE("DRC Smoothing Hysteresis Switch", WM8903_DRC_0, 0, 1, 0),
diff --git a/sound/soc/codecs/wm8978.c b/sound/soc/codecs/wm8978.c
index a682f8020eb6..73b17376ce1c 100644
--- a/sound/soc/codecs/wm8978.c
+++ b/sound/soc/codecs/wm8978.c
@@ -255,9 +255,9 @@ static const struct snd_kcontrol_new wm8978_snd_controls[] = {
/* DAC / ADC oversampling */
SOC_SINGLE("DAC 128x Oversampling Switch", WM8978_DAC_CONTROL,
- 5, 1, 0),
+ 3, 1, 0),
SOC_SINGLE("ADC 128x Oversampling Switch", WM8978_ADC_CONTROL,
- 5, 1, 0),
+ 3, 1, 0),
};
/* Mixer #1: Output (OUT1, OUT2) Mixer: mix AUX, Input mixer output and DAC */
diff --git a/sound/soc/intel/avs/pcm.c b/sound/soc/intel/avs/pcm.c
index 9d3c0ea99a29..b6730cbd1222 100644
--- a/sound/soc/intel/avs/pcm.c
+++ b/sound/soc/intel/avs/pcm.c
@@ -556,7 +556,7 @@ static int avs_dai_fe_prepare(struct snd_pcm_substream *substream, struct snd_so
data = snd_soc_dai_get_dma_data(dai, substream);
host_stream = data->host_stream;
- if (runtime->state == SNDRV_PCM_STATE_XRUN)
+ if (runtime->state != SNDRV_PCM_STATE_PREPARED)
hdac_stream(host_stream)->prepared = false;
if (hdac_stream(host_stream)->prepared)
return 0;
@@ -879,7 +879,9 @@ static int avs_soc_component_register(struct device *dev, const char *name,
return ret;
/* force name change after ASoC is done with its init */
- acomp->base.name = name;
+ acomp->base.name = devm_kstrdup(dev, name, GFP_KERNEL);
+ if (!acomp->base.name)
+ return -ENOMEM;
INIT_LIST_HEAD(&acomp->node);
return snd_soc_add_component(&acomp->base, cpu_dais, num_cpu_dais);
diff --git a/sound/soc/qcom/lpass-cdc-dma.c b/sound/soc/qcom/lpass-cdc-dma.c
index 31b9f1c22bee..97c240e8bec3 100644
--- a/sound/soc/qcom/lpass-cdc-dma.c
+++ b/sound/soc/qcom/lpass-cdc-dma.c
@@ -258,7 +258,7 @@ static int lpass_cdc_dma_daiops_trigger(struct snd_pcm_substream *substream,
int cmd, struct snd_soc_dai *dai)
{
struct snd_soc_pcm_runtime *soc_runtime = asoc_substream_to_rtd(substream);
- struct lpaif_dmactl *dmactl;
+ struct lpaif_dmactl *dmactl = NULL;
int ret = 0, id;
switch (cmd) {
diff --git a/sound/soc/qcom/lpass-cpu.c b/sound/soc/qcom/lpass-cpu.c
index b6314e924c8f..ac92f029d235 100644
--- a/sound/soc/qcom/lpass-cpu.c
+++ b/sound/soc/qcom/lpass-cpu.c
@@ -1228,8 +1228,8 @@ int asoc_qcom_lpass_cpu_platform_probe(struct platform_device *pdev)
}
if (drvdata->mi2s_playback_sd_mode[dai_id] ==
LPAIF_I2SCTL_MODE_QUAD01) {
- variant->dai_driver[dai_id].playback.channels_min = 4;
- variant->dai_driver[dai_id].playback.channels_max = 4;
+ variant->dai_driver[i].playback.channels_min = 4;
+ variant->dai_driver[i].playback.channels_max = 4;
}
}
diff --git a/sound/soc/qcom/qdsp6/q6asm.c b/sound/soc/qcom/qdsp6/q6asm.c
index 9f53cffe5184..b6bdb977029d 100644
--- a/sound/soc/qcom/qdsp6/q6asm.c
+++ b/sound/soc/qcom/qdsp6/q6asm.c
@@ -653,6 +653,11 @@ static int32_t q6asm_stream_callback(struct apr_device *adev,
goto done;
}
break;
+ case ASM_DATA_CMD_EOS:
+ case ASM_DATA_CMD_READ_V2:
+ case ASM_DATA_CMD_WRITE_V2:
+ /* response as result of close stream */
+ goto done;
default:
dev_err(ac->dev, "command[0x%x] not expecting rsp\n",
result->opcode);
diff --git a/sound/soc/samsung/i2s.c b/sound/soc/samsung/i2s.c
index 9505200f3d11..be0d6984bb86 100644
--- a/sound/soc/samsung/i2s.c
+++ b/sound/soc/samsung/i2s.c
@@ -1531,9 +1531,16 @@ static int samsung_i2s_probe(struct platform_device *pdev)
goto err_disable_pm;
priv->op_clk = clk_get_parent(priv->clk_table[CLK_I2S_RCLK_SRC]);
+ ret = clk_prepare_enable(priv->op_clk);
+ if (ret < 0) {
+ priv->op_clk = NULL;
+ goto err_unregister_clk;
+ }
return 0;
+err_unregister_clk:
+ i2s_unregister_clock_provider(priv);
err_disable_pm:
pm_runtime_disable(&pdev->dev);
err_del_sec:
@@ -1556,6 +1563,7 @@ static int samsung_i2s_remove(struct platform_device *pdev)
i2s_unregister_clock_provider(priv);
i2s_delete_secondary_device(priv);
+ clk_disable_unprepare(priv->op_clk);
clk_disable_unprepare(priv->clk);
pm_runtime_put_noidle(&pdev->dev);
diff --git a/sound/soc/soc-topology.c b/sound/soc/soc-topology.c
index 77296c950376..369435b92646 100644
--- a/sound/soc/soc-topology.c
+++ b/sound/soc/soc-topology.c
@@ -1573,9 +1573,24 @@ static int soc_tplg_dapm_complete(struct soc_tplg *tplg)
return 0;
}
+static int soc_tplg_check_name(const char *name)
+{
+ if (strnlen(name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN) ==
+ SNDRV_CTL_ELEM_ID_NAME_MAXLEN)
+ return -EINVAL;
+
+ return 0;
+}
+
static int set_stream_info(struct soc_tplg *tplg, struct snd_soc_pcm_stream *stream,
struct snd_soc_tplg_stream_caps *caps)
{
+ int ret;
+
+ ret = soc_tplg_check_name(caps->name);
+ if (ret)
+ return ret;
+
stream->stream_name = devm_kstrdup(tplg->dev, caps->name, GFP_KERNEL);
if (!stream->stream_name)
return -ENOMEM;
@@ -1624,7 +1639,11 @@ static int soc_tplg_dai_create(struct soc_tplg *tplg,
if (dai_drv == NULL)
return -ENOMEM;
- if (strlen(pcm->dai_name)) {
+ ret = soc_tplg_check_name(pcm->dai_name);
+ if (ret)
+ goto err;
+
+ if (pcm->dai_name[0]) {
dai_drv->name = devm_kstrdup(tplg->dev, pcm->dai_name, GFP_KERNEL);
if (!dai_drv->name) {
ret = -ENOMEM;
@@ -1733,7 +1752,11 @@ static int soc_tplg_fe_link_create(struct soc_tplg *tplg,
link->dobj.ops = tplg->ops;
link->dobj.type = SND_SOC_DOBJ_DAI_LINK;
- if (strlen(pcm->pcm_name)) {
+ ret = soc_tplg_check_name(pcm->pcm_name);
+ if (ret)
+ goto err;
+
+ if (pcm->pcm_name[0]) {
link->name = devm_kstrdup(tplg->dev, pcm->pcm_name, GFP_KERNEL);
link->stream_name = devm_kstrdup(tplg->dev, pcm->pcm_name, GFP_KERNEL);
if (!link->name || !link->stream_name) {
@@ -1743,7 +1766,11 @@ static int soc_tplg_fe_link_create(struct soc_tplg *tplg,
}
link->id = le32_to_cpu(pcm->pcm_id);
- if (strlen(pcm->dai_name)) {
+ ret = soc_tplg_check_name(pcm->dai_name);
+ if (ret)
+ goto err;
+
+ if (pcm->dai_name[0]) {
link->cpus->dai_name = devm_kstrdup(tplg->dev, pcm->dai_name, GFP_KERNEL);
if (!link->cpus->dai_name) {
ret = -ENOMEM;
@@ -2226,6 +2253,10 @@ static int soc_tplg_dai_config(struct soc_tplg *tplg,
memset(&dai_component, 0, sizeof(dai_component));
+ ret = soc_tplg_check_name(d->dai_name);
+ if (ret)
+ return ret;
+
dai_component.dai_name = d->dai_name;
dai = snd_soc_find_dai(&dai_component);
if (!dai) {
diff --git a/sound/soc/sof/intel/hda.c b/sound/soc/sof/intel/hda.c
index b7d6a54bbb63..4ba7c4be670d 100644
--- a/sound/soc/sof/intel/hda.c
+++ b/sound/soc/sof/intel/hda.c
@@ -128,6 +128,11 @@ static int sdw_params_stream(struct device *dev,
struct snd_soc_dapm_widget *w;
w = snd_soc_dai_get_widget(d, params_data->stream);
+ if (!w) {
+ dev_err(dev, "%s widget not found, check amp link num in the topology\n",
+ d->name);
+ return -EINVAL;
+ }
data.dai_index = (params_data->link_id << 8) | d->id;
data.dai_data = params_data->alh_stream_id;
diff --git a/sound/soc/sof/ipc4-pcm.c b/sound/soc/sof/ipc4-pcm.c
index 732872395980..2a8501d14e0a 100644
--- a/sound/soc/sof/ipc4-pcm.c
+++ b/sound/soc/sof/ipc4-pcm.c
@@ -86,7 +86,9 @@ static int sof_ipc4_trigger_pipelines(struct snd_soc_component *component,
if (ret < 0) {
dev_err(sdev->dev, "failed to pause pipeline %d\n",
swidget->pipeline_id);
- return ret;
+ if (sdev->fw_state != SOF_FW_CRASHED ||
+ state != SOF_IPC4_PIPE_RESET)
+ return ret;
}
}
@@ -100,7 +102,18 @@ static int sof_ipc4_trigger_pipelines(struct snd_soc_component *component,
if (ret < 0) {
dev_err(sdev->dev, "failed to set state %d for pipeline %d\n",
state, swidget->pipeline_id);
- break;
+ /*
+ * workaround: if the firmware is crashed while setting the
+ * pipelines to reset state we must ignore the error code and
+ * reset it to 0.
+ * Since the firmware is crashed we will not send IPC messages
+ * and we are going to see errors printed, but the state of the
+ * widgets will be correct for the next boot.
+ */
+ if (sdev->fw_state != SOF_FW_CRASHED || state != SOF_IPC4_PIPE_RESET)
+ break;
+
+ ret = 0;
}
pipeline->state = state;
diff --git a/sound/soc/sof/ipc4-topology.c b/sound/soc/sof/ipc4-topology.c
index ee6f6edec970..413dc2d8afb6 100644
--- a/sound/soc/sof/ipc4-topology.c
+++ b/sound/soc/sof/ipc4-topology.c
@@ -515,7 +515,7 @@ static int sof_ipc4_widget_setup_comp_dai(struct snd_sof_widget *swidget)
}
list_for_each_entry(w, &sdev->widget_list, list) {
- if (w->widget->sname &&
+ if (!WIDGET_IS_DAI(w->id) || !w->widget->sname ||
strcmp(w->widget->sname, swidget->widget->sname))
continue;
@@ -1237,17 +1237,20 @@ sof_ipc4_prepare_copier_module(struct snd_sof_widget *swidget,
*/
i = 0;
list_for_each_entry(w, &sdev->widget_list, list) {
- if (w->widget->sname &&
+ if (!WIDGET_IS_DAI(w->id) || !w->widget->sname ||
strcmp(w->widget->sname, swidget->widget->sname))
continue;
dai = w->private;
alh_copier = (struct sof_ipc4_copier *)dai->private;
+ if (alh_copier->dai_type != SOF_DAI_INTEL_ALH)
+ continue;
alh_data = &alh_copier->data;
blob->alh_cfg.mapping[i].alh_id = alh_data->gtw_cfg.node_id;
blob->alh_cfg.mapping[i].channel_mask = ch_mask;
i++;
}
+ blob->alh_cfg.count = i;
if (blob->alh_cfg.count > 1) {
int group_id;
diff --git a/sound/soc/stm/stm32_spdifrx.c b/sound/soc/stm/stm32_spdifrx.c
index e66382680e09..6caa6fd118e9 100644
--- a/sound/soc/stm/stm32_spdifrx.c
+++ b/sound/soc/stm/stm32_spdifrx.c
@@ -728,7 +728,7 @@ static irqreturn_t stm32_spdifrx_isr(int irq, void *devid)
if (err) {
regmap_read(spdifrx->regmap, STM32_SPDIFRX_CR, &cr);
- sync_state = FIELD_GET(SPDIFRX_CR_SPDIFEN_MASK, cr) &&
+ sync_state = FIELD_GET(SPDIFRX_CR_SPDIFEN_MASK, cr) ==
SPDIFRX_SPDIFEN_SYNC;
/* SPDIFRX is in STATE_STOP. Disable SPDIFRX to clear errors */
diff --git a/sound/soc/tegra/tegra186_asrc.c b/sound/soc/tegra/tegra186_asrc.c
index 9f12faaa609d..df25f696efed 100644
--- a/sound/soc/tegra/tegra186_asrc.c
+++ b/sound/soc/tegra/tegra186_asrc.c
@@ -831,7 +831,7 @@ static const struct snd_kcontrol_new tegra186_asrc_controls[] = {
tegra186_asrc_put_input_threshold),
SOC_SINGLE_EXT("Stream6 Input Threshold",
- ASRC_STREAM_REG(TEGRA186_ASRC_RX_THRESHOLD, 4), 0, 3, 0,
+ ASRC_STREAM_REG(TEGRA186_ASRC_RX_THRESHOLD, 5), 0, 3, 0,
tegra186_asrc_get_input_threshold,
tegra186_asrc_put_input_threshold),
diff --git a/sound/usb/caiaq/device.c b/sound/usb/caiaq/device.c
index b4b4765725ce..bc35c36148ac 100644
--- a/sound/usb/caiaq/device.c
+++ b/sound/usb/caiaq/device.c
@@ -562,6 +562,9 @@ static int snd_probe(struct usb_interface *intf,
ret = init_card(caiaqdev(card));
if (ret < 0) {
dev_err(&usb_dev->dev, "unable to init card! (ret=%d)\n", ret);
+#ifdef CONFIG_SND_USB_CAIAQ_INPUT
+ snd_usb_caiaq_input_disconnect(caiaqdev(card));
+#endif
snd_card_free(card);
return ret;
}
diff --git a/sound/usb/card.h b/sound/usb/card.h
index 6ec95b2edf86..0d860c57d9d9 100644
--- a/sound/usb/card.h
+++ b/sound/usb/card.h
@@ -89,7 +89,7 @@ struct snd_usb_endpoint {
struct snd_urb_ctx urb[MAX_URBS];
struct snd_usb_packet_info {
- uint32_t packet_size[MAX_PACKS_HS];
+ int packet_size[MAX_PACKS_HS];
int packets;
} next_packet[MAX_URBS];
unsigned int next_packet_head; /* ring buffer offset to read */
@@ -115,6 +115,7 @@ struct snd_usb_endpoint {
unsigned int phase; /* phase accumulator */
unsigned int maxpacksize; /* max packet size in bytes */
unsigned int maxframesize; /* max packet size in frames */
+ unsigned int max_urb_packs; /* packets allocated per data URB */
unsigned int max_urb_frames; /* max URB size in frames */
unsigned int curpacksize; /* current packet size in bytes (for capture) */
unsigned int curframesize; /* current packet size in frames (for capture) */
diff --git a/sound/usb/endpoint.c b/sound/usb/endpoint.c
index 778466624f03..dd81c94cbb52 100644
--- a/sound/usb/endpoint.c
+++ b/sound/usb/endpoint.c
@@ -478,7 +478,7 @@ int snd_usb_queue_pending_output_urbs(struct snd_usb_endpoint *ep,
unsigned long flags;
struct snd_usb_packet_info *packet;
struct snd_urb_ctx *ctx = NULL;
- int err, i;
+ int err;
spin_lock_irqsave(&ep->lock, flags);
if ((!implicit_fb || ep->next_packet_queued > 0) &&
@@ -497,9 +497,10 @@ int snd_usb_queue_pending_output_urbs(struct snd_usb_endpoint *ep,
/* copy over the length information */
if (implicit_fb) {
- ctx->packets = packet->packets;
- for (i = 0; i < packet->packets; i++)
- ctx->packet_size[i] = packet->packet_size[i];
+ ctx->packets = min_t(int, packet->packets,
+ ep->max_urb_packs);
+ memcpy(ctx->packet_size, packet->packet_size,
+ ctx->packets * sizeof(packet->packet_size[0]));
}
/* call the data handler to fill in playback data */
@@ -1252,15 +1253,16 @@ static int data_ep_set_params(struct snd_usb_endpoint *ep)
ep->nurbs = min(max_urbs, urbs_per_period * ep->cur_buffer_periods);
}
+ if (fmt->fmt_type == UAC_FORMAT_TYPE_II)
+ urb_packs++; /* for transfer delimiter */
+ ep->max_urb_packs = urb_packs;
+
/* allocate and initialize data urbs */
for (i = 0; i < ep->nurbs; i++) {
struct snd_urb_ctx *u = &ep->urb[i];
u->index = i;
u->ep = ep;
u->packets = urb_packs;
-
- if (fmt->fmt_type == UAC_FORMAT_TYPE_II)
- u->packets++; /* for transfer delimiter */
u->buffer_size = maxsize * u->packets;
u->urb = usb_alloc_urb(u->packets, GFP_KERNEL);
if (!u->urb)
diff --git a/sound/usb/line6/playback.c b/sound/usb/line6/playback.c
index 8233c61e23f1..0da1998dd5ad 100644
--- a/sound/usb/line6/playback.c
+++ b/sound/usb/line6/playback.c
@@ -182,6 +182,13 @@ static int submit_audio_out_urb(struct snd_line6_pcm *line6pcm)
fsize *= bytes_per_frame;
+ if (fsize > line6pcm->max_packet_size_out) {
+ dev_err(line6pcm->line6->ifcdev,
+ "playback packet too large: %d > %d\n",
+ fsize, line6pcm->max_packet_size_out);
+ return -EMSGSIZE;
+ }
+
fout->offset = urb_size;
fout->length = fsize;
urb_size += fsize;
diff --git a/sound/usb/misc/ua101.c b/sound/usb/misc/ua101.c
index 4d23ec97475d..c43f4e4a8d67 100644
--- a/sound/usb/misc/ua101.c
+++ b/sound/usb/misc/ua101.c
@@ -1022,6 +1022,15 @@ static int detect_usb_format(struct ua101 *ua)
}
ua->playback.usb_pipe = usb_sndisocpipe(ua->dev, usb_endpoint_num(epd));
ua->playback.max_packet_bytes = usb_endpoint_maxp(epd);
+
+ if (ua->capture.max_packet_bytes / ua->capture.frame_bytes *
+ ua->playback.frame_bytes > ua->playback.max_packet_bytes) {
+ dev_err(&ua->dev->dev,
+ "playback packet size %u too small for %u capture frames\n",
+ ua->playback.max_packet_bytes,
+ ua->capture.max_packet_bytes / ua->capture.frame_bytes);
+ return -ENXIO;
+ }
return 0;
}
diff --git a/sound/usb/pcm.c b/sound/usb/pcm.c
index f2c798c4506b..17e539b1fdb4 100644
--- a/sound/usb/pcm.c
+++ b/sound/usb/pcm.c
@@ -1461,7 +1461,8 @@ static int prepare_playback_urb(struct snd_usb_substream *subs,
for (i = 0; i < ctx->packets; i++) {
counts = snd_usb_endpoint_next_packet_size(ep, ctx, i, avail);
- if (counts < 0)
+ if (counts < 0 ||
+ (frames + counts) * stride > ctx->buffer_size)
break;
/* set up descriptor */
urb->iso_frame_desc[i].offset = frames * stride;
diff --git a/sound/usb/quirks.c b/sound/usb/quirks.c
index 22e69a6528df..e2f37a09a8a8 100644
--- a/sound/usb/quirks.c
+++ b/sound/usb/quirks.c
@@ -2263,8 +2263,6 @@ static const struct usb_audio_quirk_flags_table quirk_flags_table[] = {
QUIRK_FLAG_SET_IFACE_FIRST),
DEVICE_FLG(0x262a, 0x9302, /* ddHiFi TC44C */
QUIRK_FLAG_DSD_RAW),
- DEVICE_FLG(0x2708, 0x0002, /* Audient iD14 */
- QUIRK_FLAG_IGNORE_CTL_ERROR),
DEVICE_FLG(0x2772, 0x0502, /* Musical Fidelity M6s DAC */
0), /* for avoiding QUIRK_FLAG_DSD_RAW with vendor match */
DEVICE_FLG(0x2912, 0x30c8, /* Audioengine D1 */
@@ -2330,6 +2328,8 @@ static const struct usb_audio_quirk_flags_table quirk_flags_table[] = {
QUIRK_FLAG_DSD_RAW),
VENDOR_FLG(0x2622, /* IAG Limited devices */
QUIRK_FLAG_DSD_RAW),
+ VENDOR_FLG(0x2708, /* Audient devices */
+ QUIRK_FLAG_IGNORE_CTL_ERROR),
VENDOR_FLG(0x278b, /* Rotel? */
QUIRK_FLAG_DSD_RAW),
VENDOR_FLG(0x292b, /* Gustard/Ess based devices */
diff --git a/tools/gpio/Makefile b/tools/gpio/Makefile
index 342e056c8c66..1cd7636b9cac 100644
--- a/tools/gpio/Makefile
+++ b/tools/gpio/Makefile
@@ -7,7 +7,13 @@ bindir ?= /usr/bin
# isn't set and when invoked from selftests build, where srctree
# is set to ".". building_out_of_srctree is undefined for in srctree
# builds
+ifeq ($(srctree),)
+update_srctree := 1
+endif
ifndef building_out_of_srctree
+update_srctree := 1
+endif
+ifeq ($(update_srctree),1)
srctree := $(patsubst %/,%,$(dir $(CURDIR)))
srctree := $(patsubst %/,%,$(dir $(srctree)))
endif
diff --git a/tools/perf/tests/bpf.c b/tools/perf/tests/bpf.c
index 6a4235a9cf57..d5bf0e47a48a 100644
--- a/tools/perf/tests/bpf.c
+++ b/tools/perf/tests/bpf.c
@@ -29,11 +29,12 @@
static int epoll_pwait_loop(void)
{
+ struct epoll_event events;
int i;
/* Should fail NR_ITERS times */
for (i = 0; i < NR_ITERS; i++)
- epoll_pwait(-(i + 1), NULL, 0, 0, NULL);
+ epoll_pwait(-(i + 1), &events, 0, 0, NULL);
return 0;
}
diff --git a/tools/perf/util/parse-events.c b/tools/perf/util/parse-events.c
index 5973f46c2375..b77686a7bd26 100644
--- a/tools/perf/util/parse-events.c
+++ b/tools/perf/util/parse-events.c
@@ -1585,7 +1585,7 @@ int parse_events_multi_pmu_add(struct parse_events_state *parse_state,
if (parse_events_term__num(&term,
PARSE_EVENTS__TERM_TYPE_USER,
- config, 1, false, &config,
+ config, 1, false, NULL,
NULL) < 0) {
free(config);
goto out_err;
diff --git a/tools/testing/cxl/test/cxl.c b/tools/testing/cxl/test/cxl.c
index 7be8bc4e4500..37bd29e55590 100644
--- a/tools/testing/cxl/test/cxl.c
+++ b/tools/testing/cxl/test/cxl.c
@@ -533,26 +533,22 @@ static int mock_decoder_commit(struct cxl_decoder *cxld)
return 0;
}
-static int mock_decoder_reset(struct cxl_decoder *cxld)
+static void mock_decoder_reset(struct cxl_decoder *cxld)
{
struct cxl_port *port = to_cxl_port(cxld->dev.parent);
int id = cxld->id;
if ((cxld->flags & CXL_DECODER_F_ENABLE) == 0)
- return 0;
+ return;
dev_dbg(&port->dev, "%s reset\n", dev_name(&cxld->dev));
- if (port->commit_end != id) {
+ if (port->commit_end == id)
+ cxl_port_commit_reap(cxld);
+ else
dev_dbg(&port->dev,
"%s: out of order reset, expected decoder%d.%d\n",
dev_name(&cxld->dev), port->id, port->commit_end);
- return -EBUSY;
- }
-
- port->commit_end--;
cxld->flags &= ~CXL_DECODER_F_ENABLE;
-
- return 0;
}
static void default_mock_decoder(struct cxl_decoder *cxld)
diff --git a/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c b/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c
index 92267abb462f..be540970fcc1 100644
--- a/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c
+++ b/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c
@@ -103,9 +103,9 @@ static int dump_tcp_sock(struct seq_file *seq, struct tcp_sock *tp,
} else if (icsk->icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = icsk->icsk_timeout;
- } else if (timer_pending(&sp->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
- timer_expires = sp->sk_timer.expires;
+ timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = bpf_jiffies64();
diff --git a/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c b/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c
index 943f7bba180e..9f72398fc3cc 100644
--- a/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c
+++ b/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c
@@ -103,9 +103,9 @@ static int dump_tcp6_sock(struct seq_file *seq, struct tcp6_sock *tp,
} else if (icsk->icsk_pending == ICSK_TIME_PROBE0) {
timer_active = 4;
timer_expires = icsk->icsk_timeout;
- } else if (timer_pending(&sp->sk_timer)) {
+ } else if (timer_pending(&icsk->icsk_keepalive_timer)) {
timer_active = 2;
- timer_expires = sp->sk_timer.expires;
+ timer_expires = icsk->icsk_keepalive_timer.expires;
} else {
timer_active = 0;
timer_expires = bpf_jiffies64();
diff --git a/tools/testing/selftests/bpf/progs/test_global_func16.c b/tools/testing/selftests/bpf/progs/test_global_func16.c
index e7206304632e..e3e64bc472cd 100644
--- a/tools/testing/selftests/bpf/progs/test_global_func16.c
+++ b/tools/testing/selftests/bpf/progs/test_global_func16.c
@@ -13,7 +13,7 @@ __noinline int foo(int (*arr)[10])
}
SEC("cgroup_skb/ingress")
-__failure __msg("invalid indirect read from stack")
+__success
int global_func16(struct __sk_buff *skb)
{
int array[10];
diff --git a/tools/testing/selftests/bpf/verifier/atomic_cmpxchg.c b/tools/testing/selftests/bpf/verifier/atomic_cmpxchg.c
index b39665f33524..096219311951 100644
--- a/tools/testing/selftests/bpf/verifier/atomic_cmpxchg.c
+++ b/tools/testing/selftests/bpf/verifier/atomic_cmpxchg.c
@@ -83,17 +83,6 @@
.result = REJECT,
.errstr = "!read_ok",
},
-{
- "Can't use cmpxchg on uninit memory",
- .insns = {
- BPF_MOV64_IMM(BPF_REG_0, 3),
- BPF_MOV64_IMM(BPF_REG_2, 4),
- BPF_ATOMIC_OP(BPF_DW, BPF_CMPXCHG, BPF_REG_10, BPF_REG_2, -8),
- BPF_EXIT_INSN(),
- },
- .result = REJECT,
- .errstr = "invalid read from stack",
-},
{
"BPF_W cmpxchg should zero top 32 bits",
.insns = {
diff --git a/tools/testing/selftests/bpf/verifier/basic_stack.c b/tools/testing/selftests/bpf/verifier/basic_stack.c
index f995777dddb3..fb79c6112df5 100644
--- a/tools/testing/selftests/bpf/verifier/basic_stack.c
+++ b/tools/testing/selftests/bpf/verifier/basic_stack.c
@@ -17,8 +17,10 @@
BPF_EXIT_INSN(),
},
.fixup_map_hash_8b = { 2 },
- .errstr = "invalid indirect read from stack",
- .result = REJECT,
+ .result = ACCEPT,
+ .retval = POINTER_VALUE,
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "invalid indirect read from stack",
},
{
"uninitialized stack2",
@@ -27,8 +29,10 @@
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_2, -8),
BPF_EXIT_INSN(),
},
- .errstr = "invalid read from stack",
- .result = REJECT,
+ .result = ACCEPT,
+ .retval = POINTER_VALUE,
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "invalid read from stack",
},
{
"invalid fp arithmetic",
diff --git a/tools/testing/selftests/bpf/verifier/calls.c b/tools/testing/selftests/bpf/verifier/calls.c
index a201d2871bfb..a8ce7f534ba6 100644
--- a/tools/testing/selftests/bpf/verifier/calls.c
+++ b/tools/testing/selftests/bpf/verifier/calls.c
@@ -1501,7 +1501,9 @@
.prog_type = BPF_PROG_TYPE_XDP,
.fixup_map_hash_8b = { 23 },
.result = REJECT,
- .errstr = "invalid read from stack R7 off=-16 size=8",
+ .errstr = "R0 invalid mem access 'scalar'",
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "invalid read from stack R7 off=-16 size=8",
},
{
"calls: two calls that receive map_value via arg=ptr_stack_of_caller. test1",
diff --git a/tools/testing/selftests/bpf/verifier/raw_stack.c b/tools/testing/selftests/bpf/verifier/raw_stack.c
index eb5ed936580b..36bc7f7b81ea 100644
--- a/tools/testing/selftests/bpf/verifier/raw_stack.c
+++ b/tools/testing/selftests/bpf/verifier/raw_stack.c
@@ -10,9 +10,11 @@
BPF_LDX_MEM(BPF_DW, BPF_REG_0, BPF_REG_6, 0),
BPF_EXIT_INSN(),
},
- .result = REJECT,
- .errstr = "invalid read from stack R6 off=-8 size=8",
- .prog_type = BPF_PROG_TYPE_SCHED_CLS,
+ .result = ACCEPT,
+ .retval = POINTER_VALUE,
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "invalid read from stack R6 off=-8 size=8",
+ .prog_type = BPF_PROG_TYPE_SOCKET_FILTER,
},
{
"raw_stack: skb_load_bytes, negative len",
diff --git a/tools/testing/selftests/bpf/verifier/var_off.c b/tools/testing/selftests/bpf/verifier/var_off.c
index b183e26c03f1..5da76d054986 100644
--- a/tools/testing/selftests/bpf/verifier/var_off.c
+++ b/tools/testing/selftests/bpf/verifier/var_off.c
@@ -58,11 +58,15 @@
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
},
- .result = REJECT,
- .errstr = "invalid variable-offset read from stack R2",
- .prog_type = BPF_PROG_TYPE_LWT_IN,
+ .result = ACCEPT,
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "R2 variable stack access prohibited for !root",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
},
{
+/* Check that the maximum stack depth is correctly maintained according to the
+ * maximum possible variable offset.
+ */
"variable-offset stack write, priv vs unpriv",
.insns = {
/* Get an unknown value */
@@ -75,13 +79,43 @@
*/
BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_10),
/* Dereference it for a stack write */
+ BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
- /* Now read from the address we just wrote. This shows
- * that, after a variable-offset write, a priviledged
- * program can read the slots that were in the range of
- * that write (even if the verifier doesn't actually know
- * if the slot being read was really written to or not.
+ BPF_EXIT_INSN(),
+ },
+ /* Variable stack access is rejected for unprivileged.
*/
+ .errstr_unpriv = "R2 variable stack access prohibited for !root",
+ .result_unpriv = REJECT,
+ .result = ACCEPT,
+},
+{
+/* Similar to the previous test, but this time also perform a read from the
+ * address written to with a variable offset. The read is allowed, showing that,
+ * after a variable-offset write, a priviledged program can read the slots that
+ * were in the range of that write (even if the verifier doesn't actually know if
+ * the slot being read was really written to or not.
+ *
+ * Despite this test being mostly a superset, the previous test is also kept for
+ * the sake of it checking the stack depth in the case where there is no read.
+ */
+/* Check that the maximum stack depth is correctly maintained according to the
+ * maximum possible variable offset.
+ */
+ "variable-offset stack write, followed by read",
+ .insns = {
+ /* Get an unknown value */
+ BPF_LDX_MEM(BPF_W, BPF_REG_2, BPF_REG_1, 0),
+ /* Make it small and 8-byte aligned */
+ BPF_ALU64_IMM(BPF_AND, BPF_REG_2, 8),
+ BPF_ALU64_IMM(BPF_SUB, BPF_REG_2, 16),
+ /* Add it to fp. We now have either fp-8 or fp-16, but
+ * we don't know which
+ */
+ BPF_ALU64_REG(BPF_ADD, BPF_REG_2, BPF_REG_10),
+ /* Dereference it for a stack write */
+ BPF_ST_MEM(BPF_DW, BPF_REG_2, 0, 0),
+ /* Now read from the address we just wrote. */
BPF_LDX_MEM(BPF_DW, BPF_REG_3, BPF_REG_2, 0),
BPF_MOV64_IMM(BPF_REG_0, 0),
BPF_EXIT_INSN(),
@@ -233,9 +267,10 @@
BPF_EXIT_INSN(),
},
.fixup_map_hash_8b = { 5 },
- .errstr = "invalid indirect read from stack R2 var_off",
- .result = REJECT,
- .prog_type = BPF_PROG_TYPE_LWT_IN,
+ .result = ACCEPT,
+ .result_unpriv = REJECT,
+ .errstr_unpriv = "R2 variable stack access prohibited for !root",
+ .prog_type = BPF_PROG_TYPE_CGROUP_SKB,
},
{
"indirect variable-offset stack access, priv vs unpriv",
diff --git a/tools/testing/selftests/ftrace/test.d/filter/event-filter-incomplete.tc b/tools/testing/selftests/ftrace/test.d/filter/event-filter-incomplete.tc
new file mode 100644
index 000000000000..cc6df762845f
--- /dev/null
+++ b/tools/testing/selftests/ftrace/test.d/filter/event-filter-incomplete.tc
@@ -0,0 +1,24 @@
+#!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# description: event filter - reject incomplete logical expressions
+# requires: events/sched/sched_switch/filter
+# flags: instance
+
+filter=events/sched/sched_switch/filter
+
+for base in \
+ 'common_pid == 1' \
+ 'common_pid == 1 || common_pid == 2' \
+ 'common_pid == 1 && common_pid == 2' \
+ '!(common_pid == 1 || common_pid == 2)'
+do
+ printf '%s' "$base" > "$filter"
+ for tail in '&&' '||' '&& !' '|| !' '&& !! ' '|| !! '; do
+ if printf '%s' "$base $tail" > "$filter" 2>/dev/null; then
+ echo "Accepted incomplete filter: $base $tail"
+ exit_fail
+ fi
+ done
+done
+
+echo 0 > "$filter"
diff --git a/tools/testing/selftests/net/forwarding/tc_actions.sh b/tools/testing/selftests/net/forwarding/tc_actions.sh
index b0f5e55d2d0b..589629636502 100755
--- a/tools/testing/selftests/net/forwarding/tc_actions.sh
+++ b/tools/testing/selftests/net/forwarding/tc_actions.sh
@@ -235,9 +235,6 @@ mirred_egress_to_ingress_tcp_test()
check_err $? "didn't mirred redirect ICMP"
tc_check_packets "dev $h1 ingress" 102 10
check_err $? "didn't drop mirred ICMP"
- local overlimits=$(tc_rule_stats_get ${h1} 101 egress .overlimits)
- test ${overlimits} = 10
- check_err $? "wrong overlimits, expected 10 got ${overlimits}"
tc filter del dev $h1 egress protocol ip pref 100 handle 100 flower
tc filter del dev $h1 egress protocol ip pref 101 handle 101 flower
diff --git a/tools/testing/selftests/net/tls.c b/tools/testing/selftests/net/tls.c
index d89ee6e1926c..a23fcd103a27 100644
--- a/tools/testing/selftests/net/tls.c
+++ b/tools/testing/selftests/net/tls.c
@@ -29,6 +29,7 @@ static int fips_enabled;
struct tls_crypto_info_keys {
union {
+ struct tls_crypto_info crypto_info;
struct tls12_crypto_info_aes_gcm_128 aes128;
struct tls12_crypto_info_chacha20_poly1305 chacha20;
struct tls12_crypto_info_sm4_gcm sm4gcm;
@@ -1460,6 +1461,589 @@ TEST_F(tls, shutdown_reuse)
EXPECT_EQ(errno, EISCONN);
}
+TEST_F(tls, getsockopt)
+{
+ struct tls_crypto_info_keys expect, get;
+ socklen_t len;
+
+ /* get only the version/cipher */
+ len = sizeof(struct tls_crypto_info);
+ memrnd(&get, sizeof(get));
+ EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &get, &len), 0);
+ EXPECT_EQ(len, sizeof(struct tls_crypto_info));
+ EXPECT_EQ(get.crypto_info.version, variant->tls_version);
+ EXPECT_EQ(get.crypto_info.cipher_type, variant->cipher_type);
+
+ /* get the full crypto_info */
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &expect);
+ len = expect.len;
+ memrnd(&get, sizeof(get));
+ EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &get, &len), 0);
+ EXPECT_EQ(len, expect.len);
+ EXPECT_EQ(get.crypto_info.version, variant->tls_version);
+ EXPECT_EQ(get.crypto_info.cipher_type, variant->cipher_type);
+ EXPECT_EQ(memcmp(&get, &expect, expect.len), 0);
+
+ /* short get should fail */
+ len = sizeof(struct tls_crypto_info) - 1;
+ EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &get, &len), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* partial get of the cipher data should fail */
+ len = expect.len - 1;
+ EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &get, &len), -1);
+ EXPECT_EQ(errno, EINVAL);
+}
+
+TEST_F(tls, recv_efault)
+{
+ char *rec1 = "1111111111";
+ char *rec2 = "2222222222";
+ struct msghdr hdr = {};
+ struct iovec iov[2];
+ char recv_mem[12];
+ int ret;
+
+ if (self->notls)
+ SKIP(return, "no TLS support");
+
+ EXPECT_EQ(send(self->fd, rec1, 10, 0), 10);
+ EXPECT_EQ(send(self->fd, rec2, 10, 0), 10);
+
+ iov[0].iov_base = recv_mem;
+ iov[0].iov_len = sizeof(recv_mem);
+ iov[1].iov_base = NULL; /* broken iov to make process_rx_list fail */
+ iov[1].iov_len = 1;
+
+ hdr.msg_iovlen = 2;
+ hdr.msg_iov = iov;
+
+ EXPECT_EQ(recv(self->cfd, recv_mem, 1, 0), 1);
+ EXPECT_EQ(recv_mem[0], rec1[0]);
+
+ ret = recvmsg(self->cfd, &hdr, 0);
+ EXPECT_LE(ret, sizeof(recv_mem));
+ EXPECT_GE(ret, 9);
+ EXPECT_EQ(memcmp(rec1, recv_mem, 9), 0);
+ if (ret > 9)
+ EXPECT_EQ(memcmp(rec2, recv_mem + 9, ret - 9), 0);
+}
+
+#define TLS_RECORD_TYPE_HANDSHAKE 0x16
+
+TEST_F(tls_basic, recvmsg_nopad_retry_iov)
+{
+ char payload[32];
+ char first_iov[sizeof(payload)];
+ char later_iov[sizeof(payload) * 2];
+ char expected_later_iov[sizeof(later_iov)];
+ char cbuf[CMSG_SPACE(sizeof(char))];
+ struct tls_crypto_info_keys tls13;
+ struct iovec iov[] = {
+ { .iov_base = first_iov, .iov_len = sizeof(first_iov) },
+ { .iov_base = later_iov, .iov_len = sizeof(later_iov) },
+ };
+ struct msghdr msg = {
+ .msg_iov = iov,
+ .msg_iovlen = ARRAY_SIZE(iov),
+ .msg_control = cbuf,
+ .msg_controllen = sizeof(cbuf),
+ };
+ int one = 1;
+ int ret;
+ int i;
+
+ if (self->notls)
+ SKIP(return, "no TLS support");
+
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls13, 0);
+
+ ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls13, tls13.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls13, tls13.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX_EXPECT_NO_PAD,
+ &one, sizeof(one));
+ ASSERT_EQ(ret, 0);
+
+ for (i = 0; i < sizeof(payload); i++)
+ payload[i] = 0x40 + i;
+ memset(first_iov, 0xa5, sizeof(first_iov));
+ memset(later_iov, 0x5a, sizeof(later_iov));
+ memset(expected_later_iov, 0x5a, sizeof(expected_later_iov));
+
+ /* A control record forces optimistic TLS 1.3 RX to retry. */
+ ret = tls_send_cmsg(self->fd, TLS_RECORD_TYPE_HANDSHAKE,
+ payload, sizeof(payload), 0);
+ ASSERT_EQ(ret, sizeof(payload));
+
+ ret = recvmsg(self->cfd, &msg, 0);
+ ASSERT_EQ(ret, sizeof(payload));
+ EXPECT_EQ(memcmp(first_iov, payload, sizeof(payload)), 0);
+ EXPECT_EQ(memcmp(later_iov, expected_later_iov,
+ sizeof(later_iov)), 0);
+}
+
+/* key_update, length 1, update_not_requested */
+static const char key_update_msg[] = "\x18\x00\x00\x01\x00";
+static void tls_send_keyupdate(struct __test_metadata *_metadata, int fd)
+{
+ size_t len = sizeof(key_update_msg);
+
+ EXPECT_EQ(tls_send_cmsg(fd, TLS_RECORD_TYPE_HANDSHAKE,
+ (char *)key_update_msg, len, 0),
+ len);
+}
+
+static void tls_recv_keyupdate(struct __test_metadata *_metadata, int fd, int flags)
+{
+ char buf[100];
+
+ EXPECT_EQ(tls_recv_cmsg(_metadata, fd, TLS_RECORD_TYPE_HANDSHAKE, buf, sizeof(buf), flags),
+ sizeof(key_update_msg));
+ EXPECT_EQ(memcmp(buf, key_update_msg, sizeof(key_update_msg)), 0);
+}
+
+/* set the key to 0 then 1 for RX, immediately to 1 for TX */
+TEST_F(tls_basic, rekey_rx)
+{
+ struct tls_crypto_info_keys tls12_0, tls12_1;
+ char const *test_str = "test_message";
+ int send_len = strlen(test_str) + 1;
+ char buf[20];
+ int ret;
+
+ if (self->notls)
+ return;
+
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls12_0, 0);
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls12_1, 1);
+
+ ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_1, tls12_1.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_0, tls12_0.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_1, tls12_1.len);
+ EXPECT_EQ(ret, 0);
+
+ EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len);
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str, send_len), 0);
+}
+
+/* set the key to 0 then 1 for TX, immediately to 1 for RX */
+TEST_F(tls_basic, rekey_tx)
+{
+ struct tls_crypto_info_keys tls12_0, tls12_1;
+ char const *test_str = "test_message";
+ int send_len = strlen(test_str) + 1;
+ char buf[20];
+ int ret;
+
+ if (self->notls)
+ return;
+
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls12_0, 0);
+ tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128,
+ &tls12_1, 1);
+
+ ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_0, tls12_0.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_1, tls12_1.len);
+ ASSERT_EQ(ret, 0);
+
+ ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_1, tls12_1.len);
+ EXPECT_EQ(ret, 0);
+
+ EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len);
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str, send_len), 0);
+}
+
+TEST_F(tls, rekey)
+{
+ char const *test_str_1 = "test_message_before_rekey";
+ char const *test_str_2 = "test_message_after_rekey";
+ struct tls_crypto_info_keys tls12;
+ int send_len;
+ char buf[100];
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ /* initial send/recv */
+ send_len = strlen(test_str_1) + 1;
+ EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len);
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ /* send after rekey */
+ send_len = strlen(test_str_2) + 1;
+ EXPECT_EQ(send(self->fd, test_str_2, send_len, 0), send_len);
+
+ /* can't receive the KeyUpdate without a control message */
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), -1);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* recv blocking -> -EKEYEXPIRED */
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), 0), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* recv non-blocking -> -EKEYEXPIRED */
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_DONTWAIT), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* update RX key */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ /* recv after rekey */
+ EXPECT_NE(recv(self->cfd, buf, send_len, 0), -1);
+ EXPECT_EQ(memcmp(buf, test_str_2, send_len), 0);
+}
+
+TEST_F(tls, rekey_fail)
+{
+ char const *test_str_1 = "test_message_before_rekey";
+ char const *test_str_2 = "test_message_after_rekey";
+ struct tls_crypto_info_keys tls12;
+ int send_len;
+ char buf[100];
+
+ /* initial send/recv */
+ send_len = strlen(test_str_1) + 1;
+ EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len);
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+
+ if (variant->tls_version != TLS_1_3_VERSION) {
+ /* just check that rekey is not supported and return */
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1);
+ EXPECT_EQ(errno, EBUSY);
+ return;
+ }
+
+ /* successful update */
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ /* invalid update: change of version */
+ tls_crypto_info_init(TLS_1_2_VERSION, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* invalid update (RX socket): change of version */
+ tls_crypto_info_init(TLS_1_2_VERSION, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* invalid update: change of cipher */
+ if (variant->cipher_type == TLS_CIPHER_AES_GCM_256)
+ tls_crypto_info_init(variant->tls_version, TLS_CIPHER_CHACHA20_POLY1305, &tls12, 1);
+ else
+ tls_crypto_info_init(variant->tls_version, TLS_CIPHER_AES_GCM_256, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* send after rekey, the invalid updates shouldn't have an effect */
+ send_len = strlen(test_str_2) + 1;
+ EXPECT_EQ(send(self->fd, test_str_2, send_len, 0), send_len);
+
+ /* can't receive the KeyUpdate without a control message */
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), -1);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* recv blocking -> -EKEYEXPIRED */
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), 0), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* recv non-blocking -> -EKEYEXPIRED */
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_DONTWAIT), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* update RX key */
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ /* recv after rekey */
+ EXPECT_NE(recv(self->cfd, buf, send_len, 0), -1);
+ EXPECT_EQ(memcmp(buf, test_str_2, send_len), 0);
+}
+
+TEST_F(tls, rekey_peek)
+{
+ char const *test_str_1 = "test_message_before_rekey";
+ struct tls_crypto_info_keys tls12;
+ int send_len;
+ char buf[100];
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ send_len = strlen(test_str_1) + 1;
+ EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_PEEK), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ /* can't receive the KeyUpdate without a control message */
+ EXPECT_EQ(recv(self->cfd, buf, send_len, MSG_PEEK), -1);
+
+ /* peek KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, MSG_PEEK);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* update RX key */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+}
+
+TEST_F(tls, splice_rekey)
+{
+ int send_len = TLS_PAYLOAD_MAX_LEN / 2;
+ char mem_send[TLS_PAYLOAD_MAX_LEN];
+ char mem_recv[TLS_PAYLOAD_MAX_LEN];
+ struct tls_crypto_info_keys tls12;
+ int p[2];
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ memrnd(mem_send, sizeof(mem_send));
+
+ ASSERT_GE(pipe(p), 0);
+ EXPECT_EQ(send(self->fd, mem_send, send_len, 0), send_len);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ EXPECT_EQ(send(self->fd, mem_send, send_len, 0), send_len);
+
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len);
+ EXPECT_EQ(read(p[0], mem_recv, send_len), send_len);
+ EXPECT_EQ(memcmp(mem_send, mem_recv, send_len), 0);
+
+ /* can't splice the KeyUpdate */
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), -1);
+ EXPECT_EQ(errno, EINVAL);
+
+ /* peek KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, MSG_PEEK);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* can't splice before updating the key */
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), -1);
+ EXPECT_EQ(errno, EKEYEXPIRED);
+
+ /* update RX key */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len);
+ EXPECT_EQ(read(p[0], mem_recv, send_len), send_len);
+ EXPECT_EQ(memcmp(mem_send, mem_recv, send_len), 0);
+}
+
+TEST_F(tls, rekey_peek_splice)
+{
+ char const *test_str_1 = "test_message_before_rekey";
+ struct tls_crypto_info_keys tls12;
+ int send_len;
+ char buf[100];
+ char mem_recv[TLS_PAYLOAD_MAX_LEN];
+ int p[2];
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ ASSERT_GE(pipe(p), 0);
+
+ send_len = strlen(test_str_1) + 1;
+ EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len);
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_PEEK), send_len);
+ EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0);
+
+ EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len);
+ EXPECT_EQ(read(p[0], mem_recv, send_len), send_len);
+ EXPECT_EQ(memcmp(mem_recv, test_str_1, send_len), 0);
+}
+
+TEST_F(tls, rekey_getsockopt)
+{
+ struct tls_crypto_info_keys tls12;
+ struct tls_crypto_info_keys tls12_get;
+ socklen_t len;
+
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 0);
+
+ len = tls12.len;
+ EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_get, &len), 0);
+ EXPECT_EQ(len, tls12.len);
+ EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0);
+
+ len = tls12.len;
+ EXPECT_EQ(getsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_get, &len), 0);
+ EXPECT_EQ(len, tls12.len);
+ EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0);
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ len = tls12.len;
+ EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_get, &len), 0);
+ EXPECT_EQ(len, tls12.len);
+ EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0);
+
+ len = tls12.len;
+ EXPECT_EQ(getsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_get, &len), 0);
+ EXPECT_EQ(len, tls12.len);
+ EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0);
+}
+
+TEST_F(tls, rekey_poll_pending)
+{
+ char const *test_str = "test_message_after_rekey";
+ struct tls_crypto_info_keys tls12;
+ struct pollfd pfd = { };
+ int send_len;
+ int ret;
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ /* send immediately after rekey */
+ send_len = strlen(test_str) + 1;
+ EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len);
+
+ /* key hasn't been updated, expect cfd to be non-readable */
+ pfd.fd = self->cfd;
+ pfd.events = POLLIN;
+ EXPECT_EQ(poll(&pfd, 1, 0), 0);
+
+ ret = fork();
+ ASSERT_GE(ret, 0);
+
+ if (ret) {
+ int pid2, status;
+
+ /* wait before installing the new key */
+ sleep(1);
+
+ /* update RX key while poll() is sleeping */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ pid2 = wait(&status);
+ EXPECT_EQ(pid2, ret);
+ EXPECT_EQ(status, 0);
+ } else {
+ pfd.fd = self->cfd;
+ pfd.events = POLLIN;
+ EXPECT_EQ(poll(&pfd, 1, 5000), 1);
+
+ exit(!__test_passed(_metadata));
+ }
+}
+
+TEST_F(tls, rekey_poll_delay)
+{
+ char const *test_str = "test_message_after_rekey";
+ struct tls_crypto_info_keys tls12;
+ struct pollfd pfd = { };
+ int send_len;
+ int ret;
+
+ if (variant->tls_version != TLS_1_3_VERSION)
+ return;
+
+ /* update TX key */
+ tls_send_keyupdate(_metadata, self->fd);
+ tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1);
+ EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0);
+
+ /* get KeyUpdate */
+ tls_recv_keyupdate(_metadata, self->cfd, 0);
+
+ ret = fork();
+ ASSERT_GE(ret, 0);
+
+ if (ret) {
+ int pid2, status;
+
+ /* wait before installing the new key */
+ sleep(1);
+
+ /* update RX key while poll() is sleeping */
+ EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0);
+
+ sleep(1);
+ send_len = strlen(test_str) + 1;
+ EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len);
+
+ pid2 = wait(&status);
+ EXPECT_EQ(pid2, ret);
+ EXPECT_EQ(status, 0);
+ } else {
+ pfd.fd = self->cfd;
+ pfd.events = POLLIN;
+ EXPECT_EQ(poll(&pfd, 1, 5000), 1);
+ exit(!__test_passed(_metadata));
+ }
+}
+
FIXTURE(tls_err)
{
int fd, cfd;
diff --git a/tools/testing/selftests/vm/memfd_secret.c b/tools/testing/selftests/vm/memfd_secret.c
index 9b298f6a04b3..348cc976726b 100644
--- a/tools/testing/selftests/vm/memfd_secret.c
+++ b/tools/testing/selftests/vm/memfd_secret.c
@@ -56,33 +56,6 @@ static void test_file_apis(int fd)
pass("file IO is blocked as expected\n");
}
-static void test_mlock_limit(int fd)
-{
- size_t len;
- char *mem;
-
- len = mlock_limit_cur;
- if (len % page_size != 0)
- len = (len/page_size) * page_size;
-
- mem = mmap(NULL, len, prot, mode, fd, 0);
- if (mem == MAP_FAILED) {
- fail("unable to mmap secret memory\n");
- return;
- }
- munmap(mem, len);
-
- len = mlock_limit_max * 2;
- mem = mmap(NULL, len, prot, mode, fd, 0);
- if (mem != MAP_FAILED) {
- fail("unexpected mlock limit violation\n");
- munmap(mem, len);
- return;
- }
-
- pass("mlock limit is respected\n");
-}
-
static void try_process_vm_read(int fd, int pipefd[2])
{
struct iovec liov, riov;
@@ -258,7 +231,7 @@ static void prepare(void)
strerror(errno));
}
-#define NUM_TESTS 4
+#define NUM_TESTS 3
int main(int argc, char *argv[])
{
@@ -278,7 +251,6 @@ int main(int argc, char *argv[])
strerror(errno));
}
- test_mlock_limit(fd);
test_file_apis(fd);
test_process_vm_read(fd);
test_ptrace(fd);
diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c
index c4d17ded3619..ed6a90523d4c 100644
--- a/virt/kvm/kvm_main.c
+++ b/virt/kvm/kvm_main.c
@@ -3321,7 +3321,8 @@ void mark_page_dirty_in_slot(struct kvm *kvm,
struct kvm_vcpu *vcpu = kvm_get_running_vcpu();
#ifdef CONFIG_HAVE_KVM_DIRTY_RING
- if (WARN_ON_ONCE(!vcpu) || WARN_ON_ONCE(vcpu->kvm != kvm))
+ if (WARN_ON_ONCE(!vcpu && refcount_read(&kvm->users_count)) || !vcpu ||
+ WARN_ON_ONCE(vcpu->kvm != kvm))
return;
#endif
^ permalink raw reply [flat|nested] 2+ messages in thread